[petsc-users] KSPGMRESOrthog costs too much time

[Rongliang Chen]

Hi Jed,

Thank you for your reply.
I am using the composite pc now not the PCMG:

      ierr = PCCompositeAddPC(finepc,PCSHELL);CHKERRQ(ierr);
      ierr = PCCompositeAddPC(finepc,PCASM);CHKERRQ(ierr);

      ierr = PCCompositeGetPC(finepc,0,&coarsesolve);CHKERRQ(ierr);
      ierr = PCShellSetContext(coarsesolve,ctx);CHKERRQ(ierr);
      ierr = PCShellSetApply(coarsesolve,
CoarseSolvePCApply);CHKERRQ(ierr);

      ierr = PCCompositeGetPC(finepc,1,&asmpc);CHKERRQ(ierr);
      ierr = PCSetOptionsPrefix(asmpc,"fine_");CHKERRQ(ierr);
      ierr = PCSetFromOptions(asmpc);CHKERRQ(ierr);

      ierr = PCSetType(asmpc,PCASM);CHKERRQ(ierr);
      ierr = PCASMSetOverlap(asmpc,0);CHKERRQ(ierr);
      ierr = PCASMSetLocalSubdomains(asmpc,1,&grid->df_global_asm,
PETSC_NULL);CHKERRQ(ierr);


I just use two level method now and it is not very easy to try more levels
since I am using unstructure meshes.
I tried to solve the coarse level exactly by LU and it works well if the
coarse problem is small. When the coarse problem is large, LU is also very
slow (when the fine level problem is large, I can not use very small coarse
level problem). I found that nearly 90% of the time is spent on the coarse
level when the number of processor is large (np >512), so I want to know
which step of the coarse level solver costs the most of the time. Thanks.

Best,
Rongliang




> Message: 4
> Date: Thu, 17 Nov 2011 11:43:34 -0600
> From: Jed Brown <jedbrown at mcs.anl.gov>
> Subject: Re: [petsc-users] KSPGMRESOrthog costs too much time
> To: PETSc users list <petsc-users at mcs.anl.gov>
> Message-ID:
>        <CAM9tzSnFp81RUbHUXSOVN2nqvAqOyFp6HYU8A7qjR11Z7KXOJg at mail.gmail.com
> >
> Content-Type: text/plain; charset="utf-8"
>
> On Thu, Nov 17, 2011 at 11:17, Rongliang Chen <rongliang.chan at gmail.com
> >wrote:
>
> > In my log_summary output, I found that nearly 80% of the total time is
> > spent on KSPGMRESOrthog. I think this does not make sense ( the
> log_summary
> > output followed). Who has any idea about this?
> >
>
> Reductions are very expensive relative to everything else on the coarse
> level. You can try more levels or a different coarse level solver. You can
> also likely get away with solving the coarse problem inexactly.
>
> Alternatively, you can try getting Chebychev to help you out. Use
> -ksp_chebychev_estimate_eigenvalues to tune Chebychev (possibly to target a
> specific part of the spectrum).
>
>
> http://www.mcs.anl.gov/petsc/snapshots/petsc-dev/docs/manualpages/KSP/KSPChebychevSetEstimateEigenvalues.html
>
>
> >
> > Another question, I am using the two-level asm precondtioner.  On the
> > coarse level I use one-level asm preconditioned GMRES to solve a coarse
> > problem. So both the fine level solver and coarse level solver call the
> > function KSPGMRESOrthog. In the log_summary output, I just know the total
> > time spent on KSPGMRESOrthog and how can I know how much time is spent on
> > the coarse level KSPGMRESOrthog and how much is spent on fine level
> > KSPGMRESOrthog? Thanks.
> >
>
> I assume you are using PCMG for this, so you can add -pc_mg_log to profile
> the time on each level independently. You seem to have many KSPGMRESOrthog
> steps per fine-level PCApply, so I think most of the time is in the coarse
> level.
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20111117/f6c1ca41/attachment.htm
> >
>
> ------------------------------
>
> _______________________________________________
> petsc-users mailing list
> petsc-users at mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/petsc-users
>
>
> End of petsc-users Digest, Vol 35, Issue 56
> *******************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20111117/4483b0a3/attachment.htm>