[petsc-dev] Telescope usage

Loic Gouarin loic.gouarin at polytechnique.edu
Tue May 17 02:07:50 CDT 2022

Le 17/05/2022 à 01:43, Dave May a écrit :
> On Tue, 17 May 2022 at 00:12, Matthew Knepley <knepley at gmail.com> wrote:
>     On Mon, May 16, 2022 at 9:59 AM Loic Gouarin
>     <loic.gouarin at polytechnique.edu> wrote:
>         Le 16/05/2022 à 21:54, Barry Smith a écrit :
>>>         On May 16, 2022, at 3:50 PM, Loic Gouarin
>>>         <loic.gouarin at polytechnique.edu> wrote:
>>>         Thanks Barry for your quick reply.
>>>         Le 16/05/2022 à 21:41, Barry Smith a écrit :
>>>>           Loic,
>>>>             From your code it looks like you are using a DM. Is it
>>>>         a DMDA or a shell DM? If it is a DMDA then the process is
>>>>         intended to be pretty straightforward. PCTELESCOPE should
>>>>         create a new DMDA that has the same properties as the
>>>>         coarse grid DMDA but lives on a smaller number of ranks.
>>>>         From this you can just provide multiple levels of
>>>>         coarsening of the DMDA to produce the smaller multigrid
>>>>         problems.
>>>         It's a DMDA. So what you mean is to take the KSP of
>>>         PCTELESCOPE, get the DM which is the same of the coarse grid
>>>         and build all the levels as previously for the levels of
>>>         PCTELESCOPE ?
>>           Yes, I think you should be able to pull out of the coarse
>>         level PC (on the subset of ranks) a DM that and then setup
>>         coarse levels from that. But you need to look at an example
>>         that that uses telescope and probably even the telescope code
>>         itself to see all the details. I don't know them anymore.
>         I looked at the source code but It didn't help. I will try to
>         explain more clearly my experiments to understand what I make
>         wrong and you can help more easily.
>     Hi Loic!
>     Here is my guess. When TELESCOPE redistributes the coarse grid
>     DMDA, it creates a new DMDA and copies over all the information it
>     can see. However, I think you may have state that is attached to
>     the old DMDA that is not getting redistributed. If this is true, I
>     can help you write a hook that redistributes that state when
>     TELESCOPE maps between distributions.
> If you have state you need to propagate between different 
> communicators, you will have to use the telescope type which is 
> internally is referred to as
> To activate this type you need the option
> -pc_telescope_use_coarse_dm
> Or to call PCTelescopeSetUseCoarseDM()
> The assumptions of usng this options are described here
> https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM
> and source associated with the COARSEDM
> https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html
> Using TELESCOPE_COARSEDM you will have access to the hooks Matt is 
> referring to.
Thanks Matthew and Dave.

I used TELESCOPE_COARSEDM in my previous tests but it tells me that 
"Zero instances of a coarse DM were found". And indeed, my coarse solver 
only has the DM on the coarse mesh and not on the fine level with a 
coarse DM atttached. I think it's because I use DMKSPSetComputeOperators 
to create the KSP on each level.

So for now, I just added

KSP coarse;
ierr=PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr);
PC pc_coarse;
ierr=KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr);
ierr=PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr);
ierr=PCTelescopeSetUseCoarseDM(pc_coarse, PETSC_TRUE);CHKERRQ(ierr);

but it's not enough.



> Cheers,
> Dave
>        Thanks,
>          Matt
>         Thanks,
>         Loic
>>           Barry
>>>>            Can you let us know more details of what exactly goes
>>>>         wrong with attempts? It is likely straightforward to get
>>>>         things working, perhaps due to our unclear documentation.
>>>         I essentially have wrong state or NULL matrices into the
>>>         PCTELESCOPE.
>>>         Loic
>>>>           Barry
>>>>>         On May 16, 2022, at 3:28 PM, Loic Gouarin
>>>>>         <loic.gouarin at polytechnique.edu> wrote:
>>>>>         Hello,
>>>>>         I could have posted my message on the user list but it
>>>>>         seems to me that it's more in the petsc development part.
>>>>>         Don't hesitate to tell me if I'm wrong.
>>>>>         I am developing a code called cafes that studies
>>>>>         fluid-particle interactions in a Stokes fluid. For that,
>>>>>         we implemented the whole solver in matrix free on a
>>>>>         cartesian grid. The preconditioner is naturally a
>>>>>         geometrical multigrid for the velocity. The set of matrix
>>>>>         free for each level is set by hand (I worked with Matthew
>>>>>         on this a long time ago now!).
>>>>>         I would have liked to implement telescope on this part. I
>>>>>         started it but I didn't get out of it. So I would like to
>>>>>         have your help.
>>>>>         What I tried to do was to describe my classical multigrid
>>>>>         up to a level N1 and then take the coarse solver and apply
>>>>>         the telescope preconditioner to it up to a level N2. I
>>>>>         don't know if this is the best way but it seemed the most
>>>>>         intuitive. The problem is that I could not get the coarse
>>>>>         matrix because the setup is not done yet (I use a
>>>>>         DMKSPSetComputeOperators for the matrices).
>>>>>         The construction of the matrix free for each level is
>>>>>         here:
>>>>>         https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L59-L88
>>>>>         The description of the Stokes preconditioner is described
>>>>>         here :
>>>>>         https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L109-L162
>>>>>         I have tried multiple implementations without success. For
>>>>>         me, the start is to add at line 105 something like
>>>>>         KSP coarse;
>>>>>         ierr = PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr);
>>>>>         PC pc_coarse;
>>>>>         ierr = KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr);
>>>>>         ierr = PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr);
>>>>>         Then, to create the several matrices of the levels of
>>>>>         TELESCOPE, I thought using the same process as before with
>>>>>         DMKSPSetComputeOperators. But it doesn't work...
>>>>>         Could you tell me how I can make it work ?
>>>>>         Thanks,
>>>>>         Loic
>     -- 
>     What most experimenters take for granted before they begin their
>     experiments is infinitely more interesting than any results to
>     which their experiments lead.
>     -- Norbert Wiener
>     https://www.cse.buffalo.edu/~knepley/
>     <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20220517/3992e442/attachment-0001.html>

More information about the petsc-dev mailing list