[petsc-dev] Telescope usage
Loic Gouarin
loic.gouarin at polytechnique.edu
Tue May 17 06:18:15 CDT 2022
Le 17/05/2022 à 12:03, Dave May a écrit :
>
>
> On Tue 17. May 2022 at 11:56, Loic Gouarin
> <loic.gouarin at polytechnique.edu> wrote:
>
> Hi Dave,
>
> could you explain what you mean by state ?
>
>
> Ah - by state Matt and I mean any auxiliary data stored in the user
> context passed to KSPSetComputeOperators which is required to define
> your operator and is distributed. For example, a Vec which is used
> store coefficients of your PDE.
>
Yes, I have a context which gives the operators (mult and diagonal) and
some other informations to build the matrix free on each multigrid level.
You can find the idea here:
https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L49-L91
It's the definition of the operator used inside DMKSPComputeOperators.
Loic
>
>
>
> Thanks,
>
> Loic
>
> Le 17/05/2022 à 11:50, Dave May a écrit :
>> Hi Loic,
>>
>> Can you confirm if your problem has stored state which needs to
>> be repartitioned?
>>
>> Thanks,
>> Dave
>>
>> On Tue 17. May 2022 at 09:07, Loic Gouarin
>> <loic.gouarin at polytechnique.edu> wrote:
>>
>>
>> Le 17/05/2022 à 01:43, Dave May a écrit :
>>>
>>>
>>> On Tue, 17 May 2022 at 00:12, Matthew Knepley
>>> <knepley at gmail.com> wrote:
>>>
>>> On Mon, May 16, 2022 at 9:59 AM Loic Gouarin
>>> <loic.gouarin at polytechnique.edu> wrote:
>>>
>>> Le 16/05/2022 à 21:54, Barry Smith a écrit :
>>>
>>>>
>>>>
>>>>> On May 16, 2022, at 3:50 PM, Loic Gouarin
>>>>> <loic.gouarin at polytechnique.edu> wrote:
>>>>>
>>>>> Thanks Barry for your quick reply.
>>>>>
>>>>> Le 16/05/2022 à 21:41, Barry Smith a écrit :
>>>>>>
>>>>>> Loic,
>>>>>>
>>>>>> From your code it looks like you are using a
>>>>>> DM. Is it a DMDA or a shell DM? If it is a DMDA
>>>>>> then the process is intended to be pretty
>>>>>> straightforward. PCTELESCOPE should create a new
>>>>>> DMDA that has the same properties as the coarse
>>>>>> grid DMDA but lives on a smaller number of ranks.
>>>>>> From this you can just provide multiple levels of
>>>>>> coarsening of the DMDA to produce the smaller
>>>>>> multigrid problems.
>>>>> It's a DMDA. So what you mean is to take the KSP
>>>>> of PCTELESCOPE, get the DM which is the same of
>>>>> the coarse grid and build all the levels as
>>>>> previously for the levels of PCTELESCOPE ?
>>>>
>>>> Yes, I think you should be able to pull out of
>>>> the coarse level PC (on the subset of ranks) a DM
>>>> that and then setup coarse levels from that. But
>>>> you need to look at an example that that uses
>>>> telescope and probably even the telescope code
>>>> itself to see all the details. I don't know them
>>>> anymore.
>>>
>>> I looked at the source code but It didn't help. I
>>> will try to explain more clearly my experiments to
>>> understand what I make wrong and you can help more
>>> easily.
>>>
>>> Hi Loic!
>>>
>>> Here is my guess. When TELESCOPE redistributes the
>>> coarse grid DMDA, it creates a new DMDA and copies over
>>> all the information it can see. However, I think you may
>>> have state that is attached to the old DMDA that is not
>>> getting redistributed. If this is true, I can help you
>>> write a hook that redistributes that state when
>>> TELESCOPE maps between distributions.
>>>
>>>
>>> If you have state you need to propagate between different
>>> communicators, you will have to use the telescope type which
>>> is internally is referred to as
>>> TELESCOPE_COARSEDM
>>>
>>> To activate this type you need the option
>>> -pc_telescope_use_coarse_dm
>>> Or to call PCTelescopeSetUseCoarseDM()
>>> The assumptions of usng this options are described here
>>> https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM
>>>
>>> and source associated with the COARSEDM
>>> https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html
>>>
>>> Using TELESCOPE_COARSEDM you will have access to the hooks
>>> Matt is referring to.
>>>
>> Thanks Matthew and Dave.
>>
>> I used TELESCOPE_COARSEDM in my previous tests but it tells
>> me that "Zero instances of a coarse DM were found". And
>> indeed, my coarse solver only has the DM on the coarse mesh
>> and not on the fine level with a coarse DM atttached. I think
>> it's because I use DMKSPSetComputeOperators to create the KSP
>> on each level.
>>
>> So for now, I just added
>>
>> KSP coarse;
>> ierr=PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr);
>> PC pc_coarse;
>> ierr=KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr);
>> ierr=PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr);
>> ierr=PCTelescopeSetUseCoarseDM(pc_coarse,
>> PETSC_TRUE);CHKERRQ(ierr);
>>
>> but it's not enough.
>>
>> Cheers,
>>
>> Loic
>>
>>
>>> Cheers,
>>> Dave
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>> Thanks,
>>>
>>> Loic
>>>
>>>>
>>>> Barry
>>>>
>>>>>>
>>>>>> Can you let us know more details of what
>>>>>> exactly goes wrong with attempts? It is likely
>>>>>> straightforward to get things working, perhaps
>>>>>> due to our unclear documentation.
>>>>>
>>>>> I essentially have wrong state or NULL matrices
>>>>> into the PCTELESCOPE.
>>>>>
>>>>> Loic
>>>>>
>>>>>>
>>>>>> Barry
>>>>>>
>>>>>>
>>>>>>> On May 16, 2022, at 3:28 PM, Loic Gouarin
>>>>>>> <loic.gouarin at polytechnique.edu> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I could have posted my message on the user list
>>>>>>> but it seems to me that it's more in the petsc
>>>>>>> development part. Don't hesitate to tell me if
>>>>>>> I'm wrong.
>>>>>>> I am developing a code called cafes that studies
>>>>>>> fluid-particle interactions in a Stokes fluid.
>>>>>>> For that, we implemented the whole solver in
>>>>>>> matrix free on a cartesian grid. The
>>>>>>> preconditioner is naturally a geometrical
>>>>>>> multigrid for the velocity. The set of matrix
>>>>>>> free for each level is set by hand (I worked
>>>>>>> with Matthew on this a long time ago now!).
>>>>>>> I would have liked to implement telescope on
>>>>>>> this part. I started it but I didn't get out of
>>>>>>> it. So I would like to have your help.
>>>>>>> What I tried to do was to describe my classical
>>>>>>> multigrid up to a level N1 and then take the
>>>>>>> coarse solver and apply the telescope
>>>>>>> preconditioner to it up to a level N2. I don't
>>>>>>> know if this is the best way but it seemed the
>>>>>>> most intuitive. The problem is that I could not
>>>>>>> get the coarse matrix because the setup is not
>>>>>>> done yet (I use a DMKSPSetComputeOperators for
>>>>>>> the matrices).
>>>>>>>
>>>>>>> The construction of the matrix free for each
>>>>>>> level is here:
>>>>>>> https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L59-L88
>>>>>>>
>>>>>>> The description of the Stokes preconditioner is
>>>>>>> described here :
>>>>>>> https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L109-L162
>>>>>>>
>>>>>>> I have tried multiple implementations without
>>>>>>> success. For me, the start is to add at line 105
>>>>>>> something like
>>>>>>>
>>>>>>> KSP coarse;
>>>>>>> ierr = PCMGGetCoarseSolve(pc_i,
>>>>>>> &coarse);CHKERRQ(ierr);
>>>>>>> PC pc_coarse;
>>>>>>> ierr = KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr);
>>>>>>> ierr = PCSetType(pc_coarse,
>>>>>>> PCTELESCOPE);CHKERRQ(ierr);
>>>>>>>
>>>>>>> Then, to create the several matrices of the
>>>>>>> levels of TELESCOPE, I thought using the same
>>>>>>> process as before with DMKSPSetComputeOperators.
>>>>>>> But it doesn't work...
>>>>>>> Could you tell me how I can make it work ?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Loic
>>>>>>>
>>>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they
>>> begin their experiments is infinitely more interesting
>>> than any results to which their experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20220517/a2a7240b/attachment-0001.html>
More information about the petsc-dev
mailing list