[petsc-dev] Telescope usage
Matthew Knepley
knepley at gmail.com
Tue May 17 10:53:33 CDT 2022
On Tue, May 17, 2022 at 4:53 AM Dave May <dave.mayhem23 at gmail.com> wrote:
> Dear Loic,
>
> I can confirm that PCTELESCOPE works nicely when using
> KSPSetComputeOperators.
> Here is an example
>
> petsc-3.17.1/src/ksp/ksp/tutorials $ mpiexec -n 4 ./ex29 -pc_type mg
> -pc_mg_levels 2 -ksp_view -mg_coarse_pc_type telescope
> -mg_coarse_pc_telescope_reduction_factor 4 -da_grid_x 65 -da_grid_y 65
> -mg_coarse_telescope_pc_type mg -mg_coarse_telescope_pc_mg_levels 2
>
> There is no special code in ex29.c to make the above options run.
>
> I must also apologize for sending you down the garden path with COARSEDM.
> That really isn't what you want to use - I am sorry for the confusion on my
> part.
> COARSEDM is intended for the case when YOU (or rather your coarse DM)
> define the sub-communicator, rather than having PCTELESCOPE define the
> sub-communicator.
>
> I believe that PCTELESCOPE should works with DMKSPSetComputeOperators().
>
> The function you provide to DMKSPSetComputeOperators() is unusual in my
> opinion.
> Normally the method provided to DMKSPSetComputeOperators() just
> "assembles" (assembles meaning insert non-zero values for an AIJ or just
> calls MatAssemblyBegin/End for a matrix-free operator).
> However your method seems to set sizes, set methods and call
> SetFromOptions. These are typically things which should done when the
> matrix is created - not when it is being "assembled".
>
> Maybe this is part of the reason your code isn't playing nicely with
> telescope.
> I think the code would be cleaner and if you overloaded your
> DMCreateMatrix() with a method which would return your matrix-free MATSHELL.
>
> Yes I realize I am not directly helping solve your problem but maybe
> indirectly I am. Matt?
>
Yes, I agree with Dave. This refactoring, even if it does not solve the
problem, should allow us to understand it better.
Does the change make sense?
Thanks,
Matt
> Cheers,
> Dave
>
>
> On Tue, 17 May 2022 at 13:18, Loic Gouarin <loic.gouarin at polytechnique.edu>
> wrote:
>
>>
>> Le 17/05/2022 à 12:03, Dave May a écrit :
>>
>>
>>
>> On Tue 17. May 2022 at 11:56, Loic Gouarin <
>> loic.gouarin at polytechnique.edu> wrote:
>>
>>> Hi Dave,
>>>
>>> could you explain what you mean by state ?
>>>
>>
>> Ah - by state Matt and I mean any auxiliary data stored in the user
>> context passed to KSPSetComputeOperators which is required to define your
>> operator and is distributed. For example, a Vec which is used store
>> coefficients of your PDE.
>>
>> Yes, I have a context which gives the operators (mult and diagonal) and
>> some other informations to build the matrix free on each multigrid level.
>>
>> You can find the idea here:
>> https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L49-L91
>>
>> It's the definition of the operator used inside DMKSPComputeOperators.
>>
>> Loic
>>
>>
>>
>>
>> Thanks,
>>>
>>> Loic
>>> Le 17/05/2022 à 11:50, Dave May a écrit :
>>>
>>> Hi Loic,
>>>
>>> Can you confirm if your problem has stored state which needs to be
>>> repartitioned?
>>>
>>> Thanks,
>>> Dave
>>>
>>> On Tue 17. May 2022 at 09:07, Loic Gouarin <
>>> loic.gouarin at polytechnique.edu> wrote:
>>>
>>>>
>>>> Le 17/05/2022 à 01:43, Dave May a écrit :
>>>>
>>>>
>>>>
>>>> On Tue, 17 May 2022 at 00:12, Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Mon, May 16, 2022 at 9:59 AM Loic Gouarin <
>>>>> loic.gouarin at polytechnique.edu> wrote:
>>>>>
>>>>>> Le 16/05/2022 à 21:54, Barry Smith a écrit :
>>>>>>
>>>>>>
>>>>>>
>>>>>> On May 16, 2022, at 3:50 PM, Loic Gouarin <
>>>>>> loic.gouarin at polytechnique.edu> wrote:
>>>>>>
>>>>>> Thanks Barry for your quick reply.
>>>>>> Le 16/05/2022 à 21:41, Barry Smith a écrit :
>>>>>>
>>>>>>
>>>>>> Loic,
>>>>>>
>>>>>> From your code it looks like you are using a DM. Is it a DMDA or
>>>>>> a shell DM? If it is a DMDA then the process is intended to be pretty
>>>>>> straightforward. PCTELESCOPE should create a new DMDA that has the same
>>>>>> properties as the coarse grid DMDA but lives on a smaller number of ranks.
>>>>>> From this you can just provide multiple levels of coarsening of the DMDA to
>>>>>> produce the smaller multigrid problems.
>>>>>>
>>>>>> It's a DMDA. So what you mean is to take the KSP of PCTELESCOPE, get
>>>>>> the DM which is the same of the coarse grid and build all the levels as
>>>>>> previously for the levels of PCTELESCOPE ?
>>>>>>
>>>>>>
>>>>>> Yes, I think you should be able to pull out of the coarse level PC
>>>>>> (on the subset of ranks) a DM that and then setup coarse levels from that.
>>>>>> But you need to look at an example that that uses telescope and probably
>>>>>> even the telescope code itself to see all the details. I don't know them
>>>>>> anymore.
>>>>>>
>>>>>> I looked at the source code but It didn't help. I will try to explain
>>>>>> more clearly my experiments to understand what I make wrong and you can
>>>>>> help more easily.
>>>>>>
>>>>> Hi Loic!
>>>>>
>>>>> Here is my guess. When TELESCOPE redistributes the coarse grid DMDA,
>>>>> it creates a new DMDA and copies over all the information it can see.
>>>>> However, I think you may have state that is attached to the old DMDA that
>>>>> is not getting redistributed. If this is true, I can help you write a hook
>>>>> that redistributes that state when TELESCOPE maps between distributions.
>>>>>
>>>>
>>>> If you have state you need to propagate between different
>>>> communicators, you will have to use the telescope type which is internally
>>>> is referred to as
>>>>
>>>> TELESCOPE_COARSEDM
>>>>
>>>>
>>>> To activate this type you need the option
>>>>
>>>> -pc_telescope_use_coarse_dm
>>>>
>>>> Or to call PCTelescopeSetUseCoarseDM()
>>>> The assumptions of usng this options are described here
>>>> https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM
>>>>
>>>> and source associated with the COARSEDM
>>>>
>>>> https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html
>>>>
>>>> Using TELESCOPE_COARSEDM you will have access to the hooks Matt is
>>>> referring to.
>>>>
>>>> Thanks Matthew and Dave.
>>>>
>>>> I used TELESCOPE_COARSEDM in my previous tests but it tells me that
>>>> "Zero instances of a coarse DM were found". And indeed, my coarse solver
>>>> only has the DM on the coarse mesh and not on the fine level with a coarse
>>>> DM atttached. I think it's because I use DMKSPSetComputeOperators to create
>>>> the KSP on each level.
>>>>
>>>> So for now, I just added
>>>> KSP coarse;
>>>> ierr = PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr);
>>>> PC pc_coarse;
>>>> ierr = KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr);
>>>> ierr = PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr);
>>>> ierr = PCTelescopeSetUseCoarseDM(pc_coarse, PETSC_TRUE);CHKERRQ(ierr);
>>>>
>>>> but it's not enough.
>>>>
>>>> Cheers,
>>>>
>>>> Loic
>>>>
>>>>
>>>> Cheers,
>>>> Dave
>>>>
>>>>
>>>>
>>>>> Thanks,
>>>>>
>>>>> Matt
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Loic
>>>>>>
>>>>>>
>>>>>> Barry
>>>>>>
>>>>>>
>>>>>> Can you let us know more details of what exactly goes wrong with
>>>>>> attempts? It is likely straightforward to get things working, perhaps due
>>>>>> to our unclear documentation.
>>>>>>
>>>>>> I essentially have wrong state or NULL matrices into the PCTELESCOPE.
>>>>>>
>>>>>> Loic
>>>>>>
>>>>>>
>>>>>> Barry
>>>>>>
>>>>>>
>>>>>> On May 16, 2022, at 3:28 PM, Loic Gouarin <
>>>>>> loic.gouarin at polytechnique.edu> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I could have posted my message on the user list but it seems to me
>>>>>> that it's more in the petsc development part. Don't hesitate to tell me if
>>>>>> I'm wrong.
>>>>>> I am developing a code called cafes that studies fluid-particle
>>>>>> interactions in a Stokes fluid. For that, we implemented the whole solver
>>>>>> in matrix free on a cartesian grid. The preconditioner is naturally a
>>>>>> geometrical multigrid for the velocity. The set of matrix free for each
>>>>>> level is set by hand (I worked with Matthew on this a long time ago now!).
>>>>>> I would have liked to implement telescope on this part. I started it
>>>>>> but I didn't get out of it. So I would like to have your help.
>>>>>> What I tried to do was to describe my classical multigrid up to a
>>>>>> level N1 and then take the coarse solver and apply the telescope
>>>>>> preconditioner to it up to a level N2. I don't know if this is the best way
>>>>>> but it seemed the most intuitive. The problem is that I could not get the
>>>>>> coarse matrix because the setup is not done yet (I use a
>>>>>> DMKSPSetComputeOperators for the matrices).
>>>>>>
>>>>>> The construction of the matrix free for each level is here:
>>>>>> https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L59-L88
>>>>>>
>>>>>> The description of the Stokes preconditioner is described here :
>>>>>> https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L109-L162
>>>>>>
>>>>>> I have tried multiple implementations without success. For me, the
>>>>>> start is to add at line 105 something like
>>>>>> KSP coarse;
>>>>>> ierr = PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr);
>>>>>> PC pc_coarse;
>>>>>> ierr = KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr);
>>>>>> ierr = PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr);
>>>>>>
>>>>>> Then, to create the several matrices of the levels of TELESCOPE, I
>>>>>> thought using the same process as before with DMKSPSetComputeOperators. But
>>>>>> it doesn't work...
>>>>>> Could you tell me how I can make it work ?
>>>>>>
>>>>>> Thanks,
>>>>>> Loic
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20220517/7468eca5/attachment-0001.html>
More information about the petsc-dev
mailing list