[petsc-dev] Telescope usage
Dave May
dave.mayhem23 at gmail.com
Tue May 17 09:53:23 CDT 2022
Dear Loic,
I can confirm that PCTELESCOPE works nicely when using
KSPSetComputeOperators.
Here is an example
petsc-3.17.1/src/ksp/ksp/tutorials $ mpiexec -n 4 ./ex29 -pc_type mg
-pc_mg_levels 2 -ksp_view -mg_coarse_pc_type telescope
-mg_coarse_pc_telescope_reduction_factor 4 -da_grid_x 65 -da_grid_y 65
-mg_coarse_telescope_pc_type mg -mg_coarse_telescope_pc_mg_levels 2
There is no special code in ex29.c to make the above options run.
I must also apologize for sending you down the garden path with COARSEDM.
That really isn't what you want to use - I am sorry for the confusion on my
part.
COARSEDM is intended for the case when YOU (or rather your coarse DM)
define the sub-communicator, rather than having PCTELESCOPE define the
sub-communicator.
I believe that PCTELESCOPE should works with DMKSPSetComputeOperators().
The function you provide to DMKSPSetComputeOperators() is unusual in my
opinion.
Normally the method provided to DMKSPSetComputeOperators() just "assembles"
(assembles meaning insert non-zero values for an AIJ or just calls
MatAssemblyBegin/End for a matrix-free operator).
However your method seems to set sizes, set methods and call
SetFromOptions. These are typically things which should done when the
matrix is created - not when it is being "assembled".
Maybe this is part of the reason your code isn't playing nicely with
telescope.
I think the code would be cleaner and if you overloaded your
DMCreateMatrix() with a method which would return your matrix-free MATSHELL.
Yes I realize I am not directly helping solve your problem but maybe
indirectly I am. Matt?
Cheers,
Dave
On Tue, 17 May 2022 at 13:18, Loic Gouarin <loic.gouarin at polytechnique.edu>
wrote:
>
> Le 17/05/2022 à 12:03, Dave May a écrit :
>
>
>
> On Tue 17. May 2022 at 11:56, Loic Gouarin <loic.gouarin at polytechnique.edu>
> wrote:
>
>> Hi Dave,
>>
>> could you explain what you mean by state ?
>>
>
> Ah - by state Matt and I mean any auxiliary data stored in the user
> context passed to KSPSetComputeOperators which is required to define your
> operator and is distributed. For example, a Vec which is used store
> coefficients of your PDE.
>
> Yes, I have a context which gives the operators (mult and diagonal) and
> some other informations to build the matrix free on each multigrid level.
>
> You can find the idea here:
> https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L49-L91
>
> It's the definition of the operator used inside DMKSPComputeOperators.
>
> Loic
>
>
>
>
> Thanks,
>>
>> Loic
>> Le 17/05/2022 à 11:50, Dave May a écrit :
>>
>> Hi Loic,
>>
>> Can you confirm if your problem has stored state which needs to be
>> repartitioned?
>>
>> Thanks,
>> Dave
>>
>> On Tue 17. May 2022 at 09:07, Loic Gouarin <
>> loic.gouarin at polytechnique.edu> wrote:
>>
>>>
>>> Le 17/05/2022 à 01:43, Dave May a écrit :
>>>
>>>
>>>
>>> On Tue, 17 May 2022 at 00:12, Matthew Knepley <knepley at gmail.com> wrote:
>>>
>>>> On Mon, May 16, 2022 at 9:59 AM Loic Gouarin <
>>>> loic.gouarin at polytechnique.edu> wrote:
>>>>
>>>>> Le 16/05/2022 à 21:54, Barry Smith a écrit :
>>>>>
>>>>>
>>>>>
>>>>> On May 16, 2022, at 3:50 PM, Loic Gouarin <
>>>>> loic.gouarin at polytechnique.edu> wrote:
>>>>>
>>>>> Thanks Barry for your quick reply.
>>>>> Le 16/05/2022 à 21:41, Barry Smith a écrit :
>>>>>
>>>>>
>>>>> Loic,
>>>>>
>>>>> From your code it looks like you are using a DM. Is it a DMDA or a
>>>>> shell DM? If it is a DMDA then the process is intended to be pretty
>>>>> straightforward. PCTELESCOPE should create a new DMDA that has the same
>>>>> properties as the coarse grid DMDA but lives on a smaller number of ranks.
>>>>> From this you can just provide multiple levels of coarsening of the DMDA to
>>>>> produce the smaller multigrid problems.
>>>>>
>>>>> It's a DMDA. So what you mean is to take the KSP of PCTELESCOPE, get
>>>>> the DM which is the same of the coarse grid and build all the levels as
>>>>> previously for the levels of PCTELESCOPE ?
>>>>>
>>>>>
>>>>> Yes, I think you should be able to pull out of the coarse level PC
>>>>> (on the subset of ranks) a DM that and then setup coarse levels from that.
>>>>> But you need to look at an example that that uses telescope and probably
>>>>> even the telescope code itself to see all the details. I don't know them
>>>>> anymore.
>>>>>
>>>>> I looked at the source code but It didn't help. I will try to explain
>>>>> more clearly my experiments to understand what I make wrong and you can
>>>>> help more easily.
>>>>>
>>>> Hi Loic!
>>>>
>>>> Here is my guess. When TELESCOPE redistributes the coarse grid DMDA, it
>>>> creates a new DMDA and copies over all the information it can see. However,
>>>> I think you may have state that is attached to the old DMDA that is not
>>>> getting redistributed. If this is true, I can help you write a hook that
>>>> redistributes that state when TELESCOPE maps between distributions.
>>>>
>>>
>>> If you have state you need to propagate between different communicators,
>>> you will have to use the telescope type which is internally is referred to
>>> as
>>>
>>> TELESCOPE_COARSEDM
>>>
>>>
>>> To activate this type you need the option
>>>
>>> -pc_telescope_use_coarse_dm
>>>
>>> Or to call PCTelescopeSetUseCoarseDM()
>>> The assumptions of usng this options are described here
>>> https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM
>>>
>>> and source associated with the COARSEDM
>>>
>>> https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html
>>>
>>> Using TELESCOPE_COARSEDM you will have access to the hooks Matt is
>>> referring to.
>>>
>>> Thanks Matthew and Dave.
>>>
>>> I used TELESCOPE_COARSEDM in my previous tests but it tells me that
>>> "Zero instances of a coarse DM were found". And indeed, my coarse solver
>>> only has the DM on the coarse mesh and not on the fine level with a coarse
>>> DM atttached. I think it's because I use DMKSPSetComputeOperators to create
>>> the KSP on each level.
>>>
>>> So for now, I just added
>>> KSP coarse;
>>> ierr = PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr);
>>> PC pc_coarse;
>>> ierr = KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr);
>>> ierr = PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr);
>>> ierr = PCTelescopeSetUseCoarseDM(pc_coarse, PETSC_TRUE);CHKERRQ(ierr);
>>>
>>> but it's not enough.
>>>
>>> Cheers,
>>>
>>> Loic
>>>
>>>
>>> Cheers,
>>> Dave
>>>
>>>
>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>>
>>>>> Thanks,
>>>>>
>>>>> Loic
>>>>>
>>>>>
>>>>> Barry
>>>>>
>>>>>
>>>>> Can you let us know more details of what exactly goes wrong with
>>>>> attempts? It is likely straightforward to get things working, perhaps due
>>>>> to our unclear documentation.
>>>>>
>>>>> I essentially have wrong state or NULL matrices into the PCTELESCOPE.
>>>>>
>>>>> Loic
>>>>>
>>>>>
>>>>> Barry
>>>>>
>>>>>
>>>>> On May 16, 2022, at 3:28 PM, Loic Gouarin <
>>>>> loic.gouarin at polytechnique.edu> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I could have posted my message on the user list but it seems to me
>>>>> that it's more in the petsc development part. Don't hesitate to tell me if
>>>>> I'm wrong.
>>>>> I am developing a code called cafes that studies fluid-particle
>>>>> interactions in a Stokes fluid. For that, we implemented the whole solver
>>>>> in matrix free on a cartesian grid. The preconditioner is naturally a
>>>>> geometrical multigrid for the velocity. The set of matrix free for each
>>>>> level is set by hand (I worked with Matthew on this a long time ago now!).
>>>>> I would have liked to implement telescope on this part. I started it
>>>>> but I didn't get out of it. So I would like to have your help.
>>>>> What I tried to do was to describe my classical multigrid up to a
>>>>> level N1 and then take the coarse solver and apply the telescope
>>>>> preconditioner to it up to a level N2. I don't know if this is the best way
>>>>> but it seemed the most intuitive. The problem is that I could not get the
>>>>> coarse matrix because the setup is not done yet (I use a
>>>>> DMKSPSetComputeOperators for the matrices).
>>>>>
>>>>> The construction of the matrix free for each level is here:
>>>>> https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L59-L88
>>>>>
>>>>> The description of the Stokes preconditioner is described here :
>>>>> https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L109-L162
>>>>>
>>>>> I have tried multiple implementations without success. For me, the
>>>>> start is to add at line 105 something like
>>>>> KSP coarse;
>>>>> ierr = PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr);
>>>>> PC pc_coarse;
>>>>> ierr = KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr);
>>>>> ierr = PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr);
>>>>>
>>>>> Then, to create the several matrices of the levels of TELESCOPE, I
>>>>> thought using the same process as before with DMKSPSetComputeOperators. But
>>>>> it doesn't work...
>>>>> Could you tell me how I can make it work ?
>>>>>
>>>>> Thanks,
>>>>> Loic
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20220517/87f367d3/attachment-0001.html>
More information about the petsc-dev
mailing list