[petsc-users] Nesting splits based on IS

Karin&NiKo niko.karin at gmail.com
Tue Apr 27 08:57:56 CDT 2021 

I will have a look at PetscSection (that I have never used before).
The FE I use are P2-P1-P1-P1 Lagrange elements. The point is that the
numbering of the unknowns is done by the application (I do not have the
hand on it). At the moment, I build IS in order to define the different
fields and I nest them programmatically in order to set some recursive
preconditioner.
To enable the use of the command line options in order to nest the splits
is a matter of elegance and smoothness since the nesting of splits I have
programmed are fully fixed.

Nicolas

Le mar. 27 avr. 2021 à 14:15, Matthew Knepley <knepley at gmail.com> a écrit :

> On Tue, Apr 27, 2021 at 2:51 AM Karin&NiKo <niko.karin at gmail.com> wrote:
>
>> Dear PETSc Team,
>>
>> I am coming back to you to get some clue on the (missing?) Fortran
>> interface to the DMShellSetCreateSubDM routine.
>>
>
> Yes, we will have to put that in. Fortran bindings for callbacks are
> somewhat involved.
>
> However, it is easier if you layout can be described by a
> PetscSection (which has all the Fortran bindings). Then it will
> work automatically from the DMShell (I think). What kind of layout were
> you looking for?
>
>   Thanks,
>
>      Matt
>
>
>> Thanks,
>> Nicolas
>>
>> Le dim. 25 avr. 2021 à 18:20, Karin&NiKo <niko.karin at gmail.com> a écrit :
>>
>>> Dear Petsc Team,
>>>
>>> I have begun to write a small test to illustrate the use of a DMShell in
>>> order to nest splits from the command line.
>>> I am writing it in Fortran because the targeted application is also in
>>> Fortran.
>>> => I am afraid that DMShellSetCreateSubDM lacks a Fortran interface.
>>> Am I wrong ? Can't the interface be generated by bfort ?
>>>
>>> Thanks,
>>> Nicolas
>>>
>>> Le lun. 19 avr. 2021 à 12:28, Matthew Knepley <knepley at gmail.com> a
>>> écrit :
>>>
>>>> On Sun, Apr 18, 2021 at 11:54 PM Barry Smith <bsmith at petsc.dev> wrote:
>>>>
>>>>>
>>>>>   Matt,
>>>>>
>>>>>    I agree it can be done with DMs, but I am not convinced that is
>>>>> necessarily a superior way when one has to make DMSHELLS and mess around
>>>>> with their behavior.
>>>>>
>>>>>    One way to look at PCFIELDSPLIT is that its basic splitting tool is
>>>>> IS and DM is a way of generating IS that can be used for PCFIELDSPLIT. In
>>>>> this approach PCFIELDSPLIT needs a bit more support for IS and nesting. To
>>>>> also help using multiple levels of DM.
>>>>>
>>>>>    The other way is to view DM and subDM as the basic splitting tool
>>>>> for PCFIELDSPLIT but then one has to ask the question how does DM
>>>>> communicate its splits to PCFIELDSPLIT?   I am too lazy to look at the code
>>>>> but presumably with IS, hence the approach above.
>>>>>
>>>>>    In the traditional Unix software stack approach to code, each layer
>>>>> uses the simpler layer below to communicate; so DM uses IS to communicate
>>>>> to the layer below (the linear algebra and preconditioners).  DM is a
>>>>> hugely complicated layer and making it communicate directly with the vector
>>>>> level is complex; I like having the IS level in between to simplify the
>>>>> software stack and programming model.
>>>>>
>>>>>    PetscSection makes live more complicated since it is a bit disjoint
>>>>> from IS. I've never resolved in my mind what role PetscSection plays, is it
>>>>> a level above IS in the software stack that generates IS, does it sometimes
>>>>> "skip over" IS to directly talk to linear algebra?
>>>>>
>>>>>    If you cannot make a cartoon picture of the software stack, with
>>>>> all the objects, then I think the software stack is not well designed or
>>>>> defined. I fear we cannot make such a cartoon currently.
>>>>>
>>>>
>>>> DM _does_ communicate with PCFIELDSPLIT using IS. I agree with you that
>>>> IS is a good object for communication. In PETSc, IS is just a nice way to
>>>> pass a list of integers.
>>>>
>>>> I don't think DM is hugely complicated. It does a few simple jobs. Here
>>>> it's job is to remember which field each dof belongs to. That is all we
>>>> have to worry about.
>>>>
>>>> PetscSection semantically is a linear space with some structure. We
>>>> already know we want some structure like this, since we break all linear
>>>> spaces in PETSc into processes. Section allows you to break it down a
>>>> little finer into the pieces of the space for each "point", where you can
>>>> use a point to mark anything you want, like a process, cell, edge, another
>>>> dof, etc. Sections can make an IS when asked a question, such as "which
>>>> dofs lie in the closure of this cell", or "which dofs are in this field",
>>>> or "which dofs are owned by this process". I have written this in the
>>>> manual years ago.
>>>>
>>>>     Matt
>>>>
>>>>
>>>>>  Barry
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Apr 18, 2021, at 8:54 AM, Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>> On Sat, Apr 17, 2021 at 6:13 PM Barry Smith <bsmith at petsc.dev> wrote:
>>>>>
>>>>>>
>>>>>>   So you would like to be able to create three IS in your code and
>>>>>> attach them with names to the PC.  Then have -pc_fieldsplit_XXX_fields be
>>>>>> able to utilize the attached IS by name and use them to define the blocks.
>>>>>>
>>>>>>
>>>>>>   This is all doable and could be added to PCFIELDSPLIT without too
>>>>>> much code, new code. The code would be largely like
>>>>>> PCFieldSplitSetRuntimeSplits_Private.
>>>>>>
>>>>>>    The recursive part may also be doable but I think your syntax
>>>>>> below is not exactly right. You would need something like
>>>>>>
>>>>>> -fieldsplit_0_pc_type fieldsplit   // split the first PC into a
>>>>>> fieldsplit
>>>>>>      -fieldsplit_0_pc_fieldsplit_0_fields xxx
>>>>>> -fieldsplit_0_fieldsplit_0_pc_type jacobi
>>>>>>      -fieldsplit_0_pc_fieldsplit_1_fields yyy
>>>>>>      etc
>>>>>>
>>>>>> this would split the first field into two fields and use jacobi on
>>>>>> the first field.
>>>>>>
>>>>>> The problem is what to use for labelling the xxx and the yyy?
>>>>>>
>>>>>> I think one could achieve this by having the PCFIELDPLIT attach to
>>>>>> each of its split PCs the appropriate modified IS with names attached to
>>>>>> them.  There are two cases,
>>>>>>
>>>>>>   when building the split uses first all the entries from
>>>>>> fieldsplit_v_, then from fieldsplit_p_ then the new ISs it needs to attach
>>>>>> to the first split PC are two sets of integers the first from 0 to the
>>>>>> len(v)-1 and the second from len(v) to len(v)+len(p)-1.
>>>>>>
>>>>>>   when building the split it interlaces the indices from v and p
>>>>>> (interlacing only make sense if the size of v and p is the same). Then the
>>>>>> new v would be {0,2,4,...} and p would be {1,3,...}.
>>>>>>
>>>>>>   If you are ambitious and would like to add this to fieldsplit.c
>>>>>> we'd be very happy to receive an MR. It might even lead to allowing us to
>>>>>> simply how the PCFIELDPLIT interacts with DMs. If all the split type,
>>>>>> stride, named, etc are handle in a single consistent manner.
>>>>>>
>>>>>
>>>>> Barry, this is already working with DMs, which I think is the right
>>>>> way to do this.
>>>>>
>>>>> Here is the code:
>>>>>
>>>>>
>>>>> https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/fieldsplit/fieldsplit.c#L420
>>>>>
>>>>> The DM must respond to DMCreateSubDM(). The interface is that the call
>>>>> provides a list of fields [f_0, f_1, ...]
>>>>> and the DM returns an IS for that combination and a subdm for it. The
>>>>> subdm part is what allows you to do this
>>>>> recursively, since you make the same call on the subdm.
>>>>>
>>>>> If you use a PetscSection to keep track of the data layout, the code
>>>>> to do field selection is already done:
>>>>>
>>>>>
>>>>> https://gitlab.com/petsc/petsc/-/blob/main/src/dm/interface/dmi.c#L105
>>>>>
>>>>> which can just be called from the DMCreateSubDM() implementation.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>>   Barry
>>>>>>
>>>>>>
>>>>>>
>>>>>> > On Apr 17, 2021, at 11:53 AM, Karin&NiKo <niko.karin at gmail.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > Dear PETSc users,
>>>>>> >
>>>>>> > I use the fieldsplit PC in an application where the splits are
>>>>>> programmatically defined by IS using PCFieldSplitSetIS. Then the user can
>>>>>> specify its own PC at runtime using PETSc options.
>>>>>> > My question : is it possible to define nested splits in this case
>>>>>> as it can be done with strided splits (see snes/examples/tutorials/ex19.c
>>>>>> with test suffix fieldsplit_4).
>>>>>> >
>>>>>> > In order to be perfectly clear : let's say I have a 3 fields
>>>>>> problem : velocity (v split), pressure (p split) and temperature (t split).
>>>>>> > What I would like to do is something like the following but it
>>>>>> fails :
>>>>>> >
>>>>>> > -ksp_type fgmres
>>>>>> > -pc_fieldsplit_type multiplicative
>>>>>> > -pc_type fieldsplit    -pc_fieldsplit_0_fields fieldsplit_v_,
>>>>>> fieldsplit_p_    -pc_fieldsplit_1_fields fieldsplit_t_
>>>>>> >
>>>>>> > -prefix_push  fieldsplit_0_
>>>>>> > -ksp_type fgmres
>>>>>> > -pc_fieldsplit_schur_factorization_type upper
>>>>>> > -pc_type fieldsplit
>>>>>> > -pc_fieldsplit_type schur
>>>>>> > -pc_fieldsplit_schur_precondition a11
>>>>>> > -prefix_pop
>>>>>> >
>>>>>> > -prefix_push  fieldsplit_1_
>>>>>> > -ksp_type fgmres
>>>>>> > -pc_type jacobi
>>>>>> > -prefix_pop
>>>>>> >
>>>>>> > -prefix_push  fieldsplit_v_
>>>>>> > -ksp_type fgmres
>>>>>> > -pc_type gamg
>>>>>> > -prefix_pop
>>>>>> >
>>>>>> > -prefix_push  fieldsplit_p_
>>>>>> > -ksp_type fgmres
>>>>>> > -pc_type jacobi
>>>>>> > -prefix_pop
>>>>>> >
>>>>>> > I thank you for your help,
>>>>>> > Nicolas
>>>>>> >
>>>>>> >
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20210427/d8a1ae59/attachment.html>

More information about the petsc-users mailing list