[petsc-users] Create a DM given sets of IS's

Matthew Knepley knepley at gmail.com
Thu Jan 3 08:04:23 CST 2019 

Looks like it is converging great. There are two things:

- The inner FS is additive, which seems to work fine here, but maybe not
when there is more coupling with temperature
- The BJacobi/ILU might not scale well.

  Thanks,

     Matt

On Wed, Jan 2, 2019 at 8:38 PM Justin Chang <jychang48 at gmail.com> wrote:

> So I was slightly incorrect - these options I originally posted:
>
> -fieldsplit_1_pc_fieldsplit_0_fields 3,4
> -fieldsplit_1_pc_fieldsplit_1_fields 5
> -fieldsplit_2_pc_fieldsplit_0_fields 6,7
> -fieldsplit_2_pc_fieldsplit_1_fields 8
>
> should instead be these:
>
> -fieldsplit_1_pc_fieldsplit_0_fields 0,1
> -fieldsplit_1_pc_fieldsplit_1_fields 2
> -fieldsplit_2_pc_fieldsplit_0_fields 0,1
> -fieldsplit_2_pc_fieldsplit_1_fields 2
>
> So for a coupled two domain problem with three fields total: velocity,
> pressure, and temperature where I'd want to schur complement the velocity
> and pressure, I would use these options:
>
> -ksp_monitor_true_residual
> -ksp_view
> -ksp_type gmres
> -pc_type fieldsplit
> -pc_fieldsplit_0_fields 0,1,2
> -fieldsplit_0_ksp_type preonly
> -fieldsplit_0_pc_type fieldsplit
> -fieldsplit_0_pc_fieldsplit_type additive
> -fieldsplit_0_pc_fieldsplit_0_fields 0,1
> -fieldsplit_0_fieldsplit_0_pc_type fieldsplit
> -fieldsplit_0_fieldsplit_0_pc_fieldsplit_type schur
> -fieldsplit_0_fieldsplit_0_pc_fieldsplit_fact_type full
> -fieldsplit_0_fieldsplit_0_pc_fieldsplit_schur_precondition selfp
> -fieldsplit_0_fieldsplit_0_fieldsplit_v1_ksp_type preonly
> -fieldsplit_0_fieldsplit_0_fieldsplit_p1_ksp_type preonly
> -fieldsplit_0_fieldsplit_0_fieldsplit_v1_pc_type bjacobi
> -fieldsplit_0_fieldsplit_0_fieldsplit_p1_pc_type hypre
> -fieldsplit_0_pc_fieldsplit_1_fields 2
> -fieldsplit_0_fieldsplit_t1_ksp_type preonly
> -fieldsplit_0_fieldsplit_t1_pc_type hypre
> -pc_fieldsplit_1_fields 3,4,5
> -fieldsplit_1_ksp_type preonly
> -fieldsplit_1_pc_type fieldsplit
> -fieldsplit_1_pc_fieldsplit_type additive
> -fieldsplit_1_pc_fieldsplit_0_fields 0,1
> -fieldsplit_1_fieldsplit_0_pc_type fieldsplit
> -fieldsplit_1_fieldsplit_0_pc_fieldsplit_type schur
> -fieldsplit_1_fieldsplit_0_pc_fieldsplit_fact_type full
> -fieldsplit_1_fieldsplit_0_pc_fieldsplit_schur_precondition selfp
> -fieldsplit_1_fieldsplit_0_fieldsplit_v2_ksp_type preonly
> -fieldsplit_1_fieldsplit_0_fieldsplit_p2_ksp_type preonly
> -fieldsplit_1_fieldsplit_0_fieldsplit_v2_pc_type bjacobi
> -fieldsplit_1_fieldsplit_0_fieldsplit_p2_pc_type hypre
> -fieldsplit_1_pc_fieldsplit_1_fields 2
> -fieldsplit_1_fieldsplit_t2_ksp_type preonly
> -fieldsplit_1_fieldsplit_t2_pc_type hypre
>
> Attached is the output of ksp_view
>
> On Wed, Jan 2, 2019 at 3:39 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Wed, Jan 2, 2019 at 4:55 PM Justin Chang <jychang48 at gmail.com> wrote:
>>
>>> Okay that's fine.
>>>
>>> Now consider one more case:
>>>
>>> Suppose I create a PetscSection with 9 fields:
>>>
>>> section = PETSc.Section().create()
>>> section.setNumFields(9)
>>> section.setFieldName(0,'u1')
>>> section.setFieldName(1,'c1')
>>> section.setFieldName(2,'t1')
>>> section.setFieldName(3,'u2')
>>> section.setFieldName(4,'c2')
>>> section.setFieldName(5,'t2')
>>> section.setFieldName(6,'u3')
>>> section.setFieldName(7,'c3')
>>> section.setFieldName(8,'t3')
>>> section.setFieldComponents(0,1)
>>> section.setFieldComponents(1,1)
>>> section.setFieldComponents(2,1)
>>> section.setFieldComponents(3,1)
>>> section.setFieldComponents(4,1)
>>> section.setFieldComponents(5,1)
>>> section.setFieldComponents(6,1)
>>> section.setFieldComponents(7,1)
>>> section.setFieldComponents(8,1)
>>>
>>> ....
>>>
>>> where u is potential, c is concentration, and t is temperature. The
>>> numbers refer to different domains of a battery (anode, electrolyte,
>>> cathode).
>>>
>>> I want three levels of splits:
>>> 1) Split by domain
>>> 2) Split potential and concentration from temperature.
>>> 3) Split potential and concentration for schur complementing purpose.
>>>
>>> Do the following PETSc command line options describe the split I
>>> mentioned above:
>>>
>>> -pc_type fieldsplit
>>> -pc_fieldsplit_0_fields 0,1,2
>>> -pc_fieldsplit_1_fields 3,4,5
>>> -pc_fieldsplit_2_fields 6,7,8
>>> -fieldsplit_0_pc_type fieldsplit
>>> -fieldsplit_1_pc_type fieldsplit
>>> -fieldsplit_2_pc_type fieldsplit
>>> -fieldsplit_0_pc_fieldsplit_0_fields 0,1
>>> -fieldsplit_0_pc_fieldsplit_1_fields 2
>>> -fieldsplit_1_pc_fieldsplit_0_fields 3,4
>>> -fieldsplit_1_pc_fieldsplit_1_fields 5
>>> -fieldsplit_2_pc_fieldsplit_0_fields 6,7
>>> -fieldsplit_2_pc_fieldsplit_1_fields 8
>>>
>>> I don't have an actual code to test these out with yet as I've never
>>> done anything beyond two levels of split. So I am wondering if the above
>>> will break anything as far as you can tell.
>>>
>>
>> This should work as you expect. Send me the -ksp_view output when you run
>> it. It would be very cool to see 3-level in action.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Thanks,
>>> Justin
>>>
>>> On Tue, Jan 1, 2019 at 7:10 AM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Tue, Jan 1, 2019 at 5:06 AM Justin Chang <jychang48 at gmail.com>
>>>> wrote:
>>>>
>>>>> Okay so I managed to successfully do this by translating the IS's into
>>>>> a PetscSection, assigning it to an empty DMShell, and then assigning that
>>>>> to KSP.
>>>>>
>>>>> However, I seem to be unable to rename the outer fields that aggregate
>>>>> two or more inner fields.
>>>>>
>>>>
>>>> Right now you cannot do that because of the way that I check for the
>>>> option:
>>>>
>>>>
>>>> https://bitbucket.org/petsc/petsc/src/0a5f382fff62a328ec919e3e9fe959e6cbbcf413/src/ksp/pc/impls/fieldsplit/fieldsplit.c#lines-369
>>>>
>>>> I guess we could replace this by some search code that looked for any
>>>> option of that form and then read
>>>> out the splitname using scanf. Right now, I do not see a pattern search
>>>> function for the options database.
>>>>
>>>>   Thanks,
>>>>
>>>>     Matt
>>>>
>>>>
>>>>> Consider this snippet of a four-field dual porosity/permeability
>>>>> FEniCS/petsc4py code:
>>>>>
>>>>> ...
>>>>>
>>>>> ## Extract FEniCS dof layout, global indices ##
>>>>> dof_total = np.array(W.dofmap().dofs())
>>>>> dof_v1 = np.array(W.sub(0).dofmap().dofs())
>>>>> dof_p1 = np.array(W.sub(1).dofmap().dofs())
>>>>> dof_v2 = np.array(W.sub(2).dofmap().dofs())
>>>>> dof_p2 = np.array(W.sub(3).dofmap().dofs())
>>>>> offset = np.min(dof_total)
>>>>>
>>>>> ## Create PetscSection ##
>>>>> section = PETSc.Section().create()
>>>>> section.setNumFields(4)
>>>>> section.setFieldName(0,'v1')
>>>>> section.setFieldName(1,'p1')
>>>>> section.setFieldName(2,'v2')
>>>>> section.setFieldName(3,'p2')
>>>>> section.setFieldComponents(0,1)
>>>>> section.setFieldComponents(1,1)
>>>>> section.setFieldComponents(2,1)
>>>>> section.setFieldComponents(3,1)
>>>>> section.setChart(0,len(dof_total))
>>>>> for i in np.nditer(dof_v1):
>>>>>   section.setDof(i-offset,1)
>>>>>   section.setFieldDof(i-offset,0,1)
>>>>> for i in np.nditer(dof_p1):
>>>>>   section.setDof(i-offset,1)
>>>>>   section.setFieldDof(i-offset,1,1)
>>>>> for i in np.nditer(dof_v2):
>>>>>   section.setDof(i-offset,1)
>>>>>   section.setFieldDof(i-offset,2,1)
>>>>> for i in np.nditer(dof_p2):
>>>>>   section.setDof(i-offset,1)
>>>>>   section.setFieldDof(i-offset,3,1)
>>>>> section.setUp()
>>>>>
>>>>> ## Create DM and assign PetscSection ##
>>>>> dm = PETSc.DMShell().create()
>>>>> dm.setDefaultSection(section)
>>>>> dm.setUp()
>>>>>
>>>>> ## Create KSP and assign DM ##
>>>>> ksp = PETSc.KSP().create()
>>>>> ksp.setDM(dm)
>>>>> ksp.setDMActive(False)
>>>>>
>>>>> ### PETSc Command-line options ##
>>>>> PETScOptions.set('ksp_monitor_true_residual')
>>>>> PETScOptions.set('ksp_view')
>>>>> PETScOptions.set('ksp_type','gmres')
>>>>> PETScOptions.set('pc_type','fieldsplit')
>>>>> PETScOptions.set('pc_fieldsplit_0_fields','0,1')
>>>>> PETScOptions.set('pc_fieldsplit_1_fields','2,3')
>>>>> PETScOptions.set('pc_fieldsplit_type','additive')
>>>>> PETScOptions.set('fieldsplit_0_ksp_type','preonly')
>>>>> PETScOptions.set('fieldsplit_0_pc_type','fieldsplit')
>>>>> PETScOptions.set('fieldsplit_0_pc_fieldsplit_type','schur')
>>>>> PETScOptions.set('fieldsplit_0_pc_fieldsplit_schur_fact_type','full')
>>>>>
>>>>> PETScOptions.set('fieldsplit_0_pc_fieldsplit_schur_precondition','selfp')
>>>>> PETScOptions.set('fieldsplit_0_fieldsplit_v1_ksp_type','preonly')
>>>>> PETScOptions.set('fieldsplit_0_fieldsplit_v1_pc_type','bjacobi')
>>>>> PETScOptions.set('fieldsplit_0_fieldsplit_p1_ksp_type','preonly')
>>>>> PETScOptions.set('fieldsplit_0_fieldsplit_p1_pc_type','hypre')
>>>>> PETScOptions.set('fieldsplit_1_ksp_type','preonly')
>>>>> PETScOptions.set('fieldsplit_1_pc_type','fieldsplit')
>>>>> PETScOptions.set('fieldsplit_1_pc_fieldsplit_type','schur')
>>>>> PETScOptions.set('fieldsplit_1_pc_fieldsplit_schur_fact_type','full')
>>>>>
>>>>> PETScOptions.set('fieldsplit_1_pc_fieldsplit_schur_precondition','selfp')
>>>>> PETScOptions.set('fieldsplit_1_fieldsplit_v2_ksp_type','preonly')
>>>>> PETScOptions.set('fieldsplit_1_fieldsplit_v2_pc_type','bjacobi')
>>>>> PETScOptions.set('fieldsplit_1_fieldsplit_p2_ksp_type','preonly')
>>>>> PETScOptions.set('fieldsplit_1_fieldsplit_p2_pc_type','hypre')
>>>>>
>>>>> ...
>>>>>
>>>>> Is it possible to rename the outer splits (aka make
>>>>> '-pc_fieldsplit_0_fields 0,1' be something like
>>>>> '-pc_fieldsplit_micro_fields 0,1')?
>>>>>
>>>>> Thanks,
>>>>> Justin
>>>>>
>>>>>
>>>>> On Mon, Dec 31, 2018 at 7:40 AM Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Mon, Dec 31, 2018 at 2:40 AM Justin Chang via petsc-users <
>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I am solving a six field battery problem (concentration and
>>>>>>> potential for each of the two solid and one electrolyte domains) and I want
>>>>>>> to experiment with nested/recursice fieldsplitting. I have the IS's and am
>>>>>>> able to use these to define my PCFieldSplitsSetIS(). However, I can imagine
>>>>>>> this getting really messy from a programming standpoint, especially once I
>>>>>>> need to add temperature into the mix, so it is my hope that I can translate
>>>>>>> these index sets and fields into a DM (maybe DMShell?) so that I can just
>>>>>>> rely on command line options to play around with various combinations of
>>>>>>> field assignments and splits (e.g. -pc_fieldsplit_X_fields)
>>>>>>>
>>>>>>> However, it doesn't seem clear to me how I would create a DM when
>>>>>>> you already know the IS's for each fields. If I understand functions like
>>>>>>> DMCreateFieldDecomposition() correctly, it seems that it returns to you the
>>>>>>> IS's and sub DM's associated with the original DM, whereas I want to do it
>>>>>>> the other way around. Perhaps the "reversal" of something like
>>>>>>> DMCreateFieldIS()
>>>>>>> <https://www.mcs.anl.gov/petsc/petsc-dev/src/dm/interface/dm.c.html#DMCreateFieldIS>,
>>>>>>> where you convert the IS into a PetscSection or is there an easier/better
>>>>>>> way?
>>>>>>>
>>>>>>> Any thoughts/help appreciated!
>>>>>>>
>>>>>>
>>>>>> Paul has recently done this for LibMesh. I believe that constructing
>>>>>> a PetscSection is enough to get you minimally started. That allows
>>>>>> DMCreateSubDM() to work by subsetting the Section, and that should
>>>>>> allow the command line to work. CreateFieldDecomposition() should
>>>>>> be removed I think.
>>>>>>
>>>>>>   Thanks,
>>>>>>
>>>>>>     Matt
>>>>>>
>>>>>>
>>>>>>> Justin
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190103/f8b2ac02/attachment.html>

More information about the petsc-users mailing list