[petsc-dev] [petsc4py] Broken tests for DMStag

[Patrick Sanan]

Sorry about the delay on this - I'm still trying to decide on what's an
elegant solution. I'll attempt to describe the issue and potential
solutions, in case anyone has interest/ideas:

*My understanding of the way things work now*
- DMLocalToGlobal() sends data from "local" vectors to "global" vectors.
- This "l2g" map is in general surjective (onto) but not injective (into)
because local subdomains can overlap.
- When DMLocalToGlobal() is called with INSERT_VALUES, there needs to be
some mechanism to resolve this non-injectivity.
- The mechanism is to ignore the parts of the l2g map that aren't
rank-local.
- This usually works very well because "local" vectors can usually be
constructed to consist of degrees of freedom consisting of rank-local dof
in the global representation (I like to call these "native"), plus
additional dof corresponding to other dof, which are either on other ranks
or not part of the global representation at all (ghosted boundaries).
- This breaks down for a DMDA with a single rank in a given coordinate
direction, and periodic boundary conditions. You'll get an error trying due
to the non-injectivity of the map, even after excluding off-rank
communication (since the ghost region maps to native dof, which never
happens otherwise)
- However, this isn't an issue in practice, because with DMDA you can just
operate directly on a global vector. Local-to-global inserts aren't very
useful in general.

*Issue related to DMStag*
- With DMStag, the same issue occurs as with DMDA (and this is the topic of
this thread).
- However, operating directly on a global vector isn't currently convenient
with DMStag, because global vectors aren't regularly blocked, while local
vectors are.
- This is perhaps a design flaw to be addressed with DMStag, because the
local representation is being used for two orthogonal purposes: padding out
to obtain regular blocking, and padding out to include ghost regions.

*Potential solutions*
I can think of a few avenues to pursue, including
1. Whenever this could potentially arise, pad all local vectors with the
necessary 0's and use ADD_VALUES [gross]
2. Add a dedicated "padded global" representation for DMStag, and tools for
working with it. [elegant in that it resolves the design flaw above, but
adds quite a lot of code]

but here's the idea  that I'm also intrigued by:
- It's an abuse of notation to use INSERT_VALUES for DMLocalToGlobal() in
the first place
- Rather, it'd be more meaningful to use a new InsertMode value,
INSERT_VALUES_LOCAL, deprecating the use of INSERT_VALUES for
DMLocalToGlobal()*
- This name corresponds to SCATTER_REVERSE_LOCAL which gets called in e.g.
DMLocalToGlobalBegin_DA(), or the equivalent logic in
DMLocalToGlobalBegin() using PetscSection
- In most cases, this works just as before, but now the error message is
more accurate - you can't use INSERT_VALUES_LOCAL when the l2g map isn't
injective, even when ignoring parallel communication
- Add another new InsertMode value, INSERT_VALUES_INJECTIVE, which asks the
DM implementation to resolve the non-injectivity of the l2g map
- In most cases, this would behave just like INSERT_VALUES_LOCAL and most
users wouldn't know or care
- However, for the case of interest here, the implementation would be free
to use its implementation-dependent data to "do the right thing"
- For DMStag, the right thing would be to behave consistently with the case
with 2 or more ranks in the periodic direction (ignoring dof on ghost
elements).
- This could be done by setting up another VecScatter during
DMSetUp_Stag(), and using it with INSERT_VALUES_INJECTIVE

* eventually this could become an error if the l2g map was non-injective,
but probably not for a long time if ever

Am Do., 13. Juni 2019 um 15:09 Uhr schrieb Patrick Sanan <
patrick.sanan at gmail.com>:

> I will attempt to just fix the underlying issue here, that is allow
> DMStagSetUniformCoordinates() to work for period boundary conditions with a
> single rank in the corresponding dimension.
>
> Am Mi., 12. Juni 2019 um 19:10 Uhr schrieb Patrick Sanan <
> patrick.sanan at gmail.com>:
>
>> Ah, okay, so unless something weird is going on, that probably should
>> never have passed (if it's asking for periodic BCs with one rank in a given
>> direction, and INSERT_VALUES, so it's not well-defined which of the
>> multiple local dof mapping to a given global dof should be used). I'll try
>> to reproduce and fix it and/or talk to Chris.
>>
>> Am Mi., 12. Juni 2019 um 18:27 Uhr schrieb Balay, Satish via petsc-dev <
>> petsc-dev at mcs.anl.gov>:
>>
>>> On Wed, 12 Jun 2019, Lisandro Dalcin via petsc-dev wrote:
>>>
>>> > $ python test/runtests.py -v -i dmstag TestDMStag_2D_PXY
>>> > [0 at kl-18232] Python 2.7 (/usr/local/opt/python at 2/bin/python2.7)
>>> > [0 at kl-18232] PETSc 3.11.2 development (conf: 'arch-darwin-c-debug')
>>> > [0 at kl-18232] petsc4py 3.11.0
>>> (build/lib.macosx-10.14-x86_64-2.7/petsc4py)
>>> > testCoordinates (test_dmstag.TestDMStag_2D_PXY) ... ERROR
>>> > testDMDAInterface (test_dmstag.TestDMStag_2D_PXY) ... ERROR
>>> > testDof (test_dmstag.TestDMStag_2D_PXY) ... ok
>>> > testGetOther (test_dmstag.TestDMStag_2D_PXY) ... ok
>>> > testGetVec (test_dmstag.TestDMStag_2D_PXY) ... ERROR
>>> > testMigrateVec (test_dmstag.TestDMStag_2D_PXY) ... ERROR
>>>
>>> I get:
>>>
>>> ======================================================================
>>> ERROR: testCoordinates (test_dmstag.TestDMStag_2D_PXY)
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>>   File "test/test_dmstag.py", line 39, in testCoordinates
>>>     self.da.setUniformCoordinates(0,1,0,1,0,1)
>>>   File "PETSc/DMStag.pyx", line 255, in
>>> petsc4py.PETSc.DMStag.setUniformCoordinates
>>> Error: error code 56
>>> [0] DMStagSetUniformCoordinates() line 1077 in
>>> /home/balay/petsc.z/src/dm/impls/stag/stagutils.c
>>> [0] DMStagSetUniformCoordinatesExplicit() line 1118 in
>>> /home/balay/petsc.z/src/dm/impls/stag/stagutils.c
>>> [0] DMStagSetUniformCoordinatesExplicit_2d() line 135 in
>>> /home/balay/petsc.z/src/dm/impls/stag/stag2d.c
>>> [0] DMLocalToGlobalBegin() line 2614 in
>>> /home/balay/petsc.z/src/dm/interface/dm.c
>>> [0] DMLocalToGlobalBegin_Stag() line 230 in
>>> /home/balay/petsc.z/src/dm/impls/stag/stag.c
>>> [0] No support for this operation for this object type
>>> [0] Local to Global scattering with INSERT_VALUES is not supported for
>>> single rank in a direction with boundary conditions (e.g. periodic)
>>> inducing a non-injective local->global map. Either change the boundary
>>> conditions, use a stencil width of zero, or use more than one rank in the
>>> relevant direction (e.g. -stag_ranks_x 2)
>>>
>>> Satish
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20190618/67747ffe/attachment-0001.html>