[petsc-dev] [petsc4py] Broken tests for DMStag

[Patrick Sanan]

Am Di., 18. Juni 2019 um 20:30 Uhr schrieb Smith, Barry F. <
bsmith at mcs.anl.gov>:

>
>   Patrick,
>
>     I understand the problem with DMDA and periodicity with only a single
> skinny processor in one direction, local to global won't work without lots
> of additional code. I don't care because PETSc is for big problems and that
> problem is not big.


Agreed, as far as performance is concerned, but I also very much value
being able to run and debug small problems, in serial at first, on my
laptop. This skinny-periodic case doesn't matter much for DMDA, because you
don't really need to do a lot of local->global scatters with INSERT_VALUES
(just access the global vector directly). The way DMStag is right now, I
use this to construct coordinates and to do data migration, because of the
regular blocking of the local vectors.

>
>     Could you please explain to me the problem with Stag unrelated to a
> skinny domain with one processor and periodic boundary conditions?


>     Each unknown in the global vector presumably has an unknown on the
> local vector that "lives" at the same point. For insert why can't the local
> to global just use the value from that location? Meanwhile for ADD_VALUES
> why can't you just use the reverse of the global to local?
>
I don't think there's a way to know (from the point of view of the
VecScatter) which of several local unknowns is the right one to map to a
given global unknown. Usually you saying "it's the one coming from this
rank", but that breaks down for the skinny periodic case.


>     If the problem is just with skinny domain periodic one process in that
> direction just generate the error and remove any test cases for that
> situation. It seems absurd to me to waste time worrying about a totally
> irrelevant special case and introducing all kinds of special constructs for
> it is just making a mountain out of a mole hill.
>
Potential users have noticed and complained, so I need to fix it somehow :
( Sounds like playing with higher-level DM stuff isn't worth the churn, so
I should focus on solutions internal to DMStag.


>    Barry
>
>
> > On Jun 18, 2019, at 10:51 AM, Patrick Sanan via petsc-dev <
> petsc-dev at mcs.anl.gov> wrote:
> >
> > Sorry about the delay on this - I'm still trying to decide on what's an
> elegant solution. I'll attempt to describe the issue and potential
> solutions, in case anyone has interest/ideas:
> >
> > My understanding of the way things work now
> > - DMLocalToGlobal() sends data from "local" vectors to "global" vectors.
> > - This "l2g" map is in general surjective (onto) but not injective
> (into) because local subdomains can overlap.
> > - When DMLocalToGlobal() is called with INSERT_VALUES, there needs to be
> some mechanism to resolve this non-injectivity.
> > - The mechanism is to ignore the parts of the l2g map that aren't
> rank-local.
> > - This usually works very well because "local" vectors can usually be
> constructed to consist of degrees of freedom consisting of rank-local dof
> in the global representation (I like to call these "native"), plus
> additional dof corresponding to other dof, which are either on other ranks
> or not part of the global representation at all (ghosted boundaries).
> > - This breaks down for a DMDA with a single rank in a given coordinate
> direction, and periodic boundary conditions. You'll get an error trying due
> to the non-injectivity of the map, even after excluding off-rank
> communication (since the ghost region maps to native dof, which never
> happens otherwise)
> > - However, this isn't an issue in practice, because with DMDA you can
> just operate directly on a global vector. Local-to-global inserts aren't
> very useful in general.
> >
> > Issue related to DMStag
> > - With DMStag, the same issue occurs as with DMDA (and this is the topic
> of this thread).
> > - However, operating directly on a global vector isn't currently
> convenient with DMStag, because global vectors aren't regularly blocked,
> while local vectors are.
> > - This is perhaps a design flaw to be addressed with DMStag, because the
> local representation is being used for two orthogonal purposes: padding out
> to obtain regular blocking, and padding out to include ghost regions.
> >
> > Potential solutions
> > I can think of a few avenues to pursue, including
> > 1. Whenever this could potentially arise, pad all local vectors with the
> necessary 0's and use ADD_VALUES [gross]
> > 2. Add a dedicated "padded global" representation for DMStag, and tools
> for working with it. [elegant in that it resolves the design flaw above,
> but adds quite a lot of code]
> >
> > but here's the idea  that I'm also intrigued by:
> > - It's an abuse of notation to use INSERT_VALUES for DMLocalToGlobal()
> in the first place
> > - Rather, it'd be more meaningful to use a new InsertMode value,
> INSERT_VALUES_LOCAL, deprecating the use of INSERT_VALUES for
> DMLocalToGlobal()*
> > - This name corresponds to SCATTER_REVERSE_LOCAL which gets called in
> e.g. DMLocalToGlobalBegin_DA(), or the equivalent logic in
> DMLocalToGlobalBegin() using PetscSection
> > - In most cases, this works just as before, but now the error message is
> more accurate - you can't use INSERT_VALUES_LOCAL when the l2g map isn't
> injective, even when ignoring parallel communication
> > - Add another new InsertMode value, INSERT_VALUES_INJECTIVE, which asks
> the DM implementation to resolve the non-injectivity of the l2g map
> > - In most cases, this would behave just like INSERT_VALUES_LOCAL and
> most users wouldn't know or care
> > - However, for the case of interest here, the implementation would be
> free to use its implementation-dependent data to "do the right thing"
> > - For DMStag, the right thing would be to behave consistently with the
> case with 2 or more ranks in the periodic direction (ignoring dof on ghost
> elements).
> > - This could be done by setting up another VecScatter during
> DMSetUp_Stag(), and using it with INSERT_VALUES_INJECTIVE
> >
> > * eventually this could become an error if the l2g map was
> non-injective, but probably not for a long time if ever
> >
> > Am Do., 13. Juni 2019 um 15:09 Uhr schrieb Patrick Sanan <
> patrick.sanan at gmail.com>:
> > I will attempt to just fix the underlying issue here, that is allow
> DMStagSetUniformCoordinates() to work for period boundary conditions with a
> single rank in the corresponding dimension.
> >
> > Am Mi., 12. Juni 2019 um 19:10 Uhr schrieb Patrick Sanan <
> patrick.sanan at gmail.com>:
> > Ah, okay, so unless something weird is going on, that probably should
> never have passed (if it's asking for periodic BCs with one rank in a given
> direction, and INSERT_VALUES, so it's not well-defined which of the
> multiple local dof mapping to a given global dof should be used). I'll try
> to reproduce and fix it and/or talk to Chris.
> >
> > Am Mi., 12. Juni 2019 um 18:27 Uhr schrieb Balay, Satish via petsc-dev <
> petsc-dev at mcs.anl.gov>:
> > On Wed, 12 Jun 2019, Lisandro Dalcin via petsc-dev wrote:
> >
> > > $ python test/runtests.py -v -i dmstag TestDMStag_2D_PXY
> > > [0 at kl-18232] Python 2.7 (/usr/local/opt/python at 2/bin/python2.7)
> > > [0 at kl-18232] PETSc 3.11.2 development (conf: 'arch-darwin-c-debug')
> > > [0 at kl-18232] petsc4py 3.11.0
> (build/lib.macosx-10.14-x86_64-2.7/petsc4py)
> > > testCoordinates (test_dmstag.TestDMStag_2D_PXY) ... ERROR
> > > testDMDAInterface (test_dmstag.TestDMStag_2D_PXY) ... ERROR
> > > testDof (test_dmstag.TestDMStag_2D_PXY) ... ok
> > > testGetOther (test_dmstag.TestDMStag_2D_PXY) ... ok
> > > testGetVec (test_dmstag.TestDMStag_2D_PXY) ... ERROR
> > > testMigrateVec (test_dmstag.TestDMStag_2D_PXY) ... ERROR
> >
> > I get:
> >
> > ======================================================================
> > ERROR: testCoordinates (test_dmstag.TestDMStag_2D_PXY)
> > ----------------------------------------------------------------------
> > Traceback (most recent call last):
> >   File "test/test_dmstag.py", line 39, in testCoordinates
> >     self.da.setUniformCoordinates(0,1,0,1,0,1)
> >   File "PETSc/DMStag.pyx", line 255, in
> petsc4py.PETSc.DMStag.setUniformCoordinates
> > Error: error code 56
> > [0] DMStagSetUniformCoordinates() line 1077 in
> /home/balay/petsc.z/src/dm/impls/stag/stagutils.c
> > [0] DMStagSetUniformCoordinatesExplicit() line 1118 in
> /home/balay/petsc.z/src/dm/impls/stag/stagutils.c
> > [0] DMStagSetUniformCoordinatesExplicit_2d() line 135 in
> /home/balay/petsc.z/src/dm/impls/stag/stag2d.c
> > [0] DMLocalToGlobalBegin() line 2614 in
> /home/balay/petsc.z/src/dm/interface/dm.c
> > [0] DMLocalToGlobalBegin_Stag() line 230 in
> /home/balay/petsc.z/src/dm/impls/stag/stag.c
> > [0] No support for this operation for this object type
> > [0] Local to Global scattering with INSERT_VALUES is not supported for
> single rank in a direction with boundary conditions (e.g. periodic)
> inducing a non-injective local->global map. Either change the boundary
> conditions, use a stencil width of zero, or use more than one rank in the
> relevant direction (e.g. -stag_ranks_x 2)
> >
> > Satish
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20190618/91185b68/attachment.html>