[petsc-users] DMPlexDistributeField
Adrian Croucher
a.croucher at auckland.ac.nz
Mon Jun 24 18:22:13 CDT 2019
hi
Thanks Matt for the explanation about this.
I have been trying a test which does the following:
1) read in DMPlex from file
2) distribute it, with overlap = 1, using DMPlexDistribute()
3) create FVM cell and face geometry vectors using
DMPlexComputeGeometryFVM()
4) re-distribute, again with overlap = 1, using DMPlexDistribute()
5) distribute the cell and face geometry vectors using
DMPlexDistributeField()
Steps 4) and 5) should do essentially nothing, because the mesh has
already been distributed (but in my actual non-test code, there is
additional stuff between steps 3) and 4) where dual porosity cells are
added to the DM).
So I expect the cell and face geometry vectors to be essentially
unchanged from the redistribution. And the redistribution SF (from the
second distribution) should be just an identity mapping on the cell and
face points (except for the overlap ghost points).
This is true for the cells, but not the faces. I've attached the example
code and mesh. It is a simple mesh with 10 cells in a horizontal line,
each cell 50x50x50 m.
If I run on 2 processes, there are 5 cells (points 0 - 4) on each rank,
with centroids at 25, 75, 125, 175 and 225 m on rank 0, and 275, 325,
375, 425 and 475 m on rank 1. The internal faces are the points 36, 42,
47 and 52 on rank 0, and 34, 37, 42, 47 and 52 on rank 1. On rank 0
these should have centroids at 50, 100, 150 and 200 m respectively; on
rank 1 they should be at 250, 300, 350 and 400 m. This is true before
redistribution.
After redistribution, the cells centroids are still correct, and the
face data on rank 1 are OK, but the face data on rank 0 are all wrong.
If you look at the redistribution SF the entries for the rank 0 face
data are 36 <- (0,40), 42 <- (0,46), 47 <- (0,51), 52 <- (0,56), instead
of the expected 36 <- (0,36), 42 <- (0,42), 47 <- (0,47), 52 <- (0,52).
The SF for the rank 1 faces is OK.
If you change the overlap from 1 to 0, it works as expected. So it looks
to me like something isn't quite right with the SF for faces when there
is overlap. On rank 0 all the entries seem to be shifted up by 4.
I know you originally recommended using overlap = 0 for the initial
distribution and only adding overlap for the redistribution. But then
Stefano indicated that it should work with overlap now. And it would
simplify my code if I could use overlap for the initial distribution
(because if dual porosity cells are not being used, then there is no
second redistribution).
Is this a bug or is there something I'm doing wrong?
- Adrian
On 23/06/19 4:39 PM, Matthew Knepley wrote:
> On Fri, Jun 21, 2019 at 12:49 AM Adrian Croucher via petsc-users
> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>
> I have been trying to get this FVM geometry data re-distribution
> to work
> using DMPlexDistributeField().
>
> It seems to be working OK for the cell geometry data (cell volumes
> and
> centroids). But it is making a mess of the face geometry data (face
> normals and centroids).
>
> Should I even expect DMPlexDistributeField() to work for
> redistributing
> a vector of data defined on mesh faces? Or is there some reason I
> haven't thought of, which means that will never work?
>
>
> Sorry this took a long time. The place I was in France did not have
> internet.
> Here is how this stuff works:
>
> 1) You start with a Section and local vector. The Section describes
> layout of data in
> the local vector by mapping mesh points to {# dof, offset}. For
> a small example,
> suppose I had two triangles sharing an edge on a sequential
> mesh for 2 procs.
> The mesh points would be
>
> [0, 1]: Cells
> [2, 5]: Vertices, where 3,4 are shared
> [6, 10]: Edges, where 8 is shared
>
> A Section for face normals would then look like
>
> Process 0
> [0, 5]: {0, 0} Meaning no variables lie on cells or vertices
> 6: {2, 0} One vector per face
> 7: {2, 2}
> 8: {2, 4}
> 9: {2, 6}
> 10 {2, 8}
> Process 1
> empty
>
> The vector would just have the face normal values in the
> canonical order. You can use
> PetscSectionView() to check that yours looks similar.
>
> 2) Now we add a PetscSF describing the redistribution. An SF is a
> map from a given set of
> integers (leaves) to pairs (int, rank) called roots. Many leaves
> can point to one root. To begin,
> we provide an SF mapping mesh points to the new distribution
>
> Process 0
> 0 -> {0, 0}
> 1 -> {2, 0}
> 2 -> {3, 0}
> 3 -> {4, 0}
> 4 -> {6, 0}
> 5 -> {7, 0}
> 6 -> {8, 0}
> Process 1
> 0 -> {1, 0}
> 1 -> {4, 0}
> 2 -> {3, 0}
> 3 -> {5, 0}
> 4 -> {8, 0}
> 5 -> {9, 0}
> 6 -> {10, 0}
>
> This tells Petsc how to move the points to the parallel
> distribution. You can use PetscSFView() to check
> that your point SF is similar. We first use the SF to move the
> Section data to make a new parallel Section,
>
> Process 0
> [0, 3]: {0, 0}
> 4: {2, 0}
> 5: {2, 2}
> 6: {2, 4}
> Process 1
> [0, 3]: {0, 0}
> 4: {2, 0}
> 5: {2, 2}
> 6: {2, 4}
>
> and then we create a new SF that maps dofs, instead of points,
> using this new Section
>
> Process 0:
> 0 -> {0, 0} First face normal
> 1 -> {1, 0}
> 2 -> {2, 0} Second face normal
> 3 -> {3, 0}
> 4 -> {4, 0} Third face normal
> 5 -> {5, 0}
> Process 1:
> 0 -> {4, 0} First face normal
> 1 -> {5, 0}
> 2 -> {6, 0} Second face normal
> 3 -> {7, 0}
> 4 -> {8, 0} Third face normal
> 5 -> {9, 0}
>
> and then this moves the face normal data to the new layout just by
> broadcasting.
>
> You can View the structures you get back to see what Petsc thought it
> should do.
>
> Does this make sense?
>
> Thanks,
>
> Matt
>
> The only example I could find anywhere of DMPlexDistributeField()
> being
> used is in DMPlexDistributeCoordinates() so I've been basing what I'm
> doing on that.
>
> (I think I have answered my own questions below by experiment- 1)
> local
> vectors should work (they do in DMPlexDistributeCoordinates); 2)
> probably doesn't matter; 3) yes.)
>
> - Adrian
>
> On 6/06/19 1:42 PM, Adrian Croucher wrote:
> > hi
> >
> > I have some questions about using the DMPlexDistributeField()
> > function. I have finite volume mesh geometry data stored in two
> local
> > vectors created using DMPlexComputeGeometryFVM(), and I need to
> > redistribute these after calling DMPlexDistribute() to
> redistribute my
> > mesh. (I need the geometry data before redistribution, so I
> can't just
> > wait until after redistribution to create them.)
> >
> > So I figured DMPlexDistributeField() looks like the thing to use
> for
> > that, using the SF that comes out of DMPlexDistribute().
> >
> > 1) Does DMPlexDistributeField() work on local vectors or do they
> have
> > to be global ones?
> >
> > 2) It takes a 'dm' parameter and the documentation says this is
> "The
> > DMPlex object", but is that the original DM (before
> redistribution) or
> > the redistributed one, or does it not matter? It looks like it only
> > uses the DM to get the vector type.
> >
> > 3) It looks like you need to manually create the newSection and
> newVec
> > output parameters before passing them in to this routine, is that
> > correct?
> >
> > - Adrian
> >
> --
> Dr Adrian Croucher
> Senior Research Fellow
> Department of Engineering Science
> University of Auckland, New Zealand
> email: a.croucher at auckland.ac.nz <mailto:a.croucher at auckland.ac.nz>
> tel: +64 (0)9 923 4611
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
--
Dr Adrian Croucher
Senior Research Fellow
Department of Engineering Science
University of Auckland, New Zealand
email: a.croucher at auckland.ac.nz
tel: +64 (0)9 923 4611
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190625/5b86a727/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: distgeom.F90
Type: text/x-fortran
Size: 4649 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190625/5b86a727/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gminc_1d.exo
Type: application/octet-stream
Size: 21882 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190625/5b86a727/attachment-0001.obj>
More information about the petsc-users
mailing list