<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>hi</p>
<p>Thanks Matt for the explanation about this.</p>
<p>I have been trying a test which does the following:</p>
<p>1) read in DMPlex from file</p>
<p>2) distribute it, with overlap = 1, using DMPlexDistribute()<br>
</p>
<p>3) create FVM cell and face geometry vectors using
DMPlexComputeGeometryFVM()</p>
<p>4) re-distribute, again with overlap = 1, using
DMPlexDistribute()</p>
<p>5) distribute the cell and face geometry vectors using
DMPlexDistributeField()</p>
<p><br>
</p>
<p>Steps 4) and 5) should do essentially nothing, because the mesh
has already been distributed (but in my actual non-test code,
there is additional stuff between steps 3) and 4) where dual
porosity cells are added to the DM).</p>
<p>So I expect the cell and face geometry vectors to be essentially
unchanged from the redistribution. And the redistribution SF (from
the second distribution) should be just an identity mapping on the
cell and face points (except for the overlap ghost points).</p>
<p>This is true for the cells, but not the faces. I've attached the
example code and mesh. It is a simple mesh with 10 cells in a
horizontal line, each cell 50x50x50 m.</p>
<p>If I run on 2 processes, there are 5 cells (points 0 - 4) on each
rank, with centroids at 25, 75, 125, 175 and 225 m on rank 0, and
275, 325, 375, 425 and 475 m on rank 1. The internal faces are the
points 36, 42, 47 and 52 on rank 0, and 34, 37, 42, 47 and 52 on
rank 1. On rank 0 these should have centroids at 50, 100, 150 and
200 m respectively; on rank 1 they should be at 250, 300, 350 and
400 m. This is true before redistribution.</p>
<p>After redistribution, the cells centroids are still correct, and
the face data on rank 1 are OK, but the face data on rank 0 are
all wrong.</p>
<p>If you look at the redistribution SF the entries for the rank 0
face data are 36 <- (0,40), 42 <- (0,46), 47 <- (0,51),
52 <- (0,56), instead of the expected 36 <- (0,36), 42 <-
(0,42), 47 <- (0,47), 52 <- (0,52). The SF for the rank 1
faces is OK.</p>
<p>If you change the overlap from 1 to 0, it works as expected. So
it looks to me like something isn't quite right with the SF for
faces when there is overlap. On rank 0 all the entries seem to be
shifted up by 4.<br>
</p>
<p>I know you originally recommended using overlap = 0 for the
initial distribution and only adding overlap for the
redistribution. But then Stefano indicated that it should work
with overlap now. And it would simplify my code if I could use
overlap for the initial distribution (because if dual porosity
cells are not being used, then there is no second redistribution).</p>
<p>Is this a bug or is there something I'm doing wrong?<br>
</p>
<p>- Adrian<br>
</p>
<div class="moz-cite-prefix">On 23/06/19 4:39 PM, Matthew Knepley
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAMYG4Gmg50fA17NESHZiAC4FNVoBfodwZ-ySUqTfH3LeR=sERA@mail.gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr">On Fri, Jun 21, 2019 at 12:49 AM Adrian Croucher
via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov"
moz-do-not-send="true">petsc-users@mcs.anl.gov</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">I have been trying to get
this FVM geometry data re-distribution to work <br>
using DMPlexDistributeField().<br>
<br>
It seems to be working OK for the cell geometry data (cell
volumes and <br>
centroids). But it is making a mess of the face geometry
data (face <br>
normals and centroids).<br>
<br>
Should I even expect DMPlexDistributeField() to work for
redistributing <br>
a vector of data defined on mesh faces? Or is there some
reason I <br>
haven't thought of, which means that will never work?<br>
</blockquote>
<div><br>
</div>
<div>Sorry this took a long time. The place I was in France
did not have internet.</div>
<div>Here is how this stuff works:<br>
</div>
<div><br>
</div>
<div> 1) You start with a Section and local vector. The
Section describes layout of data in</div>
<div> the local vector by mapping mesh points to {# dof,
offset}. For a small example,</div>
<div> suppose I had two triangles sharing an edge on a
sequential mesh for 2 procs.</div>
<div> The mesh points would be</div>
<div><br>
</div>
<div> [0, 1]: Cells</div>
<div> [2, 5]: Vertices, where 3,4 are shared</div>
<div> [6, 10]: Edges, where 8 is shared</div>
<div><br>
</div>
<div> A Section for face normals would then look like</div>
<div><br>
</div>
<div> Process 0</div>
<div> [0, 5]: {0, 0} Meaning no variables lie on
cells or vertices</div>
<div> 6: {2, 0} One vector per face</div>
<div> 7: {2, 2}</div>
<div> 8: {2, 4}</div>
<div> 9: {2, 6}</div>
<div> 10 {2, 8}</div>
<div> Process 1</div>
<div> empty</div>
<div><br>
</div>
<div> The vector would just have the face normal values in
the canonical order. You can use</div>
<div> PetscSectionView() to check that yours looks
similar.</div>
<div><br>
</div>
<div> 2) Now we add a PetscSF describing the redistribution.
An SF is a map from a given set of</div>
<div> integers (leaves) to pairs (int, rank) called
roots. Many leaves can point to one root. To begin,</div>
<div> we provide an SF mapping mesh points to the new
distribution</div>
<div><br>
</div>
<div> Process 0</div>
<div> 0 -> {0, 0}</div>
<div> 1 -> {2, 0}</div>
<div> 2 -> {3, 0}</div>
<div> 3 -> {4, 0}</div>
<div> 4 -> {6, 0}</div>
<div> 5 -> {7, 0}</div>
<div> 6 -> {8, 0}</div>
<div> Process 1</div>
<div> 0 -> {1, 0}</div>
<div> 1 -> {4, 0}</div>
<div> 2 -> {3, 0}</div>
<div> 3 -> {5, 0}</div>
<div> 4 -> {8, 0}</div>
<div> 5 -> {9, 0}</div>
<div> 6 -> {10, 0}</div>
<div class="gmail_quote"><br>
</div>
This tells Petsc how to move the points to the parallel
distribution. You can use PetscSFView() to check</div>
<div class="gmail_quote"> that your point SF is similar. We
first use the SF to move the Section data to make a new
parallel Section,</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote"> Process 0</div>
<div class="gmail_quote"> [0, 3]: {0, 0}</div>
<div class="gmail_quote"> 4: {2, 0}<br
class="gmail-Apple-interchange-newline">
5: {2, 2}<br
class="gmail-Apple-interchange-newline">
6: {2, 4}</div>
<div class="gmail_quote"> Process 1<br
class="gmail-Apple-interchange-newline">
<div class="gmail_quote"> [0, 3]: {0, 0}</div>
<div class="gmail_quote"> 4: {2, 0}<br
class="gmail-Apple-interchange-newline">
5: {2, 2}<br
class="gmail-Apple-interchange-newline">
6: {2, 4}</div>
<br class="gmail-Apple-interchange-newline">
<div> and then we create a new SF that maps dofs, instead
of points, using this new Section</div>
<div><br>
</div>
<div> Process 0:</div>
<div> 0 -> {0, 0} First face normal</div>
<div> 1 -> {1, 0}</div>
<div> 2 -> {2, 0} Second face normal</div>
<div> 3 -> {3, 0}</div>
<div> 4 -> {4, 0} Third face normal</div>
<div> 5 -> {5, 0}</div>
<div> Process 1:</div>
<div> 0 -> {4, 0} First face normal</div>
<div> 1 -> {5, 0}</div>
<div> 2 -> {6, 0} Second face normal</div>
<div> 3 -> {7, 0}</div>
<div> 4 -> {8, 0} Third face normal</div>
<div> 5 -> {9, 0}</div>
<br class="gmail-Apple-interchange-newline">
and then this moves the face normal data to the new layout
just by broadcasting.</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">You can View the structures you get
back to see what Petsc thought it should do.</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Does this make sense?</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote"> Thanks,</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote"> Matt</div>
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
The only example I could find anywhere of
DMPlexDistributeField() being <br>
used is in DMPlexDistributeCoordinates() so I've been basing
what I'm <br>
doing on that.<br>
<br>
(I think I have answered my own questions below by
experiment- 1) local <br>
vectors should work (they do in
DMPlexDistributeCoordinates); 2) <br>
probably doesn't matter; 3) yes.)<br>
<br>
- Adrian<br>
<br>
On 6/06/19 1:42 PM, Adrian Croucher wrote:<br>
> hi<br>
><br>
> I have some questions about using the
DMPlexDistributeField() <br>
> function. I have finite volume mesh geometry data
stored in two local <br>
> vectors created using DMPlexComputeGeometryFVM(), and I
need to <br>
> redistribute these after calling DMPlexDistribute() to
redistribute my <br>
> mesh. (I need the geometry data before redistribution,
so I can't just <br>
> wait until after redistribution to create them.)<br>
><br>
> So I figured DMPlexDistributeField() looks like the
thing to use for <br>
> that, using the SF that comes out of
DMPlexDistribute().<br>
><br>
> 1) Does DMPlexDistributeField() work on local vectors
or do they have <br>
> to be global ones?<br>
><br>
> 2) It takes a 'dm' parameter and the documentation says
this is "The <br>
> DMPlex object", but is that the original DM (before
redistribution) or <br>
> the redistributed one, or does it not matter? It looks
like it only <br>
> uses the DM to get the vector type.<br>
><br>
> 3) It looks like you need to manually create the
newSection and newVec <br>
> output parameters before passing them in to this
routine, is that <br>
> correct?<br>
><br>
> - Adrian<br>
><br>
-- <br>
Dr Adrian Croucher<br>
Senior Research Fellow<br>
Department of Engineering Science<br>
University of Auckland, New Zealand<br>
email: <a href="mailto:a.croucher@auckland.ac.nz"
target="_blank" moz-do-not-send="true">a.croucher@auckland.ac.nz</a><br>
tel: +64 (0)9 923 4611<br>
<br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before
they begin their experiments is infinitely more
interesting than any results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/"
target="_blank" moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Dr Adrian Croucher
Senior Research Fellow
Department of Engineering Science
University of Auckland, New Zealand
email: <a class="moz-txt-link-abbreviated" href="mailto:a.croucher@auckland.ac.nz">a.croucher@auckland.ac.nz</a>
tel: +64 (0)9 923 4611
</pre>
</body>
</html>