<div dir="ltr"><div dir="ltr">On Fri, Jun 21, 2019 at 12:49 AM Adrian Croucher via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I have been trying to get this FVM geometry data re-distribution to work <br>
using DMPlexDistributeField().<br>
<br>
It seems to be working OK for the cell geometry data (cell volumes and <br>
centroids). But it is making a mess of the face geometry data (face <br>
normals and centroids).<br>
<br>
Should I even expect DMPlexDistributeField() to work for redistributing <br>
a vector of data defined on mesh faces? Or is there some reason I <br>
haven't thought of, which means that will never work?<br></blockquote><div><br></div><div>Sorry this took a long time. The place I was in France did not have internet.</div><div>Here is how this stuff works:<br></div><div><br></div><div> 1) You start with a Section and local vector. The Section describes layout of data in</div><div> the local vector by mapping mesh points to {# dof, offset}. For a small example,</div><div> suppose I had two triangles sharing an edge on a sequential mesh for 2 procs.</div><div> The mesh points would be</div><div><br></div><div> [0, 1]: Cells</div><div> [2, 5]: Vertices, where 3,4 are shared</div><div> [6, 10]: Edges, where 8 is shared</div><div><br></div><div> A Section for face normals would then look like</div><div><br></div><div> Process 0</div><div> [0, 5]: {0, 0} Meaning no variables lie on cells or vertices</div><div> 6: {2, 0} One vector per face</div><div> 7: {2, 2}</div><div> 8: {2, 4}</div><div> 9: {2, 6}</div><div> 10 {2, 8}</div><div> Process 1</div><div> empty</div><div><br></div><div> The vector would just have the face normal values in the canonical order. You can use</div><div> PetscSectionView() to check that yours looks similar.</div><div><br></div><div> 2) Now we add a PetscSF describing the redistribution. An SF is a map from a given set of</div><div> integers (leaves) to pairs (int, rank) called roots. Many leaves can point to one root. To begin,</div><div> we provide an SF mapping mesh points to the new distribution</div><div><br></div><div> Process 0</div><div> 0 -> {0, 0}</div><div> 1 -> {2, 0}</div><div> 2 -> {3, 0}</div><div> 3 -> {4, 0}</div><div> 4 -> {6, 0}</div><div> 5 -> {7, 0}</div><div> 6 -> {8, 0}</div><div> Process 1</div><div> 0 -> {1, 0}</div><div> 1 -> {4, 0}</div><div> 2 -> {3, 0}</div><div> 3 -> {5, 0}</div><div> 4 -> {8, 0}</div><div> 5 -> {9, 0}</div><div> 6 -> {10, 0}</div><div class="gmail_quote"><br></div> This tells Petsc how to move the points to the parallel distribution. You can use PetscSFView() to check</div><div class="gmail_quote"> that your point SF is similar. We first use the SF to move the Section data to make a new parallel Section,</div><div class="gmail_quote"><br></div><div class="gmail_quote"> Process 0</div><div class="gmail_quote"> [0, 3]: {0, 0}</div><div class="gmail_quote"> 4: {2, 0}<br class="gmail-Apple-interchange-newline"> 5: {2, 2}<br class="gmail-Apple-interchange-newline"> 6: {2, 4}</div><div class="gmail_quote"> Process 1<br class="gmail-Apple-interchange-newline"><div class="gmail_quote"> [0, 3]: {0, 0}</div><div class="gmail_quote"> 4: {2, 0}<br class="gmail-Apple-interchange-newline"> 5: {2, 2}<br class="gmail-Apple-interchange-newline"> 6: {2, 4}</div><br class="gmail-Apple-interchange-newline"><div> and then we create a new SF that maps dofs, instead of points, using this new Section</div><div><br></div><div> Process 0:</div><div> 0 -> {0, 0} First face normal</div><div> 1 -> {1, 0}</div><div> 2 -> {2, 0} Second face normal</div><div> 3 -> {3, 0}</div><div> 4 -> {4, 0} Third face normal</div><div> 5 -> {5, 0}</div><div> Process 1:</div><div> 0 -> {4, 0} First face normal</div><div> 1 -> {5, 0}</div><div> 2 -> {6, 0} Second face normal</div><div> 3 -> {7, 0}</div><div> 4 -> {8, 0} Third face normal</div><div> 5 -> {9, 0}</div><br class="gmail-Apple-interchange-newline"> and then this moves the face normal data to the new layout just by broadcasting.</div><div class="gmail_quote"><br></div><div class="gmail_quote">You can View the structures you get back to see what Petsc thought it should do.</div><div class="gmail_quote"><br></div><div class="gmail_quote">Does this make sense?</div><div class="gmail_quote"><br></div><div class="gmail_quote"> Thanks,</div><div class="gmail_quote"><br></div><div class="gmail_quote"> Matt</div><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
The only example I could find anywhere of DMPlexDistributeField() being <br>
used is in DMPlexDistributeCoordinates() so I've been basing what I'm <br>
doing on that.<br>
<br>
(I think I have answered my own questions below by experiment- 1) local <br>
vectors should work (they do in DMPlexDistributeCoordinates); 2) <br>
probably doesn't matter; 3) yes.)<br>
<br>
- Adrian<br>
<br>
On 6/06/19 1:42 PM, Adrian Croucher wrote:<br>
> hi<br>
><br>
> I have some questions about using the DMPlexDistributeField() <br>
> function. I have finite volume mesh geometry data stored in two local <br>
> vectors created using DMPlexComputeGeometryFVM(), and I need to <br>
> redistribute these after calling DMPlexDistribute() to redistribute my <br>
> mesh. (I need the geometry data before redistribution, so I can't just <br>
> wait until after redistribution to create them.)<br>
><br>
> So I figured DMPlexDistributeField() looks like the thing to use for <br>
> that, using the SF that comes out of DMPlexDistribute().<br>
><br>
> 1) Does DMPlexDistributeField() work on local vectors or do they have <br>
> to be global ones?<br>
><br>
> 2) It takes a 'dm' parameter and the documentation says this is "The <br>
> DMPlex object", but is that the original DM (before redistribution) or <br>
> the redistributed one, or does it not matter? It looks like it only <br>
> uses the DM to get the vector type.<br>
><br>
> 3) It looks like you need to manually create the newSection and newVec <br>
> output parameters before passing them in to this routine, is that <br>
> correct?<br>
><br>
> - Adrian<br>
><br>
-- <br>
Dr Adrian Croucher<br>
Senior Research Fellow<br>
Department of Engineering Science<br>
University of Auckland, New Zealand<br>
email: <a href="mailto:a.croucher@auckland.ac.nz" target="_blank">a.croucher@auckland.ac.nz</a><br>
tel: +64 (0)9 923 4611<br>
<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>