<div dir="ltr"><div dir="ltr">On Thu, Aug 13, 2020 at 9:38 AM Cameron Smith <<a href="mailto:smithc11@rpi.edu">smithc11@rpi.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello,<br>
<br>
We have a partitioned mesh that we want to create a DMPlex from that has <br>
the same distribution of elements (i.e., assignment of elements to <br>
processes) and vertices 'interior' to a process (i.e., vertices not on <br>
the inter-process boundary).<br>
<br>
We were trying to use DMPlexCreateFromCellListParallelPetsc() or <br>
DMPlexBuildFromCellListParallel() and found that the vertex ownership <br>
(roots in the returned Vertex SF) appears to be sequentially assigned to <br>
processes based on global vertex id. In general, this will not match <br>
our mesh distribution. As we understand, to subsequently set vertex <br>
coordinates (or other vertex data) we would have to utilize a star <br>
forest (SF) communication API to send data to the correct process. Is <br>
that correct?<br>
<br>
Alternatively, if we create a dmplex object from the elements that exist <br>
on each process using DMCreateFromCellList(), and then create a SF from <br>
mesh vertices on inter-process boundaries (using the mapping from local <br>
to global vertex ids provided by our mesh library), could we then <br>
associate the dmplex objects with the SF? Is it as simple as calling <br>
DMSetPointSF()?<br></blockquote><div><br></div><div>Yes. If you have all the distribution information, this is the easiest thing to do.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
If manually defining the PointSF is a way forward, we would like some <br>
help understanding its definition; i.e., which entities become roots and <br>
which become leaves. In DMPlexBuildFromCellListParallel()<br></blockquote><div><br></div><div>Short explanation of SF:</div><div><br></div><div>SF stands for Star-Forest. It is a star graph because you have a single root that points to multiple leaves. It is</div><div>a forest because you have several of these stars. We use this construct in many places in PETSc, and where it</div><div>is used determines the semantics of the indices.</div><div><br></div><div>The DMPlex point SF is an SF in which root indices are "owned" mesh points and leaf indices are "ghost" mesh</div><div>points. You can take any set of local Plexes and add an SF to make them a parallel Plex.</div><div><br></div><div>The SF is constructed with one-sided data. Locally, each process specifies two things:</div><div><br></div><div> 1) The root space: The set of indices [0, Nr) which refers to possible roots on this process. For the pointSF, this is [0, Np) where Np is the number of local mesh points.</div><div><br></div><div> 2) The leaves: Each leaf is a pair (local mesh point lp, remote mesh point rp) which says that local mesh point lp is a "ghost" of remote point rp. The remote point is</div><div> given by (rank r, local mesh point rlp) where rlp is the local mesh point number on process r.</div><div><br></div><div>With this, the Plex will automatically create all the other structures it needs.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<a href="https://gitlab.com/petsc/petsc/-/blob/753428fdb0644bc4cb7be6429ce8776c05405d40/src/dm/impls/plex/plexcreate.c#L2875-2899" rel="noreferrer" target="_blank">https://gitlab.com/petsc/petsc/-/blob/753428fdb0644bc4cb7be6429ce8776c05405d40/src/dm/impls/plex/plexcreate.c#L2875-2899</a><br>
<br>
the PointSF appears to contain roots for elements and vertices and <br>
leaves for owned vertices on the inter-process boundary. Is that correct?<br></blockquote><div><br></div><div>No, the leaves are ghost vertices. They point back to the owner.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Thank-you,<br>
Cameron<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>