<div dir="ltr"><div dir="ltr">On Thu, Oct 28, 2021 at 10:37 AM Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><br><div><br><blockquote type="cite"><div>On Oct 28, 2021, at 10:31 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div><br><div><div dir="ltr"><div dir="ltr">On Thu, Oct 28, 2021 at 9:37 AM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Matt,<br>
<br>
How difficult would it be to rework DMPLEX to allow the use of VecGhost? We have performance problems with GPUs with simple DMNETWORK models because the code spends more time uselessly copying the local part of the vector to another vector in global to local and local to global; more than 1/2 the time of the total simulation.<br></blockquote><div><br></div><div>Firedrake already does this because they "vec ghost" their vectors by default. Here is what you need:</div><div><br></div><div> When you create the PetscSection, by default it orders the unknowns according to the default point numbering. This</div><div> is what causes the ghost unknowns to be mixed in with the local unknowns. However, PetscSection allows you to set</div><div> a point permutation</div><div><br></div><div> <a href="https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermutation.html" target="_blank">https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermutation.html</a></div><div><br></div><div> This determines the order of dogs by iterating through points in this permutation, and you can put all shared points at the end.</div></div></div></div></blockquote><div><br></div> How do I know what are shared points to put at the end? Couldn't DMPLEX do this automatically with an option? Where is the Firedrake code that does this with DMPLEX so I can see it?</div></div></blockquote><div><br></div><div>Yes, it could. I was thinking this as I wrote you, and also thinking how tired I am :)</div><div><br></div><div>Here is how to do it. Suppose you only care about ghost points (Firedrake wants three layers: totalling internal, roots, and ghosts), then there is a simple test</div><div><br></div><div> DMGetLocalSection(dm, &s)</div><div> PetscSectionGetChart(s, &pStart, &pEnd);</div><div> DMGetPointSF(dm, &sf);</div><div> PetscSFGetGraph(sf, NULL, &nleaves, &leaves, NULL) </div><div> for (p = pStart; p < pEnd; ++p) {</div><div> PetscFindInt(p, nleaves, leaves, &loc);</div><div> if (loc >= 0) {I am a ghost point}</div><div> }</div><div><br></div><div>If you also care about roots (shared points that you own), you also need</div><div><br></div><div> PetscSFComputeDegreeBegin(sf, &rootdegree);</div><div> PetscSFComputeDegreeEnd(sf, &rootdegree);<br></div><div><br></div><div>and then also check</div><div><br></div><div> rootdegree[p] > 0</div><div><br></div><div>If you get it working, I will put it inside Plex and make an option to turn it on.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div><blockquote type="cite"><div><div dir="ltr"><div class="gmail_quote"><div>Does this make sense?</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Barry<br></blockquote></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</div></blockquote></div><br></div></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>