<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div><div class=""> Jed, Matt, Junchao, Lawrence,</div><div class=""><br class=""></div> I need some expert advice with this <a href="https://gitlab.com/petsc/petsc/-/merge_requests/4542" class="">https://gitlab.com/petsc/petsc/-/merge_requests/4542</a>. I was able to translate the cython to PETSc C for moving the ghost points to the end in the local section.<div class=""><br class=""></div><div class=""> But I don't know </div><div class=""><br class=""></div><div class=""> * the best way to generate the ghosts list needed by VecCreateGhost()? or should I not use VecCreateGhost() </div><div class=""><br class=""></div><div class=""> * but somehow instead use the existing DMGetSectionSF() to be able to PetscSF between the owned part and the ghosted part of a hybrid vector (that is, don't use VecCreateGhost() but create an equivalent somehow using the sf that already exists?)</div><div class=""><br class=""></div><div class=""> Thanks</div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Oct 28, 2021, at 6:37 PM, Jed Brown <<a href="mailto:jed@jedbrown.org" class="">jed@jedbrown.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Yeah, I think that's about true. Should the interface for this be to set a flag before creating fields (and perhaps that flag can become true by default at some point)?<br class=""><br class="">Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>> writes:<br class=""><br class=""><blockquote type="cite" class=""> Seems like it should be possible to simply "rewrite" the cython code into PETSc C.<br class=""><br class=""> Barry<br class=""><br class=""><br class=""><blockquote type="cite" class="">On Oct 28, 2021, at 12:36 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> wrote:<br class=""><br class="">I agree it is.<br class=""><br class=""> Matt<br class=""><br class="">On Thu, Oct 28, 2021 at 11:53 AM Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a> <<a href="mailto:bsmith@petsc.dev" class="">mailto:bsmith@petsc.dev</a>>> wrote:<br class=""><br class=""> Thanks, sounds like something DMPLEX should be automatically optionally doing directly.<br class=""><br class=""><blockquote type="cite" class="">On Oct 28, 2021, at 10:58 AM, Lawrence Mitchell <<a href="mailto:wence@gmx.li" class="">wence@gmx.li</a> <<a href="mailto:wence@gmx.li" class="">mailto:wence@gmx.li</a>>> wrote:<br class=""><br class="">Hi Barry et al.,<br class=""><br class=""><blockquote type="cite" class="">On 28 Oct 2021, at 15:37, Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a> <<a href="mailto:bsmith@petsc.dev" class="">mailto:bsmith@petsc.dev</a>>> wrote:<br class=""><br class=""><br class=""><br class=""><blockquote type="cite" class="">On Oct 28, 2021, at 10:31 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a> <<a href="mailto:knepley@gmail.com" class="">mailto:knepley@gmail.com</a>>> wrote:<br class=""><br class="">On Thu, Oct 28, 2021 at 9:37 AM Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a> <<a href="mailto:bsmith@petsc.dev" class="">mailto:bsmith@petsc.dev</a>>> wrote:<br class=""><br class=""> Matt,<br class=""><br class=""> How difficult would it be to rework DMPLEX to allow the use of VecGhost? We have performance problems with GPUs with simple DMNETWORK models because the code spends more time uselessly copying the local part of the vector to another vector in global to local and local to global; more than 1/2 the time of the total simulation.<br class=""><br class="">Firedrake already does this because they "vec ghost" their vectors by default. Here is what you need:<br class=""><br class=""> When you create the PetscSection, by default it orders the unknowns according to the default point numbering. This<br class=""> is what causes the ghost unknowns to be mixed in with the local unknowns. However, PetscSection allows you to set<br class=""> a point permutation<br class=""><br class=""> <a href="https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermutation.html" class="">https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermutation.html</a> <<a href="https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermutation.html" class="">https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermutation.html</a>><br class=""><br class=""> This determines the order of dogs by iterating through points in this permutation, and you can put all shared points at the end.<br class=""></blockquote><br class=""> How do I know what are shared points to put at the end? Couldn't DMPLEX do this automatically with an option? Where is the Firedrake code that does this with DMPLEX so I can see it?<br class=""></blockquote><br class="">The DM's "point sf" indicates shared points. To label them, do something like:<br class=""><br class="">DMCreateLabel(dm, "ghost_points");<br class="">DMGetPointSF(dm, &pointsf);<br class="">PetscSFGetGraph(pointsf, NULL, &nleaves, &ilocal, NULL);<br class="">for (PetscInt p = 0; p < nleaves; p++) {<br class=""> DMSetLabelValue(dm, "ghost_points", p, 1);<br class="">}<br class=""><br class="">Then you do something like (this is more pseudo-codey):<br class=""><br class="">PetscInt *permutation;<br class="">DMPlexGetChart(dm, &pStart, &pEnd);<br class="">PetscMalloc1(pEnd - pStart, &permutation);<br class="">PetscInt offsets[2] = {0, pEnd - pStart - nleaves};<br class="">DMGetLabel(dm, &ghosts);<br class="">DMLabelCreateIndex(ghosts, pStart, pEnd);<br class="">for (PetscInt p = pStart, p < pEnd; p++) {<br class=""> DMLabelHasPoint(ghosts, p, &has);<br class=""> if (has) {<br class=""> // this point is ghost point<br class=""> permutation[offsets[1]++] = p;<br class=""> } else {<br class=""> permutation[offsets[0]++] = p;<br class=""> }<br class="">}<br class="">DMLabelDestroyIndex(ghosts, pStart, pEnd);<br class="">ISCreateGeneral(..., permutation, &isperm);<br class=""><br class="">// Now whenever your do PetscSectionCreate, do<br class="">PetscSectionSetPermutation(..., isperm);<br class=""><br class="">And now ghost point dofs will appear after local ones.<br class=""><br class="">Note that this is probably more complicated for multi-field setups, depending whether you are point major or field major.<br class=""><br class="">You can see what we actually do (if you like reading Cython) here: <a href="https://github.com/firedrakeproject/firedrake/blob/master/firedrake/cython/dmcommon.pyx#L1734" class="">https://github.com/firedrakeproject/firedrake/blob/master/firedrake/cython/dmcommon.pyx#L1734</a> <<a href="https://github.com/firedrakeproject/firedrake/blob/master/firedrake/cython/dmcommon.pyx#L1734" class="">https://github.com/firedrakeproject/firedrake/blob/master/firedrake/cython/dmcommon.pyx#L1734</a>><br class=""><br class="">It's more complicated because here we are doing some additional things:<br class=""><br class="">1. We compute an RCM-order traversal for cells with DMPlexGetOrdering;<br class="">2. Rather than ordering all plex points in the permutation, we walk the cells in the RCM order and then greedily number points in the transitive closure (so that in a section cell and vertex dofs from the same cell will be "close" to each other in the final Vec).<br class=""><br class="">Thanks,<br class=""><br class="">Lawrence<br class=""></blockquote><br class=""><br class=""><br class="">-- <br class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener<br class=""><br class=""><a href="https://www.cse.buffalo.edu/~knepley/" class="">https://www.cse.buffalo.edu/~knepley/</a> <<a href="http://www.cse.buffalo.edu/~knepley/" class="">http://www.cse.buffalo.edu/~knepley/</a>><br class=""></blockquote></blockquote></div></div></blockquote></div><br class=""></div></body></html>