<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Toby and Matt:</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Thank you for your helpful replies.<br>
In principle, I have what I need, however, I ran into a bug with PetscSFReduce.<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
When I run the following on the pointSF from a distributed plex (2 MPI ranks on a small mesh).<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
//==============================================================================================<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
PetscSFGetGraph(point_sf,&nroots,&nleaves,&ilocal,&iremote);<br>
PetscCalloc2(nleaves,&leafdata,nroots,&rootdata);</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
\* Code that populates leafdata*/<br>
</div>
<div id="signature_bookmark"></div>
<div id="appendonsend"></div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
PetscSFReduceBegin(point_sf,MPIU_INT,leafdata, rootdata,MPI_SUM);
<div>PetscSFReduceEnd(point_sf,MPIU_INT,leafdata, rootdata,MPI_SUM);</div>
<div><br>
</div>
<div>PetscSFView(point_sf,0);</div>
<div>PetscViewerASCIIPrintf(PETSC_VIEWER_STDOUT_WORLD,"## Reduce Leafdata\n"); //I copied this from a PetscSF example.<br>
</div>
<div>PetscIntView(nleaves,leafdata,PETSC_VIEWER_STDOUT_WORLD);</div>
<div>PetscViewerASCIIPrintf(PETSC_VIEWER_STDOUT_WORLD,"## Reduce Rootdata\n");</div>
PetscIntView(nroots,rootdata,PETSC_VIEWER_STDOUT_WORLD);</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
PetscFree2(leafdata,rootdata);</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
//==============================================================================================</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
.... I get the following printout :<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
//======================================<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
PetscSF Object: 2 MPI processes
<div> type: basic</div>
<div> [0] Number of roots=29, leaves=5, remote ranks=1</div>
<div> [0] 9 <- (1,9)</div>
<div> [0] 11 <- (1,10)</div>
<div> [0] 12 <- (1,13)</div>
<div> [0] 20 <- (1,20)</div>
<div> [0] 27 <- (1,27)</div>
<div> [1] Number of roots=29, leaves=2, remote ranks=1</div>
<div> [1] 14 <- (0,13)</div>
<div> [1] 19 <- (0,18)</div>
<div> MultiSF sort=rank-order</div>
<div>## Reduce Leafdata</div>
<div>[0] 0: 2 2 2 0 0</div>
<div>[1] 0: 3 0</div>
<div>## Reduce Rootdata</div>
<div>[0] 0: 0 0 0 0 0 0 0 0 0 0 0 0 0 -686563120 0 0 0 0 0 0</div>
<div>[0] 20: 0 0 0 0 0 0 0 0 0</div>
<div>[1] 0: 0 0 0 0 0 0 0 0 0 0 0 0 0 128 0 0 0 0 0 0</div>
<div>[1] 20: -527386800 0 0 0 0 0 0 32610 0</div>
//======================================</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
The good news is that the rootdata on both processors has the correct number of nonzeros after reduction.<br>
The bad news is that the nonzeros are garbage (like what one gets when a variable isn't initialized).</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Any ideas as to what could cause this? Could something like a previous call to a PetscSF or DMPlex function do this?<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
I am still using PETSc version 3.16, but I looked at the patch notes of 3.17 and did not see any updates on PetscSFReduce().<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size: 11pt;" face="Calibri, sans-serif" color="#000000"><b>From:</b> Matthew Knepley <knepley@gmail.com><br>
<b>Sent:</b> Wednesday, May 18, 2022 2:09 AM<br>
<b>To:</b> Toby Isaac <toby.isaac@gmail.com><br>
<b>Cc:</b> Ferrand, Jesus A. <FERRANJ2@my.erau.edu>; petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> [EXTERNAL] Re: [petsc-users] DMPlex/PetscSF How to determine if local topology is other rank's ghost?</font>
<div> </div>
</div>
<div>
<table role="presentation" style="min-width: 100%; background-color: rgb(255, 235, 156); border: 1px solid rgb(136, 136, 136); font-size: 14px; line-height: 1.5; font-family: Arial, sans-serif; color: black; text-align: left;" width="100%" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="padding:5px 10px"><span style="color: rgb(214, 0, 0);"><b>CAUTION:</b></span> This email originated outside of Embry-Riddle Aeronautical University. Do not click links or open attachments unless you recognize the sender and know the content is safe.</td>
</tr>
</tbody>
</table>
<br>
<div>
<div dir="ltr">
<div dir="ltr">On Tue, May 17, 2022 at 6:47 PM Toby Isaac <<a href="mailto:toby.isaac@gmail.com" data-auth="NotApplicable">toby.isaac@gmail.com</a>> wrote:<br>
</div>
<div class="x_gmail_quote">
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div>A leaf point is attached to a root point (in a star forest there are only leaves and roots), so that means that a root point would be the point that owns a degree of freedom and a leaf point would have a ghost value.</div>
<div><br>
</div>
<div>For a "point SF" of a DMPlex:</div>
<div><br>
</div>
<div>- Each process has a local numbering of mesh points (cells + edges + faces + vertices): they are all potential roots, so the number of these is what is returned by `nroots`.</div>
<div><br>
</div>
<div>- The number of ghost mesh points is `nleaves`.</div>
<div><br>
</div>
<div>- `ilocal` would be a list of the mesh points that are leaves (using the local numbering).</div>
<div><br>
</div>
<div>- For each leaf in `ilocal`, `iremote` describes the root it is attached to: which process it belongs to, and its id in *that* process's local numbering.</div>
<div><br>
</div>
<div>If you're trying to create dof numberings on your own, please consider PetscSectionCreateGlobalSection: <<a href="https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionCreateGlobalSection/" data-auth="NotApplicable">https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionCreateGlobalSection/</a>>.
You supply the PetscSF and a PetscSection which says how many dofs there are for each point and whether any have essential boundary conditions, and it computes a global PetscSection that tells you what the global id is for each dof on this process.</div>
</div>
</blockquote>
<div><br>
</div>
<div>Toby is exactly right. Also, if you want global numbering of points you can use</div>
<div><br>
</div>
<div> <a href="https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexCreatePointNumbering/" data-auth="NotApplicable">https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexCreatePointNumbering/</a></div>
<div><br>
</div>
<div>and there is a similar thing for jsut cells or vertices.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Tue, May 17, 2022 at 7:26 PM Ferrand, Jesus A. <<a href="mailto:FERRANJ2@my.erau.edu" data-auth="NotApplicable">FERRANJ2@my.erau.edu</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Dear PETSc team:<br>
<br>
I am working with a non-overlapping distributed plex (i.e., when I call DMPlexDistribute(), I input overlap = 0), so only vertices and edges appear as ghosts to the local ranks.</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
For preallocation of a parallel global stiffness matrix for FEA, I want to determine which locally owned vertices are ghosts to another rank.<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
From reading the paper on PetscSF (<a href="https://ieeexplore.ieee.org/document/9442258" data-auth="NotApplicable" id="x_gmail-m_-7288957893647796504gmail-m_2804447925891900737LPlnkOWALinkPreview">https://ieeexplore.ieee.org/document/9442258</a>) I think I
can answer my question by inspecting the PetscSF returned by DMPlexDistribute() with PetscSFGetGraph(). I am just confused by the root/leaf and ilocal/iremote terminology.
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
I read the manual page on PetscSFGetGraph() (<a href="https://petsc.org/release/docs/manualpages/PetscSF/PetscSFGetGraph.html" data-auth="NotApplicable" id="x_gmail-m_-7288957893647796504gmail-m_2804447925891900737LPlnk343018">https://petsc.org/release/docs/manualpages/PetscSF/PetscSFGetGraph.html</a>)
and that gave me the impression that I need to PetscSFBcast() the point IDs from foreign ranks to the local ones.
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Is this correct?<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div>
<div id="x_gmail-m_-7288957893647796504gmail-m_2804447925891900737LPBorder_GTaHR0cHM6Ly9pZWVleHBsb3JlLmllZWUub3JnL2RvY3VtZW50Lzk0NDIyNTg." style="width:100%; margin-top:16px; margin-bottom:16px; max-width:800px; min-width:424px">
<table id="x_gmail-m_-7288957893647796504gmail-m_2804447925891900737LPContainer149289" role="presentation" style="padding:12px 36px 12px 12px; width:100%; border-width:1px; border-style:solid; border-color:rgb(200,200,200); border-radius:2px">
<tbody>
<tr style="border-spacing:0px" valign="top">
<td>
<div id="x_gmail-m_-7288957893647796504gmail-m_2804447925891900737LPImageContainer149289" style="margin-right:12px; height:160px; overflow:hidden">
<a href="https://ieeexplore.ieee.org/document/9442258" data-auth="NotApplicable" id="x_gmail-m_-7288957893647796504gmail-m_2804447925891900737LPImageAnchor149289"><img id="x_gmail-m_-7288957893647796504gmail-m_2804447925891900737LPThumbnailImageId149289" alt="" style="display:block" width="160" height="160" src="https://ieeexplore.ieee.org/assets/img/ieee_logo_smedia_200X200.png"></a></div>
</td>
<td style="width:100%">
<div id="x_gmail-m_-7288957893647796504gmail-m_2804447925891900737LPTitle149289" style="font-size:21px; font-weight:300; margin-right:8px; font-family:wf_segoe-ui_light,"Segoe UI Light","Segoe WP Light","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif; margin-bottom:12px">
<a href="https://ieeexplore.ieee.org/document/9442258" data-auth="NotApplicable" id="x_gmail-m_-7288957893647796504gmail-m_2804447925891900737LPUrlAnchor149289" style="text-decoration:none">The PetscSF Scalable Communication Layer | IEEE Journals & Magazine
| IEEE Xplore</a></div>
<div id="x_gmail-m_-7288957893647796504gmail-m_2804447925891900737LPDescription149289" style="font-size: 14px; max-height: 100px; color: rgb(102, 102, 102); font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; margin-bottom: 12px; margin-right: 8px; overflow: hidden;">
PetscSF, the communication component of the Portable, Extensible Toolkit for Scientific Computation (PETSc), is designed to provide PETSc's communication infrastructure suitable for exascale computers that utilize GPUs and other accelerators. PetscSF provides
a simple application programming interface (API) for managing common communication patterns in scientific computations by using a star ...</div>
<div id="x_gmail-m_-7288957893647796504gmail-m_2804447925891900737LPMetadata149289" style="font-size: 14px; font-weight: 400; color: rgb(166, 166, 166); font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif;">
<a href="http://ieeexplore.ieee.org" data-auth="NotApplicable">ieeexplore.ieee.org</a></div>
</td>
</tr>
</tbody>
</table>
<div id="x_gmail-m_-7288957893647796504gmail-m_2804447925891900737LPCloseButtonContainer149289" title="Remove link preview" role="button">
<i aria-hidden="true" id="x_gmail-m_-7288957893647796504gmail-m_2804447925891900737LPCloseButton149289"></i></div>
</div>
</div>
<br>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="x_gmail-m_-7288957893647796504gmail-m_2804447925891900737Signature">
<div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);">
</div>
<div></div>
<div></div>
<div id="x_gmail-m_-7288957893647796504gmail-m_2804447925891900737divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif;">
<p style="margin-top:0px; margin-bottom:0px"></p>
<div>
<p style="margin:0px 0px 10.66px"></p>
<div>
<p style="background: white none repeat scroll 0% 0%; margin: 0px 0px 11px; line-height: normal;">
<span style="margin: 0px; color: black; font-family: "Times New Roman", Times, serif; font-size: 12pt;">Sincerely:</span></p>
<p style="background: white none repeat scroll 0% 0%; margin: 0px; line-height: normal;">
<span style="margin: 0px; color: red; font-family: "Courier New", monospace; font-size: 14pt;"><b><span style="font-family:"Courier New",monospace">J.A. Ferrand</span></b></span><u><span style="margin: 0px; color: red; font-family: "Swis721 BlkCn BT", sans-serif; font-size: 12pt;"></span></u></p>
<p style="background: white none repeat scroll 0% 0%; margin: 0px; line-height: normal;">
<span style="margin: 0px; color: black; font-family: "Times New Roman", Times, serif; font-size: 12pt;">Embry-Riddle Aeronautical University - Daytona Beach FL</span></p>
<p style="background: white none repeat scroll 0% 0%; margin: 0px; line-height: normal;">
<span style="font-family:"Times New Roman",Times,serif"></span><span style="margin: 0px; color: black; font-family: "Times New Roman", Times, serif; font-size: 12pt;">M.Sc.</span><span style="margin: 0px; color: black; font-family: "Times New Roman", Times, serif; font-size: 12pt;">
Aerospace Engineering | May 2022</span></p>
<p style="background: white none repeat scroll 0% 0%; margin: 0px; line-height: normal;">
<span style="font-family:"Times New Roman",Times,serif"></span><span style="margin: 0px; color: black; font-family: "Times New Roman", Times, serif; font-size: 12pt;">B.Sc.</span><span style="margin: 0px; color: black; font-family: "Times New Roman", Times, serif; font-size: 12pt;">
Aerospace Engineering</span></p>
<p style="background: white none repeat scroll 0% 0%; margin: 0px; line-height: normal;">
<span style="font-family:"Times New Roman",Times,serif"></span><span style="margin: 0px; color: black; font-family: "Times New Roman", Times, serif; font-size: 12pt;">B.Sc.</span><span style="margin: 0px; color: black; font-family: "Times New Roman", Times, serif; font-size: 12pt;">
Computational Mathematics</span></p>
<p style="background: white none repeat scroll 0% 0%; margin: 0px; line-height: normal;">
<span style="font-family:"Times New Roman",Times,serif"></span><span style="margin: 0px; color: rgb(0, 0, 0); font-family: "Times New Roman", Times, serif; font-size: 12pt;"> </span></p>
<p style="background: white none repeat scroll 0% 0%; margin: 0px; line-height: normal;">
<span style="font-family:"Times New Roman",Times,serif"></span><span style="margin: 0px; color: black; font-family: Courant; font-size: 12pt;"><span style="color: rgb(0, 0, 0); font-family: "Times New Roman", Times, serif; font-size: 12pt; display: inline; background-color: rgb(255, 255, 255);">Sigma
Gamma Tau</span><span style="color: rgb(0, 0, 0); font-family: "Times New Roman", Times, serif; font-size: 12pt; display: inline; background-color: rgb(255, 255, 255);"> </span><br>
</span></p>
<span style="font-family:"Times New Roman",Times,serif"></span>
<p style="background: white none repeat scroll 0% 0%; margin: 0px; line-height: normal;">
<span style="margin: 0px; color: rgb(0, 0, 0); font-family: "Times New Roman", Times, serif; font-size: 12pt;">Tau Beta Pi</span></p>
<p style="background: white none repeat scroll 0% 0%; margin: 0px; line-height: normal;">
<span style="font-family:"Times New Roman",Times,serif"></span><span style="margin: 0px; color: black; font-family: "Times New Roman", Times, serif; font-size: 12pt;"> </span></p>
<p style="background: white none repeat scroll 0% 0%; margin: 0px; line-height: normal;">
<span style="font-family:"Times New Roman",Times,serif"></span><u><span style="margin: 0px; color: black; font-family: "Times New Roman", Times, serif; font-size: 12pt;">Phone:</span></u><span style="margin: 0px; color: black; font-family: "Times New Roman", Times, serif; font-size: 12pt;">
(386)-843-1829</span></p>
<p style="background: white none repeat scroll 0% 0%; margin: 0px; line-height: normal;">
<span style="font-family:"Times New Roman",Times,serif"></span><u><span style="margin: 0px; color: black; font-family: "Times New Roman", Times, serif; font-size: 12pt;">Email(s):</span></u><span style="margin: 0px; color: black; font-family: "Times New Roman", Times, serif; font-size: 12pt;">
<a href="mailto:ferranj2@my.erau.edu" data-auth="NotApplicable">ferranj2@my.erau.edu</a></span></p>
<p style="background: white none repeat scroll 0% 0%; margin: 0px; line-height: normal; text-indent: 0.5in;">
<span style="font-family:"Times New Roman",Times,serif"></span><span style="margin: 0px; color: black; font-family: Courant; font-size: 12pt;"><span style="margin:0px; font-family:"Times New Roman",Times,serif">
</span></span><span style="margin: 0px; color: black; font-family: "Times New Roman", Times, serif; font-size: 12pt;"><a href="mailto:jesus.ferrand@gmail.com" data-auth="NotApplicable">jesus.ferrand@gmail.com</a></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" data-auth="NotApplicable">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>