<div dir="ltr"><div dir="ltr">On Thu, Jan 19, 2023 at 10:12 PM Nicholas Arnold-Medabalimi <<a href="mailto:narnoldm@umich.edu">narnoldm@umich.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Ok thanks for the clarification. In theory, if before the Reduction back to the original distribution, if I call DMGlobaltoLocal then even with MPI_REPLACE all the leafs corresponding to the original root should have the same value so I won't have an ambiguity, correct? <br></div></blockquote><div><br></div><div>That is right, so it should give you the result you expect.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 19, 2023 at 9:28 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Thu, Jan 19, 2023 at 9:13 PM Nicholas Arnold-Medabalimi <<a href="mailto:narnoldm@umich.edu" target="_blank">narnoldm@umich.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Matt<br><br>Yep, that makes sense and is consistent.<br><br>My question is a little more specific. So let's say I take an initial mesh and distribute it and get the distribution SF with an overlap of one. Consider a cell that is a root on process 0 and a leaf on process 1 after the distribution. <br><br>Will the distribution pointSF have an entry for the cell that is a leaf in the ghost cell sense?<br><br>I guess, in short does the distribution SF only have entries for the movement of points that are roots in the ghost SF? <br></div></blockquote><div><br></div><div>I do not understand the question. Suppose that a certain cell, say 0, in the original distribution goes to two different processes, say 0 and 1, and will happen when you distribute with overlap. Then the migration SF has two leaf entries for that cell, one from process 0 and one from process 1. They both point to root cell 0 on process 0.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Sorry if this is a little unclear. <br><br>Maybe my usage will be a bit clearer. I am generating a distributionSF (type 2 in your desc) then using that to generate a dof distribution(type 3) using the section information. I then pass the information from the initial distribution to new distribution with PetscSFBcast with MPI_REPLACE. That scatters the vector to the new distribution. I then do "stuff" and now want to redistribute back. So I pass the same dof distributionSF but call PetscSFReduce with MPI_REPLACE. My concern is I am only setting the root cell values on each partition. So if the ghost cells are part of the distribution SF there will be multiple cells reducing to the original distribution cell? <br></div></blockquote><div><br></div><div>Yes, definitely.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thanks<br>Nicholas <br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 19, 2023 at 8:28 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Thu, Jan 19, 2023 at 11:58 AM Nicholas Arnold-Medabalimi <<a href="mailto:narnoldm@umich.edu" target="_blank">narnoldm@umich.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Petsc Users<br><br>I'm working with a distribution start forest generated by DMPlexDistribute and PetscSFBcast and Reduce to move data between the initial distribution and the distribution generated by DMPlex Distribute. <br><br>I'm trying to debug some values that aren't being copied properly and wanted to verify I understand how a redistribution SF works compared with a SF that describes overlapped points. <br><br> [0] 0 <- (0,7) point 0 on the distributed plex is point 7 on process 0 on the initial distribution<br> [0] 1 <- (0,8)
point 1 on the distributed plex is point 8 on process 0 on the initial distribution<br> [0] 2 <- (0,9)<br> [0] 3 <- (0,10)<br> [0] 4 <- (0,11)<br><br> [1] 0 <- (1,0)
point 0 on the distributed plex is point 0 on process 1 on the initial distribution<br> [1] 1 <- (1,1)<br> [1] 2 <- (1,2)<br> [1] 3 <- (0,0)
point 3 on the distributed plex is point 0 on process 0 on the initial distribution<br> [1] 4 <- (0,1)<br> [1] 5 <- (0,2)<br clear="all"><div><br> my confusion I think is how does the distributionSF inform of what cells will be leafs on the distribution? <br></div></div></blockquote><div><br></div><div>I should eventually write something to clarify this. I am using SF in (at least) two different ways.</div><div><br></div><div>First, there is a familiar SF that we use for dealing with "ghost" points. These are replicated points where one process</div><div>is said to "own" the point and another process is said to hold a "ghost". The ghost points are leaves in the SF which</div><div>point back to the root point owned by another process. We call this the pointSF for a DM.</div><div><br></div><div>Second, we have a migration SF. Here the root points give the original point distribution. The leaf points give the new</div><div>point distribution. Thus a PetscSFBcast() pushes points from the original to new distribution, which is what we mean</div><div>by a migration.</div><div><br></div><div>Third, instead of point values, we might want to communicate fields over those points. For this we make new SFes,</div><div>where the numbering does not refer to points, but rather to dofs.</div><div><br></div><div>Does this make sense?</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Sincerely<br>Nicholas<br><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div style="font-family:arial;font-size:small"><font color="#000000">Nicholas Arnold-Medabalimi<br><br></font><span style="font-family:sans-serif;font-size:14px">Ph.D. Candidate</span><font color="#000000"><br>Computational Aeroscience Lab<br>University of Michigan</font></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div style="font-family:arial;font-size:small"><font color="#000000">Nicholas Arnold-Medabalimi<br><br></font><span style="font-family:sans-serif;font-size:14px">Ph.D. Candidate</span><font color="#000000"><br>Computational Aeroscience Lab<br>University of Michigan</font></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div style="font-family:arial;font-size:small"><font color="#000000">Nicholas Arnold-Medabalimi<br><br></font><span style="font-family:sans-serif;font-size:14px">Ph.D. Candidate</span><font color="#000000"><br>Computational Aeroscience Lab<br>University of Michigan</font></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>