<div dir="ltr"><div dir="ltr">On Fri, May 12, 2023 at 10:34 AM Matthew Young <<a href="mailto:myoung.space.science@gmail.com">myoung.space.science@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Okay, cool.</div><div>Should I do that by explicitly setting each particle's DMSwarmField_rank when I assign its position?</div><div>What role does DMSwarmMigrate play in all of this?</div></div></blockquote><div><br></div><div>If you are using a DMDA or a DMPlex, then PETSc now how to do parallel point location (at least for neighbors). So if you add particles on the process which owns that part of the grid, and then call</div><div>DMSwarmMigrate(), it will locate every particle, and set the 'rank' and 'cellid' correctly. Then if you move</div><div>the particles (change the coordinates) and call DMSwarmMigrate(), it will move them to the correct process</div><div>and reset those entries.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div>--Matt</div><div dir="ltr">==========================<br></div><div dir="ltr">Matthew Young, PhD (he/him)<div><div><div><div>Research Scientist II</div><div>Space Science Center<br></div></div></div><div>University of New Hampshire</div><div><a href="mailto:Matthew.Young@unh.edu" target="_blank">Matthew.Young@unh.edu</a><br></div></div><div><div dir="ltr"><div dir="ltr">==========================<br></div></div></div></div></div></div></div></div></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 12, 2023 at 10:05 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Fri, May 12, 2023 at 9:40 AM Matthew Young <<a href="mailto:myoung.space.science@gmail.com" target="_blank">myoung.space.science@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Got it.</div><div><br></div><div>I'm specifically thinking about this in terms of the gather stage of my PIC code, where I loop over local particles to fill local density and flux arrays by linearly interpolating particle positions to the grid. The gather function currently assumes that the coordinates (i.e., the array representation of DMSwarmPICField_coor) of all particles on a given rank would correspond to only the global indices owned by that rank, via the relationship between indices and coordinates in the associated cell DM. Based on what you described, it sounds like I need to make sure that I initially lay down the particles so that their rank matches their coordinates.</div></div></blockquote><div><br></div><div>Right now, yes. I will fix that before August.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div>--Matt</div><div dir="ltr">==========================<br></div><div dir="ltr">Matthew Young, PhD (he/him)<div><div><div><div>Research Scientist II</div><div>Space Science Center<br></div></div></div><div>University of New Hampshire</div><div><a href="mailto:Matthew.Young@unh.edu" target="_blank">Matthew.Young@unh.edu</a><br></div></div><div><div dir="ltr"><div dir="ltr">==========================<br></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 12, 2023 at 5:15 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Thu, May 11, 2023 at 9:15 PM Matthew Young <<a href="mailto:myoung.space.science@gmail.com" target="_blank">myoung.space.science@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Does setting up a PIC-type DMSWARM with an associated cell DM guarantee that each MPI rank will own the particles with coordinates inside the bounds of the portion of the grid it owns?</div></blockquote><div><br></div><div>There is a caveat that we are currently fixing. Swarm communication is setup to be nearest neighbor (since there is no coarse grid of bounding boxes). So if your particles are initially in the right place, and only move nearest neighbor, everything is fine. We are adding a hierarchy of bounding boxes so that we can communicate anywhere.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div>--Matt</div><div dir="ltr">==========================<br></div><div dir="ltr">Matthew Young, PhD (he/him)<div><div><div><div>Research Scientist II</div><div>Space Science Center<br></div></div></div><div>University of New Hampshire</div><div><a href="mailto:Matthew.Young@unh.edu" target="_blank">Matthew.Young@unh.edu</a><br></div></div><div><div dir="ltr"><div dir="ltr">==========================<br></div></div></div></div></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div><span>-- </span><br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div><span>-- </span><br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>