<html aria-label="message body"><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">Thank you Matt<div><br></div><div>Indeed the right way was to use the rank property of the SF. </div><div><br></div><div>Thanks, </div><div>Miguel</div><div><br></div><div><div style="color: rgb(46, 46, 46); background-color: rgb(255, 255, 255); font-family: Menlo, Monaco, "Courier New", monospace; line-height: 18px; white-space: pre;"><div> <span style="color: rgb(69, 69, 69);">PetscCall</span><span style="color: rgb(94, 94, 94);">(</span></div><div> <span style="color: rgb(99, 99, 36);">DMLocatePoints</span><span style="color: rgb(94, 94, 94);">(</span><span style="color: rgb(9, 89, 132);">FE_Mesh</span><span style="color: rgb(94, 94, 94);">,</span> <span style="color: rgb(69, 69, 69);">coordinates</span><span style="color: rgb(94, 94, 94);">,</span> <span style="color: rgb(9, 89, 132);">DM_POINTLOCATION_NONE</span><span style="color: rgb(94, 94, 94);">,</span> <span style="color: rgb(94, 94, 94);">&</span><span style="color: rgb(69, 69, 69);">cellSF</span><span style="color: rgb(94, 94, 94);">));</span></div><div> <span style="color: rgb(69, 69, 69);">PetscCall</span><span style="color: rgb(94, 94, 94);">(</span><span style="color: rgb(99, 99, 36);">PetscSFGetGraph</span><span style="color: rgb(94, 94, 94);">(</span><span style="color: rgb(69, 69, 69);">cellSF</span><span style="color: rgb(94, 94, 94);">,</span> <span style="color: rgb(94, 94, 94);">&</span><span style="color: rgb(9, 89, 132);">nroots</span><span style="color: rgb(94, 94, 94);">,</span> <span style="color: rgb(19, 10, 100);">NULL</span><span style="color: rgb(94, 94, 94);">,</span> <span style="color: rgb(19, 10, 100);">NULL</span><span style="color: rgb(94, 94, 94);">,</span> <span style="color: rgb(94, 94, 94);">&</span><span style="color: rgb(69, 69, 69);">LA_sfcell</span><span style="color: rgb(94, 94, 94);">));</span></div><br><div> <span style="color: rgb(69, 69, 69);">PetscCall</span><span style="color: rgb(94, 94, 94);">(</span><span style="color: rgb(99, 99, 36);">VecGetArray</span><span style="color: rgb(94, 94, 94);">(</span><span style="color: rgb(69, 69, 69);">coordinates</span><span style="color: rgb(94, 94, 94);">,</span> <span style="color: rgb(94, 94, 94);">&</span><span style="color: rgb(69, 69, 69);">coordinates_ptr</span><span style="color: rgb(94, 94, 94);">));</span></div><br><div><span style="color: rgb(118, 217, 102);"> // Note: mean_q_ptr should be filled based on n_atoms_local increment</span></div><div> <span style="color: rgb(116, 46, 125);">for</span> <span style="color: rgb(94, 94, 94);">(</span><span style="color: rgb(9, 89, 132);">PetscInt</span> <span style="color: rgb(9, 89, 132);">site_i</span> <span style="color: rgb(94, 94, 94);">=</span> <span style="color: rgb(73, 104, 57);">0</span><span style="color: rgb(94, 94, 94);">;</span> <span style="color: rgb(9, 89, 132);">site_i</span> <span style="color: rgb(94, 94, 94);"><</span> <span style="color: rgb(9, 89, 132);">n_atoms</span><span style="color: rgb(94, 94, 94);">;</span> <span style="color: rgb(9, 89, 132);">site_i</span><span style="color: rgb(94, 94, 94);">++)</span> <span style="color: rgb(94, 94, 94);">{</span></div><div><span style="color: rgb(118, 217, 102);"> // Check if the point is found in the local process</span></div><div> <span style="color: rgb(116, 46, 125);">if</span> <span style="color: rgb(94, 94, 94);">(</span><span style="color: rgb(9, 89, 132);">LA_sfcell</span><span style="color: rgb(94, 94, 94);">[</span><span style="color: rgb(9, 89, 132);">site_i</span><span style="color: rgb(94, 94, 94);">].</span>rank <span style="color: rgb(94, 94, 94);">==</span> <span style="color: rgb(9, 89, 132);">rank_MPI</span><span style="color: rgb(94, 94, 94);">)</span> <span style="color: rgb(94, 94, 94);">{</span></div><div> <span style="color: rgb(9, 89, 132);">idx_ptr</span><span style="color: rgb(94, 94, 94);">[</span><span style="color: rgb(9, 89, 132);">n_atoms_local</span><span style="color: rgb(94, 94, 94);">]</span> <span style="color: rgb(94, 94, 94);">=</span> <span style="color: rgb(9, 89, 132);">site_i</span><span style="color: rgb(94, 94, 94);">;</span></div><div> <span style="color: rgb(116, 46, 125);">for</span> <span style="color: rgb(94, 94, 94);">(</span><span style="color: rgb(63, 151, 223);">int</span> <span style="color: rgb(9, 89, 132);">alpha</span> <span style="color: rgb(94, 94, 94);">=</span> <span style="color: rgb(73, 104, 57);">0</span><span style="color: rgb(94, 94, 94);">;</span> <span style="color: rgb(9, 89, 132);">alpha</span> <span style="color: rgb(94, 94, 94);"><</span> <span style="color: rgb(9, 89, 132);">dim</span><span style="color: rgb(94, 94, 94);">;</span> <span style="color: rgb(9, 89, 132);">alpha</span><span style="color: rgb(94, 94, 94);">++)</span> <span style="color: rgb(94, 94, 94);">{</span></div><div> <span style="color: rgb(9, 89, 132);">mean_q_ptr</span><span style="color: rgb(94, 94, 94);">[</span><span style="color: rgb(9, 89, 132);">n_atoms_local</span> <span style="color: rgb(94, 94, 94);">*</span> <span style="color: rgb(9, 89, 132);">dim</span> <span style="color: rgb(94, 94, 94);">+</span> <span style="color: rgb(9, 89, 132);">alpha</span><span style="color: rgb(94, 94, 94);">]</span> <span style="color: rgb(94, 94, 94);">=</span></div><div> <span style="color: rgb(9, 89, 132);">Simulation_file</span><span style="color: rgb(94, 94, 94);">.</span>mean_q<span style="color: rgb(94, 94, 94);">[</span><span style="color: rgb(9, 89, 132);">site_i</span> <span style="color: rgb(94, 94, 94);">*</span> <span style="color: rgb(9, 89, 132);">dim</span> <span style="color: rgb(94, 94, 94);">+</span> <span style="color: rgb(9, 89, 132);">alpha</span><span style="color: rgb(94, 94, 94);">];</span></div><div> <span style="color: rgb(94, 94, 94);">}</span></div><div> <span style="color: rgb(9, 89, 132);">n_atoms_local</span><span style="color: rgb(94, 94, 94);">++;</span></div><div> <span style="color: rgb(94, 94, 94);">}</span></div><div> <span style="color: rgb(94, 94, 94);">}</span></div><br></div><div><br><blockquote type="cite"><div>On Mar 30, 2026, at 2:50 AM, Matthew Knepley <knepley@gmail.com> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-line: none; text-decoration-thickness: auto; text-decoration-style: solid;"><div dir="ltr">On Sun, Mar 29, 2026 at 6:25 PM Miguel Molinos <<a href="mailto:m.molinos@upm.es">m.molinos@upm.es</a>> wrote:</div><div class="gmail_quote gmail_quote_container"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div><p style="">Dear all,</p><p style="">I am experiencing an issue where particles created with DMSwarm are being increased across MPI ranks when initializing a DMSwarm over a periodic DMPlex. I follow this procedure on every rank:</p><ol start="1"><li style=""><p>Create a sequential vector with the global coordinates of all particles (18,000 atoms).</p></li><li><p><font>Call DMLocatePoints(FE_Mesh, coordinates, DM_POINTLOCATION_NONE, &cellSF) to find which particles belong to the local DMPlex partition. Then call </font><span style="white-space: pre-wrap; background-color: rgb(255, 255, 255);"><font>PetscSFGetGraph.</font></span></p></li><li style=""><p>Iterate through the results and add any particle with a valid cell index (index != DMLOCATEPOINT_POINT_NOT_FOUND) to the local DMSwarm.</p></li></ol><p style="">The sum of n_atoms_local across all ranks gives 18,740 instead of the expected 18,000. It seems that particles located exactly on periodic boundaries or MPI interfaces are being identified as "found" by multiple ranks simultaneously. Does DMLocatePoints return points found in ghost/halo cells by default when the DMPlex is periodic or distributed with overlap?</p></div></blockquote><div>I do not think this should happen at periodic boundaries, but definitely if you have cell overlap it will happen. It iterates over all local cells to check particle location. You could discard those points by checking if the cell is in the point SF in Step 3.</div><div><br></div><div> <span class="Apple-converted-space"> </span>Thanks,</div><div><br></div><div> Matt </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div><p style="">Thanks,</p><p style="">Miguel</p></div></blockquote></div><div><br clear="all"></div><div><br></div><span class="gmail_signature_prefix">--<span class="Apple-converted-space"> </span></span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="https://urldefense.com/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!D9dNQwwGXtA!X58muDkArsK-iH2YdyGstzlwOwzuxXNE308JQOmlP5pY7V8oHcgLbmf1Qiazmqv1IdaROicSzLQR_o8$" target="_blank">https://www.cse.buffalo.edu/~knepley/</a></div></div></div></div></div></div></div></div></div></blockquote></div><br></div></body></html>