<div dir="auto">Hey Matt,</div><div style="background-color:rgba(0,0,0,0)!important;border-color:rgb(0,0,0)!important"><br><div class="gmail_quote" style="background-color:rgba(0,0,0,0)!important;border-color:rgb(0,0,0)!important"><div dir="ltr" class="gmail_attr">On Thu 22. Dec 2022 at 05:02, Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div dir="ltr"><div dir="ltr">On Thu, Dec 22, 2022 at 6:28 AM Matteo Semplice <<a href="mailto:matteo.semplice@uninsubria.it" target="_blank">matteo.semplice@uninsubria.it</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
<div>
<p>Dear all</p>
<p> please ignore my previous email and read this one: I have
better localized the problem. Maybe DMSwarmMigrate is designed to
migrate particles only to first neighbouring ranks?<br>
</p>
<div></div></div></blockquote><div>Yes, I believe that was the design.</div><div><br></div><div>Dave, is this correct?</div></div></div></blockquote><div dir="auto"><br></div><div dir="auto">Correct. <span style="font-size:21px;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)">DMSwarmMigrate_</span><span style="font-size:21px;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)">DMNeighborScatter() only scatter points to the neighbour ranks - where neighbours are defined by the DM provided to represent the mesh.</span><br></div><div dir="auto"><span style="font-size:21px;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)"><br></span></div><div dir="auto"><span style="font-size:21px;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)">DMSwarmMigrate_</span><span style="font-size:21px;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)">DMNeighborScatter() Is selected by default if you attach a DM.</span><br></div><div dir="auto"><span style="font-size:21px;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)"><br></span></div><div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px">The scatter method should be over ridden with </span></font></div><div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px"><span style="word-spacing:0px;font-size:medium;color:rgb(0,0,0)"><br></span></span></font></div><div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px"><span style="word-spacing:0px;font-size:medium;color:rgb(0,0,0)">DMSwarmSetMigrateType()</span><br></span></font></div><div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px"><span style="word-spacing:0px;font-size:medium;color:rgb(0,0,0)"><br></span></span></font></div><div dir="auto" style="background-color:rgba(0,0,0,0)!important;border-color:rgb(0,0,0)!important"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px">however it appears this method no longer exists.</span></font></div><div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px"><div><br></div>If one can determine the exact rank where points should should be sent and it is not going to be the neighbour rank (given by the DM), I would suggest not attaching the DM at all.</span></font></div><div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px"><br></span></font></div><div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px">However if this is not possible and one wanted to scatter to say the neighbours neighbours, we will have to add a new interface and refactor things a little bit.</span></font></div><div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px"><br></span></font></div><div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px">Cheers</span></font></div><div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px">Dave</span></font></div><div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px"><br></span></font></div><div dir="auto"><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div dir="ltr"><div class="gmail_quote"><div dir="auto"></div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div></div></div><div dir="ltr"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div><div>Il 22/12/22 11:44, Matteo Semplice ha
scritto:<br>
</div>
<blockquote type="cite">
<p>Dear everybody,</p>
<p> I have bug a bit into the code and I am able to add more
information.<br>
</p>
<div>Il 02/12/22 12:48, Matteo Semplice ha
scritto:<br>
</div>
<blockquote type="cite">
<div>Hi.</div>
<div>I am sorry to take this up again,
but further tests show that it's not right yet.<br>
</div>
<div><br>
</div>
<div>Il 04/11/22 12:48, Matthew Knepley
ha scritto:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Fri, Nov 4, 2022 at 7:46 AM Matteo
Semplice <<a href="mailto:matteo.semplice@uninsubria.it" target="_blank">matteo.semplice@uninsubria.it</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
<div> On 04/11/2022 02:43, Matthew Knepley wrote:<br>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Thu, Nov 3, 2022 at 8:36 PM
Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
<div dir="ltr">
<div dir="ltr">On Thu, Oct 27, 2022 at 11:57
AM Semplice Matteo <<a href="mailto:matteo.semplice@uninsubria.it" target="_blank">matteo.semplice@uninsubria.it</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
<div>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;background-color:rgb(255,255,255);color:rgb(0,0,0)">Dear
Petsc developers,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;background-color:rgb(255,255,255);color:rgb(0,0,0)">
I am trying to use a DMSwarm to
locate a cloud of points with
respect to a background mesh. In
the real application the points
will be loaded from disk, but I
have created a small demo in which</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;background-color:rgb(255,255,255);color:rgb(0,0,0)">
<ul style="font-family:Calibri,Arial,Helvetica,sans-serif">
<li style="font-family:Calibri,Arial,Helvetica,sans-serif"><span style="font-family:Calibri,Arial,Helvetica,sans-serif">each processor creates
Npart particles, all within
the domain covered by the
mesh, but not all in the
local portion of the mesh</span></li>
<li style="font-family:Calibri,Arial,Helvetica,sans-serif"><span style="font-family:Calibri,Arial,Helvetica,sans-serif">migrate the particles</span></li>
</ul>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif">After migration most
particles are not any more in
the DMSwarm (how many and which
ones seems to depend on the
number of cpus, but it never
happens that all particle
survive the migration process).</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><br>
</div>
</div>
</div>
</div>
</blockquote>
<div>Thanks for sending this. I found the
problem. Someone has some overly fancy
code inside DMDA to figure out the local
bounding box from the coordinates.</div>
<div>It is broken for DM_BOUNDARY_GHOSTED,
but we never tested with this. I will
fix it.</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Okay, I think this fix is correct</div>
<div><br>
</div>
<div> <a href="https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fpetsc%2Fpetsc%2F-%2Fmerge_requests%2F5802&data=05%7C01%7Cmatteo.semplice%40uninsubria.it%7Cf4d64b09df1f438437ad08dad45b342b%7C9252ed8bdffc401c86ca6237da9991fa%7C0%7C0%7C638055785720875500%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=ISLaLhhnYU4njkYfod%2F3tEiIOIV5uZvmiAlKZ2PvhmE%3D&reserved=0" target="_blank">https://gitlab.com/petsc/petsc/-/merge_requests/5802</a></div>
<div><br>
</div>
<div>I incorporated your test as
src/dm/impls/da/tests/ex1.c. Can you take a
look and see if this fixes your issue?</div>
</div>
</div>
</blockquote>
<p>Yes, we have tested 2d and 3d, with various
combinations of DM_BOUNDARY_* along different
directions and it works like a charm.</p>
<p>On a side note, neither <span style="font-family:monospace"><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(0,0,0)">DMSwarmViewXDMF</span></span>
nor <span style="font-family:monospace"><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(0,0,0)">DMSwarmMigrate</span></span>
seem to be implemented for 1d: I get</p>
<p><span style="font-family:monospace"><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(0,0,0)">[0]PETSC
ERROR: No support for this operation for this
object type</span><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(178,24,24)">
</span><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(0,0,0)"> </span>[0]PETSC
ERROR: Support not provided for 1D<br>
</span></p>
<p>However, currently I have no need for this feature.<br>
<span style="font-family:monospace"></span></p>
<p>Finally, if the test is meant to stay in the
source, you may remove the call to <span style="font-family:monospace"><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(0,0,0)">DMSwarmRegisterPetscDatatypeField</span></span>
as in the attached patch.<br>
</p>
<p>Thanks a lot!!</p>
</div>
</blockquote>
<div>Thanks! Glad it works.</div>
<div>
<p> Matt</p>
</div>
</div>
</div>
</blockquote>
<p cols="72">There are still problems when
not using 1,2 or 4 cpus. Any other number of cpus that I've
tested does not work corectly.<br>
</p>
</blockquote>
<p>I have now modified private_DMDALocatePointsIS_2D_Regular to
print out some debugging information. I see that this is called
twice during migration, once before and once after
DMSwarmMigrate_DMNeighborScatter. If I understand correctly, the
second call to private_DMDALocatePointsIS_2D_Regular should be
able to locate all particles owned by the rank but it fails for
some of them because they have been sent to the wrong rank
(despite being well away from process boundaries).<br>
</p>
<p>For example, running the example src/dm/impls/da/tests/ex1.c
with Nx=21 (20x20 Q1 elements on [-1,1]X[-1,1]) with 3
processors,</p>
<p>- the particles (-0.191,-0.462) and (0.191,-0.462) are sent
cpu2 instead of cpu0</p>
<p>- those at (-0.287,-0.693)and (0.287,-0.693) are sent to cpu1
instead of cpu0</p>
<p>- those at (0.191,0.462) and (-0.191,0.462) are sent to cpu0
instead of cpu2</p>
<p>(This is 2d and thus not affected by the 3d issue mentioned
yesterday on petsc-dev. Tests were made based on the release
branch pulled out this morning, i.e. on commit <span style="font-family:monospace"><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(178,104,24)">bebdc8d016f</span></span>).<br>
</p>
</blockquote>
<p>I see: particles are sent "all around" and not only to the
destination rank.</p>
<p>Still however, running the example src/dm/impls/da/tests/ex1.c
with Nx=21 (20x20 Q1 elements on [-1,1]X[-1,1]) with 3 processors,
there are 2 particles initially owned by rank2 (at y=-0.6929 and
x=+/-0.2870) that are sent only to rank1 and never make it to
rank0 and are thus lost in the end since rank1, correctly,
discards them.</p>
<p>Thanks</p>
<p> Matteo<br>
</p>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div></div>