<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div class="moz-cite-prefix">Hi.</div>
<div class="moz-cite-prefix">I am sorry to take this up again, but
further tests show that it's not right yet.<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Il 04/11/22 12:48, Matthew Knepley ha
scritto:<br>
</div>
<blockquote type="cite" cite="mid:CAMYG4Gnznu1ct7GhjYNcBVRqPmpuVNnz1O_4OqM-_s9fZ28=vA@mail.gmail.com">
<div dir="ltr">
<div dir="ltr">On Fri, Nov 4, 2022 at 7:46 AM Matteo Semplice
<<a href="mailto:matteo.semplice@uninsubria.it" moz-do-not-send="true" class="moz-txt-link-freetext">matteo.semplice@uninsubria.it</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div> On 04/11/2022 02:43, Matthew Knepley wrote:<br>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Thu, Nov 3, 2022 at 8:36 PM Matthew
Knepley <<a href="mailto:knepley@gmail.com" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">knepley@gmail.com</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Thu, Oct 27, 2022 at 11:57 AM
Semplice Matteo <<a href="mailto:matteo.semplice@uninsubria.it" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">matteo.semplice@uninsubria.it</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">Dear
Petsc developers,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
I am trying to use a DMSwarm to locate
a cloud of points with respect to a
background mesh. In the real
application the points will be loaded
from disk, but I have created a small
demo in which</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<ul>
<li><span>each processor creates
Npart particles, all within the
domain covered by the mesh, but
not all in the local portion of
the mesh</span></li>
<li><span>migrate the particles</span></li>
</ul>
<div>After migration most particles
are not any more in the DMSwarm (how
many and which ones seems to depend
on the number of cpus, but it never
happens that all particle survive
the migration process).</div>
<div><br>
</div>
<div>I am clearly missing some step,
since I'd expect that a DMDA would
be able to locate particles without
the need to go through a DMShell as
it is done in <a href="https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fsrc%2Fdm%2Ftutorials%2Fswarm_ex3.c.html&data=05%7C01%7Cmatteo.semplice%40uninsubria.it%7C60177cfedb2948f480d008dabe5a7e7a%7C9252ed8bdffc401c86ca6237da9991fa%7C0%7C0%7C638031593122164987%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=43%2FIiK5P%2FhBStaVKhonw0p7wb87L8jqf2MheL9E1gR0%3D&reserved=0" originalsrc="https://petsc.org/main/src/dm/tutorials/swarm_ex3.c.html" shash="aJ8CegKwWsTdHUKKsgq63G9ffl8oISK6WbwdpycPc+UKai5AmmtIgvshVu60ujJmzVmfykRBsdGZc6CZoL5HRBuH9iZPWXiHSiwnf+Md7K6vrlBsbEPGgnnjCpYFjryGgtYOu09JRTYS/0Y5X9ovizgVV8UJCdMYvpK0NsIo6n8=" target="_blank" moz-do-not-send="true">
src/dm/tutorials/swarm_ex3.c.html</a><br>
</div>
<div><br>
</div>
<div>I attach my demo code.</div>
<div><br>
</div>
<div>Could someone give me a hint?</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Thanks for sending this. I found the
problem. Someone has some overly fancy code
inside DMDA to figure out the local bounding
box from the coordinates.</div>
<div>It is broken for DM_BOUNDARY_GHOSTED, but
we never tested with this. I will fix it.</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Okay, I think this fix is correct</div>
<div><br>
</div>
<div> <a href="https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fpetsc%2Fpetsc%2F-%2Fmerge_requests%2F5802&data=05%7C01%7Cmatteo.semplice%40uninsubria.it%7C60177cfedb2948f480d008dabe5a7e7a%7C9252ed8bdffc401c86ca6237da9991fa%7C0%7C0%7C638031593122164987%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=trNMoF9qIT91tBwkyatWdl7%2BB0Jy0AtJdsL8oYmN7sU%3D&reserved=0" originalsrc="https://gitlab.com/petsc/petsc/-/merge_requests/5802" shash="GHJdakswvpmVCsAZ5YWb96XeXovPKxxtm/st5Xk8CJPKMcZ3sXiW6PcezVCQkX4YxxuXVT/vG0olMl9LtcoH8VBnY0H3qsDDqCHRB+9EqndOa2rRN44mfGvVAdzW5GXbZtDtflyYsqUXidLm9LhzABe87vxcvDqoJ3+9e8P/Ikc=" target="_blank" moz-do-not-send="true">https://gitlab.com/petsc/petsc/-/merge_requests/5802</a></div>
<div><br>
</div>
<div>I incorporated your test as
src/dm/impls/da/tests/ex1.c. Can you take a look
and see if this fixes your issue?</div>
</div>
</div>
</blockquote>
<p>Yes, we have tested 2d and 3d, with various
combinations of DM_BOUNDARY_* along different directions
and it works like a charm.</p>
<p>On a side note, neither <span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">DMSwarmViewXDMF</span></span>
nor <span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">DMSwarmMigrate</span></span>
seem to be implemented for 1d: I get</p>
<p><span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">[0]PETSC
ERROR: No support for this operation for this object
type</span><span style="color:rgb(178,24,24);background-color:rgb(255,255,255)">
</span><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)"> </span>[0]PETSC
ERROR: Support not provided for 1D<br>
</span></p>
<p>However, currently I have no need for this feature.<br>
<span style="font-family:monospace"></span></p>
<p>Finally, if the test is meant to stay in the source,
you may remove the call to <span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">DMSwarmRegisterPetscDatatypeField</span></span>
as in the attached patch.<br>
</p>
<p>Thanks a lot!!</p>
</div>
</blockquote>
<div>Thanks! Glad it works.</div>
<div>
<p> Matt</p>
</div>
</div>
</div>
</blockquote>
<p class="moz-signature" cols="72">There are still problems when not
using 1,2 or 4 cpus. Any other number of cpus that I've tested
does not work corectly.<br>
</p>
<p class="moz-signature" cols="72">I have modified <span style="font-family:monospace"><span style="color:#000000;background-color:#ffffff;">src/dm/impls/da/tests/ex1.c</span>
</span>to show the problem (see attached patch). In particular, I
have introduced a -Nx option to allow grid sizes higher than the
standard 4x4 and also suppressed by default particle coordinates
output (but there's an option -part_view that recreates the old
behaviour). Here's the output on my machine:</p>
<p class="moz-signature" cols="72"><span style="font-family:monospace"><span style="color:#000000;background-color:#ffffff;"><span style="font-family:monospace"><span style="color:#000000;background-color:#ffffff;">$ mpirun
-np 1 ./da_test_ex1 -Nx 40 </span><br>
Total of 8 particles
<br>
... calling DMSwarmMigrate ...
<br>
Total of 8 particles<br>
<br>
</span></span></span></p>
<p class="moz-signature" cols="72"><span style="font-family:monospace"><span style="color:#000000;background-color:#ffffff;">$ mpirun -np 2
./da_test_ex1 -Nx 40 </span><br>
Total of 16 particles
<br>
... calling DMSwarmMigrate ...
<br>
Total of 16 particles
<br>
<span style="font-weight:bold;color:#5454ff;background-color:#ffffff;"></span><span style="color:#000000;background-color:#ffffff;"><br>
</span></span></p>
<p class="moz-signature" cols="72"><span style="font-family:monospace"><span style="color:#000000;background-color:#ffffff;">$ mpirun -np 4
./da_test_ex1 -Nx 40 </span><br>
Total of 32 particles
<br>
... calling DMSwarmMigrate ...
<br>
Total of 32 particles
<br>
<span style="font-weight:bold;color:#5454ff;background-color:#ffffff;"></span><span style="color:#000000;background-color:#ffffff;"><br>
</span></span></p>
<p class="moz-signature" cols="72"><span style="font-family:monospace"><span style="color:#000000;background-color:#ffffff;">$ mpirun -np 3
./da_test_ex1 -Nx 40 </span><br>
Total of 24 particles
<br>
... calling DMSwarmMigrate ...
<br>
Total of 22 particles
<br>
<span style="font-weight:bold;color:#5454ff;background-color:#ffffff;"></span><span style="color:#000000;background-color:#ffffff;"><br>
</span></span></p>
<p class="moz-signature" cols="72"><span style="font-family:monospace"><span style="color:#000000;background-color:#ffffff;">$ mpirun
-oversubscribe -np 5 ./da_test_ex1 -Nx 40 </span><br>
Total of 40 particles
<br>
... calling DMSwarmMigrate ...
<br>
Total of 22 particles
<br>
</span></p>
<p class="moz-signature" cols="72"><span style="font-family:monospace">$<span style="color:#000000;background-color:#ffffff;"> mpirun
-oversubscribe -np 8 ./da_test_ex1 -Nx 40 </span><br>
Total of 64 particles
<br>
... calling DMSwarmMigrate ...
<br>
Total of 46 particles<br>
</span></p>
<p class="moz-signature" cols="72">As you see, only 1,2,4 cpus do
not lose particles.</p>
<p class="moz-signature" cols="72">(The test could be easily
modified to return nTotPart/predistribution -
nTotPart/postdistribution so that it will have a nonzero exit code
in case it loses particles)<br>
</p>
<p class="moz-signature" cols="72">Best</p>
<p class="moz-signature" cols="72"> Matteo<br>
</p>
<br>
</body>
</html>