<div dir="ltr"><div dir="ltr">On Thu, Dec 22, 2022 at 3:08 PM Matteo Semplice <<a href="mailto:matteo.semplice@uninsubria.it">matteo.semplice@uninsubria.it</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p><br>
</p>
<div>Il 22/12/22 20:06, Dave May ha scritto:<br>
</div>
<blockquote type="cite">
<div><br>
</div>
<div><br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu 22. Dec 2022 at
10:27, Matteo Semplice <<a href="mailto:matteo.semplice@uninsubria.it" target="_blank">matteo.semplice@uninsubria.it</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Dear Dave and Matt,</p>
<p> I am really dealing with two different use cases in
a code that will compute a levelset function passing
through a large set of points. If I had
DMSwarmSetMigrateType() and if it were safe to switch
the migration mode back and forth in the same swarm,
this would cover all my use cases here. Is it safe to
add it back to petsc? Details below if you are curious.<br>
</p>
<p>1) During preprocessing I am loading a point cloud from
disk (in whatever order it comes) and need to send the
particles to the right ranks. Since the background DM is
a DMDA I can easily figure out the destination rank.
This would be covered by your suggestion not to attach
the DM, except that later I need to locate these points
with respect to the background cells in order to
initialize data on the Vecs associated to the DMDA.<br>
</p>
<p>2) Then I need to implement a semilagrangian time
evolution scheme. For this I'd like to send particles
around at the "foot of characteristic", collect data
there and then send them back to the originating point.
The first migration would be based on particle
coordinates (DMSwarmMigrate_DMNeighborScatter and the
restriction to only neighbouring ranks is perfect),
while for the second move it would be easier to just
send them back to the originating rank, which I can
easily store in an Int field in the swarm. Thus at each
timestep I'd need to swap migrate types in this swarm
(DMScatter for moving them to the feet and BASIC to send
them back).</p>
</div>
</blockquote>
<div dir="auto"><br>
</div>
<div dir="auto">When you use BASIC, you would have to
explicitly call the point location routine from your code as
BASIC does not interact with the DM. </div>
<div dir="auto"><br>
</div>
<div dir="auto">Based on what I see in the code, switching
migrate modes between basic and dmneighbourscatter should
be safe.</div>
<div dir="auto"><br>
</div>
<div dir="auto">If you are fine calling the point location
from your side then what you propose should work.</div>
</div>
</div>
</blockquote>
<p>If I understood the code correctly, BASIC will just migrate
particles sending them to what is stored in DMSwarmField_rank,
right? That'd be easy since I can create a SWARM with all the data
I need and an extra int field (say "original_rank") and copy those
values into DMSwarmField_rank before calling migrate for the
"going back" step. After this backward migration I do not need to
locate particles again (e.g. I do not need DMSwarmSortGetAccess
after the BASIC migration, but only after the DMNeighborScatter
one).</p>
<p>Thus having back DMSwarmSetMigrateType() should be enough for me.<br>
</p>
<p></p></div></blockquote><div>Hi Matteo,</div><div><br></div><div>I have done this in</div><div><br></div><div> <a href="https://gitlab.com/petsc/petsc/-/merge_requests/5941">https://gitlab.com/petsc/petsc/-/merge_requests/5941</a></div><div><br></div><div>I also hope to get the fix for your DMDA issue in there.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><p>Thanks</p>
<p> Matteo<br>
</p>
<blockquote type="cite">
<div>
<div class="gmail_quote">
<div dir="auto"><br>
</div>
<div dir="auto">Cheers</div>
<div dir="auto">Dave</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p dir="auto"><br>
</p>
Thanks
<p> Matteo<br>
</p>
<div>Il 22/12/22 18:40, Dave May ha scritto:<br>
</div>
</div>
<div>
<blockquote type="cite">
<div dir="auto">Hey Matt,</div>
<div style="background-color:rgba(0,0,0,0);border-color:rgb(0,0,0)"><br>
<div class="gmail_quote" style="background-color:rgba(0,0,0,0);border-color:rgb(0,0,0)">
<div dir="ltr" class="gmail_attr">On Thu 22. Dec
2022 at 05:02, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Thu, Dec 22, 2022 at 6:28 AM
Matteo Semplice <<a href="mailto:matteo.semplice@uninsubria.it" target="_blank">matteo.semplice@uninsubria.it</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Dear all</p>
<p> please ignore my previous email and
read this one: I have better localized
the problem. Maybe DMSwarmMigrate is
designed to migrate particles only to
first neighbouring ranks?<br>
</p>
</div>
</blockquote>
<div>Yes, I believe that was the design.</div>
<div><br>
</div>
<div>Dave, is this correct?</div>
</div>
</div>
</blockquote>
<div dir="auto"><br>
</div>
<div dir="auto">Correct. <span style="font-size:21px;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)">DMSwarmMigrate_</span><span style="font-size:21px;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)">DMNeighborScatter()
only scatter points to the neighbour ranks -
where neighbours are defined by the DM provided
to represent the mesh.</span><br>
</div>
<div dir="auto"><span style="font-size:21px;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)"><br>
</span></div>
<div dir="auto"><span style="font-size:21px;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)">DMSwarmMigrate_</span><span style="font-size:21px;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)">DMNeighborScatter()
Is selected by default if you attach a DM.</span><br>
</div>
<div dir="auto"><span style="font-size:21px;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)"><br>
</span></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px">The
scatter method should be over ridden with </span></font></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px"><span style="word-spacing:0px;font-size:medium;color:rgb(0,0,0)"><br>
</span></span></font></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px"><span style="word-spacing:0px;font-size:medium;color:rgb(0,0,0)">DMSwarmSetMigrateType()</span><br>
</span></font></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px"><span style="word-spacing:0px;font-size:medium;color:rgb(0,0,0)"><br>
</span></span></font></div>
<div dir="auto" style="background-color:rgba(0,0,0,0);border-color:rgb(0,0,0)"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px">however
it appears this method no longer exists.</span></font></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px">
<div><br>
</div>
If one can determine the exact rank where
points should should be sent and it is not
going to be the neighbour rank (given by the
DM), I would suggest not attaching the DM at
all.</span></font></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px"><br>
</span></font></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px">However
if this is not possible and one wanted to
scatter to say the neighbours neighbours, we
will have to add a new interface and refactor
things a little bit.</span></font></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px"><br>
</span></font></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px">Cheers</span></font></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px">Dave</span></font></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px"><br>
</span></font></div>
<div dir="auto"><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
</div>
</div>
<div dir="ltr">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Il 22/12/22 11:44, Matteo Semplice ha
scritto:<br>
</div>
<blockquote type="cite">
<p>Dear everybody,</p>
<p> I have bug a bit into the code
and I am able to add more information.<br>
</p>
<div>Il 02/12/22 12:48, Matteo Semplice
ha scritto:<br>
</div>
<blockquote type="cite">
<div>Hi.</div>
<div>I am sorry to take this up again,
but further tests show that it's not
right yet.<br>
</div>
<div><br>
</div>
<div>Il 04/11/22 12:48, Matthew
Knepley ha scritto:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Fri, Nov 4, 2022
at 7:46 AM Matteo Semplice <<a href="mailto:matteo.semplice@uninsubria.it" target="_blank">matteo.semplice@uninsubria.it</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div> On 04/11/2022 02:43,
Matthew Knepley wrote:<br>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Thu,
Nov 3, 2022 at 8:36 PM
Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On
Thu, Oct 27,
2022 at 11:57 AM
Semplice Matteo
<<a href="mailto:matteo.semplice@uninsubria.it" target="_blank">matteo.semplice@uninsubria.it</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;background-color:rgb(255,255,255);color:rgb(0,0,0)">Dear
Petsc
developers,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;background-color:rgb(255,255,255);color:rgb(0,0,0)">
I am trying to
use a DMSwarm
to locate a
cloud of
points with
respect to a
background
mesh. In the
real
application
the points
will be loaded
from disk, but
I have created
a small demo
in which</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;background-color:rgb(255,255,255);color:rgb(0,0,0)">
<ul style="font-family:Calibri,Arial,Helvetica,sans-serif">
<li style="font-family:Calibri,Arial,Helvetica,sans-serif"><span style="font-family:Calibri,Arial,Helvetica,sans-serif">each processor
creates Npart
particles, all
within the
domain covered
by the mesh,
but not all in
the local
portion of the
mesh</span></li>
<li style="font-family:Calibri,Arial,Helvetica,sans-serif"><span style="font-family:Calibri,Arial,Helvetica,sans-serif">migrate the
particles</span></li>
</ul>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif">After
migration most
particles are
not any more
in the DMSwarm
(how many and
which ones
seems to
depend on the
number of
cpus, but it
never happens
that all
particle
survive the
migration
process).</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><br>
</div>
</div>
</div>
</div>
</blockquote>
<div>Thanks for
sending this.
I found the
problem.
Someone has
some overly
fancy code
inside DMDA to
figure out the
local bounding
box from the
coordinates.</div>
<div>It is
broken for
DM_BOUNDARY_GHOSTED,
but we never
tested with
this. I will
fix it.</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Okay, I think
this fix is correct</div>
<div><br>
</div>
<div> <a href="https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fpetsc%2Fpetsc%2F-%2Fmerge_requests%2F5802&data=05%7C01%7Cmatteo.semplice%40uninsubria.it%7C94569ac2a32a4838103608dae44f9c46%7C9252ed8bdffc401c86ca6237da9991fa%7C0%7C0%7C638073327837332509%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9yKNH0%2FbepSLU2PO5S%2B2GkYSmjjFIDGHpapy%2FUb43D4%3D&reserved=0" target="_blank">https://gitlab.com/petsc/petsc/-/merge_requests/5802</a></div>
<div><br>
</div>
<div>I incorporated
your test as
src/dm/impls/da/tests/ex1.c.
Can you take a look
and see if this
fixes your issue?</div>
</div>
</div>
</blockquote>
<p>Yes, we have tested 2d
and 3d, with various
combinations of
DM_BOUNDARY_* along
different directions and
it works like a charm.</p>
<p>On a side note, neither <span style="font-family:monospace"><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(0,0,0)">DMSwarmViewXDMF</span></span>
nor <span style="font-family:monospace"><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(0,0,0)">DMSwarmMigrate</span></span>
seem to be implemented for
1d: I get</p>
<p><span style="font-family:monospace"><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(0,0,0)">[0]PETSC
ERROR: No support for
this operation for
this object type</span><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(178,24,24)">
</span><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(0,0,0)">
</span>[0]PETSC ERROR:
Support not provided for
1D<br>
</span></p>
<p>However, currently I have
no need for this feature.<br>
<span style="font-family:monospace"></span></p>
<p>Finally, if the test is
meant to stay in the
source, you may remove the
call to <span style="font-family:monospace"><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(0,0,0)">DMSwarmRegisterPetscDatatypeField</span></span>
as in the attached patch.<br>
</p>
<p>Thanks a lot!!</p>
</div>
</blockquote>
<div>Thanks! Glad it works.</div>
<div>
<p> Matt</p>
</div>
</div>
</div>
</blockquote>
<p cols="72">There are still problems
when not using 1,2 or 4 cpus. Any
other number of cpus that I've
tested does not work corectly.<br>
</p>
</blockquote>
<p>I have now modified
private_DMDALocatePointsIS_2D_Regular
to print out some debugging
information. I see that this is called
twice during migration, once before
and once after
DMSwarmMigrate_DMNeighborScatter. If I
understand correctly, the second call
to
private_DMDALocatePointsIS_2D_Regular
should be able to locate all particles
owned by the rank but it fails for
some of them because they have been
sent to the wrong rank (despite being
well away from process boundaries).<br>
</p>
<p>For example, running the example
src/dm/impls/da/tests/ex1.c with Nx=21
(20x20 Q1 elements on [-1,1]X[-1,1])
with 3 processors,</p>
<p>- the particles (-0.191,-0.462) and
(0.191,-0.462) are sent cpu2 instead
of cpu0</p>
<p>- those at (-0.287,-0.693)and
(0.287,-0.693) are sent to cpu1
instead of cpu0</p>
<p>- those at (0.191,0.462) and
(-0.191,0.462) are sent to cpu0
instead of cpu2</p>
<p>(This is 2d and thus not affected by
the 3d issue mentioned yesterday on
petsc-dev. Tests were made based on
the release branch pulled out this
morning, i.e. on commit <span style="font-family:monospace"><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(178,104,24)">bebdc8d016f</span></span>).<br>
</p>
</blockquote>
<p>I see: particles are sent "all around"
and not only to the destination rank.</p>
<p>Still however, running the example
src/dm/impls/da/tests/ex1.c with Nx=21
(20x20 Q1 elements on [-1,1]X[-1,1])
with 3 processors, there are 2 particles
initially owned by rank2 (at y=-0.6929
and x=+/-0.2870) that are sent only to
rank1 and never make it to rank0 and are
thus lost in the end since rank1,
correctly, discards them.</p>
<p>Thanks</p>
<p> Matteo<br>
</p>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take
for granted before they begin
their experiments is infinitely
more interesting than any results
to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cse.buffalo.edu%2F~knepley%2F&data=05%7C01%7Cmatteo.semplice%40uninsubria.it%7C94569ac2a32a4838103608dae44f9c46%7C9252ed8bdffc401c86ca6237da9991fa%7C0%7C0%7C638073327837332509%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=LEmGbachok2Z3m9SyCASEcTWytkpxw%2FnHCIXjYXPTa8%3D&reserved=0" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<pre cols="72">--
---
Professore Associato in Analisi Numerica
Dipartimento di Scienza e Alta Tecnologia
Università degli Studi dell'Insubria
<a href="https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.google.com%2Fmaps%2Fsearch%2FVia%2BValleggio%2C%2B11%2B-%2BComo%3Fentry%3Dgmail%26source%3Dg&data=05%7C01%7Cmatteo.semplice%40uninsubria.it%7C94569ac2a32a4838103608dae44f9c46%7C9252ed8bdffc401c86ca6237da9991fa%7C0%7C0%7C638073327837332509%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=0MwkPJb5hslcrFhqrLCaN4OKs%2FoylMR3dOib2MYsNOY%3D&reserved=0" target="_blank">Via Valleggio, 11 - Como</a></pre>
</div>
</blockquote>
</div>
</div>
</blockquote>
<pre cols="72">--
---
Professore Associato in Analisi Numerica
Dipartimento di Scienza e Alta Tecnologia
Università degli Studi dell'Insubria
Via Valleggio, 11 - Como</pre>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>