<div><br></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu 22. Dec 2022 at 10:27, Matteo Semplice <<a href="mailto:matteo.semplice@uninsubria.it">matteo.semplice@uninsubria.it</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<p>Dear Dave and Matt,</p>
<p> I am really dealing with two different use cases in a code
that will compute a levelset function passing through a large set
of points. If I had DMSwarmSetMigrateType() and if it were safe to
switch the migration mode back and forth in the same swarm, this
would cover all my use cases here. Is it safe to add it back to
petsc? Details below if you are curious.<br>
</p>
<p>1) During preprocessing I am loading a point cloud from disk (in
whatever order it comes) and need to send the particles to the
right ranks. Since the background DM is a DMDA I can easily figure
out the destination rank. This would be covered by your suggestion
not to attach the DM, except that later I need to locate these
points with respect to the background cells in order to initialize
data on the Vecs associated to the DMDA.<br>
</p>
<p>2) Then I need to implement a semilagrangian time evolution
scheme. For this I'd like to send particles around at the "foot of
characteristic", collect data there and then send them back to the
originating point. The first migration would be based on particle
coordinates (DMSwarmMigrate_DMNeighborScatter and the restriction
to only neighbouring ranks is perfect), while for the second move
it would be easier to just send them back to the originating rank,
which I can easily store in an Int field in the swarm. Thus at
each timestep I'd need to swap migrate types in this swarm
(DMScatter for moving them to the feet and BASIC to send them
back).</p></div></blockquote><div dir="auto"><br></div><div dir="auto">When you use BASIC, you would have to explicitly call the point location routine from your code as BASIC does not interact with the DM. </div><div dir="auto"><br></div><div dir="auto">Based on what I see in the code, switching migrate modes between basic and dmneighbourscatter should be safe.</div><div dir="auto"><br></div><div dir="auto">If you are fine calling the point location from your side then what you propose should work.</div><div dir="auto"><br></div><div dir="auto">Cheers</div><div dir="auto">Dave</div><div dir="auto"><br></div><div dir="auto"><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><p dir="auto"><br>
</p>
Thanks
<p> Matteo<br>
</p>
<div>Il 22/12/22 18:40, Dave May ha scritto:<br>
</div></div><div>
<blockquote type="cite">
<div dir="auto">Hey Matt,</div>
<div style="background-color:rgba(0,0,0,0)!important;border-color:rgb(0,0,0)!important"><br>
<div class="gmail_quote" style="background-color:rgba(0,0,0,0)!important;border-color:rgb(0,0,0)!important">
<div dir="ltr" class="gmail_attr">On Thu 22. Dec 2022 at
05:02, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
<div dir="ltr">
<div dir="ltr">On Thu, Dec 22, 2022 at 6:28 AM Matteo
Semplice <<a href="mailto:matteo.semplice@uninsubria.it" target="_blank">matteo.semplice@uninsubria.it</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
<div>
<p>Dear all</p>
<p> please ignore my previous email and read this
one: I have better localized the problem. Maybe
DMSwarmMigrate is designed to migrate particles
only to first neighbouring ranks?<br>
</p>
</div>
</blockquote>
<div>Yes, I believe that was the design.</div>
<div><br>
</div>
<div>Dave, is this correct?</div>
</div>
</div>
</blockquote>
<div dir="auto"><br>
</div>
<div dir="auto">Correct. <span style="font-size:21px;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)">DMSwarmMigrate_</span><span style="font-size:21px;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)">DMNeighborScatter()
only scatter points to the neighbour ranks - where
neighbours are defined by the DM provided to represent the
mesh.</span><br>
</div>
<div dir="auto"><span style="font-size:21px;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)"><br>
</span></div>
<div dir="auto"><span style="font-size:21px;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)">DMSwarmMigrate_</span><span style="font-size:21px;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)">DMNeighborScatter()
Is selected by default if you attach a DM.</span><br>
</div>
<div dir="auto"><span style="font-size:21px;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)"><br>
</span></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px">The scatter
method should be over ridden with </span></font></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px"><span style="word-spacing:0px;font-size:medium;color:rgb(0,0,0)"><br>
</span></span></font></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px"><span style="word-spacing:0px;font-size:medium;color:rgb(0,0,0)">DMSwarmSetMigrateType()</span><br>
</span></font></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px"><span style="word-spacing:0px;font-size:medium;color:rgb(0,0,0)"><br>
</span></span></font></div>
<div dir="auto" style="background-color:rgba(0,0,0,0)!important;border-color:rgb(0,0,0)!important"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px">however it
appears this method no longer exists.</span></font></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px">
<div><br>
</div>
If one can determine the exact rank where points should
should be sent and it is not going to be the neighbour
rank (given by the DM), I would suggest not attaching
the DM at all.</span></font></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px"><br>
</span></font></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px">However if this
is not possible and one wanted to scatter to say the
neighbours neighbours, we will have to add a new
interface and refactor things a little bit.</span></font></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px"><br>
</span></font></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px">Cheers</span></font></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px">Dave</span></font></div>
<div dir="auto"><font style="color:rgb(49,49,49)"><span style="font-size:21px;word-spacing:1px"><br>
</span></font></div>
<div dir="auto"><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
</div>
</div>
<div dir="ltr">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
<div>
<div>Il 22/12/22 11:44, Matteo Semplice ha scritto:<br>
</div>
<blockquote type="cite">
<p>Dear everybody,</p>
<p> I have bug a bit into the code and I am
able to add more information.<br>
</p>
<div>Il 02/12/22 12:48, Matteo Semplice ha
scritto:<br>
</div>
<blockquote type="cite">
<div>Hi.</div>
<div>I am sorry to take this up again, but
further tests show that it's not right yet.<br>
</div>
<div><br>
</div>
<div>Il 04/11/22 12:48, Matthew Knepley ha
scritto:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Fri, Nov 4, 2022 at 7:46
AM Matteo Semplice <<a href="mailto:matteo.semplice@uninsubria.it" target="_blank">matteo.semplice@uninsubria.it</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
<div> On 04/11/2022 02:43, Matthew
Knepley wrote:<br>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Thu, Nov 3, 2022
at 8:36 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
<div dir="ltr">
<div dir="ltr">On Thu, Oct
27, 2022 at 11:57 AM
Semplice Matteo <<a href="mailto:matteo.semplice@uninsubria.it" target="_blank">matteo.semplice@uninsubria.it</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
<div>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;background-color:rgb(255,255,255);color:rgb(0,0,0)">Dear
Petsc developers,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;background-color:rgb(255,255,255);color:rgb(0,0,0)">
I am trying to use
a DMSwarm to
locate a cloud of
points with
respect to a
background mesh.
In the real
application the
points will be
loaded from disk,
but I have created
a small demo in
which</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;background-color:rgb(255,255,255);color:rgb(0,0,0)">
<ul style="font-family:Calibri,Arial,Helvetica,sans-serif">
<li style="font-family:Calibri,Arial,Helvetica,sans-serif"><span style="font-family:Calibri,Arial,Helvetica,sans-serif">each processor
creates Npart
particles, all
within the
domain covered
by the mesh,
but not all in
the local
portion of the
mesh</span></li>
<li style="font-family:Calibri,Arial,Helvetica,sans-serif"><span style="font-family:Calibri,Arial,Helvetica,sans-serif">migrate the
particles</span></li>
</ul>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif">After
migration most
particles are
not any more in
the DMSwarm (how
many and which
ones seems to
depend on the
number of cpus,
but it never
happens that all
particle survive
the migration
process).</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif"><br>
</div>
</div>
</div>
</div>
</blockquote>
<div>Thanks for sending
this. I found the
problem. Someone has
some overly fancy code
inside DMDA to figure
out the local bounding
box from the
coordinates.</div>
<div>It is broken for
DM_BOUNDARY_GHOSTED, but
we never tested with
this. I will fix it.</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Okay, I think this fix is
correct</div>
<div><br>
</div>
<div> <a href="https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fpetsc%2Fpetsc%2F-%2Fmerge_requests%2F5802&data=05%7C01%7Cmatteo.semplice%40uninsubria.it%7C49a33e96a2144e96728008dae44394ca%7C9252ed8bdffc401c86ca6237da9991fa%7C0%7C0%7C638073276161185276%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=sahWnsix2F7anVnUYz2ANyivon%2F6q33geOlaRJD5iEY%3D&reserved=0" target="_blank">https://gitlab.com/petsc/petsc/-/merge_requests/5802</a></div>
<div><br>
</div>
<div>I incorporated your test as
src/dm/impls/da/tests/ex1.c.
Can you take a look and see if
this fixes your issue?</div>
</div>
</div>
</blockquote>
<p>Yes, we have tested 2d and 3d, with
various combinations of
DM_BOUNDARY_* along different
directions and it works like a
charm.</p>
<p>On a side note, neither <span style="font-family:monospace"><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(0,0,0)">DMSwarmViewXDMF</span></span>
nor <span style="font-family:monospace"><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(0,0,0)">DMSwarmMigrate</span></span>
seem to be implemented for 1d: I get</p>
<p><span style="font-family:monospace"><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(0,0,0)">[0]PETSC
ERROR: No support for this
operation for this object type</span><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(178,24,24)">
</span><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(0,0,0)">
</span>[0]PETSC ERROR: Support not
provided for 1D<br>
</span></p>
<p>However, currently I have no need
for this feature.<br>
<span style="font-family:monospace"></span></p>
<p>Finally, if the test is meant to
stay in the source, you may remove
the call to <span style="font-family:monospace"><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(0,0,0)">DMSwarmRegisterPetscDatatypeField</span></span>
as in the attached patch.<br>
</p>
<p>Thanks a lot!!</p>
</div>
</blockquote>
<div>Thanks! Glad it works.</div>
<div>
<p> Matt</p>
</div>
</div>
</div>
</blockquote>
<p cols="72">There are still problems when not
using 1,2 or 4 cpus. Any other number of cpus
that I've tested does not work corectly.<br>
</p>
</blockquote>
<p>I have now modified
private_DMDALocatePointsIS_2D_Regular to print
out some debugging information. I see that this
is called twice during migration, once before
and once after DMSwarmMigrate_DMNeighborScatter.
If I understand correctly, the second call to
private_DMDALocatePointsIS_2D_Regular should be
able to locate all particles owned by the rank
but it fails for some of them because they have
been sent to the wrong rank (despite being well
away from process boundaries).<br>
</p>
<p>For example, running the example
src/dm/impls/da/tests/ex1.c with Nx=21 (20x20 Q1
elements on [-1,1]X[-1,1]) with 3 processors,</p>
<p>- the particles (-0.191,-0.462) and
(0.191,-0.462) are sent cpu2 instead of cpu0</p>
<p>- those at (-0.287,-0.693)and (0.287,-0.693)
are sent to cpu1 instead of cpu0</p>
<p>- those at (0.191,0.462) and (-0.191,0.462) are
sent to cpu0 instead of cpu2</p>
<p>(This is 2d and thus not affected by the 3d
issue mentioned yesterday on petsc-dev. Tests
were made based on the release branch pulled out
this morning, i.e. on commit <span style="font-family:monospace"><span style="font-family:monospace;background-color:rgb(255,255,255);color:rgb(178,104,24)">bebdc8d016f</span></span>).<br>
</p>
</blockquote>
<p>I see: particles are sent "all around" and not
only to the destination rank.</p>
<p>Still however, running the example
src/dm/impls/da/tests/ex1.c with Nx=21 (20x20 Q1
elements on [-1,1]X[-1,1]) with 3 processors,
there are 2 particles initially owned by rank2 (at
y=-0.6929 and x=+/-0.2870) that are sent only to
rank1 and never make it to rank0 and are thus lost
in the end since rank1, correctly, discards them.</p>
<p>Thanks</p>
<p> Matteo<br>
</p>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted
before they begin their experiments is
infinitely more interesting than any results
to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cse.buffalo.edu%2F~knepley%2F&data=05%7C01%7Cmatteo.semplice%40uninsubria.it%7C49a33e96a2144e96728008dae44394ca%7C9252ed8bdffc401c86ca6237da9991fa%7C0%7C0%7C638073276161185276%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JTwzg7jNDdj6y%2BpT0fIaqllAlIvZfxrQ0IVPFRMAW6E%3D&reserved=0" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<pre cols="72">--
---
Professore Associato in Analisi Numerica
Dipartimento di Scienza e Alta Tecnologia
Università degli Studi dell'Insubria
<a href="https://www.google.com/maps/search/Via+Valleggio,+11+-+Como?entry=gmail&source=g">Via Valleggio, 11 - Como</a></pre>
</div>
</blockquote></div></div>