[petsc-users] locate DMSwarm particles with respect to a background DMDA mesh

Thu Dec 22 13:06:06 CST 2022

On Thu 22. Dec 2022 at 10:27, Matteo Semplice <matteo.semplice at uninsubria.it>
wrote:

> Dear Dave and Matt,
>
>     I am really dealing with two different use cases in a code that will
> compute a levelset function passing through a large set of points. If I had
> DMSwarmSetMigrateType() and if it were safe to switch the migration mode
> back and forth in the same swarm, this would cover all my use cases here.
> Is it safe to add it back to petsc? Details below if you are curious.
>
> 1) During preprocessing I am loading a point cloud from disk (in whatever
> order it comes) and need to send the particles to the right ranks. Since
> the background DM is a DMDA I can easily figure out the destination rank.
> This would be covered by your suggestion not to attach the DM, except that
> later I need to locate these points with respect to the background cells in
> order to initialize data on the Vecs associated to the DMDA.
>
> 2) Then I need to implement a semilagrangian time evolution scheme. For
> this I'd like to send particles around at the "foot of characteristic",
> collect data there and then send them back to the originating point. The
> first migration would be based on particle coordinates
> (DMSwarmMigrate_DMNeighborScatter and the restriction to only neighbouring
> ranks is perfect), while for the second move it would be easier to just
> send them back to the originating rank, which I can easily store in an Int
> field in the swarm. Thus at each timestep I'd need to swap migrate types in
> this swarm (DMScatter for moving them to the feet and BASIC to send them
> back).
>

When you use BASIC, you would have to explicitly call the point location
routine from your code as BASIC does not interact with the DM.

Based on what I see in the code, switching  migrate modes between basic and
dmneighbourscatter should be safe.

If you are fine calling the point location from your side then what you
propose should work.

Cheers
Dave

> Thanks
>
>     Matteo
> Il 22/12/22 18:40, Dave May ha scritto:
>
> Hey Matt,
>
> On Thu 22. Dec 2022 at 05:02, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Thu, Dec 22, 2022 at 6:28 AM Matteo Semplice <
>> matteo.semplice at uninsubria.it> wrote:
>>
>>> Dear all
>>>
>>>     please ignore my previous email and read this one: I have better
>>> localized the problem. Maybe DMSwarmMigrate is designed to migrate
>>> particles only to first neighbouring ranks?
>>>
>> Yes, I believe that was the design.
>>
>> Dave, is this correct?
>>
>
> Correct. DMSwarmMigrate_DMNeighborScatter() only scatter points to the
> neighbour ranks - where neighbours are defined by the DM provided to
> represent the mesh.
>
> DMSwarmMigrate_DMNeighborScatter() Is selected by default if you attach a
> DM.
>
> The scatter method should be over ridden with
>
> DMSwarmSetMigrateType()
>
> however it appears this method no longer exists.
>
> If one can determine the exact rank where points should should be sent and
> it is not going to be the neighbour rank (given by the DM), I would suggest
> not attaching the DM at all.
>
> However if this is not possible and one wanted to scatter to say the
> neighbours neighbours, we will have to add a new interface and refactor
> things a little bit.
>
> Cheers
> Dave
>
>
>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Il 22/12/22 11:44, Matteo Semplice ha scritto:
>>>
>>> Dear everybody,
>>>
>>>     I have bug a bit into the code and I am able to add more information.
>>> Il 02/12/22 12:48, Matteo Semplice ha scritto:
>>>
>>> Hi.
>>> I am sorry to take this up again, but further tests show that it's not
>>> right yet.
>>>
>>> Il 04/11/22 12:48, Matthew Knepley ha scritto:
>>>
>>> On Fri, Nov 4, 2022 at 7:46 AM Matteo Semplice <
>>> matteo.semplice at uninsubria.it> wrote:
>>>
>>>> On 04/11/2022 02:43, Matthew Knepley wrote:
>>>>
>>>> On Thu, Nov 3, 2022 at 8:36 PM Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Thu, Oct 27, 2022 at 11:57 AM Semplice Matteo <
>>>>> matteo.semplice at uninsubria.it> wrote:
>>>>>
>>>>>> Dear Petsc developers,
>>>>>>     I am trying to use a DMSwarm to locate a cloud of points with
>>>>>> respect to a background mesh. In the real application the points will be
>>>>>> loaded from disk, but I have created a small demo in which
>>>>>>
>>>>>>    - each processor creates Npart particles, all within the domain
>>>>>>    covered by the mesh, but not all in the local portion of the mesh
>>>>>>    - migrate the particles
>>>>>>
>>>>>> After migration most particles are not any more in the DMSwarm (how
>>>>>> many and which ones seems to depend on the number of cpus, but it never
>>>>>> happens that all particle survive the migration process).
>>>>>>
>>>>>> Thanks for sending this. I found the problem. Someone has some overly
>>>>> fancy code inside DMDA to figure out the local bounding box from the
>>>>> coordinates.
>>>>> It is broken for DM_BOUNDARY_GHOSTED, but we never tested with this. I
>>>>> will fix it.
>>>>>
>>>>
>>>> Okay, I think this fix is correct
>>>>
>>>>   https://gitlab.com/petsc/petsc/-/merge_requests/5802
>>>> <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fpetsc%2Fpetsc%2F-%2Fmerge_requests%2F5802&data=05%7C01%7Cmatteo.semplice%40uninsubria.it%7C49a33e96a2144e96728008dae44394ca%7C9252ed8bdffc401c86ca6237da9991fa%7C0%7C0%7C638073276161185276%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=sahWnsix2F7anVnUYz2ANyivon%2F6q33geOlaRJD5iEY%3D&reserved=0>
>>>>
>>>> I incorporated your test as src/dm/impls/da/tests/ex1.c. Can you take a
>>>> look and see if this fixes your issue?
>>>>
>>>> Yes, we have tested 2d and 3d, with various combinations of
>>>> DM_BOUNDARY_* along different directions and it works like a charm.
>>>>
>>>> On a side note, neither DMSwarmViewXDMF nor DMSwarmMigrate seem to be
>>>> implemented for 1d: I get
>>>>
>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>                                                                                                                                             [0]PETSC
>>>> ERROR: Support not provided for 1D
>>>>
>>>> However, currently I have no need for this feature.
>>>>
>>>> Finally, if the test is meant to stay in the source, you may remove the
>>>> call to DMSwarmRegisterPetscDatatypeField as in the attached patch.
>>>>
>>>> Thanks a lot!!
>>>>
>>> Thanks! Glad it works.
>>>
>>>    Matt
>>>
>>> There are still problems when not using 1,2 or 4 cpus. Any other number
>>> of cpus that I've tested does not work corectly.
>>>
>>> I have now modified private_DMDALocatePointsIS_2D_Regular to print out
>>> some debugging information. I see that this is called twice during
>>> migration, once before and once after DMSwarmMigrate_DMNeighborScatter. If
>>> I understand correctly, the second call to
>>> private_DMDALocatePointsIS_2D_Regular should be able to locate all
>>> particles owned by the rank but it fails for some of them because they have
>>> been sent to the wrong rank (despite being well away from process
>>> boundaries).
>>>
>>> For example, running the example src/dm/impls/da/tests/ex1.c with Nx=21
>>> (20x20 Q1 elements on [-1,1]X[-1,1]) with 3 processors,
>>>
>>> - the particles (-0.191,-0.462) and (0.191,-0.462) are sent cpu2 instead
>>> of cpu0
>>>
>>> - those at (-0.287,-0.693)and (0.287,-0.693) are sent to cpu1 instead of
>>> cpu0
>>>
>>> - those at (0.191,0.462) and (-0.191,0.462) are sent to cpu0 instead of
>>> cpu2
>>>
>>> (This is 2d and thus not affected by the 3d issue mentioned yesterday on
>>> petsc-dev. Tests were made based on the release branch pulled out this
>>> morning, i.e. on commit bebdc8d016f).
>>>
>>> I see: particles are sent "all around" and not only to the destination
>>> rank.
>>>
>>> Still however, running the example src/dm/impls/da/tests/ex1.c with
>>> Nx=21 (20x20 Q1 elements on [-1,1]X[-1,1]) with 3 processors, there are 2
>>> particles initially owned by rank2 (at y=-0.6929 and x=+/-0.2870) that are
>>> sent only to rank1 and never make it to rank0 and are thus lost in the end
>>> since rank1, correctly, discards them.
>>>
>>> Thanks
>>>
>>>     Matteo
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cse.buffalo.edu%2F~knepley%2F&data=05%7C01%7Cmatteo.semplice%40uninsubria.it%7C49a33e96a2144e96728008dae44394ca%7C9252ed8bdffc401c86ca6237da9991fa%7C0%7C0%7C638073276161185276%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JTwzg7jNDdj6y%2BpT0fIaqllAlIvZfxrQ0IVPFRMAW6E%3D&reserved=0>
>>
> --
> ---
> Professore Associato in Analisi Numerica
> Dipartimento di Scienza e Alta Tecnologia
> Università degli Studi dell'InsubriaVia Valleggio, 11 - Como <https://www.google.com/maps/search/Via+Valleggio,+11+-+Como?entry=gmail&source=g>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20221222/e7e00e70/attachment-0001.html>