[petsc-users] locate DMSwarm particles with respect to a background DMDA mesh

[Matteo Semplice]

Dear everybody,

     I have bug a bit into the code and I am able to add more information.

Il 02/12/22 12:48, Matteo Semplice ha scritto:
> Hi.
> I am sorry to take this up again, but further tests show that it's not 
> right yet.
>
> Il 04/11/22 12:48, Matthew Knepley ha scritto:
>> On Fri, Nov 4, 2022 at 7:46 AM Matteo Semplice 
>> <matteo.semplice at uninsubria.it> wrote:
>>
>>     On 04/11/2022 02:43, Matthew Knepley wrote:
>>>     On Thu, Nov 3, 2022 at 8:36 PM Matthew Knepley
>>>     <knepley at gmail.com> wrote:
>>>
>>>         On Thu, Oct 27, 2022 at 11:57 AM Semplice Matteo
>>>         <matteo.semplice at uninsubria.it> wrote:
>>>
>>>             Dear Petsc developers,
>>>             I am trying to use a DMSwarm to locate a cloud of points
>>>             with respect to a background mesh. In the real
>>>             application the points will be loaded from disk, but I
>>>             have created a small demo in which
>>>
>>>               * each processor creates Npart particles, all within
>>>                 the domain covered by the mesh, but not all in the
>>>                 local portion of the mesh
>>>               * migrate the particles
>>>
>>>             After migration most particles are not any more in the
>>>             DMSwarm (how many and which ones seems to depend on the
>>>             number of cpus, but it never happens that all particle
>>>             survive the migration process).
>>>
>>>         Thanks for sending this. I found the problem. Someone has
>>>         some overly fancy code inside DMDA to figure out the local
>>>         bounding box from the coordinates.
>>>         It is broken for DM_BOUNDARY_GHOSTED, but we never tested
>>>         with this. I will fix it.
>>>
>>>
>>>     Okay, I think this fix is correct
>>>
>>>     https://gitlab.com/petsc/petsc/-/merge_requests/5802
>>>     <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fpetsc%2Fpetsc%2F-%2Fmerge_requests%2F5802&data=05%7C01%7Cmatteo.semplice%40uninsubria.it%7Cf4d64b09df1f438437ad08dad45b342b%7C9252ed8bdffc401c86ca6237da9991fa%7C0%7C0%7C638055785720875500%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=ISLaLhhnYU4njkYfod%2F3tEiIOIV5uZvmiAlKZ2PvhmE%3D&reserved=0>
>>>
>>>     I incorporated your test as src/dm/impls/da/tests/ex1.c. Can you
>>>     take a look and see if this fixes your issue?
>>
>>     Yes, we have tested 2d and 3d, with various combinations of
>>     DM_BOUNDARY_* along different directions and it works like a charm.
>>
>>     On a side note, neither DMSwarmViewXDMF nor DMSwarmMigrate seem
>>     to be implemented for 1d: I get
>>
>>     [0]PETSC ERROR: No support for this operation for this object
>>     type[0]PETSC ERROR: Support not provided for 1D
>>
>>     However, currently I have no need for this feature.
>>
>>     Finally, if the test is meant to stay in the source, you may
>>     remove the call to DMSwarmRegisterPetscDatatypeField as in the
>>     attached patch.
>>
>>     Thanks a lot!!
>>
>> Thanks! Glad it works.
>>
>>    Matt
>>
> There are still problems when not using 1,2 or 4 cpus. Any other 
> number of cpus that I've tested does not work corectly.
>
I have now modified private_DMDALocatePointsIS_2D_Regular to print out 
some debugging information. I see that this is called twice during 
migration, once before and once after DMSwarmMigrate_DMNeighborScatter. 
If I understand correctly, the second call to 
private_DMDALocatePointsIS_2D_Regular should be able to locate all 
particles owned by the rank but it fails for some of them because they 
have been sent to the wrong rank (despite being well away from process 
boundaries).

For example, running the example src/dm/impls/da/tests/ex1.c with Nx=21 
(20x20 Q1 elements on [-1,1]X[-1,1]) with 3 processors,

- the particles (-0.191,-0.462) and (0.191,-0.462) are sent cpu2 instead 
of cpu0

- those at (-0.287,-0.693)and (0.287,-0.693) are sent to cpu1 instead of 
cpu0

- those at (0.191,0.462) and (-0.191,0.462) are sent to cpu0 instead of cpu2

(This is 2d and thus not affected by the 3d issue mentioned yesterday on 
petsc-dev. Tests were made based on the release branch pulled out this 
morning, i.e. on commit bebdc8d016f).

I attach the output separated by process.

If you have any hints, they would be appreciated.

Thanks

     Matteo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20221222/ba260714/attachment-0001.html>
-------------- next part --------------

... calling DMSwarmMigrate ...

=== private_DMDALocatePointsIS_2D_Regular
0] local domain (-1.000,-1.000) ... (1.000,-0.400) 
0] local domain 20X6 cells with dx=(0.100,0.100)
0] locating particle 0 at (0.231,0.096)
0] locating particle 1 at (0.096,0.231)
0] locating particle 2 at (-0.096,0.231)
0] locating particle 3 at (-0.231,0.096)
0] locating particle 4 at (-0.231,-0.096)
0] locating particle 5 at (-0.096,-0.231)
0] locating particle 6 at (0.096,-0.231)
0] locating particle 7 at (0.231,-0.096)
=== DMSwarmMigrate_DMNeighborScatter
=== private_DMDALocatePointsIS_2D_Regular
0] local domain (-1.000,-1.000) ... (1.000,-0.400) 
0] local domain 20X6 cells with dx=(0.100,0.100)
0] locating particle 0 at (0.191,0.462)
0] locating particle 1 at (-0.191,0.462)
0] locating particle 2 at (-0.191,-0.462)
0] ---->ok, particle 2 is in cell (8,5)-->108
0] locating particle 3 at (0.191,-0.462)
0] ---->ok, particle 3 is in cell (11,5)-->111
-------------- next part --------------
=== private_DMDALocatePointsIS_2D_Regular
1] local domain (-1.000,-0.400) ... (1.000,0.300) 
1] local domain 20X7 cells with dx=(0.100,0.100)
1] locating particle 0 at (0.462,0.191)
1] ---->ok, particle 0 is in cell (14,11)-->114
1] locating particle 1 at (0.191,0.462)
1] locating particle 2 at (-0.191,0.462)
1] locating particle 3 at (-0.462,0.191)
1] ---->ok, particle 3 is in cell (5,11)-->105
1] locating particle 4 at (-0.462,-0.191)
1] ---->ok, particle 4 is in cell (5,8)-->45
1] locating particle 5 at (-0.191,-0.462)
1] locating particle 6 at (0.191,-0.462)
1] locating particle 7 at (0.462,-0.191)
1] ---->ok, particle 7 is in cell (14,8)-->54
=== DMSwarmMigrate_DMNeighborScatter
=== private_DMDALocatePointsIS_2D_Regular
1] local domain (-1.000,-0.400) ... (1.000,0.300) 
1] local domain 20X7 cells with dx=(0.100,0.100)
1] locating particle 0 at (0.231,0.096)
1] ---->ok, particle 0 is in cell (12,10)-->92
1] locating particle 1 at (0.096,0.231)
1] ---->ok, particle 1 is in cell (10,12)-->130
1] locating particle 2 at (-0.096,0.231)
1] ---->ok, particle 2 is in cell (9,12)-->129
1] locating particle 3 at (-0.231,0.096)
1] ---->ok, particle 3 is in cell (7,10)-->87
1] locating particle 4 at (-0.231,-0.096)
1] ---->ok, particle 4 is in cell (7,9)-->67
1] locating particle 5 at (-0.096,-0.231)
1] ---->ok, particle 5 is in cell (9,7)-->29
1] locating particle 6 at (0.096,-0.231)
1] ---->ok, particle 6 is in cell (10,7)-->30
1] locating particle 7 at (0.231,-0.096)
1] ---->ok, particle 7 is in cell (12,9)-->72
1] locating particle 8 at (0.693,0.287)
1] ---->ok, particle 8 is in cell (16,12)-->136
1] locating particle 9 at (-0.693,0.287)
1] ---->ok, particle 9 is in cell (3,12)-->123
1] locating particle 10 at (-0.693,-0.287)
1] ---->ok, particle 10 is in cell (3,7)-->23
1] locating particle 11 at (-0.287,-0.693)
1] locating particle 12 at (0.287,-0.693)
1] locating particle 13 at (0.693,-0.287)
1] ---->ok, particle 13 is in cell (16,7)-->36
-------------- next part --------------
=== private_DMDALocatePointsIS_2D_Regular
2] local domain (-1.000,0.300) ... (1.000,1.000) 
2] local domain 20X7 cells with dx=(0.100,0.100)
2] locating particle 0 at (0.693,0.287)
2] locating particle 1 at (0.287,0.693)
2] ---->ok, particle 1 is in cell (12,16)-->72
2] locating particle 2 at (-0.287,0.693)
2] ---->ok, particle 2 is in cell (7,16)-->67
2] locating particle 3 at (-0.693,0.287)
2] locating particle 4 at (-0.693,-0.287)
2] locating particle 5 at (-0.287,-0.693)
2] locating particle 6 at (0.287,-0.693)
2] locating particle 7 at (0.693,-0.287)
=== DMSwarmMigrate_DMNeighborScatter
=== private_DMDALocatePointsIS_2D_Regular
2] local domain (-1.000,0.300) ... (1.000,1.000) 
2] local domain 20X7 cells with dx=(0.100,0.100)
2] locating particle 0 at (0.191,0.462)
2] ---->ok, particle 0 is in cell (11,14)-->31
2] locating particle 1 at (-0.191,0.462)
2] ---->ok, particle 1 is in cell (8,14)-->28
2] locating particle 2 at (-0.191,-0.462)
2] locating particle 3 at (0.191,-0.462)