[petsc-users] Question about DMSwarmMigrate

[Eduardo Fernandez]

Dear PETSc community,

I have some questions about the DMSWARM structure and a request for
suggestions in order to speed up a task that I require in my code.

I am developing a numerical method that is similar to the PIC, MPM, etc,
however, all cells of the DM mesh must contain particles. In this method,
about 10 particles are included in each cell (element), particles are
displaced at each time step, and carry information that must be projected
onto/from the mesh.

Searching on the internet I came across the DMSWARM structure. Indeed, the
structure allows easy initialization of a cloud of particles, easy
detection of the element that holds a specific particle and easy
particle-mesh projection. So it turns out to be quite convenient for my
purpose.

However, in a test implementation I performed (based on the PETSC
examples), I notice a significant bottleneck (IMHO) in the DMSwarmMigrate
function (line 198 of the attached code). Given that I am not sure if my
implementation is correct or optimal, I am attaching a sample code. Here,
some extra info: 3D implementation (tetra and hexa); cpu, mpich & open mpi;
DMPlex (from gmsh file, also attached). The following tables are obtained
with the attached code using 9329 tetra elements and 128085 particles (but
the desired goal is to reach millions of elements and particles).

TABLE 1 : using the same mesh and particles

..................................processors: - - - - 1 - - - - - - 2 - - -
- - - 4 - - - - - 8 - - - - - 16
......num. of particles per process: - -128000 - - 64000 - - 32000 - -
16000 - - 8000
...time per DMSwarmMigrate call: - - 200 s. - - - - 90 s. - - - 17 s. - - -
5 s. - - - 3 s.


TABLE 2 : using the same amount of processors = 16 (all cases)

...................................elements: - - - 9329  - - - 18730 - - -
- 45432
....................................particles: - - 128085 - - 260554 - - -
631430
time per DMSwarmMigrate call: - - - - 3 s. - - - - 13 s. - - - - - 88 s.


My questions:

- Is this performance of DMSwarmMigrate normal ? (I have the feeling that
it is quite time consuming, e.g. when compared to the time spent by
KSPSolve in linear elasticity using an iterative solver).

- The DMSwarm functionalities I use are: create a particle field, associate
it to a DMPlex, and obtain Vecs that are dynamic in size as a result of the
migration. ---> Is the implementation of the attached code optimal for the
purpose I mention?

Note that I did some tests with the DMSwarmSetLocalSizes function but saw
no difference. Also, I set DMSWARM_MIGRATE_BASIC, which is faster but did
not relocate particles.

If the answer to both questions is YES, I would appreciate any suggestions
on how I can reach my target (millions of particles).

Thank you very much for taking the time to read this email.

Kind regards,

Eduardo.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240718/72e23a78/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: problem.cpp
Type: text/x-c++src
Size: 9218 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240718/72e23a78/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: geometry_tetra.geo
Type: application/octet-stream
Size: 1833 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240718/72e23a78/attachment-0001.obj>