<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannervau65c6 { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonvau65c6:hover, .pfptPrimaryButtonvau65c6:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonvau65c6:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
</head><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Dear PETSc community, I have some questions about the DMSWARM structure and a request for suggestions in order to speed up a task that I require in my code. I am developing a numerical method that is similar to the PIC, MPM, etc, however, all
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannervau65c6" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannervau65c6" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannervau65c6" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannervau65c6" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763">Dear PETSc community,<br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763">I have some questions about the DMSWARM structure and a request for suggestions in order to speed up a task that I require in my code.</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763">I am developing a numerical method that is similar to the PIC, MPM, etc, however, all cells of the DM mesh must contain particles. In this method, about 10 particles are included in each cell (element), particles are displaced at each time step, and carry information that must be projected onto/from the mesh.</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763">Searching on the internet I came across the DMSWARM structure. Indeed, the structure allows easy initialization of a cloud of particles, easy detection of the element that holds a specific particle and easy particle-mesh projection. So it turns out to be quite convenient for my purpose. </div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763">However, in a test implementation I performed (based on the PETSC examples), I notice a significant bottleneck (IMHO) in the DMSwarmMigrate function (line 198 of the attached code). Given that I am not sure if my implementation is correct or optimal, I am attaching a sample code. Here, some extra info: 3D implementation (tetra and hexa); cpu, mpich & open mpi; DMPlex (from gmsh file, also attached). The following tables are obtained with the attached code using 9329 tetra elements and 128085 particles (but the desired goal is to reach millions of elements and particles).</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763">TABLE 1 : using the same mesh and particles</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763">..................................processors: - - - - 1 - - - - - - 2 - - - - - - 4 - - - - - 8 - - - - - 16</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763">......num. of particles per process: - -128000 - - 64000 - - 32000 - - 16000 - - 8000<br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763">...time per DMSwarmMigrate call: - - 200 s. - - - - 90 s. - - - 17 s. - - - 5 s. - - - 3 s.</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763">TABLE 2 : using the same amount of processors = 16 (all cases)</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><div class="gmail_default">...................................elements: - - - 9329 - - - 18730 - - - - 45432</div><div class="gmail_default">....................................particles: - - 128085 - - 260554 - - - 631430</div><div class="gmail_default">time per DMSwarmMigrate call: - - - - 3 s. - - - - 13 s. - - - - - 88 s.</div></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763">My questions:<br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763">- Is this performance of DMSwarmMigrate normal ? (I have the feeling that it is quite time consuming, e.g. when compared to the time spent by KSPSolve in linear elasticity using an iterative solver).</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763">- The DMSwarm functionalities I use are: create a particle field, associate it to a DMPlex, and obtain Vecs that are dynamic in size as a result of the migration. ---> Is the implementation of the attached code optimal for the purpose I mention? </div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763">Note that I did some tests with the DMSwarmSetLocalSizes function but saw no difference. Also, I set DMSWARM_MIGRATE_BASIC, which is faster but did not relocate particles.</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763">If the answer to both questions is YES, I would appreciate any suggestions on how I can reach my target (millions of particles).</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763">Thank you very much for taking the time to read this email. <br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763">Kind regards,</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763">Eduardo.</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:#073763"><br></div></div>