<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">So, I tried a distributed input file with our legacy implementation, down to 10 seconds.<div class="">I’ll try to figure out the CreateFromFile + DMView stuff.</div><div class="">Stefano, I can’t say whether it’s a regression or not, I’m just starting to interface DMPlex in our code to see if it’s a viable alternative.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Pierre<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 28 Apr 2020, at 1:03 PM, Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" class="">stefano.zampini@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">I think the slowdown in the second test (interpolate 0) is coming from the fact that Plex has to compute the dual graph on the fly, see here <a href="https://gitlab.com/petsc/petsc/-/blob/master/src/dm/impls/plex/plexpartition.c#L469" class="">https://gitlab.com/petsc/petsc/-/blob/master/src/dm/impls/plex/plexpartition.c#L469</a>.<div class=""><br class=""></div><div class="">Are you having a performance regression with DMPLEX on this second case, or is it just the first time you noticed it?</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno mar 28 apr 2020 alle ore 13:57 Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> ha scritto:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class="">On Tue, Apr 28, 2020 at 5:19 AM Pierre Jolivet <<a href="mailto:pierre.jolivet@enseeiht.fr" target="_blank" class="">pierre.jolivet@enseeiht.fr</a>> wrote:</div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><blockquote type="cite" class=""><div class="">On 14 Apr 2020, at 2:36 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><div dir="ltr" class="">On Tue, Apr 14, 2020 at 6:36 AM Pierre Jolivet <<a href="mailto:pierre.jolivet@enseeiht.fr" target="_blank" class="">pierre.jolivet@enseeiht.fr</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello,<br class="">I’d like to call DMPlexInterpolate after DMPlexDistribute and not the other way around for performance reasons (please stop me here if this is equivalent).<br class="">When there is no overlap in DMPlexDistribute, it goes through fine.<br class="">If there is overlap, I run into an error.<br class="">Is this the expected behavior?</blockquote></div></div></div></blockquote><div class=""><br class=""></div><div class="">Sorry for taking so long to get back at this.</div><div class="">I thought I got everything setup, but when looking at the performance, I’m quite surprised.</div><div class="">I rewound and looked at src/dm/impls/plex/tutorials/ex2.c by just adding the DMPlexDistribute step. I guess this is equivalent to your steps #1 and #2.</div><div class=""><div class="">diff --git a/src/dm/impls/plex/tutorials/ex2.c b/src/dm/impls/plex/tutorials/ex2.c</div><div class="">index a069d922b2..5e5c4fb584 100644</div><div class="">--- a/src/dm/impls/plex/tutorials/ex2.c</div><div class="">+++ b/src/dm/impls/plex/tutorials/ex2.c</div><div class="">@@ -65,2 +65,5 @@ static PetscErrorCode CreateMesh(MPI_Comm comm, AppCtx *user, DM *dm)</div><div class=""> ierr = DMPlexCreateFromFile(comm, user->filename, user->interpolate, dm);CHKERRQ(ierr);</div><div class="">+ DM dmParallel;</div><div class="">+ ierr = DMPlexDistribute(*dm, 0, NULL, &dmParallel);CHKERRQ(ierr);</div><div class="">+ ierr = DMDestroy(&dmParallel);CHKERRQ(ierr);</div><div class=""> }</div><div class=""><br class=""></div><div class="">With a cube of 741000 nodes 4574068 elements.</div><div class="">$ cat untitled.geo && ~/gmsh-4.5.4-Linux64/bin/gmsh untitled.geo -bin -3</div><div class="">//+</div><div class="">SetFactory("OpenCASCADE");</div><div class="">Box(1) = {-0.5, -0.5, 0, 1, 1, 1};</div><div class="">Characteristic Length {:} = 0.01;</div><div class=""><br class=""></div><div class="">I get the following timings (optimized build, scalar-type=real, 64-bit-indices=false, SKL cluster).</div><div class="">$ mpirun -n 120 ./ex2 -filename untitled.msh -log_view -interpolate true</div><div class="">DMPlexInterp 1 1.0 1.0444e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 73 0 0 0 5 73 0 0 0 6 0</div><div class=""><div class="">DMPlexDistribute 1 1.0 2.8793e+01 1.0 0.00e+00 0.0 3.6e+03 6.2e+05 3.3e+01 21 0100100 60 21 0100100 69 0</div><div class="">$ mpirun -n 120 ./ex2 -filename untitled.msh -log_view -interpolate false</div><div class="">DMPlexDistribute 1 1.0 7.0265e+02 1.0 0.00e+00 0.0 3.1e+03 2.0e+05 2.6e+01 99 0100100 58 99 0100100 68 0</div></div><div class=""><br class=""></div></div><div class="">Do you think I messed up something else?</div><div class="">Our legacy implementation, on the same mesh, takes around 20 seconds (accounting for the ParMETIS call) to partition and distribute the mesh and generate the underlying structures for our FEM kernels (kd tree, distributed numbering, so on and so forth).</div><div class="">Of course, DMPlex is much more versatile and generic, so I’d understand if it’s a little bit slower, but that’s quite a lot slower right now.</div><div class="">Are there any option I could try to play with to speed things up?</div></div></div></blockquote><div class=""><br class=""></div><div class="">Yes, something is messed up. Vaclav's latest paper has almost the exact setup for the cube benchmark (<a href="https://arxiv.org/pdf/2004.08729.pdf" target="_blank" class="">https://arxiv.org/pdf/2004.08729.pdf</a>), and you can see that the interpolation time is an order of magnitude smaller for 128 procs and also scales linearly. What you show above in ex2 looks serial, in that the mesh is loaded on 1 proc, and then interpolation is also done on 1 proc, which corresponds to the time he shows as "Serial Startup". I cannot explain the poor performance of your second run, however the interpolation time in the first can be greatly reduced by interpolating in parallel. Thus, the right answer is to load in parallel, interpolation, and redistribute. That is what we do in that paper.</div><div class=""><br class=""></div><div class="">Short answer: If you want scalable load and interpolation, I think you should be using the parallel stuff that Vaclav just put in. I think that means first converting your mesh to the HDF5 format using CreateFromFile and then DMView to hdf5. Then you can load it in parallel. Vaclav, has everything been pushed for this?</div><div class=""><br class=""></div><div class=""> Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><div class="">Thanks in advance for your help,</div><div class="">Pierre</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><div class="gmail_quote"><div class="">Yes. What we want you to do is:</div><div class=""><br class=""></div><div class="">1) Load/generate mesh</div><div class=""><br class=""></div><div class="">2) Distribute (this can be done at the same time as load with parallel load)</div><div class=""><br class=""></div><div class="">3) Interpolate (this is also an option from parallel load)</div><div class=""><br class=""></div><div class="">4) If necessary, redistribute for load balance</div><div class=""><br class=""></div><div class="">5) Construct overlap</div><div class=""><br class=""></div><div class=""> When you pass '1' below to DMDistribute(), it distributes as normal and then calls</div><div class=""><br class=""></div><div class=""> <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexDistributeOverlap.html" target="_blank" class="">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexDistributeOverlap.html</a></div><div class=""><br class=""></div><div class=""> at the end. So you just postpone calling that until you have interpolated.</div><div class=""><br class=""></div><div class=""> Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Here is a MWE.<br class="">$ patch -p1 < patch.txt<br class="">$ cd src/dm/impls/plex/tests/<br class="">$ make ex18<br class="">$ mpirun -n 2 ./ex18 -distribute -interpolate after_distribute<br class="">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br class="">[0]PETSC ERROR: Petsc has generated inconsistent data<br class="">[0]PETSC ERROR: Point SF contains 1 which is a cell<br class=""><br class="">Thanks,<br class="">Pierre<br class=""><br class="">diff --git a/src/dm/impls/plex/tests/ex18.c b/src/dm/impls/plex/tests/ex18.c<br class="">index 07421b3522..dd62be58e5 100644<br class="">--- a/src/dm/impls/plex/tests/ex18.c<br class="">+++ b/src/dm/impls/plex/tests/ex18.c<br class="">@@ -806 +806 @@ static PetscErrorCode CreateMesh(MPI_Comm comm, AppCtx *user, DM *dm)<br class="">- ierr = DMPlexDistribute(*dm, 0, NULL, &pdm);CHKERRQ(ierr);<br class="">+ ierr = DMPlexDistribute(*dm, 1, NULL, &pdm);CHKERRQ(ierr);</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>--<span class=""> </span><br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a></div></div></div></div></div></div></div></div></div></blockquote></div><br class=""></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class="gmail_signature">Stefano</div>
</div></blockquote></div><br class=""></div></body></html>