<div dir="ltr"><div dir="ltr">On Sun, Mar 7, 2021 at 8:20 AM Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">Is phase 1 the old method and 2 the new?
<div>Is this 128^3 mesh per process?</div></div></div></blockquote><div><br></div><div>Both phases are the same method. My interpretation is that there is severe message congestion. Sending fewer bigger messages in a phase 1, and then</div><div>rebalancing locally in a phase 2, both using the same communication method, is much faster than just directly communicating the data in a single phase.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sun, Mar 7, 2021 at 7:27 AM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_quote">
<div>[2] On the robustness and performance of entropy stable discontinuous collocation methods for the compressible Navier-Stokes equations, ROjas .<a href="http://et.al" target="_blank">et.al</a>.</div>
<div> <a href="https://arxiv.org/abs/1911.10966" target="_blank">https://arxiv.org/abs/1911.10966</a></div>
<div></div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>This is not the proper reference, here is the correct one <a href="https://www.sciencedirect.com/science/article/pii/S0021999120306185?dgcid=rss_sd_all" target="_blank">
https://www.sciencedirect.com/science/article/pii/S0021999120306185?dgcid=rss_sd_all</a></div>
<div>However, there the algorithm is only outlined, and performances related to the mesh distribution are not really reported.</div>
<div>We observed a large gain for large core counts and one to all distributions (from minutes to seconds) by splitting the several communication rounds needed by DMPlex into stages: from rank 0 to 1 rank per node, and then decomposing independently within
the node.</div>
<div>Attached the total time for one-to-all DMPlexDistrbute for a 128^3 mesh<br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
?<br>
<br>
The attached plots suggest (A), (B), and (C) is happening for<br>
Cahn-Hilliard problem (from firedrake-bench repo) on a 2D 8Kx8K<br>
unit-square mesh. The implementation is here [1]. Versions are<br>
Firedrake, PyOp2: 20200204.0; PETSc 3.13.1; ParMETIS 4.0.3.<br>
<br>
Two questions, one on (A) and the other on (B)+(C):<br>
<br>
1. Is (A) result expected? Given (A), any effort to improve the quality<br>
of the compiled assembly kernels (or anything else other than mesh<br>
distribution) appears futile since it takes 1% of end-to-end execution<br>
time, or am I missing something?<br>
<br>
1a. Is mesh distribution fundamentally necessary for any FEM framework,<br>
or is it only needed by Firedrake? If latter, then how do other<br>
frameworks partition the mesh and execute in parallel with MPI but avoid<br>
the non-scalable mesh destribution step?<br>
<br>
2. Results (B) and (C) suggest that the mesh distribution step does<br>
not scale. Is it a fundamental property of the mesh distribution problem<br>
that it has a central bottleneck in the master process, or is it<br>
a limitation of the current implementation in PETSc-DMPlex?<br>
<br>
2a. Our (B) result seems to agree with Figure 4(left) of [2]. Fig 6 of [2]<br>
suggests a way to reduce the time spent on sequential bottleneck by<br>
"parallel mesh refinment" that creates high-resolution meshes from an<br>
initial coarse mesh. Is this approach implemented in DMPLex? If so, any<br>
pointers on how to try it out with Firedrake? If not, any other<br>
directions for reducing this bottleneck?<br>
<br>
2b. Fig 6 in [3] shows plots for Assembly and Solve steps that scale well up<br>
to 96 cores -- is mesh distribution included in those times? Is anyone<br>
reading this aware of any other publications with evaluations of<br>
Firedrake that measure mesh distribution (or explain how to avoid or<br>
exclude it)?<br>
<br>
Thank you for your time and any info or tips.<br>
<br>
<br>
[1] <a href="https://github.com/ISI-apex/firedrake-bench/blob/master/cahn_hilliard/firedrake_cahn_hilliard_problem.py" rel="noreferrer" target="_blank">
https://github.com/ISI-apex/firedrake-bench/blob/master/cahn_hilliard/firedrake_cahn_hilliard_problem.py</a><br>
<br>
[2] Unstructured Overlapping Mesh Distribution in Parallel, Matthew G.<br>
Knepley, Michael Lange, Gerard J. Gorman, 2015.<br>
<a href="https://arxiv.org/pdf/1506.06194.pdf" rel="noreferrer" target="_blank">https://arxiv.org/pdf/1506.06194.pdf</a><br>
<br>
[3] Efficient mesh management in Firedrake using PETSc-DMPlex, Michael<br>
Lange, Lawrence Mitchell, Matthew G. Knepley and Gerard J. Gorman, SISC,<br>
38(5), S143-S155, 2016. <a href="http://arxiv.org/abs/1506.07749" rel="noreferrer" target="_blank">
http://arxiv.org/abs/1506.07749</a><br>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<br>
-- <br>
<div dir="ltr">Stefano</div>
</div>
</blockquote>
</div>
</div>
</blockquote></div></div>