<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Oct 3, 2020, at 9:12 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div dir="ltr" class="">On Sat, Oct 3, 2020 at 1:49 PM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" class="">stefano.zampini@gmail.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class=""><div class=""><br class=""></div><div class=""> <span class="Apple-converted-space"> </span>There is a MATPARTITIONINGHIERARCH (man page) that Fande provided that helped scaling up problems he was working on significantly.</div><div class=""><br class=""></div><div class=""> Barry</div></div></blockquote><div class=""><br class=""></div><div class="">The scaling issue with DMPlex is the one-to-all pattern of communication that happens when distributing an original sequential mesh. MATPARTITIONINGHIERARCH won't fix the issue.</div><div class="">In order to get reasonable performances when distributing a sequential mesh on a large number of processes, you need at least two stages of partitioning: an initial one from the sequential mesh to a mesh with one process per node, migrate the PLEX data, then partition on each node separately, and migrate the data again.</div></div></div></blockquote><div class=""><br class=""></div><div class="">Just to make sure I understand completely, You partition a serial mesh (SELF) onto one process per node (1PROC), and then refine, and repartition the new mesh onto the whole machine (WORLD). Thus I</div><div class="">need three communicators, right? And also a method, for moving a Plex on a subcomm onto the larger comm, using 0 parts on the new ranks.</div></div></div></div></blockquote><div><br class=""></div><div>This is how the loop over stages looks like in pseudo code</div><div><br class=""></div><div>DMPlexDistributeML(DM,overlap,SF* migration, DM *newdm)</div><div>For i = 0 : nstages</div><div> DMPlexGetPartitioner(dm,&p)</div><div> PartitionerSetStage(p,i)</div><div> DMPlexDistribute(dm,0,&sft,&dmt);</div><div> SFCompose(sf,sft)</div> dm = dmt;</div><div> If (I = nstages and overlap > 1) DMPlexDistributeOverlap</div><div>end</div><div><br class=""></div><div>I thought about including the refinement step within the stages, but it turns out it is not doable now if you want to get back a migration sf which is usable (and we need it to migrate our data)</div><div>For a general solution, we need hooks for migrating user-defined data and to generate user-defined data while refining.</div><div><br class=""></div><div>The partitioner is defined with a series of MPI_Groups that identify the various processes involved per stage. But the temporary meshes that are generated in the loop are always defined on the same global communicator.</div><div><br class=""></div><div> </div><div><br class=""></div><div><br class=""></div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div class="gmail_quote"><div class=""><br class=""></div><div class=""> Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class=""><div class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Oct 3, 2020, at 10:04 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">On Sat, Oct 3, 2020 at 10:51 AM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank" class="">stefano.zampini@gmail.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr"><br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div class=""><br class=""></div><div class="">Secondly, I'd like to add a multilevel "simple" partitioning in DMPlex to optimize communication. I am thinking that I can create a mesh with 'nnodes' cells and distribute that to 'nnodes*procs_node' processes with a "spread" distribution. (the default seems to be "compact"). Then refine that enough to get 'procs_node' more cells and the use a simple partitioner again to put one cell on each process, in such a way that the locality is preserved (not sure how that would work). Then refine from there on each proc for a scaling study.</div><div class=""><br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">Mark</div><div class=""><br class=""></div><div class="">for multilevel partitioning, you need custom code, since what kills performances with one-to-all patterns in DMPlex is the actual communication of the mesh data.</div><div class="">However, you can always generate a mesh to have one cell per process, and then refine from there.</div><div class=""><br class=""></div><div class="">I have coded a multilevel partitioner that works quite well for general meshes, we have it in a private repo with Lisandro. From my experience, the benefits of using the multilevel scheme start from 4K processes on. If you plan very large runs (say > 32K cores) then you definitely want a multistage scheme.</div><div class=""><br class=""></div><div class="">We never contributed the code since it requires some boilerplate code to run through the stages of the partitioning and move the data.</div><div class="">If you are using hexas, you can always define your own "shell" partitioner producing box decompositions.</div></div></div></blockquote><div class=""><br class=""></div><div class="">I could integrate it if you want to stop maintaining it there :) It sounds really useful.</div><div class=""><br class=""></div><div class=""> <span class="Apple-converted-space"> </span>Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div class="gmail_quote"><div class="">Another option is to generate the meshes upfront in sequential, and then use the parallel HDF5 reader that Vaclav and Matt put together.</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div class=""></div><div class="">The point here is to get communication patterns that look like an (idealized) well partition application. (I suppose I could take an array of factors, the product of which is the number of processors, and generalize this in a loop for any number of memory levels, or make an oct-tree).</div><div class=""><br class=""></div><div class="">Any thoughts?</div><div class="">Thanks,</div><div class="">Mark</div><div class=""><br class=""></div><div class=""><br class=""></div></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>--<span class="Apple-converted-space"> </span><br class=""><div dir="ltr" class="">Stefano</div></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>--<span class="Apple-converted-space"> </span><br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div></div></blockquote></div><br class=""></div></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>--<span class="Apple-converted-space"> </span><br class=""><div dir="ltr" class="">Stefano</div></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>--<span class="Apple-converted-space"> </span><br class=""><div dir="ltr" class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a></div></div></div></div></div></div></div></div></div></blockquote></div><br class=""></body></html>