[petsc-dev] Scaling test with ex13 (snes)

[Matthew Knepley]

On Sat, Oct 3, 2020 at 1:49 PM Stefano Zampini <stefano.zampini at gmail.com>
wrote:

>
>>     There is a MATPARTITIONINGHIERARCH (man page) that Fande provided
>> that helped scaling up problems he was working on significantly.
>>
>>    Barry
>>
>
> The scaling issue with DMPlex is the one-to-all pattern of communication
> that happens when distributing an original sequential mesh.
> MATPARTITIONINGHIERARCH won't fix the issue.
> In order to get reasonable performances when distributing a sequential
> mesh on a large number of processes, you need at least two stages of
> partitioning: an initial one from the sequential mesh to a mesh with one
> process per node, migrate the PLEX data, then partition on each node
> separately, and migrate the data again.
>

Just to make sure I understand completely, You partition a serial mesh
(SELF) onto one process per node (1PROC), and then refine, and repartition
the new mesh onto the whole machine (WORLD). Thus I
need three communicators, right? And also a method, for moving a Plex on a
subcomm onto the larger comm, using 0 parts on the new ranks.

  Thanks,

     Matt


>
>> On Oct 3, 2020, at 10:04 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>
>> On Sat, Oct 3, 2020 at 10:51 AM Stefano Zampini <
>> stefano.zampini at gmail.com> wrote:
>>
>>>
>>>
>>>
>>>> Secondly, I'd like to add a multilevel "simple" partitioning in DMPlex
>>>> to optimize communication. I am thinking that I can create a mesh with
>>>> 'nnodes' cells and distribute that to 'nnodes*procs_node' processes with a
>>>> "spread" distribution. (the default seems to be "compact"). Then refine
>>>> that enough to get 'procs_node' more cells and the use a simple partitioner
>>>> again to put one cell on each process, in such a way that the locality is
>>>> preserved (not sure how that would work). Then refine from there on each
>>>> proc for a scaling study.
>>>>
>>>>
>>> Mark
>>>
>>> for multilevel partitioning, you need custom code, since what kills
>>> performances with one-to-all patterns in DMPlex is the actual communication
>>> of the mesh data.
>>> However, you can always generate a mesh to have one cell per process,
>>> and then refine from there.
>>>
>>> I have coded a multilevel partitioner that works quite well for
>>> general meshes, we have it in a private repo with Lisandro. From my
>>> experience, the benefits of using the multilevel scheme start from 4K
>>> processes on. If you plan very large runs (say > 32K cores) then you
>>> definitely want a multistage scheme.
>>>
>>> We never contributed the code since it requires some boilerplate code to
>>> run through the stages of the partitioning and move the data.
>>> If you are using hexas, you can always define your own "shell"
>>> partitioner producing box decompositions.
>>>
>>
>> I could integrate it if you want to stop maintaining it there :) It
>> sounds really useful.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Another option is to generate the meshes upfront in sequential, and then
>>> use the parallel HDF5 reader that Vaclav and Matt put together.
>>>
>>>
>>>> The point here is to get communication patterns that look like an
>>>> (idealized) well partition application. (I suppose I could take an array of
>>>> factors, the product of which is the number of processors, and generalize
>>>> this in a loop for any number of memory levels, or make an oct-tree).
>>>>
>>>> Any thoughts?
>>>> Thanks,
>>>> Mark
>>>>
>>>>
>>>>
>>>
>>> --
>>> Stefano
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>>
>>
>
> --
> Stefano
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20201003/280d59a4/attachment.html>