[petsc-users] Bad memory scaling with PETSc 3.10

[Matthew Knepley]

On Fri, Mar 1, 2019 at 10:53 AM Myriam Peyrounette via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi,
>
> I used to run my code with PETSc 3.6. Since I upgraded the PETSc version
> to 3.10, this code has a bad memory scaling.
>
> To report this issue, I took the PETSc script ex42.c and slightly
> modified it so that the KSP and PC configurations are the same as in my
> code. In particular, I use a "personnalised" multi-grid method. The
> modifications are indicated by the keyword "TopBridge" in the attached
> scripts.
>
> To plot the memory (weak) scaling, I ran four calculations for each
> script with increasing problem sizes and computations cores:
>
> 1. 100,000 elts on 4 cores
> 2. 1 million elts on 40 cores
> 3. 10 millions elts on 400 cores
> 4. 100 millions elts on 4,000 cores
>
> The resulting graph is also attached. The scaling using PETSc 3.10
> clearly deteriorates for large cases, while the one using PETSc 3.6 is
> robust.
>
> After a few tests, I found that the scaling is mostly sensitive to the
> use of the AMG method for the coarse grid (line 1780 in
> main_ex42_petsc36.cc). In particular, the performance strongly
> deteriorates when commenting lines 1777 to 1790 (in main_ex42_petsc36.cc).
>
> Do you have any idea of what changed between version 3.6 and version
> 3.10 that may imply such degradation?
>

I believe the default values for PCGAMG changed between versions. It sounds
like the coarsening rate
is not great enough, so that these grids are too large. This can be set
using:


https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html

There is some explanation of this effect on that page. Let us know if
setting this does not correct the situation.

  Thanks,

     Matt


> Let me know if you need further information.
>
> Best,
>
> Myriam Peyrounette
>
>
> --
> Myriam Peyrounette
> CNRS/IDRIS - HLST
> --
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190301/f4de7a98/attachment.html>