[petsc-users] Bad memory scaling with PETSc 3.10

[Mark Adams]

On Tue, Mar 5, 2019 at 8:06 AM Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Mar 5, 2019 at 7:14 AM Myriam Peyrounette <
> myriam.peyrounette at idris.fr> wrote:
>
>> Hi Matt,
>>
>> I plotted the memory scalings using different threshold values. The two
>> scalings are slightly translated (from -22 to -88 mB) but this gain is
>> neglectable. The 3.6-scaling keeps being robust while the 3.10-scaling
>> deteriorates.
>>
>> Do you have any other suggestion?
>>
>> Mark, what is the option she can give to output all the GAMG data?
>

I think we did this and it was fine, or am I getting threads mixed up?

Use -info and grep on GAMG. This will print out the average nnz/row on each
level, which is way of seeing if the coarse grids are getting out of
control. But coarse grids are smaller so they should not be a big deal.


>
> Also, run using -ksp_view. GAMG will report all the sizes of its grids, so
> it should be easy to see
> if the coarse grid sizes are increasing, and also what the effect of the
> threshold value is.
>

The coarse grid "sizes" will go way down, that is what MG does unless
something is very wrong. The nnz/row will go up in many cases. If ksp_view
prints out the nnz for the operator on each level then you can compute the
average nnz/row (-info just does that for you).

The only change that I can think of in GAMG wrt coarsening was the
treatment of the square graph parameters. It used to be a bool (square
first level or not). Now it is an integer q (square the first q levels).

If you suspect GAMG, can you test with Hypre?


>
>   Thanks,
>
>      Matt
>
>> Thanks
>> Myriam
>>
>> Le 03/02/19 à 02:27, Matthew Knepley a écrit :
>>
>> On Fri, Mar 1, 2019 at 10:53 AM Myriam Peyrounette via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Hi,
>>>
>>> I used to run my code with PETSc 3.6. Since I upgraded the PETSc version
>>> to 3.10, this code has a bad memory scaling.
>>>
>>> To report this issue, I took the PETSc script ex42.c and slightly
>>> modified it so that the KSP and PC configurations are the same as in my
>>> code. In particular, I use a "personnalised" multi-grid method. The
>>> modifications are indicated by the keyword "TopBridge" in the attached
>>> scripts.
>>>
>>> To plot the memory (weak) scaling, I ran four calculations for each
>>> script with increasing problem sizes and computations cores:
>>>
>>> 1. 100,000 elts on 4 cores
>>> 2. 1 million elts on 40 cores
>>> 3. 10 millions elts on 400 cores
>>> 4. 100 millions elts on 4,000 cores
>>>
>>> The resulting graph is also attached. The scaling using PETSc 3.10
>>> clearly deteriorates for large cases, while the one using PETSc 3.6 is
>>> robust.
>>>
>>> After a few tests, I found that the scaling is mostly sensitive to the
>>> use of the AMG method for the coarse grid (line 1780 in
>>> main_ex42_petsc36.cc). In particular, the performance strongly
>>> deteriorates when commenting lines 1777 to 1790 (in
>>> main_ex42_petsc36.cc).
>>>
>>> Do you have any idea of what changed between version 3.6 and version
>>> 3.10 that may imply such degradation?
>>>
>>
>> I believe the default values for PCGAMG changed between versions. It
>> sounds like the coarsening rate
>> is not great enough, so that these grids are too large. This can be set
>> using:
>>
>>
>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html
>>
>> There is some explanation of this effect on that page. Let us know if
>> setting this does not correct the situation.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Let me know if you need further information.
>>>
>>> Best,
>>>
>>> Myriam Peyrounette
>>>
>>>
>>> --
>>> Myriam Peyrounette
>>> CNRS/IDRIS - HLST
>>> --
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/%7Eknepley/>
>>
>>
>> --
>> Myriam Peyrounette
>> CNRS/IDRIS - HLST
>> --
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190430/575688b1/attachment.html>