[petsc-users] Bad memory scaling with PETSc 3.10
Myriam Peyrounette
myriam.peyrounette at idris.fr
Mon Mar 11 08:32:07 CDT 2019
The code I am using here is the example 42 of PETSc
(https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html).
Indeed it solves the Stokes equation. I thought it was a good idea to
use an example you might know (and didn't find any that uses GAMG
functions). I just changed the PCMG setup so that the memory problem
appears. And it appears when adding PCGAMG.
I don't care about the performance or even the result rightness here,
but only about the difference in memory use between 3.6 and 3.10. Do you
think finding a more adapted script would help?
I used the threshold of 0.1 only once, at the beginning, to test its
influence. I used the default threshold (of 0, I guess) for all the
other runs.
Myriam
Le 03/11/19 à 13:52, Mark Adams a écrit :
> In looking at this larger scale run ...
>
> * Your eigen estimates are much lower than your tiny test problem.
> But this is Stokes apparently and it should not work anyway. Maybe you
> have a small time step that adds a lot of mass that brings the eigen
> estimates down. And your min eigenvalue (not used) is positive. I
> would expect negative for Stokes ...
>
> * You seem to be setting a threshold value of 0.1 -- that is very high
>
> * v3.6 says "using nonzero initial guess" but this is not in v3.10.
> Maybe we just stopped printing that.
>
> * There were some changes to coasening parameters in going from v3.6
> but it does not look like your problem was effected. (The coarsening
> algo is non-deterministic by default and you can see small difference
> on different runs)
>
> * We may have also added a "noisy" RHS for eigen estimates by default
> from v3.6.
>
> * And for non-symetric problems you can try -pc_gamg_agg_nsmooths 0,
> but again GAMG is not built for Stokes anyway.
>
>
> On Tue, Mar 5, 2019 at 11:53 AM Myriam Peyrounette
> <myriam.peyrounette at idris.fr <mailto:myriam.peyrounette at idris.fr>> wrote:
>
> I used PCView to display the size of the linear system in each
> level of the MG. You'll find the outputs attached to this mail
> (zip file) for both the default threshold value and a value of
> 0.1, and for both 3.6 and 3.10 PETSc versions.
>
> For convenience, I summarized the information in a graph, also
> attached (png file).
>
> As you can see, there are slight differences between the two
> versions but none is critical, in my opinion. Do you see anything
> suspicious in the outputs?
>
> + I can't find the default threshold value. Do you know where I
> can find it?
>
> Thanks for the follow-up
>
> Myriam
>
>
> Le 03/05/19 à 14:06, Matthew Knepley a écrit :
>> On Tue, Mar 5, 2019 at 7:14 AM Myriam Peyrounette
>> <myriam.peyrounette at idris.fr
>> <mailto:myriam.peyrounette at idris.fr>> wrote:
>>
>> Hi Matt,
>>
>> I plotted the memory scalings using different threshold
>> values. The two scalings are slightly translated (from -22 to
>> -88 mB) but this gain is neglectable. The 3.6-scaling keeps
>> being robust while the 3.10-scaling deteriorates.
>>
>> Do you have any other suggestion?
>>
>> Mark, what is the option she can give to output all the GAMG data?
>>
>> Also, run using -ksp_view. GAMG will report all the sizes of its
>> grids, so it should be easy to see
>> if the coarse grid sizes are increasing, and also what the effect
>> of the threshold value is.
>>
>> Thanks,
>>
>> Matt
>>
>> Thanks
>>
>> Myriam
>>
>> Le 03/02/19 à 02:27, Matthew Knepley a écrit :
>>> On Fri, Mar 1, 2019 at 10:53 AM Myriam Peyrounette via
>>> petsc-users <petsc-users at mcs.anl.gov
>>> <mailto:petsc-users at mcs.anl.gov>> wrote:
>>>
>>> Hi,
>>>
>>> I used to run my code with PETSc 3.6. Since I upgraded
>>> the PETSc version
>>> to 3.10, this code has a bad memory scaling.
>>>
>>> To report this issue, I took the PETSc script ex42.c and
>>> slightly
>>> modified it so that the KSP and PC configurations are
>>> the same as in my
>>> code. In particular, I use a "personnalised" multi-grid
>>> method. The
>>> modifications are indicated by the keyword "TopBridge"
>>> in the attached
>>> scripts.
>>>
>>> To plot the memory (weak) scaling, I ran four
>>> calculations for each
>>> script with increasing problem sizes and computations cores:
>>>
>>> 1. 100,000 elts on 4 cores
>>> 2. 1 million elts on 40 cores
>>> 3. 10 millions elts on 400 cores
>>> 4. 100 millions elts on 4,000 cores
>>>
>>> The resulting graph is also attached. The scaling using
>>> PETSc 3.10
>>> clearly deteriorates for large cases, while the one
>>> using PETSc 3.6 is
>>> robust.
>>>
>>> After a few tests, I found that the scaling is mostly
>>> sensitive to the
>>> use of the AMG method for the coarse grid (line 1780 in
>>> main_ex42_petsc36.cc). In particular, the performance
>>> strongly
>>> deteriorates when commenting lines 1777 to 1790 (in
>>> main_ex42_petsc36.cc).
>>>
>>> Do you have any idea of what changed between version 3.6
>>> and version
>>> 3.10 that may imply such degradation?
>>>
>>>
>>> I believe the default values for PCGAMG changed between
>>> versions. It sounds like the coarsening rate
>>> is not great enough, so that these grids are too large. This
>>> can be set using:
>>>
>>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html
>>>
>>> There is some explanation of this effect on that page. Let
>>> us know if setting this does not correct the situation.
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>>
>>> Let me know if you need further information.
>>>
>>> Best,
>>>
>>> Myriam Peyrounette
>>>
>>>
>>> --
>>> Myriam Peyrounette
>>> CNRS/IDRIS - HLST
>>> --
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin
>>> their experiments is infinitely more interesting than any
>>> results to which their experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/%7Eknepley/>
>>
>> --
>> Myriam Peyrounette
>> CNRS/IDRIS - HLST
>> --
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to
>> which their experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/%7Eknepley/>
>
> --
> Myriam Peyrounette
> CNRS/IDRIS - HLST
> --
>
--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190311/58a8d1b6/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2975 bytes
Desc: Signature cryptographique S/MIME
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190311/58a8d1b6/attachment-0001.p7s>
More information about the petsc-users
mailing list