[petsc-users] Bad memory scaling with PETSc 3.10
Myriam Peyrounette
myriam.peyrounette at idris.fr
Mon Mar 11 08:53:22 CDT 2019
There is a small difference in memory usage already (of 135mB). It is
not a big deal but it will be for larger problems (as shown by the
memory scaling). If we find the origin of this small gap for a small
case, we probably find the reason why the memory scaling is so bad with
3.10.
I am currently looking for the exact commit where the problem arises,
using git bisect. I'll let you know about the result.
Le 03/11/19 à 14:40, Mark Adams a écrit :
> Is there a difference in memory usage on your tiny problem? I assume no.
>
> I don't see anything that could come from GAMG other than the RAP
> stuff that you have discussed already.
>
> On Mon, Mar 11, 2019 at 9:32 AM Myriam Peyrounette
> <myriam.peyrounette at idris.fr <mailto:myriam.peyrounette at idris.fr>> wrote:
>
> The code I am using here is the example 42 of PETSc
> (https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html).
> Indeed it solves the Stokes equation. I thought it was a good idea
> to use an example you might know (and didn't find any that uses
> GAMG functions). I just changed the PCMG setup so that the memory
> problem appears. And it appears when adding PCGAMG.
>
> I don't care about the performance or even the result rightness
> here, but only about the difference in memory use between 3.6 and
> 3.10. Do you think finding a more adapted script would help?
>
> I used the threshold of 0.1 only once, at the beginning, to test
> its influence. I used the default threshold (of 0, I guess) for
> all the other runs.
>
> Myriam
>
>
> Le 03/11/19 à 13:52, Mark Adams a écrit :
>> In looking at this larger scale run ...
>>
>> * Your eigen estimates are much lower than your tiny test
>> problem. But this is Stokes apparently and it should not work
>> anyway. Maybe you have a small time step that adds a lot of mass
>> that brings the eigen estimates down. And your min eigenvalue
>> (not used) is positive. I would expect negative for Stokes ...
>>
>> * You seem to be setting a threshold value of 0.1 -- that is very
>> high
>>
>> * v3.6 says "using nonzero initial guess" but this is not in
>> v3.10. Maybe we just stopped printing that.
>>
>> * There were some changes to coasening parameters in going from
>> v3.6 but it does not look like your problem was effected. (The
>> coarsening algo is non-deterministic by default and you can see
>> small difference on different runs)
>>
>> * We may have also added a "noisy" RHS for eigen estimates by
>> default from v3.6.
>>
>> * And for non-symetric problems you can try -pc_gamg_agg_nsmooths
>> 0, but again GAMG is not built for Stokes anyway.
>>
>>
>> On Tue, Mar 5, 2019 at 11:53 AM Myriam Peyrounette
>> <myriam.peyrounette at idris.fr
>> <mailto:myriam.peyrounette at idris.fr>> wrote:
>>
>> I used PCView to display the size of the linear system in
>> each level of the MG. You'll find the outputs attached to
>> this mail (zip file) for both the default threshold value and
>> a value of 0.1, and for both 3.6 and 3.10 PETSc versions.
>>
>> For convenience, I summarized the information in a graph,
>> also attached (png file).
>>
>> As you can see, there are slight differences between the two
>> versions but none is critical, in my opinion. Do you see
>> anything suspicious in the outputs?
>>
>> + I can't find the default threshold value. Do you know where
>> I can find it?
>>
>> Thanks for the follow-up
>>
>> Myriam
>>
>>
>> Le 03/05/19 à 14:06, Matthew Knepley a écrit :
>>> On Tue, Mar 5, 2019 at 7:14 AM Myriam Peyrounette
>>> <myriam.peyrounette at idris.fr
>>> <mailto:myriam.peyrounette at idris.fr>> wrote:
>>>
>>> Hi Matt,
>>>
>>> I plotted the memory scalings using different threshold
>>> values. The two scalings are slightly translated (from
>>> -22 to -88 mB) but this gain is neglectable. The
>>> 3.6-scaling keeps being robust while the 3.10-scaling
>>> deteriorates.
>>>
>>> Do you have any other suggestion?
>>>
>>> Mark, what is the option she can give to output all the GAMG
>>> data?
>>>
>>> Also, run using -ksp_view. GAMG will report all the sizes of
>>> its grids, so it should be easy to see
>>> if the coarse grid sizes are increasing, and also what the
>>> effect of the threshold value is.
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>> Thanks
>>>
>>> Myriam
>>>
>>> Le 03/02/19 à 02:27, Matthew Knepley a écrit :
>>>> On Fri, Mar 1, 2019 at 10:53 AM Myriam Peyrounette via
>>>> petsc-users <petsc-users at mcs.anl.gov
>>>> <mailto:petsc-users at mcs.anl.gov>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I used to run my code with PETSc 3.6. Since I
>>>> upgraded the PETSc version
>>>> to 3.10, this code has a bad memory scaling.
>>>>
>>>> To report this issue, I took the PETSc script
>>>> ex42.c and slightly
>>>> modified it so that the KSP and PC configurations
>>>> are the same as in my
>>>> code. In particular, I use a "personnalised"
>>>> multi-grid method. The
>>>> modifications are indicated by the keyword
>>>> "TopBridge" in the attached
>>>> scripts.
>>>>
>>>> To plot the memory (weak) scaling, I ran four
>>>> calculations for each
>>>> script with increasing problem sizes and
>>>> computations cores:
>>>>
>>>> 1. 100,000 elts on 4 cores
>>>> 2. 1 million elts on 40 cores
>>>> 3. 10 millions elts on 400 cores
>>>> 4. 100 millions elts on 4,000 cores
>>>>
>>>> The resulting graph is also attached. The scaling
>>>> using PETSc 3.10
>>>> clearly deteriorates for large cases, while the one
>>>> using PETSc 3.6 is
>>>> robust.
>>>>
>>>> After a few tests, I found that the scaling is
>>>> mostly sensitive to the
>>>> use of the AMG method for the coarse grid (line 1780 in
>>>> main_ex42_petsc36.cc). In particular, the
>>>> performance strongly
>>>> deteriorates when commenting lines 1777 to 1790 (in
>>>> main_ex42_petsc36.cc).
>>>>
>>>> Do you have any idea of what changed between
>>>> version 3.6 and version
>>>> 3.10 that may imply such degradation?
>>>>
>>>>
>>>> I believe the default values for PCGAMG changed between
>>>> versions. It sounds like the coarsening rate
>>>> is not great enough, so that these grids are too large.
>>>> This can be set using:
>>>>
>>>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html
>>>>
>>>> There is some explanation of this effect on that page.
>>>> Let us know if setting this does not correct the situation.
>>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>>
>>>> Let me know if you need further information.
>>>>
>>>> Best,
>>>>
>>>> Myriam Peyrounette
>>>>
>>>>
>>>> --
>>>> Myriam Peyrounette
>>>> CNRS/IDRIS - HLST
>>>> --
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they
>>>> begin their experiments is infinitely more interesting
>>>> than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/%7Eknepley/>
>>>
>>> --
>>> Myriam Peyrounette
>>> CNRS/IDRIS - HLST
>>> --
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin
>>> their experiments is infinitely more interesting than any
>>> results to which their experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/%7Eknepley/>
>>
>> --
>> Myriam Peyrounette
>> CNRS/IDRIS - HLST
>> --
>>
>
> --
> Myriam Peyrounette
> CNRS/IDRIS - HLST
> --
>
--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190311/a322fa37/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2975 bytes
Desc: Signature cryptographique S/MIME
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190311/a322fa37/attachment-0001.p7s>
More information about the petsc-users
mailing list