[petsc-users] Bad memory scaling with PETSc 3.10

Myriam Peyrounette myriam.peyrounette at idris.fr
Wed Mar 20 07:48:38 CDT 2019


Hi all,

I used git bisect to determine when the memory need increased. I found
that the first "bad" commit is   aa690a28a7284adb519c28cb44eae20a2c131c85.

Barry was right, this commit seems to be about an evolution of
MatPtAPSymbolic_MPIAIJ_MPIAIJ. You mentioned the option "-matptap_via
scalable" but I can't find any information about it. Can you tell me more?

Thanks

Myriam


Le 03/11/19 à 14:40, Mark Adams a écrit :
> Is there a difference in memory usage on your tiny problem? I assume no.
>
> I don't see anything that could come from GAMG other than the RAP
> stuff that you have discussed already.
>
> On Mon, Mar 11, 2019 at 9:32 AM Myriam Peyrounette
> <myriam.peyrounette at idris.fr <mailto:myriam.peyrounette at idris.fr>> wrote:
>
>     The code I am using here is the example 42 of PETSc
>     (https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html).
>     Indeed it solves the Stokes equation. I thought it was a good idea
>     to use an example you might know (and didn't find any that uses
>     GAMG functions). I just changed the PCMG setup so that the memory
>     problem appears. And it appears when adding PCGAMG.
>
>     I don't care about the performance or even the result rightness
>     here, but only about the difference in memory use between 3.6 and
>     3.10. Do you think finding a more adapted script would help?
>
>     I used the threshold of 0.1 only once, at the beginning, to test
>     its influence. I used the default threshold (of 0, I guess) for
>     all the other runs.
>
>     Myriam
>
>
>     Le 03/11/19 à 13:52, Mark Adams a écrit :
>>     In looking at this larger scale run ...
>>
>>     * Your eigen estimates are much lower than your tiny test
>>     problem.  But this is Stokes apparently and it should not work
>>     anyway. Maybe you have a small time step that adds a lot of mass
>>     that brings the eigen estimates down. And your min eigenvalue
>>     (not used) is positive. I would expect negative for Stokes ...
>>
>>     * You seem to be setting a threshold value of 0.1 -- that is very
>>     high
>>
>>     * v3.6 says "using nonzero initial guess" but this is not in
>>     v3.10. Maybe we just stopped printing that.
>>
>>     * There were some changes to coasening parameters in going from
>>     v3.6 but it does not look like your problem was effected. (The
>>     coarsening algo is non-deterministic by default and you can see
>>     small difference on different runs)
>>
>>     * We may have also added a "noisy" RHS for eigen estimates by
>>     default from v3.6.
>>
>>     * And for non-symetric problems you can try -pc_gamg_agg_nsmooths
>>     0, but again GAMG is not built for Stokes anyway.
>>
>>
>>     On Tue, Mar 5, 2019 at 11:53 AM Myriam Peyrounette
>>     <myriam.peyrounette at idris.fr
>>     <mailto:myriam.peyrounette at idris.fr>> wrote:
>>
>>         I used PCView to display the size of the linear system in
>>         each level of the MG. You'll find the outputs attached to
>>         this mail (zip file) for both the default threshold value and
>>         a value of 0.1, and for both 3.6 and 3.10 PETSc versions.
>>
>>         For convenience, I summarized the information in a graph,
>>         also attached (png file).
>>
>>         As you can see, there are slight differences between the two
>>         versions but none is critical, in my opinion. Do you see
>>         anything suspicious in the outputs?
>>
>>         + I can't find the default threshold value. Do you know where
>>         I can find it?
>>
>>         Thanks for the follow-up
>>
>>         Myriam
>>
>>
>>         Le 03/05/19 à 14:06, Matthew Knepley a écrit :
>>>         On Tue, Mar 5, 2019 at 7:14 AM Myriam Peyrounette
>>>         <myriam.peyrounette at idris.fr
>>>         <mailto:myriam.peyrounette at idris.fr>> wrote:
>>>
>>>             Hi Matt,
>>>
>>>             I plotted the memory scalings using different threshold
>>>             values. The two scalings are slightly translated (from
>>>             -22 to -88 mB) but this gain is neglectable. The
>>>             3.6-scaling keeps being robust while the 3.10-scaling
>>>             deteriorates.
>>>
>>>             Do you have any other suggestion?
>>>
>>>         Mark, what is the option she can give to output all the GAMG
>>>         data?
>>>
>>>         Also, run using -ksp_view. GAMG will report all the sizes of
>>>         its grids, so it should be easy to see
>>>         if the coarse grid sizes are increasing, and also what the
>>>         effect of the threshold value is.
>>>
>>>           Thanks,
>>>
>>>              Matt 
>>>
>>>             Thanks
>>>
>>>             Myriam
>>>
>>>             Le 03/02/19 à 02:27, Matthew Knepley a écrit :
>>>>             On Fri, Mar 1, 2019 at 10:53 AM Myriam Peyrounette via
>>>>             petsc-users <petsc-users at mcs.anl.gov
>>>>             <mailto:petsc-users at mcs.anl.gov>> wrote:
>>>>
>>>>                 Hi,
>>>>
>>>>                 I used to run my code with PETSc 3.6. Since I
>>>>                 upgraded the PETSc version
>>>>                 to 3.10, this code has a bad memory scaling.
>>>>
>>>>                 To report this issue, I took the PETSc script
>>>>                 ex42.c and slightly
>>>>                 modified it so that the KSP and PC configurations
>>>>                 are the same as in my
>>>>                 code. In particular, I use a "personnalised"
>>>>                 multi-grid method. The
>>>>                 modifications are indicated by the keyword
>>>>                 "TopBridge" in the attached
>>>>                 scripts.
>>>>
>>>>                 To plot the memory (weak) scaling, I ran four
>>>>                 calculations for each
>>>>                 script with increasing problem sizes and
>>>>                 computations cores:
>>>>
>>>>                 1. 100,000 elts on 4 cores
>>>>                 2. 1 million elts on 40 cores
>>>>                 3. 10 millions elts on 400 cores
>>>>                 4. 100 millions elts on 4,000 cores
>>>>
>>>>                 The resulting graph is also attached. The scaling
>>>>                 using PETSc 3.10
>>>>                 clearly deteriorates for large cases, while the one
>>>>                 using PETSc 3.6 is
>>>>                 robust.
>>>>
>>>>                 After a few tests, I found that the scaling is
>>>>                 mostly sensitive to the
>>>>                 use of the AMG method for the coarse grid (line 1780 in
>>>>                 main_ex42_petsc36.cc). In particular, the
>>>>                 performance strongly
>>>>                 deteriorates when commenting lines 1777 to 1790 (in
>>>>                 main_ex42_petsc36.cc).
>>>>
>>>>                 Do you have any idea of what changed between
>>>>                 version 3.6 and version
>>>>                 3.10 that may imply such degradation?
>>>>
>>>>
>>>>             I believe the default values for PCGAMG changed between
>>>>             versions. It sounds like the coarsening rate
>>>>             is not great enough, so that these grids are too large.
>>>>             This can be set using:
>>>>
>>>>               https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html
>>>>
>>>>             There is some explanation of this effect on that page.
>>>>             Let us know if setting this does not correct the situation.
>>>>
>>>>               Thanks,
>>>>
>>>>                  Matt
>>>>              
>>>>
>>>>                 Let me know if you need further information.
>>>>
>>>>                 Best,
>>>>
>>>>                 Myriam Peyrounette
>>>>
>>>>
>>>>                 -- 
>>>>                 Myriam Peyrounette
>>>>                 CNRS/IDRIS - HLST
>>>>                 --
>>>>
>>>>
>>>>
>>>>             -- 
>>>>             What most experimenters take for granted before they
>>>>             begin their experiments is infinitely more interesting
>>>>             than any results to which their experiments lead.
>>>>             -- Norbert Wiener
>>>>
>>>>             https://www.cse.buffalo.edu/~knepley/
>>>>             <http://www.cse.buffalo.edu/%7Eknepley/>
>>>
>>>             -- 
>>>             Myriam Peyrounette
>>>             CNRS/IDRIS - HLST
>>>             --
>>>
>>>
>>>
>>>         -- 
>>>         What most experimenters take for granted before they begin
>>>         their experiments is infinitely more interesting than any
>>>         results to which their experiments lead.
>>>         -- Norbert Wiener
>>>
>>>         https://www.cse.buffalo.edu/~knepley/
>>>         <http://www.cse.buffalo.edu/%7Eknepley/>
>>
>>         -- 
>>         Myriam Peyrounette
>>         CNRS/IDRIS - HLST
>>         --
>>
>
>     -- 
>     Myriam Peyrounette
>     CNRS/IDRIS - HLST
>     --
>

-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190320/3dfa902b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2975 bytes
Desc: Signature cryptographique S/MIME
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190320/3dfa902b/attachment-0001.p7s>


More information about the petsc-users mailing list