[petsc-users] Bad memory scaling with PETSc 3.10

Tue Mar 5 10:53:47 CST 2019

I used PCView to display the size of the linear system in each level of
the MG. You'll find the outputs attached to this mail (zip file) for
both the default threshold value and a value of 0.1, and for both 3.6
and 3.10 PETSc versions.

For convenience, I summarized the information in a graph, also attached
(png file).

As you can see, there are slight differences between the two versions
but none is critical, in my opinion. Do you see anything suspicious in
the outputs?

+ I can't find the default threshold value. Do you know where I can find it?

Thanks for the follow-up

Myriam

Le 03/05/19 à 14:06, Matthew Knepley a écrit :
> On Tue, Mar 5, 2019 at 7:14 AM Myriam Peyrounette
> <myriam.peyrounette at idris.fr <mailto:myriam.peyrounette at idris.fr>> wrote:
>
>     Hi Matt,
>
>     I plotted the memory scalings using different threshold values.
>     The two scalings are slightly translated (from -22 to -88 mB) but
>     this gain is neglectable. The 3.6-scaling keeps being robust while
>     the 3.10-scaling deteriorates.
>
>     Do you have any other suggestion?
>
> Mark, what is the option she can give to output all the GAMG data?
>
> Also, run using -ksp_view. GAMG will report all the sizes of its
> grids, so it should be easy to see
> if the coarse grid sizes are increasing, and also what the effect of
> the threshold value is.
>
>   Thanks,
>
>      Matt 
>
>     Thanks
>
>     Myriam
>
>     Le 03/02/19 à 02:27, Matthew Knepley a écrit :
>>     On Fri, Mar 1, 2019 at 10:53 AM Myriam Peyrounette via
>>     petsc-users <petsc-users at mcs.anl.gov
>>     <mailto:petsc-users at mcs.anl.gov>> wrote:
>>
>>         Hi,
>>
>>         I used to run my code with PETSc 3.6. Since I upgraded the
>>         PETSc version
>>         to 3.10, this code has a bad memory scaling.
>>
>>         To report this issue, I took the PETSc script ex42.c and slightly
>>         modified it so that the KSP and PC configurations are the
>>         same as in my
>>         code. In particular, I use a "personnalised" multi-grid
>>         method. The
>>         modifications are indicated by the keyword "TopBridge" in the
>>         attached
>>         scripts.
>>
>>         To plot the memory (weak) scaling, I ran four calculations
>>         for each
>>         script with increasing problem sizes and computations cores:
>>
>>         1. 100,000 elts on 4 cores
>>         2. 1 million elts on 40 cores
>>         3. 10 millions elts on 400 cores
>>         4. 100 millions elts on 4,000 cores
>>
>>         The resulting graph is also attached. The scaling using PETSc
>>         3.10
>>         clearly deteriorates for large cases, while the one using
>>         PETSc 3.6 is
>>         robust.
>>
>>         After a few tests, I found that the scaling is mostly
>>         sensitive to the
>>         use of the AMG method for the coarse grid (line 1780 in
>>         main_ex42_petsc36.cc). In particular, the performance strongly
>>         deteriorates when commenting lines 1777 to 1790 (in
>>         main_ex42_petsc36.cc).
>>
>>         Do you have any idea of what changed between version 3.6 and
>>         version
>>         3.10 that may imply such degradation?
>>
>>
>>     I believe the default values for PCGAMG changed between versions.
>>     It sounds like the coarsening rate
>>     is not great enough, so that these grids are too large. This can
>>     be set using:
>>
>>       https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html
>>
>>     There is some explanation of this effect on that page. Let us
>>     know if setting this does not correct the situation.
>>
>>       Thanks,
>>
>>          Matt
>>      
>>
>>         Let me know if you need further information.
>>
>>         Best,
>>
>>         Myriam Peyrounette
>>
>>
>>         -- 
>>         Myriam Peyrounette
>>         CNRS/IDRIS - HLST
>>         --
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>>
>>     https://www.cse.buffalo.edu/~knepley/
>>     <http://www.cse.buffalo.edu/%7Eknepley/>
>
>     -- 
>     Myriam Peyrounette
>     CNRS/IDRIS - HLST
>     --
>
>
>
> -- 
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/%7Eknepley/>

-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190305/8ab9215d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PCView_GAMG_threshold_investigation.zip
Type: application/zip
Size: 6924 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190305/8ab9215d/attachment-0001.zip>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pcview_gamg_threshold_investigation.png
Type: image/png
Size: 23243 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190305/8ab9215d/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2975 bytes
Desc: Signature cryptographique S/MIME
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190305/8ab9215d/attachment-0001.p7s>