<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi all,</p>
<p>I used git bisect to determine when the memory need increased. I
found that the first "bad" commit is
aa690a28a7284adb519c28cb44eae20a2c131c85.</p>
<p>Barry was right, this commit seems to be about an evolution of <span
class="blob-code-inner"><span class="pl-en x x-first x-last">MatPtAPSymbolic_MPIAIJ_MPIAIJ.
You mentioned the option "-matptap_via scalable" but I can't
find any information about it. Can you tell me more?</span></span></p>
<p><span class="blob-code-inner"><span class="pl-en x x-first
x-last">Thanks</span></span></p>
<p><span class="blob-code-inner"><span class="pl-en x x-first
x-last">Myriam</span></span></p>
<p><span class="blob-code-inner"><span class="pl-en x x-first
x-last"></span></span></p>
<br>
<div class="moz-cite-prefix">Le 03/11/19 à 14:40, Mark Adams a
écrit :<br>
</div>
<blockquote type="cite"
cite="mid:CADOhEh5MHFQOoHF24hOWhoAiLnqYuZO29hwKzHmMndicF1xj0g@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<div dir="ltr">Is there a difference in memory usage on your tiny
problem? I assume no.
<div><br>
</div>
<div>I don't see anything that could come from GAMG other than
the RAP stuff that you have discussed already.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Mar 11, 2019 at 9:32
AM Myriam Peyrounette <<a
href="mailto:myriam.peyrounette@idris.fr"
moz-do-not-send="true">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>The code I am using here is the example 42 of PETSc
(<a
class="gmail-m_4941328961016005032moz-txt-link-freetext"
href="https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html"
target="_blank" moz-do-not-send="true">https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html</a>).
Indeed it solves the Stokes equation. I thought it was a
good idea to use an example you might know (and didn't
find any that uses GAMG functions). I just changed the
PCMG setup so that the memory problem appears. And it
appears when adding PCGAMG.</p>
<p>I don't care about the performance or even the result
rightness here, but only about the difference in memory
use between 3.6 and 3.10. Do you think finding a more
adapted script would help?<br>
</p>
<p>I used the threshold of 0.1 only once, at the beginning,
to test its influence. I used the default threshold (of 0,
I guess) for all the other runs.</p>
<p>Myriam<br>
</p>
<br>
<div class="gmail-m_4941328961016005032moz-cite-prefix">Le
03/11/19 à 13:52, Mark Adams a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">In looking at this larger scale run ...
<div><br>
</div>
<div>* Your eigen estimates are much lower than your
tiny test problem. But this is Stokes apparently
and it should not work anyway. Maybe you have a
small time step that adds a lot of mass that brings
the eigen estimates down. And your min eigenvalue
(not used) is positive. I would expect negative for
Stokes ...</div>
<div><br>
</div>
<div>* You seem to be setting a threshold value of 0.1
-- that is very high</div>
<div><br>
</div>
<div>* v3.6 says "using nonzero initial guess" but
this is not in v3.10. Maybe we just stopped printing
that.</div>
<div><br>
</div>
<div>* There were some changes to coasening parameters
in going from v3.6 but it does not look like your
problem was effected. (The coarsening algo is
non-deterministic by default and you can see small
difference on different runs)</div>
<div><br>
</div>
<div>* We may have also added a "noisy" RHS for eigen
estimates by default from v3.6.</div>
<div><br>
</div>
<div>* And for non-symetric problems you can try
-pc_gamg_agg_nsmooths 0, but again GAMG is not built
for Stokes anyway.</div>
<div><br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Mar 5, 2019 at
11:53 AM Myriam Peyrounette <<a
href="mailto:myriam.peyrounette@idris.fr"
target="_blank" moz-do-not-send="true">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>I used PCView to display the size of the linear
system in each level of the MG. You'll find the
outputs attached to this mail (zip file) for both
the default threshold value and a value of 0.1,
and for both 3.6 and 3.10 PETSc versions. <br>
</p>
<p>For convenience, I summarized the information in
a graph, also attached (png file).</p>
<p>As you can see, there are slight differences
between the two versions but none is critical, in
my opinion. Do you see anything suspicious in the
outputs?</p>
<p>+ I can't find the default threshold value. Do
you know where I can find it?<br>
</p>
<p>Thanks for the follow-up</p>
<p>Myriam<br>
</p>
<br>
<div
class="gmail-m_4941328961016005032gmail-m_4553173887686987135moz-cite-prefix">Le
03/05/19 à 14:06, Matthew Knepley a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Tue, Mar 5, 2019 at 7:14 AM
Myriam Peyrounette <<a
href="mailto:myriam.peyrounette@idris.fr"
target="_blank" moz-do-not-send="true">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Matt,</p>
<p>I plotted the memory scalings using
different threshold values. The two
scalings are slightly translated (from
-22 to -88 mB) but this gain is
neglectable. The 3.6-scaling keeps being
robust while the 3.10-scaling
deteriorates.</p>
<p>Do you have any other suggestion?</p>
</div>
</blockquote>
<div>Mark, what is the option she can give to
output all the GAMG data?</div>
<div><br>
</div>
<div>Also, run using -ksp_view. GAMG will
report all the sizes of its grids, so it
should be easy to see</div>
<div>if the coarse grid sizes are increasing,
and also what the effect of the threshold
value is.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt <br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Thanks<br>
</p>
Myriam <br>
<br>
<div
class="gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-cite-prefix">Le
03/02/19 à 02:27, Matthew Knepley a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">On Fri, Mar 1, 2019
at 10:53 AM Myriam Peyrounette via
petsc-users <<a
href="mailto:petsc-users@mcs.anl.gov"
target="_blank"
moz-do-not-send="true">petsc-users@mcs.anl.gov</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
I used to run my code with PETSc
3.6. Since I upgraded the PETSc
version<br>
to 3.10, this code has a bad
memory scaling.<br>
<br>
To report this issue, I took the
PETSc script ex42.c and slightly<br>
modified it so that the KSP and
PC configurations are the same
as in my<br>
code. In particular, I use a
"personnalised" multi-grid
method. The<br>
modifications are indicated by
the keyword "TopBridge" in the
attached<br>
scripts.<br>
<br>
To plot the memory (weak)
scaling, I ran four calculations
for each<br>
script with increasing problem
sizes and computations cores:<br>
<br>
1. 100,000 elts on 4 cores<br>
2. 1 million elts on 40 cores<br>
3. 10 millions elts on 400 cores<br>
4. 100 millions elts on 4,000
cores<br>
<br>
The resulting graph is also
attached. The scaling using
PETSc 3.10<br>
clearly deteriorates for large
cases, while the one using PETSc
3.6 is<br>
robust.<br>
<br>
After a few tests, I found that
the scaling is mostly sensitive
to the<br>
use of the AMG method for the
coarse grid (line 1780 in<br>
main_ex42_petsc36.cc). In
particular, the performance
strongly<br>
deteriorates when commenting
lines 1777 to 1790 (in
main_ex42_petsc36.cc).<br>
<br>
Do you have any idea of what
changed between version 3.6 and
version<br>
3.10 that may imply such
degradation?<br>
</blockquote>
<div><br>
</div>
<div>I believe the default values
for PCGAMG changed between
versions. It sounds like the
coarsening rate</div>
<div>is not great enough, so that
these grids are too large. This
can be set using:</div>
<div><br>
</div>
<div> <a
href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html"
target="_blank"
moz-do-not-send="true">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html</a></div>
<div><br>
</div>
<div>There is some explanation of
this effect on that page. Let us
know if setting this does not
correct the situation.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
Let me know if you need further
information.<br>
<br>
Best,<br>
<br>
Myriam Peyrounette<br>
<br>
<br>
-- <br>
Myriam Peyrounette<br>
CNRS/IDRIS - HLST<br>
--<br>
<br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr"
class="gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most
experimenters take for
granted before they
begin their
experiments is
infinitely more
interesting than any
results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a
href="http://www.cse.buffalo.edu/%7Eknepley/"
target="_blank"
moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr"
class="gmail-m_4941328961016005032gmail-m_4553173887686987135gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take
for granted before they begin
their experiments is infinitely
more interesting than any results
to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a
href="http://www.cse.buffalo.edu/%7Eknepley/"
target="_blank"
moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_4941328961016005032gmail-m_4553173887686987135moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="gmail-m_4941328961016005032moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</body>
</html>