<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi,</p>
<p>thanks for the explanations. I tried the last PETSc version
(commit fbc5705bc518d02a4999f188aad4ccff5f754cbf), which includes
the patch you talked about. But the memory scaling shows no
improvement (see scaling attached), even when using the "scalable"
options :(</p>
<p>I had a look at the PETSc functions MatPtAPNumeric_MPIAIJ_MPIAIJ
and MatPtAPSymbolic_MPIAIJ_MPIAIJ (especially at the differences
before and after the first "bad" commit), but I can't find what
induced this memory issue.</p>
<p>Myriam<br>
</p>
<p><br>
</p>
<p><br>
</p>
<br>
<div class="moz-cite-prefix">Le 03/20/19 à 17:38, Fande Kong a
écrit :<br>
</div>
<blockquote type="cite"
cite="mid:CAN5Wd-LUGytgsGMOv-P=PqP_1Jf8OJbs9wKzUvVt3SiKHw+MAQ@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Hi Myriam,</div>
<div><br>
</div>
<div>There are three algorithms in PETSc to do PtAP
( const char *algTypes[3] =
{"scalable","nonscalable","hypre"};), and can be
specified using the petsc options: -matptap_via
xxxx.</div>
<div><br>
</div>
<div>(1) -matptap_via hypre: This call the hypre
package to do the PtAP trough an all-at-once triple
product. In our experiences, it is the most memory
efficient, but could be slow.</div>
<div><br>
</div>
<div>(2) -matptap_via scalable: This involves a
row-wise algorithm plus an outer product. This will
use more memory than hypre, but way faster. This
used to have a bug that could take all your memory,
and I have a fix at <a
href="https://bitbucket.org/petsc/petsc/pull-requests/1452/mpiptap-enable-large-scale-simulations/diff"
moz-do-not-send="true">https://bitbucket.org/petsc/petsc/pull-requests/1452/mpiptap-enable-large-scale-simulations/diff</a>.
When using this option, we may want to have extra
options such as -inner_offdiag_matmatmult_via
scalable -inner_diag_matmatmult_via scalable to
select inner scalable algorithms.</div>
<div><br>
</div>
<div>(3) -matptap_via nonscalable: Suppose to be
even faster, but use more memory. It does dense
matrix operations.</div>
<div><br>
</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Fande Kong</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Mar 20,
2019 at 10:06 AM Myriam Peyrounette via
petsc-users <<a
href="mailto:petsc-users@mcs.anl.gov"
moz-do-not-send="true">petsc-users@mcs.anl.gov</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>More precisely: something happens when
upgrading the functions
MatPtAPNumeric_MPIAIJ_MPIAIJ and/or
MatPtAPSymbolic_MPIAIJ_MPIAIJ. <br>
</p>
<p>Unfortunately, there are a lot of differences
between the old and new versions of these
functions. I keep investigating but if you
have any idea, please let me know.</p>
<p>Best,<br>
</p>
<p>Myriam<br>
</p>
<br>
<div
class="gmail-m_7961152398334556293moz-cite-prefix">Le
03/20/19 à 13:48, Myriam Peyrounette a écrit :<br>
</div>
<blockquote type="cite">
<p>Hi all,</p>
<p>I used git bisect to determine when the
memory need increased. I found that the
first "bad" commit is
aa690a28a7284adb519c28cb44eae20a2c131c85.</p>
<p>Barry was right, this commit seems to be
about an evolution of <span
class="gmail-m_7961152398334556293blob-code-inner"><span
class="gmail-m_7961152398334556293pl-en
gmail-m_7961152398334556293x
gmail-m_7961152398334556293x-first
gmail-m_7961152398334556293x-last">MatPtAPSymbolic_MPIAIJ_MPIAIJ.
You mentioned the option "-matptap_via
scalable" but I can't find any
information about it. Can you tell me
more?</span></span></p>
<p><span
class="gmail-m_7961152398334556293blob-code-inner"><span
class="gmail-m_7961152398334556293pl-en
gmail-m_7961152398334556293x
gmail-m_7961152398334556293x-first
gmail-m_7961152398334556293x-last">Thanks</span></span></p>
<p><span
class="gmail-m_7961152398334556293blob-code-inner"><span
class="gmail-m_7961152398334556293pl-en
gmail-m_7961152398334556293x
gmail-m_7961152398334556293x-first
gmail-m_7961152398334556293x-last">Myriam</span></span></p>
<p><span
class="gmail-m_7961152398334556293blob-code-inner"><span
class="gmail-m_7961152398334556293pl-en
gmail-m_7961152398334556293x
gmail-m_7961152398334556293x-first
gmail-m_7961152398334556293x-last"></span></span></p>
<br>
<div
class="gmail-m_7961152398334556293moz-cite-prefix">Le
03/11/19 à 14:40, Mark Adams a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">Is there a difference in
memory usage on your tiny problem? I
assume no.
<div><br>
</div>
<div>I don't see anything that could come
from GAMG other than the RAP stuff that
you have discussed already.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon,
Mar 11, 2019 at 9:32 AM Myriam
Peyrounette <<a
href="mailto:myriam.peyrounette@idris.fr"
target="_blank" moz-do-not-send="true">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>The code I am using here is the
example 42 of PETSc (<a
class="gmail-m_7961152398334556293gmail-m_4941328961016005032moz-txt-link-freetext"
href="https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html"
target="_blank"
moz-do-not-send="true">https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html</a>).
Indeed it solves the Stokes
equation. I thought it was a good
idea to use an example you might
know (and didn't find any that uses
GAMG functions). I just changed the
PCMG setup so that the memory
problem appears. And it appears when
adding PCGAMG.</p>
<p>I don't care about the performance
or even the result rightness here,
but only about the difference in
memory use between 3.6 and 3.10. Do
you think finding a more adapted
script would help?<br>
</p>
<p>I used the threshold of 0.1 only
once, at the beginning, to test its
influence. I used the default
threshold (of 0, I guess) for all
the other runs.</p>
<p>Myriam<br>
</p>
<br>
<div
class="gmail-m_7961152398334556293gmail-m_4941328961016005032moz-cite-prefix">Le
03/11/19 à 13:52, Mark Adams a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">In looking at this
larger scale run ...
<div><br>
</div>
<div>* Your eigen estimates are
much lower than your tiny test
problem. But this is Stokes
apparently and it should not
work anyway. Maybe you have a
small time step that adds a
lot of mass that brings the
eigen estimates down. And your
min eigenvalue (not used) is
positive. I would expect
negative for Stokes ...</div>
<div><br>
</div>
<div>* You seem to be setting a
threshold value of 0.1 -- that
is very high</div>
<div><br>
</div>
<div>* v3.6 says "using nonzero
initial guess" but this is not
in v3.10. Maybe we just
stopped printing that.</div>
<div><br>
</div>
<div>* There were some changes
to coasening parameters in
going from v3.6 but it does
not look like your problem was
effected. (The coarsening algo
is non-deterministic by
default and you can see small
difference on different runs)</div>
<div><br>
</div>
<div>* We may have also added a
"noisy" RHS for eigen
estimates by default from
v3.6.</div>
<div><br>
</div>
<div>* And for non-symetric
problems you can try
-pc_gamg_agg_nsmooths 0, but
again GAMG is not built for
Stokes anyway.</div>
<div><br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On
Tue, Mar 5, 2019 at 11:53 AM
Myriam Peyrounette <<a
href="mailto:myriam.peyrounette@idris.fr"
target="_blank"
moz-do-not-send="true">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>I used PCView to display
the size of the linear
system in each level of the
MG. You'll find the outputs
attached to this mail (zip
file) for both the default
threshold value and a value
of 0.1, and for both 3.6 and
3.10 PETSc versions. <br>
</p>
<p>For convenience, I
summarized the information
in a graph, also attached
(png file).</p>
<p>As you can see, there are
slight differences between
the two versions but none is
critical, in my opinion. Do
you see anything suspicious
in the outputs?</p>
<p>+ I can't find the default
threshold value. Do you know
where I can find it?<br>
</p>
<p>Thanks for the follow-up</p>
<p>Myriam<br>
</p>
<br>
<div
class="gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135moz-cite-prefix">Le
03/05/19 à 14:06, Matthew
Knepley a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Tue, Mar
5, 2019 at 7:14 AM
Myriam Peyrounette <<a
href="mailto:myriam.peyrounette@idris.fr" target="_blank"
moz-do-not-send="true">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote
class="gmail_quote"
style="margin:0px 0px
0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Matt,</p>
<p>I plotted the
memory scalings
using different
threshold values.
The two scalings
are slightly
translated (from
-22 to -88 mB) but
this gain is
neglectable. The
3.6-scaling keeps
being robust while
the 3.10-scaling
deteriorates.</p>
<p>Do you have any
other suggestion?</p>
</div>
</blockquote>
<div>Mark, what is the
option she can give to
output all the GAMG
data?</div>
<div><br>
</div>
<div>Also, run using
-ksp_view. GAMG will
report all the sizes
of its grids, so it
should be easy to see</div>
<div>if the coarse grid
sizes are increasing,
and also what the
effect of the
threshold value is.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt <br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px
0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Thanks<br>
</p>
Myriam <br>
<br>
<div
class="gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-cite-prefix">Le
03/02/19 à 02:27,
Matthew Knepley a
écrit :<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">On
Fri, Mar 1,
2019 at 10:53
AM Myriam
Peyrounette
via
petsc-users
<<a
href="mailto:petsc-users@mcs.anl.gov"
target="_blank" moz-do-not-send="true">petsc-users@mcs.anl.gov</a>>
wrote:<br>
</div>
<div
class="gmail_quote">
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
I used to run
my code with
PETSc 3.6.
Since I
upgraded the
PETSc version<br>
to 3.10, this
code has a bad
memory
scaling.<br>
<br>
To report this
issue, I took
the PETSc
script ex42.c
and slightly<br>
modified it so
that the KSP
and PC
configurations
are the same
as in my<br>
code. In
particular, I
use a
"personnalised"
multi-grid
method. The<br>
modifications
are indicated
by the keyword
"TopBridge" in
the attached<br>
scripts.<br>
<br>
To plot the
memory (weak)
scaling, I ran
four
calculations
for each<br>
script with
increasing
problem sizes
and
computations
cores:<br>
<br>
1. 100,000
elts on 4
cores<br>
2. 1 million
elts on 40
cores<br>
3. 10 millions
elts on 400
cores<br>
4. 100
millions elts
on 4,000 cores<br>
<br>
The resulting
graph is also
attached. The
scaling using
PETSc 3.10<br>
clearly
deteriorates
for large
cases, while
the one using
PETSc 3.6 is<br>
robust.<br>
<br>
After a few
tests, I found
that the
scaling is
mostly
sensitive to
the<br>
use of the AMG
method for the
coarse grid
(line 1780 in<br>
main_ex42_petsc36.cc). In particular, the performance strongly<br>
deteriorates
when
commenting
lines 1777 to
1790 (in
main_ex42_petsc36.cc).<br>
<br>
Do you have
any idea of
what changed
between
version 3.6
and version<br>
3.10 that may
imply such
degradation?<br>
</blockquote>
<div><br>
</div>
<div>I believe
the default
values for
PCGAMG changed
between
versions. It
sounds like
the coarsening
rate</div>
<div>is not
great enough,
so that these
grids are too
large. This
can be set
using:</div>
<div><br>
</div>
<div> <a
href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html"
target="_blank" moz-do-not-send="true">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html</a></div>
<div><br>
</div>
<div>There is
some
explanation of
this effect on
that page. Let
us know if
setting this
does not
correct the
situation.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
Let me know if
you need
further
information.<br>
<br>
Best,<br>
<br>
Myriam
Peyrounette<br>
<br>
<br>
-- <br>
Myriam
Peyrounette<br>
CNRS/IDRIS -
HLST<br>
--<br>
<br>
</blockquote>
</div>
<br
clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr"
class="gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener</div>
<div><br>
</div>
<div><a
href="http://www.cse.buffalo.edu/%7Eknepley/"
target="_blank" moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr"
class="gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener</div>
<div><br>
</div>
<div><a
href="http://www.cse.buffalo.edu/%7Eknepley/"
target="_blank" moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="gmail-m_7961152398334556293gmail-m_4941328961016005032moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="gmail-m_7961152398334556293moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</blockquote>
<br>
<pre class="gmail-m_7961152398334556293moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</body>
</html>