<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi Myriam,</div><div><br></div><div>There are three algorithms in PETSc to do PtAP ( const char *algTypes[3] = {"scalable","nonscalable","hypre"};), and can be specified using the petsc options: -matptap_via xxxx.</div><div><br></div><div>(1) -matptap_via hypre: This call the hypre package to do the PtAP trough an all-at-once triple product. In our experiences, it is the most memory efficient, but could be slow.</div><div><br></div><div>(2) -matptap_via scalable: This involves a row-wise algorithm plus an outer product. This will use more memory than hypre, but way faster. This used to have a bug that could take all your memory, and I have a fix at <a href="https://bitbucket.org/petsc/petsc/pull-requests/1452/mpiptap-enable-large-scale-simulations/diff">https://bitbucket.org/petsc/petsc/pull-requests/1452/mpiptap-enable-large-scale-simulations/diff</a>. When using this option, we may want to have extra options such as -inner_offdiag_matmatmult_via scalable -inner_diag_matmatmult_via scalable to select inner scalable algorithms.</div><div><br></div><div>(3) -matptap_via nonscalable: Suppose to be even faster, but use more memory. It does dense matrix operations.</div><div><br></div><div><br></div><div>Thanks,</div><div><br></div><div>Fande Kong</div><div><br></div><div><br></div><div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 20, 2019 at 10:06 AM Myriam Peyrounette via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>More precisely: something happens when upgrading the functions
MatPtAPNumeric_MPIAIJ_MPIAIJ and/or MatPtAPSymbolic_MPIAIJ_MPIAIJ.
<br>
</p>
<p>Unfortunately, there are a lot of differences between the old and
new versions of these functions. I keep investigating but if you
have any idea, please let me know.</p>
<p>Best,<br>
</p>
<p>Myriam<br>
</p>
<br>
<div class="gmail-m_7961152398334556293moz-cite-prefix">Le 03/20/19 à 13:48, Myriam Peyrounette
a écrit :<br>
</div>
<blockquote type="cite">
<p>Hi all,</p>
<p>I used git bisect to determine when the memory need increased.
I found that the first "bad" commit is
aa690a28a7284adb519c28cb44eae20a2c131c85.</p>
<p>Barry was right, this commit seems to be about an evolution of
<span class="gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_7961152398334556293pl-en gmail-m_7961152398334556293x gmail-m_7961152398334556293x-first gmail-m_7961152398334556293x-last">MatPtAPSymbolic_MPIAIJ_MPIAIJ. You mentioned the
option "-matptap_via scalable" but I can't find any
information about it. Can you tell me more?</span></span></p>
<p><span class="gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_7961152398334556293pl-en gmail-m_7961152398334556293x gmail-m_7961152398334556293x-first gmail-m_7961152398334556293x-last">Thanks</span></span></p>
<p><span class="gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_7961152398334556293pl-en gmail-m_7961152398334556293x gmail-m_7961152398334556293x-first gmail-m_7961152398334556293x-last">Myriam</span></span></p>
<p><span class="gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_7961152398334556293pl-en gmail-m_7961152398334556293x gmail-m_7961152398334556293x-first gmail-m_7961152398334556293x-last"></span></span></p>
<br>
<div class="gmail-m_7961152398334556293moz-cite-prefix">Le 03/11/19 à 14:40, Mark Adams a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">Is there a difference in memory usage on your
tiny problem? I assume no.
<div><br>
</div>
<div>I don't see anything that could come from GAMG other than
the RAP stuff that you have discussed already.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Mar 11, 2019 at 9:32
AM Myriam Peyrounette <<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>The code I am using here is the example 42 of PETSc (<a class="gmail-m_7961152398334556293gmail-m_4941328961016005032moz-txt-link-freetext" href="https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html</a>).
Indeed it solves the Stokes equation. I thought it was a
good idea to use an example you might know (and didn't
find any that uses GAMG functions). I just changed the
PCMG setup so that the memory problem appears. And it
appears when adding PCGAMG.</p>
<p>I don't care about the performance or even the result
rightness here, but only about the difference in memory
use between 3.6 and 3.10. Do you think finding a more
adapted script would help?<br>
</p>
<p>I used the threshold of 0.1 only once, at the
beginning, to test its influence. I used the default
threshold (of 0, I guess) for all the other runs.</p>
<p>Myriam<br>
</p>
<br>
<div class="gmail-m_7961152398334556293gmail-m_4941328961016005032moz-cite-prefix">Le
03/11/19 à 13:52, Mark Adams a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">In looking at this larger scale run ...
<div><br>
</div>
<div>* Your eigen estimates are much lower than your
tiny test problem. But this is Stokes apparently
and it should not work anyway. Maybe you have a
small time step that adds a lot of mass that
brings the eigen estimates down. And your min
eigenvalue (not used) is positive. I would expect
negative for Stokes ...</div>
<div><br>
</div>
<div>* You seem to be setting a threshold value of
0.1 -- that is very high</div>
<div><br>
</div>
<div>* v3.6 says "using nonzero initial guess" but
this is not in v3.10. Maybe we just stopped
printing that.</div>
<div><br>
</div>
<div>* There were some changes to coasening
parameters in going from v3.6 but it does not look
like your problem was effected. (The coarsening
algo is non-deterministic by default and you can
see small difference on different runs)</div>
<div><br>
</div>
<div>* We may have also added a "noisy" RHS for
eigen estimates by default from v3.6.</div>
<div><br>
</div>
<div>* And for non-symetric problems you can try
-pc_gamg_agg_nsmooths 0, but again GAMG is not
built for Stokes anyway.</div>
<div><br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Mar 5, 2019
at 11:53 AM Myriam Peyrounette <<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>I used PCView to display the size of the linear
system in each level of the MG. You'll find the
outputs attached to this mail (zip file) for
both the default threshold value and a value of
0.1, and for both 3.6 and 3.10 PETSc versions. <br>
</p>
<p>For convenience, I summarized the information
in a graph, also attached (png file).</p>
<p>As you can see, there are slight differences
between the two versions but none is critical,
in my opinion. Do you see anything suspicious in
the outputs?</p>
<p>+ I can't find the default threshold value. Do
you know where I can find it?<br>
</p>
<p>Thanks for the follow-up</p>
<p>Myriam<br>
</p>
<br>
<div class="gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135moz-cite-prefix">Le
03/05/19 à 14:06, Matthew Knepley a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Tue, Mar 5, 2019 at 7:14 AM
Myriam Peyrounette <<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Matt,</p>
<p>I plotted the memory scalings using
different threshold values. The two
scalings are slightly translated (from
-22 to -88 mB) but this gain is
neglectable. The 3.6-scaling keeps
being robust while the 3.10-scaling
deteriorates.</p>
<p>Do you have any other suggestion?</p>
</div>
</blockquote>
<div>Mark, what is the option she can give
to output all the GAMG data?</div>
<div><br>
</div>
<div>Also, run using -ksp_view. GAMG will
report all the sizes of its grids, so it
should be easy to see</div>
<div>if the coarse grid sizes are
increasing, and also what the effect of
the threshold value is.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Thanks<br>
</p>
Myriam <br>
<br>
<div class="gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-cite-prefix">Le
03/02/19 à 02:27, Matthew Knepley a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">On Fri, Mar 1, 2019
at 10:53 AM Myriam Peyrounette
via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
I used to run my code with
PETSc 3.6. Since I upgraded
the PETSc version<br>
to 3.10, this code has a bad
memory scaling.<br>
<br>
To report this issue, I took
the PETSc script ex42.c and
slightly<br>
modified it so that the KSP
and PC configurations are the
same as in my<br>
code. In particular, I use a
"personnalised" multi-grid
method. The<br>
modifications are indicated by
the keyword "TopBridge" in the
attached<br>
scripts.<br>
<br>
To plot the memory (weak)
scaling, I ran four
calculations for each<br>
script with increasing problem
sizes and computations cores:<br>
<br>
1. 100,000 elts on 4 cores<br>
2. 1 million elts on 40 cores<br>
3. 10 millions elts on 400
cores<br>
4. 100 millions elts on 4,000
cores<br>
<br>
The resulting graph is also
attached. The scaling using
PETSc 3.10<br>
clearly deteriorates for large
cases, while the one using
PETSc 3.6 is<br>
robust.<br>
<br>
After a few tests, I found
that the scaling is mostly
sensitive to the<br>
use of the AMG method for the
coarse grid (line 1780 in<br>
main_ex42_petsc36.cc). In
particular, the performance
strongly<br>
deteriorates when commenting
lines 1777 to 1790 (in
main_ex42_petsc36.cc).<br>
<br>
Do you have any idea of what
changed between version 3.6
and version<br>
3.10 that may imply such
degradation?<br>
</blockquote>
<div><br>
</div>
<div>I believe the default
values for PCGAMG changed
between versions. It sounds
like the coarsening rate</div>
<div>is not great enough, so
that these grids are too
large. This can be set using:</div>
<div><br>
</div>
<div> <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html</a></div>
<div><br>
</div>
<div>There is some explanation
of this effect on that page.
Let us know if setting this
does not correct the
situation.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Let me know if you need
further information.<br>
<br>
Best,<br>
<br>
Myriam Peyrounette<br>
<br>
<br>
-- <br>
Myriam Peyrounette<br>
CNRS/IDRIS - HLST<br>
--<br>
<br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most
experimenters take
for granted before
they begin their
experiments is
infinitely more
interesting than any
results to which
their experiments
lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/%7Eknepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take
for granted before they begin
their experiments is infinitely
more interesting than any
results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/%7Eknepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="gmail-m_7961152398334556293gmail-m_4941328961016005032moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="gmail-m_7961152398334556293moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</blockquote>
<br>
<pre class="gmail-m_7961152398334556293moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote></div></div></div></div></div></div></div>