<div dir="ltr"><div dir="ltr"><div dir="ltr"></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 26 Mar 2019 at 09:52, Myriam Peyrounette via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>How can I be sure they are indeed used? Can I print this
information in some log file? <br></p></div></blockquote><div>Yes. Re-run the job with the command line option</div><div><br></div><div>-options_left true<br></div><div><br></div><div>This will report all options parsed, and importantly, will also indicate if any options were unused.</div><div> </div><div><br></div><div>Thanks</div><div>Dave</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>
</p>
<p>Thanks in advance<br>
</p>
<p>Myriam<br>
</p>
<br>
<div class="gmail-m_-6197033154084460570moz-cite-prefix">Le 03/25/19 à 18:24, Matthew Knepley a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Mon, Mar 25, 2019 at 10:54 AM Myriam
Peyrounette via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi,</p>
<p>thanks for the explanations. I tried the last PETSc
version (commit
fbc5705bc518d02a4999f188aad4ccff5f754cbf), which
includes the patch you talked about. But the memory
scaling shows no improvement (see scaling attached),
even when using the "scalable" options :(</p>
<p>I had a look at the PETSc functions
MatPtAPNumeric_MPIAIJ_MPIAIJ and
MatPtAPSymbolic_MPIAIJ_MPIAIJ (especially at the
differences before and after the first "bad" commit),
but I can't find what induced this memory issue.</p>
</div>
</blockquote>
<div>Are you sure that the option was used? It just looks
suspicious to me that they use exactly the same amount of
memory. It should be different, even if it does not solve
the problem.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Myriam<br>
</p>
<p><br>
</p>
<p><br>
</p>
<br>
<div class="gmail-m_-6197033154084460570gmail-m_7299408251829905453moz-cite-prefix">Le
03/20/19 à 17:38, Fande Kong a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Hi Myriam,</div>
<div><br>
</div>
<div>There are three algorithms in PETSc to
do PtAP ( const char *algTypes[3]
= {"scalable","nonscalable","hypre"};),
and can be specified using the petsc
options: -matptap_via xxxx.</div>
<div><br>
</div>
<div>(1) -matptap_via hypre: This call the
hypre package to do the PtAP trough an
all-at-once triple product. In our
experiences, it is the most memory
efficient, but could be slow.</div>
<div><br>
</div>
<div>(2) -matptap_via scalable: This
involves a row-wise algorithm plus an
outer product. This will use more memory
than hypre, but way faster. This used to
have a bug that could take all your
memory, and I have a fix at <a href="https://bitbucket.org/petsc/petsc/pull-requests/1452/mpiptap-enable-large-scale-simulations/diff" target="_blank">https://bitbucket.org/petsc/petsc/pull-requests/1452/mpiptap-enable-large-scale-simulations/diff</a>.
When using this option, we may want to
have extra options such as
-inner_offdiag_matmatmult_via scalable
-inner_diag_matmatmult_via scalable to
select inner scalable algorithms.</div>
<div><br>
</div>
<div>(3) -matptap_via nonscalable: Suppose
to be even faster, but use more memory. It
does dense matrix operations.</div>
<div><br>
</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Fande Kong</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed,
Mar 20, 2019 at 10:06 AM Myriam
Peyrounette via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>More precisely: something happens
when upgrading the functions
MatPtAPNumeric_MPIAIJ_MPIAIJ and/or
MatPtAPSymbolic_MPIAIJ_MPIAIJ. <br>
</p>
<p>Unfortunately, there are a lot of
differences between the old and new
versions of these functions. I keep
investigating but if you have any
idea, please let me know.</p>
<p>Best,<br>
</p>
<p>Myriam<br>
</p>
<br>
<div class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293moz-cite-prefix">Le
03/20/19 à 13:48, Myriam Peyrounette
a écrit :<br>
</div>
<blockquote type="cite">
<p>Hi all,</p>
<p>I used git bisect to determine
when the memory need increased. I
found that the first "bad" commit
is
aa690a28a7284adb519c28cb44eae20a2c131c85.</p>
<p>Barry was right, this commit
seems to be about an evolution of
<span class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x gmail-m_7299408251829905453gmail-m_7961152398334556293x-first gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x-last">MatPtAPSymbolic_MPIAIJ_MPIAIJ.
You mentioned the option
"-matptap_via scalable" but I
can't find any information
about it. Can you tell me
more?</span></span></p>
<p><span class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x gmail-m_7299408251829905453gmail-m_7961152398334556293x-first gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x-last">Thanks</span></span></p>
<p><span class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x gmail-m_7299408251829905453gmail-m_7961152398334556293x-first gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x-last">Myriam</span></span></p>
<p><span class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x gmail-m_7299408251829905453gmail-m_7961152398334556293x-first gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x-last"></span></span></p>
<br>
<div class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293moz-cite-prefix">Le
03/11/19 à 14:40, Mark Adams a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">Is there a
difference in memory usage on
your tiny problem? I assume no.
<div><br>
</div>
<div>I don't see anything that
could come from GAMG other
than the RAP stuff that you
have discussed already.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Mar
11, 2019 at 9:32 AM Myriam
Peyrounette <<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>The code I am using here
is the example 42 of PETSc
(<a class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032moz-txt-link-freetext" href="https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html</a>).
Indeed it solves the
Stokes equation. I thought
it was a good idea to use
an example you might know
(and didn't find any that
uses GAMG functions). I
just changed the PCMG
setup so that the memory
problem appears. And it
appears when adding
PCGAMG.</p>
<p>I don't care about the
performance or even the
result rightness here, but
only about the difference
in memory use between 3.6
and 3.10. Do you think
finding a more adapted
script would help?<br>
</p>
<p>I used the threshold of
0.1 only once, at the
beginning, to test its
influence. I used the
default threshold (of 0, I
guess) for all the other
runs.</p>
<p>Myriam<br>
</p>
<br>
<div class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032moz-cite-prefix">Le
03/11/19 à 13:52, Mark
Adams a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">In
looking at this larger
scale run ...
<div><br>
</div>
<div>* Your eigen
estimates are much
lower than your tiny
test problem. But
this is Stokes
apparently and it
should not work
anyway. Maybe you
have a small time
step that adds a lot
of mass that brings
the eigen estimates
down. And your min
eigenvalue (not
used) is positive. I
would expect
negative for Stokes
...</div>
<div><br>
</div>
<div>* You seem to be
setting a threshold
value of 0.1 -- that
is very high</div>
<div><br>
</div>
<div>* v3.6 says
"using nonzero
initial guess" but
this is not in
v3.10. Maybe we just
stopped printing
that.</div>
<div><br>
</div>
<div>* There were some
changes to coasening
parameters in going
from v3.6 but it
does not look like
your problem was
effected. (The
coarsening algo is
non-deterministic by
default and you can
see small difference
on different runs)</div>
<div><br>
</div>
<div>* We may have
also added a "noisy"
RHS for eigen
estimates by default
from v3.6.</div>
<div><br>
</div>
<div>* And for
non-symetric
problems you can try
-pc_gamg_agg_nsmooths 0, but again GAMG is not built for Stokes anyway.</div>
<div><br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On
Tue, Mar 5, 2019 at
11:53 AM Myriam
Peyrounette <<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>I used PCView to
display the size
of the linear
system in each
level of the MG.
You'll find the
outputs attached
to this mail (zip
file) for both the
default threshold
value and a value
of 0.1, and for
both 3.6 and 3.10
PETSc versions. <br>
</p>
<p>For convenience,
I summarized the
information in a
graph, also
attached (png
file).</p>
<p>As you can see,
there are slight
differences
between the two
versions but none
is critical, in my
opinion. Do you
see anything
suspicious in the
outputs?</p>
<p>+ I can't find
the default
threshold value.
Do you know where
I can find it?<br>
</p>
<p>Thanks for the
follow-up</p>
<p>Myriam<br>
</p>
<br>
<div class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135moz-cite-prefix">Le
03/05/19 à 14:06,
Matthew Knepley a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On
Tue, Mar 5,
2019 at 7:14
AM Myriam
Peyrounette
<<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Matt,</p>
<p>I plotted
the memory
scalings using
different
threshold
values. The
two scalings
are slightly
translated
(from -22 to
-88 mB) but
this gain is
neglectable.
The
3.6-scaling
keeps being
robust while
the
3.10-scaling
deteriorates.</p>
<p>Do you have
any other
suggestion?</p>
</div>
</blockquote>
<div>Mark,
what is the
option she can
give to output
all the GAMG
data?</div>
<div><br>
</div>
<div>Also, run
using
-ksp_view.
GAMG will
report all the
sizes of its
grids, so it
should be easy
to see</div>
<div>if the
coarse grid
sizes are
increasing,
and also what
the effect of
the threshold
value is.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div>
Matt <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Thanks<br>
</p>
Myriam <br>
<br>
<div class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-cite-prefix">Le
03/02/19 à
02:27, Matthew
Knepley a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">On
Fri, Mar 1,
2019 at 10:53
AM Myriam
Peyrounette
via
petsc-users
<<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
I used to run
my code with
PETSc 3.6.
Since I
upgraded the
PETSc version<br>
to 3.10, this
code has a bad
memory
scaling.<br>
<br>
To report this
issue, I took
the PETSc
script ex42.c
and slightly<br>
modified it so
that the KSP
and PC
configurations
are the same
as in my<br>
code. In
particular, I
use a
"personnalised"
multi-grid
method. The<br>
modifications
are indicated
by the keyword
"TopBridge" in
the attached<br>
scripts.<br>
<br>
To plot the
memory (weak)
scaling, I ran
four
calculations
for each<br>
script with
increasing
problem sizes
and
computations
cores:<br>
<br>
1. 100,000
elts on 4
cores<br>
2. 1 million
elts on 40
cores<br>
3. 10 millions
elts on 400
cores<br>
4. 100
millions elts
on 4,000 cores<br>
<br>
The resulting
graph is also
attached. The
scaling using
PETSc 3.10<br>
clearly
deteriorates
for large
cases, while
the one using
PETSc 3.6 is<br>
robust.<br>
<br>
After a few
tests, I found
that the
scaling is
mostly
sensitive to
the<br>
use of the AMG
method for the
coarse grid
(line 1780 in<br>
main_ex42_petsc36.cc). In particular, the performance strongly<br>
deteriorates
when
commenting
lines 1777 to
1790 (in
main_ex42_petsc36.cc).<br>
<br>
Do you have
any idea of
what changed
between
version 3.6
and version<br>
3.10 that may
imply such
degradation?<br>
</blockquote>
<div><br>
</div>
<div>I believe
the default
values for
PCGAMG changed
between
versions. It
sounds like
the coarsening
rate</div>
<div>is not
great enough,
so that these
grids are too
large. This
can be set
using:</div>
<div><br>
</div>
<div> <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html</a></div>
<div><br>
</div>
<div>There is
some
explanation of
this effect on
that page. Let
us know if
setting this
does not
correct the
situation.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Let me know if
you need
further
information.<br>
<br>
Best,<br>
<br>
Myriam
Peyrounette<br>
<br>
<br>
-- <br>
Myriam
Peyrounette<br>
CNRS/IDRIS -
HLST<br>
--<br>
<br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/%7Eknepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/%7Eknepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</blockquote>
<br>
<pre class="gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_-6197033154084460570gmail-m_7299408251829905453moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail-m_-6197033154084460570gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before
they begin their experiments is infinitely more
interesting than any results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/%7Eknepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_-6197033154084460570moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote></div></div></div>