<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi all,</p>
<p>for your information, you'll find attached the comparison of the
weak memory scalings when using : <br>
</p>
<p>- PETSc 3.6.4 (reference)<br>
- PETSc 3.10.4 without specific options<br>
- PETSc 3.10.4 with the three scalability options you mentionned</p>
<p>Using the scalability options does improve the memory scaling.
However, the 3.6 version still has a better one...</p>
<p>Myriam<br>
</p>
<br>
<div class="moz-cite-prefix">Le 03/26/19 à 16:16, Myriam Peyrounette
a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:0e5a0f3b-ebb8-849c-7ce9-ec9c23108576@idris.fr">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<p>*SetFromOptions() was not called indeed... Thanks! The code
performance is better now with regard to memory usage!<br>
</p>
<p>I still have to plot the memory scaling on bigger cases to see
if it has the same good behaviour as when using the 3.6 version.</p>
<p>I'll let ou know as soon as I have plotted it.</p>
<p>Thanks again</p>
<p>Myriam<br>
</p>
<br>
<div class="moz-cite-prefix">Le 03/26/19 à 14:30, Matthew Knepley
a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:CAMYG4Gm20FaSXzkiiaEu+9JVnne-jN276zx1q7Rh5J-eGJ_mHg@mail.gmail.com">
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
<div dir="ltr">
<div dir="ltr">On Tue, Mar 26, 2019 at 9:27 AM Myriam
Peyrounette <<a href="mailto:myriam.peyrounette@idris.fr"
moz-do-not-send="true">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>I checked with -ksp_view (attached) but no prefix is
associated with the matrix. Some are associated to the
KSP and PC, but none to the Mat</p>
</div>
</blockquote>
<div>Another thing that could prevent options being used is
that *SetFromOptions() is not called for the object.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<div class="gmail-m_9164101223360987041moz-cite-prefix">Le
03/26/19 à 11:55, Dave May a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, 26 Mar
2019 at 10:36, Myriam Peyrounette <<a
href="mailto:myriam.peyrounette@idris.fr"
target="_blank" moz-do-not-send="true">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Oh you were right, the three options are
unsused (-matptap_via scalable,
-inner_offdiag_matmatmult_via scalable and
-inner_diag_matmatmult_via scalable). Does
this mean I am not using the associated
PtAP functions?</p>
</div>
</blockquote>
<div><br>
</div>
<div>No - not necessarily. All it means is the
options were not parsed. </div>
<div><br>
</div>
<div>If your matrices have an option prefix
associated with them (e.g. abc) , then you
need to provide the option as</div>
<div> -abc_matptap_via scalable</div>
<div><br>
</div>
<div>If you are not sure if you matrices have a
prefix, look at the result of -ksp_view (see
below for an example)</div>
<div><br>
</div>
<div>
<p class="gmail-m_9164101223360987041gmail-p1"
style="margin:0px;font:10px
Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span
class="gmail-m_9164101223360987041gmail-s1"
style="font-variant-ligatures:no-common-ligatures"><span
class="gmail-m_9164101223360987041gmail-Apple-converted-space"> </span>Mat
Object: 2 MPI processes</span></p>
<p class="gmail-m_9164101223360987041gmail-p1"
style="margin:0px;font:10px
Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span
class="gmail-m_9164101223360987041gmail-s1"
style="font-variant-ligatures:no-common-ligatures"><span
class="gmail-m_9164101223360987041gmail-Apple-converted-space"> </span>type:
mpiaij</span></p>
<p class="gmail-m_9164101223360987041gmail-p1"
style="margin:0px;font:10px
Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span
class="gmail-m_9164101223360987041gmail-s1"
style="font-variant-ligatures:no-common-ligatures"><span
class="gmail-m_9164101223360987041gmail-Apple-converted-space"> </span>rows=363,
cols=363, bs=3</span></p>
<p class="gmail-m_9164101223360987041gmail-p1"
style="margin:0px;font:10px
Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span
class="gmail-m_9164101223360987041gmail-s1"
style="font-variant-ligatures:no-common-ligatures"><span
class="gmail-m_9164101223360987041gmail-Apple-converted-space"> </span>total:
nonzeros=8649, allocated nonzeros=8649</span></p>
<p class="gmail-m_9164101223360987041gmail-p1"
style="margin:0px;font:10px
Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span
class="gmail-m_9164101223360987041gmail-s1"
style="font-variant-ligatures:no-common-ligatures"><span
class="gmail-m_9164101223360987041gmail-Apple-converted-space"> </span>total
number of mallocs used during MatSetValues
calls =0</span></p>
<p class="gmail-m_9164101223360987041gmail-p1"
style="margin:0px;font:10px
Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span
class="gmail-m_9164101223360987041gmail-s1"
style="font-variant-ligatures:no-common-ligatures"><span
class="gmail-m_9164101223360987041gmail-Apple-converted-space"> </span>Mat
Object: (B_) 2 MPI processes</span></p>
<p class="gmail-m_9164101223360987041gmail-p1"
style="margin:0px;font:10px
Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span
class="gmail-m_9164101223360987041gmail-s1"
style="font-variant-ligatures:no-common-ligatures"><span
class="gmail-m_9164101223360987041gmail-Apple-converted-space"> </span>type:
mpiaij</span></p>
<p class="gmail-m_9164101223360987041gmail-p1"
style="margin:0px;font:10px
Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span
class="gmail-m_9164101223360987041gmail-s1"
style="font-variant-ligatures:no-common-ligatures"><span
class="gmail-m_9164101223360987041gmail-Apple-converted-space"> </span>rows=363,
cols=363, bs=3</span></p>
<p class="gmail-m_9164101223360987041gmail-p1"
style="margin:0px;font:10px
Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span
class="gmail-m_9164101223360987041gmail-s1"
style="font-variant-ligatures:no-common-ligatures"><span
class="gmail-m_9164101223360987041gmail-Apple-converted-space"> </span>total:
nonzeros=8649, allocated nonzeros=8649</span></p>
<p class="gmail-m_9164101223360987041gmail-p1"
style="margin:0px;font:10px
Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span
class="gmail-m_9164101223360987041gmail-s1"
style="font-variant-ligatures:no-common-ligatures"><span
class="gmail-m_9164101223360987041gmail-Apple-converted-space"> </span>total
number of mallocs used during MatSetValues
calls =0</span></p>
</div>
<div><br>
</div>
<div>The first matrix has no options prefix, but
the second does and it's called "B_".</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div> <br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Myriam<br>
</p>
<br>
<div
class="gmail-m_9164101223360987041gmail-m_-8509232780790875615moz-cite-prefix">Le
03/26/19 à 11:10, Dave May a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr"><br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On
Tue, 26 Mar 2019 at 09:52, Myriam
Peyrounette via petsc-users <<a
href="mailto:petsc-users@mcs.anl.gov" target="_blank"
moz-do-not-send="true">petsc-users@mcs.anl.gov</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>How can I be sure they are
indeed used? Can I print this
information in some log file?
<br>
</p>
</div>
</blockquote>
<div>Yes. Re-run the job with the
command line option</div>
<div><br>
</div>
<div>-options_left true<br>
</div>
<div><br>
</div>
<div>This will report all options
parsed, and importantly, will also
indicate if any options were
unused.</div>
<div> </div>
<div><br>
</div>
<div>Thanks</div>
<div>Dave</div>
<div><br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> </p>
<p>Thanks in advance<br>
</p>
<p>Myriam<br>
</p>
<br>
<div
class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570moz-cite-prefix">Le
03/25/19 à 18:24, Matthew
Knepley a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Mon, Mar
25, 2019 at 10:54 AM
Myriam Peyrounette via
petsc-users <<a
href="mailto:petsc-users@mcs.anl.gov"
target="_blank"
moz-do-not-send="true">petsc-users@mcs.anl.gov</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote
class="gmail_quote"
style="margin:0px 0px
0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi,</p>
<p>thanks for the
explanations. I
tried the last PETSc
version (commit
fbc5705bc518d02a4999f188aad4ccff5f754cbf),
which includes the
patch you talked
about. But the
memory scaling shows
no improvement (see
scaling attached),
even when using the
"scalable" options
:(</p>
<p>I had a look at the
PETSc functions
MatPtAPNumeric_MPIAIJ_MPIAIJ
and
MatPtAPSymbolic_MPIAIJ_MPIAIJ
(especially at the
differences before
and after the first
"bad" commit), but I
can't find what
induced this memory
issue.</p>
</div>
</blockquote>
<div>Are you sure that the
option was used? It just
looks suspicious to me
that they use exactly
the same amount of
memory. It should be
different, even if it
does not solve the
problem.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt </div>
<blockquote
class="gmail_quote"
style="margin:0px 0px
0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Myriam<br>
</p>
<p><br>
</p>
<p><br>
</p>
<br>
<div
class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453moz-cite-prefix">Le
03/20/19 à 17:38,
Fande Kong a écrit :<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Hi Myriam,</div>
<div><br>
</div>
<div>There are
three
algorithms in
PETSc to do
PtAP ( const
char
*algTypes[3] =
{"scalable","nonscalable","hypre"};), and can be specified using the
petsc
options: -matptap_via
xxxx.</div>
<div><br>
</div>
<div>(1)
-matptap_via
hypre: This
call the hypre
package to do
the PtAP
trough an
all-at-once
triple
product. In
our
experiences,
it is the most
memory
efficient, but
could be slow.</div>
<div><br>
</div>
<div>(2)
-matptap_via
scalable: This
involves a
row-wise
algorithm plus
an outer
product. This
will use more
memory than
hypre, but way
faster. This
used to have a
bug that could
take all your
memory, and I
have a fix at <a
href="https://bitbucket.org/petsc/petsc/pull-requests/1452/mpiptap-enable-large-scale-simulations/diff"
target="_blank" moz-do-not-send="true">https://bitbucket.org/petsc/petsc/pull-requests/1452/mpiptap-enable-large-scale-simulations/diff</a>.
When using
this option,
we may want to
have extra
options such
as
-inner_offdiag_matmatmult_via
scalable
-inner_diag_matmatmult_via
scalable to
select inner
scalable
algorithms.</div>
<div><br>
</div>
<div>(3) -matptap_via
nonscalable:
Suppose to be
even faster,
but use more
memory. It
does dense
matrix
operations.</div>
<div><br>
</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Fande
Kong</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Wed, Mar 20, 2019 at 10:06 AM Myriam Peyrounette
via
petsc-users
<<a
href="mailto:petsc-users@mcs.anl.gov"
target="_blank" moz-do-not-send="true">petsc-users@mcs.anl.gov</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div
bgcolor="#FFFFFF">
<p>More
precisely:
something
happens when
upgrading the
functions
MatPtAPNumeric_MPIAIJ_MPIAIJ and/or MatPtAPSymbolic_MPIAIJ_MPIAIJ. <br>
</p>
<p>Unfortunately,
there are a
lot of
differences
between the
old and new
versions of
these
functions. I
keep
investigating
but if you
have any idea,
please let me
know.</p>
<p>Best,<br>
</p>
<p>Myriam<br>
</p>
<br>
<div
class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293moz-cite-prefix">Le
03/20/19 à
13:48, Myriam
Peyrounette a
écrit :<br>
</div>
<blockquote
type="cite">
<p>Hi all,</p>
<p>I used git
bisect to
determine when
the memory
need
increased. I
found that the
first "bad"
commit is
aa690a28a7284adb519c28cb44eae20a2c131c85.</p>
<p>Barry was
right, this
commit seems
to be about an
evolution of <span
class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span
class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en
gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x
gmail-m_7299408251829905453gmail-m_7961152398334556293x-first
gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x-last">MatPtAPSymbolic_MPIAIJ_MPIAIJ.
You mentioned
the option
"-matptap_via
scalable" but
I can't find
any
information
about it. Can
you tell me
more?</span></span></p>
<p><span
class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span
class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en
gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x
gmail-m_7299408251829905453gmail-m_7961152398334556293x-first
gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x-last">Thanks</span></span></p>
<p><span
class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span
class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en
gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x
gmail-m_7299408251829905453gmail-m_7961152398334556293x-first
gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x-last">Myriam</span></span></p>
<p><span
class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span
class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en
gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x
gmail-m_7299408251829905453gmail-m_7961152398334556293x-first
gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x-last"></span></span></p>
<br>
<div
class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293moz-cite-prefix">Le
03/11/19 à
14:40, Mark
Adams a
écrit :<br>
</div>
<blockquote
type="cite">
<div dir="ltr">Is
there a
difference in
memory usage
on your tiny
problem? I
assume no.
<div><br>
</div>
<div>I don't
see anything
that could
come from GAMG
other than the
RAP stuff that
you have
discussed
already.</div>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Mon, Mar 11, 2019 at 9:32 AM Myriam Peyrounette
<<a
href="mailto:myriam.peyrounette@idris.fr"
target="_blank" moz-do-not-send="true">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div
bgcolor="#FFFFFF">
<p>The code I
am using here
is the example
42 of PETSc (<a
class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032moz-txt-link-freetext"
href="https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html"
target="_blank" moz-do-not-send="true">https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html</a>).
Indeed it
solves the
Stokes
equation. I
thought it was
a good idea to
use an example
you might know
(and didn't
find any that
uses GAMG
functions). I
just changed
the PCMG setup
so that the
memory problem
appears. And
it appears
when adding
PCGAMG.</p>
<p>I don't
care about the
performance or
even the
result
rightness
here, but only
about the
difference in
memory use
between 3.6
and 3.10. Do
you think
finding a more
adapted script
would help?<br>
</p>
<p>I used the
threshold of
0.1 only once,
at the
beginning, to
test its
influence. I
used the
default
threshold (of
0, I guess)
for all the
other runs.</p>
<p>Myriam<br>
</p>
<br>
<div
class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032moz-cite-prefix">Le
03/11/19 à
13:52, Mark
Adams a
écrit :<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div dir="ltr">In
looking at
this larger
scale run ...
<div><br>
</div>
<div>* Your
eigen
estimates are
much lower
than your tiny
test problem.
But this is
Stokes
apparently and
it should not
work anyway.
Maybe you have
a small time
step that adds
a lot of mass
that brings
the eigen
estimates
down. And your
min eigenvalue
(not used) is
positive. I
would expect
negative for
Stokes ...</div>
<div><br>
</div>
<div>* You
seem to be
setting a
threshold
value of 0.1
-- that is
very high</div>
<div><br>
</div>
<div>* v3.6
says "using
nonzero
initial guess"
but this is
not in v3.10.
Maybe we just
stopped
printing that.</div>
<div><br>
</div>
<div>* There
were some
changes to
coasening
parameters in
going from
v3.6 but it
does not look
like your
problem was
effected. (The
coarsening
algo is
non-deterministic
by default and
you can see
small
difference on
different
runs)</div>
<div><br>
</div>
<div>* We may
have also
added a
"noisy" RHS
for eigen
estimates by
default from
v3.6.</div>
<div><br>
</div>
<div>* And for
non-symetric
problems you
can try
-pc_gamg_agg_nsmooths 0, but again GAMG is not built for Stokes anyway.</div>
<div><br>
</div>
</div>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Tue, Mar 5, 2019 at 11:53 AM Myriam Peyrounette
<<a
href="mailto:myriam.peyrounette@idris.fr"
target="_blank" moz-do-not-send="true">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div
bgcolor="#FFFFFF">
<p>I used
PCView to
display the
size of the
linear system
in each level
of the MG.
You'll find
the outputs
attached to
this mail (zip
file) for both
the default
threshold
value and a
value of 0.1,
and for both
3.6 and 3.10
PETSc
versions. <br>
</p>
<p>For
convenience, I
summarized the
information in
a graph, also
attached (png
file).</p>
<p>As you can
see, there are
slight
differences
between the
two versions
but none is
critical, in
my opinion. Do
you see
anything
suspicious in
the outputs?</p>
<p>+ I can't
find the
default
threshold
value. Do you
know where I
can find it?<br>
</p>
<p>Thanks for
the follow-up</p>
<p>Myriam<br>
</p>
<br>
<div
class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135moz-cite-prefix">Le
03/05/19 à
14:06, Matthew
Knepley a
écrit :<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div dir="ltr">On
Tue, Mar 5,
2019 at 7:14
AM Myriam
Peyrounette
<<a
href="mailto:myriam.peyrounette@idris.fr"
target="_blank" moz-do-not-send="true">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<div
class="gmail_quote">
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div
bgcolor="#FFFFFF">
<p>Hi Matt,</p>
<p>I plotted
the memory
scalings using
different
threshold
values. The
two scalings
are slightly
translated
(from -22 to
-88 mB) but
this gain is
neglectable.
The
3.6-scaling
keeps being
robust while
the
3.10-scaling
deteriorates.</p>
<p>Do you have
any other
suggestion?</p>
</div>
</blockquote>
<div>Mark,
what is the
option she can
give to output
all the GAMG
data?</div>
<div><br>
</div>
<div>Also, run
using
-ksp_view.
GAMG will
report all the
sizes of its
grids, so it
should be easy
to see</div>
<div>if the
coarse grid
sizes are
increasing,
and also what
the effect of
the threshold
value is.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div>
Matt <br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div
bgcolor="#FFFFFF">
<p>Thanks<br>
</p>
Myriam <br>
<br>
<div
class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-cite-prefix">Le
03/02/19 à
02:27, Matthew
Knepley a
écrit :<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">On
Fri, Mar 1,
2019 at 10:53
AM Myriam
Peyrounette
via
petsc-users
<<a
href="mailto:petsc-users@mcs.anl.gov"
target="_blank" moz-do-not-send="true">petsc-users@mcs.anl.gov</a>>
wrote:<br>
</div>
<div
class="gmail_quote">
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
I used to run
my code with
PETSc 3.6.
Since I
upgraded the
PETSc version<br>
to 3.10, this
code has a bad
memory
scaling.<br>
<br>
To report this
issue, I took
the PETSc
script ex42.c
and slightly<br>
modified it so
that the KSP
and PC
configurations
are the same
as in my<br>
code. In
particular, I
use a
"personnalised"
multi-grid
method. The<br>
modifications
are indicated
by the keyword
"TopBridge" in
the attached<br>
scripts.<br>
<br>
To plot the
memory (weak)
scaling, I ran
four
calculations
for each<br>
script with
increasing
problem sizes
and
computations
cores:<br>
<br>
1. 100,000
elts on 4
cores<br>
2. 1 million
elts on 40
cores<br>
3. 10 millions
elts on 400
cores<br>
4. 100
millions elts
on 4,000 cores<br>
<br>
The resulting
graph is also
attached. The
scaling using
PETSc 3.10<br>
clearly
deteriorates
for large
cases, while
the one using
PETSc 3.6 is<br>
robust.<br>
<br>
After a few
tests, I found
that the
scaling is
mostly
sensitive to
the<br>
use of the AMG
method for the
coarse grid
(line 1780 in<br>
main_ex42_petsc36.cc). In particular, the performance strongly<br>
deteriorates
when
commenting
lines 1777 to
1790 (in
main_ex42_petsc36.cc).<br>
<br>
Do you have
any idea of
what changed
between
version 3.6
and version<br>
3.10 that may
imply such
degradation?<br>
</blockquote>
<div><br>
</div>
<div>I believe
the default
values for
PCGAMG changed
between
versions. It
sounds like
the coarsening
rate</div>
<div>is not
great enough,
so that these
grids are too
large. This
can be set
using:</div>
<div><br>
</div>
<div> <a
href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html"
target="_blank" moz-do-not-send="true">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html</a></div>
<div><br>
</div>
<div>There is
some
explanation of
this effect on
that page. Let
us know if
setting this
does not
correct the
situation.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
Let me know if
you need
further
information.<br>
<br>
Best,<br>
<br>
Myriam
Peyrounette<br>
<br>
<br>
-- <br>
Myriam
Peyrounette<br>
CNRS/IDRIS -
HLST<br>
--<br>
<br>
</blockquote>
</div>
<br
clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr"
class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener</div>
<div><br>
</div>
<div><a
href="http://www.cse.buffalo.edu/%7Eknepley/"
target="_blank" moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
<br
clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr"
class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener</div>
<div><br>
</div>
<div><a
href="http://www.cse.buffalo.edu/%7Eknepley/"
target="_blank" moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</blockquote>
<br>
<pre class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr"
class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener</div>
<div><br>
</div>
<div><a
href="http://www.cse.buffalo.edu/%7Eknepley/"
target="_blank" moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_9164101223360987041gmail-m_-8509232780790875615gmail-m_-6197033154084460570moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_9164101223360987041gmail-m_-8509232780790875615moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_9164101223360987041moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted
before they begin their experiments is
infinitely more interesting than any results to
which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a
href="http://www.cse.buffalo.edu/%7Eknepley/"
target="_blank" moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</body>
</html>