<div dir="ltr"><div>Could you also post the result from -log_view?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 26 Mar 2019 at 13:27, Myriam Peyrounette <<a href="mailto:myriam.peyrounette@idris.fr">myriam.peyrounette@idris.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>I checked with -ksp_view (attached) but no prefix is associated
with the matrix. Some are associated to the KSP and PC, but none
to the Mat.<br>
</p>
<br>
<div class="gmail-m_-4814170559513831079moz-cite-prefix">Le 03/26/19 à 11:55, Dave May a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, 26 Mar 2019 at
10:36, Myriam Peyrounette <<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Oh you were right, the three options are unsused
(-matptap_via scalable, -inner_offdiag_matmatmult_via
scalable and -inner_diag_matmatmult_via scalable).
Does this mean I am not using the associated PtAP
functions?</p>
</div>
</blockquote>
<div><br>
</div>
<div>No - not necessarily. All it means is the options were
not parsed. </div>
<div><br>
</div>
<div>If your matrices have an option prefix associated with
them (e.g. abc) , then you need to provide the option as</div>
<div> -abc_matptap_via scalable</div>
<div><br>
</div>
<div>If you are not sure if you matrices have a prefix, look
at the result of -ksp_view (see below for an example)</div>
<div><br>
</div>
<div>
<p class="gmail-m_-4814170559513831079gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-m_-4814170559513831079gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4814170559513831079gmail-Apple-converted-space"> </span>Mat
Object: 2 MPI processes</span></p>
<p class="gmail-m_-4814170559513831079gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-m_-4814170559513831079gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4814170559513831079gmail-Apple-converted-space"> </span>type:
mpiaij</span></p>
<p class="gmail-m_-4814170559513831079gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-m_-4814170559513831079gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4814170559513831079gmail-Apple-converted-space"> </span>rows=363,
cols=363, bs=3</span></p>
<p class="gmail-m_-4814170559513831079gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-m_-4814170559513831079gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4814170559513831079gmail-Apple-converted-space"> </span>total:
nonzeros=8649, allocated nonzeros=8649</span></p>
<p class="gmail-m_-4814170559513831079gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-m_-4814170559513831079gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4814170559513831079gmail-Apple-converted-space"> </span>total
number of mallocs used during MatSetValues calls =0</span></p>
<p class="gmail-m_-4814170559513831079gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-m_-4814170559513831079gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4814170559513831079gmail-Apple-converted-space"> </span>Mat
Object: (B_) 2 MPI processes</span></p>
<p class="gmail-m_-4814170559513831079gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-m_-4814170559513831079gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4814170559513831079gmail-Apple-converted-space"> </span>type:
mpiaij</span></p>
<p class="gmail-m_-4814170559513831079gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-m_-4814170559513831079gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4814170559513831079gmail-Apple-converted-space"> </span>rows=363,
cols=363, bs=3</span></p>
<p class="gmail-m_-4814170559513831079gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-m_-4814170559513831079gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4814170559513831079gmail-Apple-converted-space"> </span>total:
nonzeros=8649, allocated nonzeros=8649</span></p>
<p class="gmail-m_-4814170559513831079gmail-p1" style="margin:0px;font:10px Monaco;color:rgb(245,245,245);background-color:rgb(0,0,0)"><span class="gmail-m_-4814170559513831079gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-m_-4814170559513831079gmail-Apple-converted-space"> </span>total
number of mallocs used during MatSetValues calls =0</span></p>
</div>
<div><br>
</div>
<div>The first matrix has no options prefix, but the second
does and it's called "B_".</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div> <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Myriam<br>
</p>
<br>
<div class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615moz-cite-prefix">Le
03/26/19 à 11:10, Dave May a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr"><br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, 26 Mar
2019 at 09:52, Myriam Peyrounette via
petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>How can I be sure they are indeed used?
Can I print this information in some log
file? <br>
</p>
</div>
</blockquote>
<div>Yes. Re-run the job with the command line
option</div>
<div><br>
</div>
<div>-options_left true<br>
</div>
<div><br>
</div>
<div>This will report all options parsed, and
importantly, will also indicate if any options
were unused.</div>
<div> </div>
<div><br>
</div>
<div>Thanks</div>
<div>Dave</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p> </p>
<p>Thanks in advance<br>
</p>
<p>Myriam<br>
</p>
<br>
<div class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570moz-cite-prefix">Le
03/25/19 à 18:24, Matthew Knepley a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Mon, Mar 25, 2019 at
10:54 AM Myriam Peyrounette via
petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi,</p>
<p>thanks for the explanations. I
tried the last PETSc version
(commit
fbc5705bc518d02a4999f188aad4ccff5f754cbf),
which includes the patch you
talked about. But the memory
scaling shows no improvement
(see scaling attached), even
when using the "scalable"
options :(</p>
<p>I had a look at the PETSc
functions
MatPtAPNumeric_MPIAIJ_MPIAIJ and
MatPtAPSymbolic_MPIAIJ_MPIAIJ
(especially at the differences
before and after the first "bad"
commit), but I can't find what
induced this memory issue.</p>
</div>
</blockquote>
<div>Are you sure that the option was
used? It just looks suspicious to me
that they use exactly the same
amount of memory. It should be
different, even if it does not solve
the problem.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Myriam<br>
</p>
<p><br>
</p>
<p><br>
</p>
<br>
<div class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453moz-cite-prefix">Le
03/20/19 à 17:38, Fande Kong a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Hi Myriam,</div>
<div><br>
</div>
<div>There are three
algorithms in
PETSc to do PtAP
( const char
*algTypes[3] =
{"scalable","nonscalable","hypre"};), and can be specified using the
petsc
options: -matptap_via
xxxx.</div>
<div><br>
</div>
<div>(1)
-matptap_via
hypre: This call
the hypre package
to do the PtAP
trough an
all-at-once triple
product. In our
experiences, it is
the most memory
efficient, but
could be slow.</div>
<div><br>
</div>
<div>(2)
-matptap_via
scalable: This
involves a
row-wise algorithm
plus an outer
product. This
will use more
memory than hypre,
but way faster.
This used to have
a bug that could
take all your
memory, and I have
a fix at <a href="https://bitbucket.org/petsc/petsc/pull-requests/1452/mpiptap-enable-large-scale-simulations/diff" target="_blank">https://bitbucket.org/petsc/petsc/pull-requests/1452/mpiptap-enable-large-scale-simulations/diff</a>.
When using this
option, we may
want to have extra
options such as
-inner_offdiag_matmatmult_via
scalable
-inner_diag_matmatmult_via
scalable to
select inner
scalable
algorithms.</div>
<div><br>
</div>
<div>(3) -matptap_via
nonscalable:
Suppose to be even
faster, but use
more memory. It
does dense matrix
operations.</div>
<div><br>
</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Fande Kong</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On
Wed, Mar 20,
2019 at 10:06 AM
Myriam
Peyrounette via
petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>More
precisely:
something
happens when
upgrading the
functions
MatPtAPNumeric_MPIAIJ_MPIAIJ and/or MatPtAPSymbolic_MPIAIJ_MPIAIJ. <br>
</p>
<p>Unfortunately,
there are a
lot of
differences
between the
old and new
versions of
these
functions. I
keep
investigating
but if you
have any idea,
please let me
know.</p>
<p>Best,<br>
</p>
<p>Myriam<br>
</p>
<br>
<div class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293moz-cite-prefix">Le
03/20/19 à
13:48, Myriam
Peyrounette a
écrit :<br>
</div>
<blockquote type="cite">
<p>Hi all,</p>
<p>I used git
bisect to
determine when
the memory
need
increased. I
found that the
first "bad"
commit is
aa690a28a7284adb519c28cb44eae20a2c131c85.</p>
<p>Barry was
right, this
commit seems
to be about an
evolution of <span class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x gmail-m_7299408251829905453gmail-m_7961152398334556293x-first gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x-last">MatPtAPSymbolic_MPIAIJ_MPIAIJ.
You mentioned
the option
"-matptap_via
scalable" but
I can't find
any
information
about it. Can
you tell me
more?</span></span></p>
<p><span class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x gmail-m_7299408251829905453gmail-m_7961152398334556293x-first gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x-last">Thanks</span></span></p>
<p><span class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x gmail-m_7299408251829905453gmail-m_7961152398334556293x-first gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x-last">Myriam</span></span></p>
<p><span class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293blob-code-inner"><span class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293pl-en gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x gmail-m_7299408251829905453gmail-m_7961152398334556293x-first gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293x-last"></span></span></p>
<br>
<div class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293moz-cite-prefix">Le
03/11/19 à
14:40, Mark
Adams a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">Is
there a
difference in
memory usage
on your tiny
problem? I
assume no.
<div><br>
</div>
<div>I don't
see anything
that could
come from GAMG
other than the
RAP stuff that
you have
discussed
already.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Mar 11, 2019 at 9:32 AM Myriam Peyrounette
<<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>The code I
am using here
is the example
42 of PETSc (<a class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032moz-txt-link-freetext" href="https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html</a>).
Indeed it
solves the
Stokes
equation. I
thought it was
a good idea to
use an example
you might know
(and didn't
find any that
uses GAMG
functions). I
just changed
the PCMG setup
so that the
memory problem
appears. And
it appears
when adding
PCGAMG.</p>
<p>I don't
care about the
performance or
even the
result
rightness
here, but only
about the
difference in
memory use
between 3.6
and 3.10. Do
you think
finding a more
adapted script
would help?<br>
</p>
<p>I used the
threshold of
0.1 only once,
at the
beginning, to
test its
influence. I
used the
default
threshold (of
0, I guess)
for all the
other runs.</p>
<p>Myriam<br>
</p>
<br>
<div class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032moz-cite-prefix">Le
03/11/19 à
13:52, Mark
Adams a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">In
looking at
this larger
scale run ...
<div><br>
</div>
<div>* Your
eigen
estimates are
much lower
than your tiny
test problem.
But this is
Stokes
apparently and
it should not
work anyway.
Maybe you have
a small time
step that adds
a lot of mass
that brings
the eigen
estimates
down. And your
min eigenvalue
(not used) is
positive. I
would expect
negative for
Stokes ...</div>
<div><br>
</div>
<div>* You
seem to be
setting a
threshold
value of 0.1
-- that is
very high</div>
<div><br>
</div>
<div>* v3.6
says "using
nonzero
initial guess"
but this is
not in v3.10.
Maybe we just
stopped
printing that.</div>
<div><br>
</div>
<div>* There
were some
changes to
coasening
parameters in
going from
v3.6 but it
does not look
like your
problem was
effected. (The
coarsening
algo is
non-deterministic
by default and
you can see
small
difference on
different
runs)</div>
<div><br>
</div>
<div>* We may
have also
added a
"noisy" RHS
for eigen
estimates by
default from
v3.6.</div>
<div><br>
</div>
<div>* And for
non-symetric
problems you
can try
-pc_gamg_agg_nsmooths 0, but again GAMG is not built for Stokes anyway.</div>
<div><br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Mar 5, 2019 at 11:53 AM Myriam Peyrounette
<<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>I used
PCView to
display the
size of the
linear system
in each level
of the MG.
You'll find
the outputs
attached to
this mail (zip
file) for both
the default
threshold
value and a
value of 0.1,
and for both
3.6 and 3.10
PETSc
versions. <br>
</p>
<p>For
convenience, I
summarized the
information in
a graph, also
attached (png
file).</p>
<p>As you can
see, there are
slight
differences
between the
two versions
but none is
critical, in
my opinion. Do
you see
anything
suspicious in
the outputs?</p>
<p>+ I can't
find the
default
threshold
value. Do you
know where I
can find it?<br>
</p>
<p>Thanks for
the follow-up</p>
<p>Myriam<br>
</p>
<br>
<div class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135moz-cite-prefix">Le
03/05/19 à
14:06, Matthew
Knepley a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On
Tue, Mar 5,
2019 at 7:14
AM Myriam
Peyrounette
<<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Matt,</p>
<p>I plotted
the memory
scalings using
different
threshold
values. The
two scalings
are slightly
translated
(from -22 to
-88 mB) but
this gain is
neglectable.
The
3.6-scaling
keeps being
robust while
the
3.10-scaling
deteriorates.</p>
<p>Do you have
any other
suggestion?</p>
</div>
</blockquote>
<div>Mark,
what is the
option she can
give to output
all the GAMG
data?</div>
<div><br>
</div>
<div>Also, run
using
-ksp_view.
GAMG will
report all the
sizes of its
grids, so it
should be easy
to see</div>
<div>if the
coarse grid
sizes are
increasing,
and also what
the effect of
the threshold
value is.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div>
Matt <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Thanks<br>
</p>
Myriam <br>
<br>
<div class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-cite-prefix">Le
03/02/19 à
02:27, Matthew
Knepley a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">On
Fri, Mar 1,
2019 at 10:53
AM Myriam
Peyrounette
via
petsc-users
<<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
I used to run
my code with
PETSc 3.6.
Since I
upgraded the
PETSc version<br>
to 3.10, this
code has a bad
memory
scaling.<br>
<br>
To report this
issue, I took
the PETSc
script ex42.c
and slightly<br>
modified it so
that the KSP
and PC
configurations
are the same
as in my<br>
code. In
particular, I
use a
"personnalised"
multi-grid
method. The<br>
modifications
are indicated
by the keyword
"TopBridge" in
the attached<br>
scripts.<br>
<br>
To plot the
memory (weak)
scaling, I ran
four
calculations
for each<br>
script with
increasing
problem sizes
and
computations
cores:<br>
<br>
1. 100,000
elts on 4
cores<br>
2. 1 million
elts on 40
cores<br>
3. 10 millions
elts on 400
cores<br>
4. 100
millions elts
on 4,000 cores<br>
<br>
The resulting
graph is also
attached. The
scaling using
PETSc 3.10<br>
clearly
deteriorates
for large
cases, while
the one using
PETSc 3.6 is<br>
robust.<br>
<br>
After a few
tests, I found
that the
scaling is
mostly
sensitive to
the<br>
use of the AMG
method for the
coarse grid
(line 1780 in<br>
main_ex42_petsc36.cc). In particular, the performance strongly<br>
deteriorates
when
commenting
lines 1777 to
1790 (in
main_ex42_petsc36.cc).<br>
<br>
Do you have
any idea of
what changed
between
version 3.6
and version<br>
3.10 that may
imply such
degradation?<br>
</blockquote>
<div><br>
</div>
<div>I believe
the default
values for
PCGAMG changed
between
versions. It
sounds like
the coarsening
rate</div>
<div>is not
great enough,
so that these
grids are too
large. This
can be set
using:</div>
<div><br>
</div>
<div> <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html</a></div>
<div><br>
</div>
<div>There is
some
explanation of
this effect on
that page. Let
us know if
setting this
does not
correct the
situation.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Let me know if
you need
further
information.<br>
<br>
Best,<br>
<br>
Myriam
Peyrounette<br>
<br>
<br>
-- <br>
Myriam
Peyrounette<br>
CNRS/IDRIS -
HLST<br>
--<br>
<br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/%7Eknepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail-m_-3242500023102749998moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/%7Eknepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032gmail-m_4553173887686987135moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293gmail-m_4941328961016005032moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</blockquote>
<br>
<pre class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453gmail-m_7961152398334556293moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail-m_7299408251829905453moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters
take for granted before
they begin their
experiments is infinitely
more interesting than any
results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/%7Eknepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615gmail-m_-6197033154084460570moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_-4814170559513831079gmail-m_-8509232780790875615moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_-4814170559513831079moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote></div></div>