<div dir="ltr">Hi Myriam,<div><br></div><div>Thanks for your valuable feedback. The plots are really cool! Finally all the algorithms are scalable.</div><div><br></div><div>Thanks,</div><div><br></div><div>Fande</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 29, 2019 at 8:58 AM Myriam Peyrounette <<a href="mailto:myriam.peyrounette@idris.fr">myriam.peyrounette@idris.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Oh sorry, I missed that. That's great!</p>
<p>Thanks,</p>
<p>Myriam<br>
</p>
<br>
<div class="gmail-m_-2298207637864984588moz-cite-prefix">Le 05/29/19 à 16:55, Zhang, Hong a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">Myriam:<br>
</div>
<div>This branch is merged to master.</div>
<div>Thanks for your work and patience. It helps us a lot. The
graphs are very nice :-)</div>
<div><br>
</div>
<div>We plan to re-organise the APIs of mat-mat opts, make them
easier for users.</div>
<div>Hong</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi, <br>
</p>
<p>Do you have any idea when Barry's fix (<a class="gmail-m_-2298207637864984588gmail-m_-8191395118479287483moz-txt-link-freetext" href="https://bitbucket.org/petsc/petsc/pull-requests/1606/change-handling-of-matptap_mpiaij_mpimaij/diff" target="_blank">https://bitbucket.org/petsc/petsc/pull-requests/1606/change-handling-of-matptap_mpiaij_mpimaij/diff</a>)
will be released? I can see it has been merged to the
"next" branch. Does it mean it will be soon available on
master?
<br>
</p>
<p>+for your information, I plotted a summary of the
scalings of interest (memory and time):<br>
- using petsc-3.10.2 (ref "bad" scaling)<br>
- using petsc-3.6.4 (ref "good" scaling)<br>
- using commit d330a26 + Barry's fix and different
algorithms (none, scalable, allatonce, allatonce_merged)</p>
<p>Best regards,</p>
<p>Myriam<br>
</p>
<br>
<div class="gmail-m_-2298207637864984588gmail-m_-8191395118479287483moz-cite-prefix">Le
05/13/19 à 17:20, Fande Kong a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Hi Myriam,</div>
<div><br>
</div>
<div>Thanks for your report back.</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On
Mon, May 13, 2019 at 2:01 AM
Myriam Peyrounette <<a href="mailto:myriam.peyrounette@idris.fr" target="_blank">myriam.peyrounette@idris.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
Hi all,<br>
<br>
I tried with 3.11.1 version and
Barry's fix. The good scaling is
back!<br>
See the green curve in the plot
attached. It is even better than
PETSc<br>
3.6! And it runs faster (10-15s
instead of 200-300s with 3.6).<br>
</blockquote>
<div><br>
</div>
<div>We are glad your issue was
resolved here. </div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<br>
So you were right. It seems that
not all the PtAPs used the
scalable<br>
version.<br>
<br>
I was a bit confused about the
options to set... I used the
options:<br>
-matptap_via scalable and
-mat_freeintermediatedatastructures
1. Do you<br>
think it would be even better with
allatonce?<br>
</blockquote>
<div><br>
</div>
<div>"scalable" and "allatonce"
correspond to different algorithms
respectively. ``allatonce" should
be using less memory than
"scalable". The "allatonce"
algorithm would be a good
alternative if your application is
memory sensitive and the problem
size is large. </div>
<div>We are definitely curious about
the memory usage of ``allatonce"
in your test cases but don't feel
obligated to do these tests since
your concern were resolved now. In
case you are also interested in
how our new algorithms perform, I
post petsc options here that are
used to </div>
<div>choose these algorithms:</div>
<div><br>
</div>
<div>algorithm 1: ``allatonce" </div>
<div><br>
</div>
<div>-matptap_via allatonce<br>
</div>
<div>-mat_freeintermediatedatastructures
1<br>
</div>
<div><br>
</div>
<div>algorithm 2:
``allatonce_merged" <br>
</div>
<div><br>
</div>
<div>
<div>-matptap_via allatonce_merged<br>
</div>
<div>-mat_freeintermediatedatastructures
1</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Again, thanks for your report
that help us improve PETSc.</div>
<div><br>
</div>
<div>Fande,</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<br>
It is unfortunate that this fix
can't be merged with the master
branch.<br>
But the patch works well and I can
consider the issue as solved now.<br>
<br>
Thanks a lot for your time!<br>
<br>
Myriam<br>
<br>
<br>
Le 05/04/19 à 06:54, Smith, Barry
F. a écrit :<br>
> Hmm, I had already fixed
this, I think, <br>
><br>
> <a href="https://bitbucket.org/petsc/petsc/pull-requests/1606/change-handling-of-matptap_mpiaij_mpimaij/diff" rel="noreferrer" target="_blank">
https://bitbucket.org/petsc/petsc/pull-requests/1606/change-handling-of-matptap_mpiaij_mpimaij/diff</a><br>
><br>
> but unfortunately our
backlog of pull requests kept it
out of master. We are (well Satish
and Jed) working on a new CI
infrastructure that will hopefully
be more stable than the current CI
that we are using.<br>
><br>
> Fande,<br>
> Sorry you had to spend
time on this. <br>
><br>
><br>
> Barry<br>
><br>
><br>
><br>
>> On May 3, 2019, at 11:20
PM, Fande Kong via petsc-users
<<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
wrote:<br>
>><br>
>> Hi Myriam,<br>
>><br>
>> I run the example you
attached earlier with "-mx 48 -my
48 -mz 48 -levels 3 -ksp_view
-matptap_via allatonce -log_view
".
<br>
>><br>
>> There are six PtAPs. Two
of them are sill using the
nonscalable version of the
algorithm (this might explain why
the memory still exponentially
increases) even though we have
asked PETSc to use the
``allatonce" algorithm. This is
happening because MATMAIJ does not
honor the petsc option, instead,
it uses the default setting of
MPIAIJ. I have a fix at
<a href="https://bitbucket.org/petsc/petsc/pull-requests/1623/choose-algorithms-in/diff" rel="noreferrer" target="_blank">
https://bitbucket.org/petsc/petsc/pull-requests/1623/choose-algorithms-in/diff</a>.
The PR should fix the issue.<br>
>><br>
>> Thanks again for your
report,<br>
>><br>
>> Fande,<br>
>><br>
>> <br>
<br>
-- <br>
Myriam Peyrounette<br>
CNRS/IDRIS - HLST<br>
--<br>
<br>
<br>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_-2298207637864984588gmail-m_-8191395118479287483moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_-2298207637864984588moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote></div>