[petsc-users] Bad memory scaling with PETSc 3.10

[Myriam Peyrounette]

Hi,

Do you have any idea when Barry's fix
(https://bitbucket.org/petsc/petsc/pull-requests/1606/change-handling-of-matptap_mpiaij_mpimaij/diff)
will be released? I can see it has been merged to the "next" branch.
Does it mean it will be soon available on master?

+for your information, I plotted a summary of the scalings of interest
(memory and time):
- using petsc-3.10.2 (ref "bad" scaling)
- using petsc-3.6.4 (ref "good" scaling)
- using commit d330a26 + Barry's fix and different algorithms (none,
scalable, allatonce, allatonce_merged)

Best regards,

Myriam


Le 05/13/19 à 17:20, Fande Kong a écrit :
> Hi Myriam,
>
> Thanks for your report back.
>
> On Mon, May 13, 2019 at 2:01 AM Myriam Peyrounette
> <myriam.peyrounette at idris.fr <mailto:myriam.peyrounette at idris.fr>> wrote:
>
>     Hi all,
>
>     I tried with 3.11.1 version and Barry's fix. The good scaling is back!
>     See the green curve in the plot attached. It is even better than PETSc
>     3.6! And it runs faster (10-15s instead of 200-300s with 3.6).
>
>
> We are glad your issue was resolved here. 
>  
>
>
>     So you were right. It seems that not all the PtAPs used the scalable
>     version.
>
>     I was a bit confused about the options to set... I used the options:
>     -matptap_via scalable and -mat_freeintermediatedatastructures 1.
>     Do you
>     think it would be even better with allatonce?
>
>
> "scalable" and "allatonce" correspond to different algorithms
> respectively. ``allatonce" should be using less memory than
> "scalable". The "allatonce" algorithm  would be a good alternative if
> your application is memory sensitive and the problem size is large. 
> We are definitely curious about the memory usage of ``allatonce" in
> your test cases but don't feel obligated to do these tests since your
> concern were resolved now. In case you are also interested in how our
> new algorithms perform, I post petsc options here that are used to 
> choose these algorithms:
>
> algorithm 1: ``allatonce" 
>
> -matptap_via allatonce
> -mat_freeintermediatedatastructures 1
>
> algorithm 2: ``allatonce_merged" 
>
> -matptap_via allatonce_merged
> -mat_freeintermediatedatastructures 1
>
>
> Again, thanks for your report that help us improve PETSc.
>
> Fande,
>  
>
>
>     It is unfortunate that this fix can't be merged with the master
>     branch.
>     But the patch works well and I can consider the issue as solved now.
>
>     Thanks a lot for your time!
>
>     Myriam
>
>
>     Le 05/04/19 à 06:54, Smith, Barry F. a écrit :
>     >    Hmm, I had already fixed this, I think,
>     >
>     >   
>     https://bitbucket.org/petsc/petsc/pull-requests/1606/change-handling-of-matptap_mpiaij_mpimaij/diff
>     >
>     >    but unfortunately our backlog of pull requests kept it out of
>     master. We are (well Satish and Jed) working on a new CI
>     infrastructure that will hopefully be more stable than the current
>     CI that we are using.
>     >
>     >    Fande,
>     >       Sorry you had to spend time on this.
>     >
>     >
>     >    Barry
>     >
>     >
>     >
>     >> On May 3, 2019, at 11:20 PM, Fande Kong via petsc-users
>     <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>     >>
>     >> Hi Myriam,
>     >>
>     >> I run the example you attached earlier with "-mx 48 -my 48 -mz
>     48 -levels 3 -ksp_view  -matptap_via allatonce -log_view ". 
>     >>
>     >> There are six PtAPs. Two of them are sill using the nonscalable
>     version of the algorithm (this might explain why the memory still
>     exponentially increases) even though we have asked PETSc to use
>     the ``allatonce" algorithm. This is happening because MATMAIJ does
>     not honor the petsc option, instead, it uses the default setting
>     of MPIAIJ.  I have a fix at
>     https://bitbucket.org/petsc/petsc/pull-requests/1623/choose-algorithms-in/diff.
>     The PR should fix the issue.
>     >>
>     >> Thanks again for your report,
>     >>
>     >> Fande,
>     >>
>     >> 
>
>     -- 
>     Myriam Peyrounette
>     CNRS/IDRIS - HLST
>     --
>
>

-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190529/61bb3c6c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex42_mem_scaling_ada_patched.png
Type: image/png
Size: 37597 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190529/61bb3c6c/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex42_time_scaling_ada_patched.png
Type: image/png
Size: 81468 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190529/61bb3c6c/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2975 bytes
Desc: Signature cryptographique S/MIME
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190529/61bb3c6c/attachment-0001.p7s>