[petsc-users] Bad memory scaling with PETSc 3.10

[Zhang, Hong]

Myriam:
This branch is merged to master.
Thanks for your work and patience. It helps us a lot. The graphs are very nice :-)

We plan to re-organise the APIs of mat-mat opts, make them easier for users.
Hong

Hi,

Do you have any idea when Barry's fix (https://bitbucket.org/petsc/petsc/pull-requests/1606/change-handling-of-matptap_mpiaij_mpimaij/diff) will be released? I can see it has been merged to the "next" branch. Does it mean it will be soon available on master?

+for your information, I plotted a summary of the scalings of interest (memory and time):
- using petsc-3.10.2 (ref "bad" scaling)
- using petsc-3.6.4 (ref "good" scaling)
- using commit d330a26 + Barry's fix and different algorithms (none, scalable, allatonce, allatonce_merged)

Best regards,

Myriam

Le 05/13/19 à 17:20, Fande Kong a écrit :
Hi Myriam,

Thanks for your report back.

On Mon, May 13, 2019 at 2:01 AM Myriam Peyrounette <myriam.peyrounette at idris.fr<mailto:myriam.peyrounette at idris.fr>> wrote:
Hi all,

I tried with 3.11.1 version and Barry's fix. The good scaling is back!
See the green curve in the plot attached. It is even better than PETSc
3.6! And it runs faster (10-15s instead of 200-300s with 3.6).

We are glad your issue was resolved here.


So you were right. It seems that not all the PtAPs used the scalable
version.

I was a bit confused about the options to set... I used the options:
-matptap_via scalable and -mat_freeintermediatedatastructures 1. Do you
think it would be even better with allatonce?

"scalable" and "allatonce" correspond to different algorithms respectively. ``allatonce" should be using less memory than "scalable". The "allatonce" algorithm  would be a good alternative if your application is memory sensitive and the problem size is large.
We are definitely curious about the memory usage of ``allatonce" in your test cases but don't feel obligated to do these tests since your concern were resolved now. In case you are also interested in how our new algorithms perform, I post petsc options here that are used to
choose these algorithms:

algorithm 1: ``allatonce"

-matptap_via allatonce
-mat_freeintermediatedatastructures 1

algorithm 2: ``allatonce_merged"

-matptap_via allatonce_merged
-mat_freeintermediatedatastructures 1


Again, thanks for your report that help us improve PETSc.

Fande,


It is unfortunate that this fix can't be merged with the master branch.
But the patch works well and I can consider the issue as solved now.

Thanks a lot for your time!

Myriam


Le 05/04/19 à 06:54, Smith, Barry F. a écrit :
>    Hmm, I had already fixed this, I think,
>
>    https://bitbucket.org/petsc/petsc/pull-requests/1606/change-handling-of-matptap_mpiaij_mpimaij/diff
>
>    but unfortunately our backlog of pull requests kept it out of master. We are (well Satish and Jed) working on a new CI infrastructure that will hopefully be more stable than the current CI that we are using.
>
>    Fande,
>       Sorry you had to spend time on this.
>
>
>    Barry
>
>
>
>> On May 3, 2019, at 11:20 PM, Fande Kong via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
>>
>> Hi Myriam,
>>
>> I run the example you attached earlier with "-mx 48 -my 48 -mz 48 -levels 3 -ksp_view  -matptap_via allatonce -log_view ".
>>
>> There are six PtAPs. Two of them are sill using the nonscalable version of the algorithm (this might explain why the memory still exponentially increases) even though we have asked PETSc to use the ``allatonce" algorithm. This is happening because MATMAIJ does not honor the petsc option, instead, it uses the default setting of MPIAIJ.  I have a fix at https://bitbucket.org/petsc/petsc/pull-requests/1623/choose-algorithms-in/diff. The PR should fix the issue.
>>
>> Thanks again for your report,
>>
>> Fande,
>>
>>

--
Myriam Peyrounette
CNRS/IDRIS - HLST
--




--
Myriam Peyrounette
CNRS/IDRIS - HLST
--

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190529/2524c84a/attachment.html>