[petsc-users] Bad memory scaling with PETSc 3.10

[Fande Kong]

Hi Myriam,

Thanks for your valuable feedback. The plots are really cool! Finally all
the algorithms are scalable.

Thanks,

Fande

On Wed, May 29, 2019 at 8:58 AM Myriam Peyrounette <
myriam.peyrounette at idris.fr> wrote:

> Oh sorry, I missed that. That's great!
>
> Thanks,
>
> Myriam
>
> Le 05/29/19 à 16:55, Zhang, Hong a écrit :
>
> Myriam:
> This branch is merged to master.
> Thanks for your work and patience. It helps us a lot. The graphs are very
> nice :-)
>
> We plan to re-organise the APIs of mat-mat opts, make them easier for
> users.
> Hong
>
>> Hi,
>>
>> Do you have any idea when Barry's fix (
>> https://bitbucket.org/petsc/petsc/pull-requests/1606/change-handling-of-matptap_mpiaij_mpimaij/diff)
>> will be released? I can see it has been merged to the "next" branch. Does
>> it mean it will be soon available on master?
>>
>> +for your information, I plotted a summary of the scalings of interest
>> (memory and time):
>> - using petsc-3.10.2 (ref "bad" scaling)
>> - using petsc-3.6.4 (ref "good" scaling)
>> - using commit d330a26 + Barry's fix and different algorithms (none,
>> scalable, allatonce, allatonce_merged)
>>
>> Best regards,
>>
>> Myriam
>>
>> Le 05/13/19 à 17:20, Fande Kong a écrit :
>>
>> Hi Myriam,
>>
>> Thanks for your report back.
>>
>> On Mon, May 13, 2019 at 2:01 AM Myriam Peyrounette <
>> myriam.peyrounette at idris.fr> wrote:
>>
>>> Hi all,
>>>
>>> I tried with 3.11.1 version and Barry's fix. The good scaling is back!
>>> See the green curve in the plot attached. It is even better than PETSc
>>> 3.6! And it runs faster (10-15s instead of 200-300s with 3.6).
>>>
>>
>> We are glad your issue was resolved here.
>>
>>
>>>
>>> So you were right. It seems that not all the PtAPs used the scalable
>>> version.
>>>
>>> I was a bit confused about the options to set... I used the options:
>>> -matptap_via scalable and -mat_freeintermediatedatastructures 1. Do you
>>> think it would be even better with allatonce?
>>>
>>
>> "scalable" and "allatonce" correspond to different algorithms
>> respectively. ``allatonce" should be using less memory than "scalable". The
>> "allatonce" algorithm  would be a good alternative if your application is
>> memory sensitive and the problem size is large.
>> We are definitely curious about the memory usage of ``allatonce" in your
>> test cases but don't feel obligated to do these tests since your concern
>> were resolved now. In case you are also interested in how our new
>> algorithms perform, I post petsc options here that are used to
>> choose these algorithms:
>>
>> algorithm 1: ``allatonce"
>>
>> -matptap_via allatonce
>> -mat_freeintermediatedatastructures 1
>>
>> algorithm 2: ``allatonce_merged"
>>
>> -matptap_via allatonce_merged
>> -mat_freeintermediatedatastructures 1
>>
>>
>> Again, thanks for your report that help us improve PETSc.
>>
>> Fande,
>>
>>
>>>
>>> It is unfortunate that this fix can't be merged with the master branch.
>>> But the patch works well and I can consider the issue as solved now.
>>>
>>> Thanks a lot for your time!
>>>
>>> Myriam
>>>
>>>
>>> Le 05/04/19 à 06:54, Smith, Barry F. a écrit :
>>> >    Hmm, I had already fixed this, I think,
>>> >
>>> >
>>> https://bitbucket.org/petsc/petsc/pull-requests/1606/change-handling-of-matptap_mpiaij_mpimaij/diff
>>> >
>>> >    but unfortunately our backlog of pull requests kept it out of
>>> master. We are (well Satish and Jed) working on a new CI infrastructure
>>> that will hopefully be more stable than the current CI that we are using.
>>> >
>>> >    Fande,
>>> >       Sorry you had to spend time on this.
>>> >
>>> >
>>> >    Barry
>>> >
>>> >
>>> >
>>> >> On May 3, 2019, at 11:20 PM, Fande Kong via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>> >>
>>> >> Hi Myriam,
>>> >>
>>> >> I run the example you attached earlier with "-mx 48 -my 48 -mz 48
>>> -levels 3 -ksp_view  -matptap_via allatonce -log_view ".
>>> >>
>>> >> There are six PtAPs. Two of them are sill using the nonscalable
>>> version of the algorithm (this might explain why the memory still
>>> exponentially increases) even though we have asked PETSc to use the
>>> ``allatonce" algorithm. This is happening because MATMAIJ does not honor
>>> the petsc option, instead, it uses the default setting of MPIAIJ.  I have a
>>> fix at
>>> https://bitbucket.org/petsc/petsc/pull-requests/1623/choose-algorithms-in/diff.
>>> The PR should fix the issue.
>>> >>
>>> >> Thanks again for your report,
>>> >>
>>> >> Fande,
>>> >>
>>> >>
>>>
>>> --
>>> Myriam Peyrounette
>>> CNRS/IDRIS - HLST
>>> --
>>>
>>>
>>>
>> --
>> Myriam Peyrounette
>> CNRS/IDRIS - HLST
>> --
>>
>>
> --
> Myriam Peyrounette
> CNRS/IDRIS - HLST
> --
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190529/981a57aa/attachment.html>