[petsc-dev] [petsc-users] Bad memory scaling with PETSc 3.10

[Fande Kong]

On Fri, Mar 22, 2019 at 11:19 AM Zhang, Hong <hzhang at mcs.anl.gov> wrote:

> Fande,
> The images are very interesting and helpful. How did you get these images?
>

We used gperftools https://gperftools.github.io/gperftools/heapprofile.html



>
> Petsc PtAP uses 753MB for PtAPSymbolic and only 116MB for PtAPNumeric,
> while hypre uses 215MB -- it seems hypre does not implement symbolic PtAP.
>
> When I implement PtAP, my focus was on numeric part because it was used
> repeatedly. Now we should look at symbolic part and optimize it. It should
> have room for improvement or new algorithmic approach.
> We also need understand the algorithm and implementation used hypre.
>

The reason PETSc PtAP uses a lot of memory is that the PETSc algorithm
needs to create intermediate data such as AP_loc, P_loc, P_oth, Rd, Ro,
C_oth, etc. " -mat_freeintermediatedatastructures 1" does help reduce the
memory usage  (see the attachment), but it is still twice more than that
used in HYPRE PtAP.

The way to reduce the memory is to have the all-at-once algorithm (Mark is
an expert on this). But I am not sure how efficient it could be
implemented.

Thanks,

Fande,


>
> Hong
>
> On Thu, Mar 21, 2019 at 6:37 PM Mark Adams via petsc-dev <
> petsc-dev at mcs.anl.gov> wrote:
>
>>
>>> Could you explain this more by adding some small examples?
>>>
>>>
>> Since you are considering implementing all-at-once (four nested loops,
>> right?) I'll give you my old code.
>>
>> This code is hardwired for two AMG and for a geometric-AMG, where the
>> blocks of the R (and hence P) matrices are scaled identities and I only
>> store the scale. So you ignore those branches. This code also does
>> equivalent real form complex, so more branches to ignore.
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20190322/7ab8e4d7/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: petsc_ptap_free_data.png
Type: image/png
Size: 622141 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20190322/7ab8e4d7/attachment-0001.png>