[petsc-users] GAMG scaling

[Fande Kong]

Hong,

Thanks for your reply.

On Thu, Dec 20, 2018 at 4:43 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:

> Fande:
>
>> Hong,
>> Thanks for your improvements on PtAP that is critical for MG-type
>> algorithms.
>>
>> On Wed, May 3, 2017 at 10:17 AM Hong <hzhang at mcs.anl.gov> wrote:
>>
>>> Mark,
>>> Below is the copy of my email sent to you on Feb 27:
>>>
>>> I implemented scalable MatPtAP and did comparisons of three
>>> implementations using ex56.c on alcf cetus machine (this machine has
>>> small memory, 1GB/core):
>>> - nonscalable PtAP: use an array of length PN to do dense axpy
>>> - scalable PtAP:       do sparse axpy without use of PN array
>>>
>>
>> What PN means here?
>>
> Global number of columns of P.
>
>>
>> - hypre PtAP.
>>>
>>> The results are attached. Summary:
>>> - nonscalable PtAP is 2x faster than scalable, 8x faster than hypre PtAP
>>> - scalable PtAP is 4x faster than hypre PtAP
>>> - hypre uses less memory (see job.ne399.n63.np1000.sh)
>>>
>>
>> I was wondering how much more memory PETSc PtAP uses than hypre? I am
>> implementing an AMG algorithm based on PETSc right now, and it is working
>> well. But we find some a bottleneck with PtAP. For the same P and A, PETSc
>> PtAP fails to generate a coarse matrix due to out of memory, while hypre
>> still can generates the coarse matrix.
>>
>> I do not want to just use the HYPRE one because we had to duplicate
>> matrices if I used HYPRE PtAP.
>>
>> It would be nice if you guys already have done some compassions on these
>> implementations for the memory usage.
>>
> Do you encounter memory issue with  scalable PtAP?
>

Yes, the code will crash within "MatPtAP(Mat A,Mat P,MatReuse
scall,PetscReal fill,Mat *C)", and we receive signal 9  from the
supercomputer. Signal 9 usually means out of memory.

If we decrease the problem size, the code runs just fine. BTW, we are
trying to simulation a realistic case on the supercomputer with 10K cores
for a problem with 4 billion unknowns.


> Karl had a student in the summer who improved MatPtAP(). Do you use the
> latest version of petsc?
>

I am using PETSc-3.9.4. I could upgrade to PETSc-3.10.x if necessary.


> HYPRE may use less memory than PETSc because it does not save and reuse
> the matrices.
>

What "the matrices" are referred as to ?  Do you mean intermediate
 matrices (tmp = PtA or tmp = AP)? Could we have an option to not save
them? I am using SNES, but my preconditioning matrix is fixed and then the
preconditioner is fixed during the entire simulation. There is no reason
for me cache the intermediate matrices as they may take memory and I do not
need to use them any more.



>
>
> I do not understand why generating coarse matrix fails due to out of
> memory.
>

I mean we can not get  C= PtAP done because "PtAP" takes too much memory,
 compared with HYPRE PtAP.



> Do you use direct solver at coarse grid?
>

We have a 10-level AMG, and the coarsest matrix is small (at most 100x100),
 and we use LU.


Fande,


> Hong
>
>>
>>> Based on above observation, I set the default PtAP algorithm as
>>> 'nonscalable'.
>>> When PN > local estimated nonzero of C=PtAP, then switch default to
>>> 'scalable'.
>>> User can overwrite default.
>>>
>>> For the case of np=8000, ne=599 (see job.ne599.n500.np8000.sh), I get
>>> MatPtAP                   3.6224e+01 (nonscalable for small mats,
>>> scalable for larger ones)
>>> scalable MatPtAP     4.6129e+01
>>> hypre                        1.9389e+02
>>>
>>> This work in on petsc-master. Give it a try. If you encounter any
>>> problem, let me know.
>>>
>>> Hong
>>>
>>> On Wed, May 3, 2017 at 10:01 AM, Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>> (Hong), what is the current state of optimizing RAP for scaling?
>>>>
>>>> Nate, is driving 3D elasticity problems at scaling with GAMG and we are
>>>> working out performance problems. They are hitting problems at ~1.5B dof
>>>> problems on a basic Cray (XC30 I think).
>>>>
>>>> Thanks,
>>>> Mark
>>>>
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181220/7f519984/attachment.html>