[petsc-users] Which preconditioners are scalable?

[Sebastian Steiger]

Hello PETSc developers

I'm doing some scaling benchmarks and I found that the parallel asm
preconditioner, my favorite preconditioner, has a limit in the number of
cores it can handle.

I am doing a numerical experiment where I scale up the size of my matrix
by roughly the same factor as the number of CPUs employed. When I look
at which function used how much memory using PETSc's routine
PetscMallocDumpLog, I see the following:

Function name                        N=300         N=600     increase
======================================================================
MatGetSubMatrices_MPIAIJ_Local    75'912'016   134'516'928    1.77
MatIncreaseOverlap_MPIAIJ_Once    168'288'288  346'870'832    2.06
MatIncreaseOverlap_MPIAIJ_Receive  2'918'960     5'658'160    1.94

The matrix sizes are 6'899'904 and 14'224'896, respectively. Above
N~5000 CPUs I am running out of memory.

Here's my question now: Is the asm preconditioner limited from the
algorithm point of view, or is it the implementation? I thought that
'only' the local matrices, plus some constant overlap with neighbors,
are solved, so that memory consumption should stay constant when I scale
up with a constant number of rows per process.

Best
Sebastian