[petsc-users] Memory for matrix assembly

[Jed Brown]

On Mon, Nov 21, 2011 at 22:47, Andrej Mesaros <andrej.mesaros at bc.edu> wrote:

> Dear all,
>
> I need guidance in finding the memory needed for matrix assembly.
>
> The job that fails when I reserve 3.5GB memory per node gives me the error
> output below. The job was run on 96 nodes, each storing its own part of a
> matrix (around 60k rows each, ~100M non-zero complex entries).
>
> The error occurs during assembly (similar numbers for every node):
>
> [25]PETSC ERROR: Out of memory. This could be due to allocating
> [25]PETSC ERROR: too large an object or bleeding by not properly
> [25]PETSC ERROR: destroying unneeded objects.
> [25]PETSC ERROR: Memory allocated 4565256864 Memory used by process
> 3658739712
> [25]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
> [25]PETSC ERROR: Memory requested 980025524!
> [25]PETSC ERROR:
> ------------------------------**------------------------------**
> ------------
> [25]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17
> 13:37:48 CDT 2011
> [25]PETSC ERROR: See docs/changes/index.html for recent updates.
> [25]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [25]PETSC ERROR: See docs/index.html for manual pages.
> [25]PETSC ERROR:
> ------------------------------**------------------------------**
> ------------
> [25]PETSC ERROR: Unknown Name on a linux-gnu named compute-5-54.local by
> mesaros Wed Oct 12 22:27:13 2011
> [25]PETSC ERROR: Libraries linked from
> /home/mesaros/code/petsc-3.1-**p8/linux-gnu-mpi-debug-**complex/lib
> [25]PETSC ERROR: Configure run at Thu Jun 30 12:30:13 2011
> [25]PETSC ERROR: Configure options --with-scalar-type=complex
> --with-64-bit-indices=1 --download-f-blas-lapack=yes --download-mpich=1
> --with-mpi-exec=/usr/publ$
> [25]PETSC ERROR:
> ------------------------------**------------------------------**
> ------------
> [25]PETSC ERROR: PetscMallocAlign() line 49 in src/sys/memory/mal.c
> [25]PETSC ERROR: PetscTrMallocDefault() line 192 in src/sys/memory/mtr.c
> [25]PETSC ERROR: PetscPostIrecvInt() line 250 in src/sys/utils/mpimesg.c
>

Looks like you are trying to half a billion (--with-64-bit-indices) or a
billion entries. How are you computing the nonzeros? Is it possible that
many processes are computing entries that need to go to one process?


> [25]PETSC ERROR: MatStashScatterBegin_Private() line 498 in
> src/mat/utils/matstash.c
> [25]PETSC ERROR: MatAssemblyBegin_MPIAIJ() line 474 in
> src/mat/impls/aij/mpi/mpiaij.c
> [25]PETSC ERROR: MatAssemblyBegin() line 4564 in src/mat/interface/matrix.c
>
>
> Now, how much memory would I need per node for this assembly to work? Is
> it "Memory allocated" + "Memory requested", which is around 5.5GB? And did
> it fail when "Memory used by process" reached ~3.5GB, which was the limit
> for the job? Usually, breaking the limit on memory per node kills the job,
> and PETSc then doesn't give the above "Out of memory" output.
>
> Additionally, can I simply estimate the additional memory needed for SLEPc
> to find ~100 lowest eigenvalues?
>

Start with what is typically needed by PETSc (the matrix, the setup cost is
for your preconditioner, the vectors for the Krylov method) and add
100*n*sizeof(PetscScalar).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20111121/d6de761b/attachment.htm>