[petsc-users] Memory for matrix assembly

[Andrej Mesaros]

Dear all,

I need guidance in finding the memory needed for matrix assembly.

The job that fails when I reserve 3.5GB memory per node gives me the 
error output below. The job was run on 96 nodes, each storing its own 
part of a matrix (around 60k rows each, ~100M non-zero complex entries).

The error occurs during assembly (similar numbers for every node):

[25]PETSC ERROR: Out of memory. This could be due to allocating
[25]PETSC ERROR: too large an object or bleeding by not properly
[25]PETSC ERROR: destroying unneeded objects.
[25]PETSC ERROR: Memory allocated 4565256864 Memory used by process
3658739712
[25]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
[25]PETSC ERROR: Memory requested 980025524!
[25]PETSC ERROR:
------------------------------------------------------------------------
[25]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17
13:37:48 CDT 2011
[25]PETSC ERROR: See docs/changes/index.html for recent updates.
[25]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[25]PETSC ERROR: See docs/index.html for manual pages.
[25]PETSC ERROR:
------------------------------------------------------------------------
[25]PETSC ERROR: Unknown Name on a linux-gnu named compute-5-54.local by
mesaros Wed Oct 12 22:27:13 2011
[25]PETSC ERROR: Libraries linked from
/home/mesaros/code/petsc-3.1-p8/linux-gnu-mpi-debug-complex/lib
[25]PETSC ERROR: Configure run at Thu Jun 30 12:30:13 2011
[25]PETSC ERROR: Configure options --with-scalar-type=complex
--with-64-bit-indices=1 --download-f-blas-lapack=yes --download-mpich=1
--with-mpi-exec=/usr/publ$
[25]PETSC ERROR:
------------------------------------------------------------------------
[25]PETSC ERROR: PetscMallocAlign() line 49 in src/sys/memory/mal.c
[25]PETSC ERROR: PetscTrMallocDefault() line 192 in src/sys/memory/mtr.c
[25]PETSC ERROR: PetscPostIrecvInt() line 250 in src/sys/utils/mpimesg.c
[25]PETSC ERROR: MatStashScatterBegin_Private() line 498 in
src/mat/utils/matstash.c
[25]PETSC ERROR: MatAssemblyBegin_MPIAIJ() line 474 in
src/mat/impls/aij/mpi/mpiaij.c
[25]PETSC ERROR: MatAssemblyBegin() line 4564 in src/mat/interface/matrix.c


Now, how much memory would I need per node for this assembly to work? Is 
it "Memory allocated" + "Memory requested", which is around 5.5GB? And 
did it fail when "Memory used by process" reached ~3.5GB, which was the 
limit for the job? Usually, breaking the limit on memory per node kills 
the job, and PETSc then doesn't give the above "Out of memory" output.

Additionally, can I simply estimate the additional memory needed for 
SLEPc to find ~100 lowest eigenvalues?

Any insight is highly appreciated, and thanks to developers for the 
great software!
Andrej
PS: 64bit mode should be working fine (Linux 2.6, PETSc compiled with 
64bit, giving x86_64 executable on AMD cluster). PETSc+SLEPc definitely 
works fine when parallelized matrices use ~2.5GB memory per node.