<div class="gmail_quote">On Mon, Nov 21, 2011 at 22:47, Andrej Mesaros <span dir="ltr"><<a href="mailto:andrej.mesaros@bc.edu">andrej.mesaros@bc.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Dear all,<br>
<br>
I need guidance in finding the memory needed for matrix assembly.<br>
<br>
The job that fails when I reserve 3.5GB memory per node gives me the error output below. The job was run on 96 nodes, each storing its own part of a matrix (around 60k rows each, ~100M non-zero complex entries).<br>
<br>
The error occurs during assembly (similar numbers for every node):<br>
<br>
[25]PETSC ERROR: Out of memory. This could be due to allocating<br>
[25]PETSC ERROR: too large an object or bleeding by not properly<br>
[25]PETSC ERROR: destroying unneeded objects.<br>
[25]PETSC ERROR: Memory allocated <a href="tel:4565256864" value="+14565256864" target="_blank">4565256864</a> Memory used by process<br>
3658739712<br>
[25]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.<br>
[25]PETSC ERROR: Memory requested 980025524!<br>
[25]PETSC ERROR:<br>
------------------------------<u></u>------------------------------<u></u>------------<br>
[25]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17<br>
13:37:48 CDT 2011<br>
[25]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
[25]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
[25]PETSC ERROR: See docs/index.html for manual pages.<br>
[25]PETSC ERROR:<br>
------------------------------<u></u>------------------------------<u></u>------------<br>
[25]PETSC ERROR: Unknown Name on a linux-gnu named compute-5-54.local by<br>
mesaros Wed Oct 12 22:27:13 2011<br>
[25]PETSC ERROR: Libraries linked from<br>
/home/mesaros/code/petsc-3.1-<u></u>p8/linux-gnu-mpi-debug-<u></u>complex/lib<br>
[25]PETSC ERROR: Configure run at Thu Jun 30 12:30:13 2011<br>
[25]PETSC ERROR: Configure options --with-scalar-type=complex<br>
--with-64-bit-indices=1 --download-f-blas-lapack=yes --download-mpich=1<br>
--with-mpi-exec=/usr/publ$<br>
[25]PETSC ERROR:<br>
------------------------------<u></u>------------------------------<u></u>------------<br>
[25]PETSC ERROR: PetscMallocAlign() line 49 in src/sys/memory/mal.c<br>
[25]PETSC ERROR: PetscTrMallocDefault() line 192 in src/sys/memory/mtr.c<br>
[25]PETSC ERROR: PetscPostIrecvInt() line 250 in src/sys/utils/mpimesg.c<br></blockquote><div><br></div><div>Looks like you are trying to half a billion (--with-64-bit-indices) or a billion entries. How are you computing the nonzeros? Is it possible that many processes are computing entries that need to go to one process?</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
[25]PETSC ERROR: MatStashScatterBegin_Private() line 498 in<br>
src/mat/utils/matstash.c<br>
[25]PETSC ERROR: MatAssemblyBegin_MPIAIJ() line 474 in<br>
src/mat/impls/aij/mpi/mpiaij.c<br>
[25]PETSC ERROR: MatAssemblyBegin() line 4564 in src/mat/interface/matrix.c<br>
<br>
<br>
Now, how much memory would I need per node for this assembly to work? Is it "Memory allocated" + "Memory requested", which is around 5.5GB? And did it fail when "Memory used by process" reached ~3.5GB, which was the limit for the job? Usually, breaking the limit on memory per node kills the job, and PETSc then doesn't give the above "Out of memory" output.<br>
<br>
Additionally, can I simply estimate the additional memory needed for SLEPc to find ~100 lowest eigenvalues?<br></blockquote><div><br></div><div>Start with what is typically needed by PETSc (the matrix, the setup cost is for your preconditioner, the vectors for the Krylov method) and add 100*n*sizeof(PetscScalar).</div>
</div>