[petsc-dev] Petsc memory usage w/ top

[Dave Nystrom]

Jed Brown writes:
 > On Fri, Dec 2, 2011 at 21:06, Dave Nystrom <Dave.Nystrom at tachyonlogic.com>wrote:
 > 
 > > It seems that whenever I am running my code with petsc and run top on linux
 > > it reports that my code is running something like 24-25 GB of memory, even
 > > if I am running a smallish problem.  Is this normal?  I just had a problem
 > > terminate because it ran out of memory and I had no idea that I was pushing
 > > any memory boundaries.  I actually don't think the problem should have run
 > > out of memory but this behavior with top seems to keep it from being a
 > > useful tool for getting a sense of how much memory my code is using.
 > 
 > 1. Check -log_summary to see which object types the large allocations are
 > coming from.

The output of -log_summary never seems to agree with what I see in top.  BTW,
I'm running Centos 5.x on my system.  It is probably about 5.5 or 5.6 based
on the last time I did a yum update.

 > 2. What are you running?

I'm running a variety of things.  My code currently has 6 linear solves that
are being called.  Different methods work better for different linear
solves.  And I am doing lots of experimenting with different krylov solvers
and preconditioners.  So for krylov solvers, I have been using mostly cg and
also trying minres, symmlq and gmres.  For preconditioners, I have been
trying jacobi, block jacobi, lu, cholesky, ilu, icc, sor, sacusp and
ainvcusp.  When I run my code using our native, home grown cg solver with a
cholesky preconditioner, top reports for my current problem size memory usage
of 567 MB.  My current problem size is using a 2d grid that is 200x602 - not
really all that big.  And yet when I am running petsc solvers, top is
reporting 24 to 26 GB whether or not I am using cuda.

So do you guys not see this sort of behavior with top?  I see this sort of
behavior consistently, i.e. exclusively including with a petsc example
problem.  And according to dmesg, my previous job running with petsc ran out
of physical memory and swap space.  I have 12 GB of physical memory and about
14 GB of swap space.  But this is the first time I have run out of memory on
the cpu.  I have scaled up a petsc example problem large enough to get a
failed memory allocation for the gpu however.