[petsc-users] Memory growth issue

[Sanjay Govindjee]

I am trying to track down a memory issue with my code; apologies in 
advance for the longish message.

I am solving a FEA problem with a number of load steps involving about 3000
right hand side and tangent assemblies and solves.  The program is 
mainly Fortran, with a C memory allocator.

When I run my code in strictly serial mode (no Petsc or MPI routines) 
the memory stays constant over the whole run.

When I run it in parallel mode with petsc solvers with num_processes=1, 
the memory (max resident set size) also stays constant:

PetscMalloc = 28,976, ProgramNativeMalloc = constant, Resident Size = 
24,854,528 (constant) [CG/JACOBI]

[PetscMalloc and Resident Size as reported by PetscMallocGetCurrentUsage 
and PetscMemoryGetCurrentUsage (and summed across processes as needed);
ProgramNativeMalloc reported by program memory allocator.]

When I run it in parallel mode with petsc solvers but num_processes=2, 
the resident memory grows steadily during the run:

PetscMalloc = 3,039,072 (constant), ProgramNativeMalloc = constant, 
Resident Size = (finish) 31,313,920 (start) 24,698,880 [CG/JACOBI]

When I run it in parallel mode with petsc solvers but num_processes=4, 
the resident memory grows steadily during the run:

PetscMalloc = 3,307,888 (constant), ProgramNativeMalloc = 1,427,584 
(constant), Resident Size = (finish) 70,787,072  (start) 45,801,472 
[CG/JACOBI]
PetscMalloc = 5,903,808 (constant), ProgramNativeMalloc = 1,427,584 
(constant), Resident Size = (finish) 112,410,624 (start) 52,076,544 
[GMRES/BJACOBI]
PetscMalloc = 3,188,944 (constant), ProgramNativeMalloc = 1,427,584 
(constant), Resident Size = (finish) 712,798,208 (start) 381,480,960 
[SUPERLU]
PetscMalloc = 6,539,408 (constant), ProgramNativeMalloc = 1,427,584 
(constant), Resident Size = (finish) 591,048,704 (start) 278,671,360 [MUMPS]

The memory growth feels alarming but maybe I do not understand the 
values in ru_maxrss from getrusage().

My box (MacBook Pro) has a broken Valgrind so I need to get to a system 
with a functional one; notwithstanding, the code has always been 
Valgrind clean.
There are no Fortran Pointers or Fortran Allocatable arrays in the part 
of the code being used.  The program's C memory allocator keeps track of
itself so I do not see that the problem is there.  The Petsc malloc is 
also steady.

Other random hints:

1) If I comment out the call to KSPSolve and to my MPI data-exchange 
routine (for passing solution values between processes after each solve,
use  MPI_Isend, MPI_Recv, MPI_BARRIER)  the memory growth essentially 
goes away.

2) If I comment out the call to my MPI data-exchange routine but leave 
the call to KSPSolve the problem remains but is substantially reduced
for CG/JACOBI, and is marginally reduced for the GMRES/BJACOBI, SUPERLU, 
and MUMPS runs.

3) If I comment out the call to KSPSolve but leave the call to my MPI 
data-exchange routine the problem remains.

Any suggestions/hints of where to look will be great.

-sanjay