[petsc-users] Memory usage function: output for all ranks

[Andrey Ovsyannikov]

Dear PETSc team,

I am working on optimization of Chombo-Crunch CFD code for next-generation
supercomputer architectures at NERSC (Berkeley Lab) and we use PETSc AMG
solver. During memory analysis study I faced with a difficulty to get
memory usage data from PETSc for all MPI ranks. I am looking for memory
dump function to get a detailed information on memory usage (not only
resident size and virtual memory but allso allocation by Vec, Mat, etc).
There is PetscMallocDumpLog() function but it is a collective function and
it always provides a log for 0 rank. I am wondering if it is possible to
include in PETSc a modification of PetscMallocDumpLog() which dumps the
similar log but for all MPI ranks.

I am attaching an example of my own memory function which uses PETSc
non-collective functions and it provides a resident set size and virtual
memory for all ranks. Perhaps in a similar way it is possible to modify
PetscMallocDumpLog.

Thank you,

void petscMemoryLog(const char prefix[])
{
  FILE* fd;
  char fname[PETSC_MAX_PATH_LEN];
  PetscMPIInt rank;

  MPI_Comm_rank(Chombo_MPI::comm,&rank);

  PetscLogDouble allocated;
  PetscLogDouble resident;
  PetscMallocGetCurrentUsage(&allocated);
  PetscMemoryGetCurrentUsage(&resident);
  PetscSNPrintf(fname,sizeof(fname),"%s.%d",prefix,rank);
  PetscFOpen(PETSC_COMM_SELF,fname,"a",&fd);

  PetscFPrintf(PETSC_COMM_SELF,fd,"### PETSc memory footprint for rank %d
\n",rank);
  PetscFPrintf(PETSC_COMM_SELF,fd,"[%d] Memory allocated by PetscMalloc()
%.0f bytes\n",rank,allocated);
  PetscFPrintf(PETSC_COMM_SELF,fd,"[%d] RSS usage by entire process %.0f
KB\n",rank,resident);
  PetscFClose(PETSC_COMM_SELF,fd);
}

Best regards,
Andrey Ovsyannikov, Ph.D.
Postdoctoral Fellow
NERSC Division
Lawrence Berkeley National Laboratory
510-486-7880
aovsyannikov at lbl.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151130/0bf60ae2/attachment.html>