[petsc-users] [petsc-maint] Memory usage function: output for all ranks

[Barry Smith]

  PETSc reporting of memory usage for objects is unfortunately not that great; for example distinguishing between temporary work space allocation vs memory that is kept for the life of the object is not always clear. Associating memory with particular objects requires the PETSc source code to mark each allocation appropriately and that is tedious and prone to error so we don't always do it right.

  I'm with Rich (and Matt), use valgrind on a handful of nodes, including node 0, 1,2, 4, 8, 16 that will provide the information you need. (the other processes will have very similar information that is not worth saving).

  Barry

Making PETSc track its own memory usage is actually a pretty hard problem we don't have the resources to do properly.

> On Nov 30, 2015, at 7:47 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> Andrey Ovsyannikov <aovsyannikov at lbl.gov> writes:
>> Thanks for your quick response. I like Massif tool and I have been using it
>> recently. However, I was not able to run Valgrind for large jobs. I am
>> interested in memory analysis of large scale runs with more than 1000 MPI
>> ranks. PetscMemoryGetCurrentUsage() works fine for this puprpose but it
>> does not provide details where I allocate memory. Maybe it would beneficial
>> for PETSc community to have some tool/function from PETSc itself.
> 
> Why do you want data from every rank?  PETSc usually tries to avoid
> diagnostic output that scales with number of processes because it sneaks
> up on people and causes crashes or huge IO costs that wastes their time.
> That which is okay at 1k ranks may be unacceptable at 1M ranks.  Would
> it be sufficient to compute some statistics, max/min, or the like?