[petsc-users] [EXTERNAL] Re: Question about -memory_view

[Jorti, Zakariae]

Hello Barry,


I appreciate the clarification.

I tried to check the memory usage with seff and I got the following results:


seff 7274633
Job ID: 7274633
Cluster: perlmutter
User/Group: zjorti/zjorti
State: COMPLETED (exit code 0)
Nodes: 64
Cores per node: 256
CPU Utilized: 3-13:19:59
CPU Efficiency: 3.88% of 91-14:11:12 core-walltime
Job Wall-clock time: 00:08:03
Memory Utilized: 958.68 GB (estimated maximum)
Memory Efficiency: 0.00% of 0.00 MB (0.00 MB/node)


Do you know how to interpret this?
The memory utilized seems higher than the one given by -memory_view (3.1056e+11)...

Thank you.
Best,


Zakariae

________________________________
From: Barry Smith <bsmith at petsc.dev>
Sent: Wednesday, April 12, 2023 3:32:11 PM
To: Jorti, Zakariae
Cc: petsc-users at mcs.anl.gov
Subject: [EXTERNAL] Re: [petsc-users] Question about -memory_view



On Apr 12, 2023, at 4:08 PM, Jorti, Zakariae via petsc-users <petsc-users at mcs.anl.gov> wrote:

Hello,

I am running some matrix computations on 64 nodes, using 640 MPI tasks.
And I wanted to check the memory usage with the -memory_view flag.
I get the following output:


Summary of Memory Usage in PETSc
Maximum (over computational time) process memory:        total 3.1056e+11 max 5.9918e+08 min 4.2213e+08
Current process memory:                                  total 1.9194e+11 max 3.9960e+08 min 2.2761e+08


What is the difference between maximum process memory and current process memory?

   Over computational time means the maximum it ever was (the high water mark) and current means what it is right now. For memory usage obtained from the OS (what we call  "process memory" in the output as opposed to PetscMalloc()ed memory)  often the current does not ever go below the maximum it ever was because the "extra now unneeded memory" is not returned to the OS.


What does total mean here? (in each node or total of all the nodes)

 Total is sum over all MPI ranks, max is maximum over all ranks, min is minimum over all ranks

Also, if the job fails because of memory shortage, will this -memory_view still output some information?

   Generally not if the job fails before the memory information is printed. Usually one runs with smaller memory usage increasing the problem size several times to see how the memory usage scales with the problem size (linearly, quadratically, etc) and this guides understanding the memory usage and if it can be improved. Just running with a large memory usage alone is not that useful in providing information.



Thank you.

Zakariae

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230412/1c44b30a/attachment-0001.html>