<html><head><meta http-equiv="content-type" content="text/html; charset=us-ascii"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> No idea. <div><br></div><div> You can allocate a few vectors of known size, fill them with some value VecSet(x,2.0) and check the various memory reports to what they report.</div><div><br></div><div> Barry</div><div><br><div><br><blockquote type="cite"><div>On Apr 12, 2023, at 6:21 PM, Jorti, Zakariae <zjorti@lanl.gov> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div id="divtagdefaultwrapper" dir="ltr" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;"><div style="margin-top: 0px; margin-bottom: 0px;">Hello Barry,</div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div style="margin-top: 0px; margin-bottom: 0px;">I appreciate the clarification.</div><div style="margin-top: 0px; margin-bottom: 0px;">I tried to check the memory usage with seff and I got the following results:<span class="Apple-converted-space"> </span><br></div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><p style="margin-top: 0px; margin-bottom: 0px;"></p><div>seff 7274633<br>Job ID: 7274633<br>Cluster: perlmutter<br>User/Group: zjorti/zjorti<br>State: COMPLETED (exit code 0)<br>Nodes: 64<br>Cores per node: 256<br>CPU Utilized: 3-13:19:59<br>CPU Efficiency: 3.88% of 91-14:11:12 core-walltime<br>Job Wall-clock time: 00:08:03<br>Memory Utilized: 958.68 GB (estimated maximum)<br>Memory Efficiency: 0.00% of 0.00 MB (0.00 MB/node)</div><br><p style="margin-top: 0px; margin-bottom: 0px;"></p><div style="margin-top: 0px; margin-bottom: 0px;">Do you know how to interpret this?<br>The memory utilized seems higher than the one given by -memory_view (<font size="3" face="Calibri,Helvetica,sans-serif" style="font-family: Calibri, Helvetica, sans-serif, serif, EmojiFont;"><span dir="ltr" id="divtagdefaultwrapper" style="font-size: 12pt; font-weight: normal; text-transform: none;">3.1056e+11</span></font>)...</div><div style="margin-top: 0px; margin-bottom: 0px;">Thank you.<br>Best,</div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div style="margin-top: 0px; margin-bottom: 0px;">Zakariae<br></div></div><hr tabindex="-1" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 1026.046875px;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;"></span><div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><font face="Calibri, sans-serif" style="font-size: 11pt;"><b>From:</b><span class="Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>><br><b>Sent:</b><span class="Apple-converted-space"> </span>Wednesday, April 12, 2023 3:32:11 PM<br><b>To:</b><span class="Apple-converted-space"> </span>Jorti, Zakariae<br><b>Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a><br><b>Subject:</b><span class="Apple-converted-space"> </span>[EXTERNAL] Re: [petsc-users] Question about -memory_view</font><div> </div></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br><div><br><blockquote type="cite"><div>On Apr 12, 2023, at 4:08 PM, Jorti, Zakariae via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div><div id="divtagdefaultwrapper" dir="ltr" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;"><div style="margin-top: 0px; margin-bottom: 0px;">Hello,</div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div style="margin-top: 0px; margin-bottom: 0px;">I am running some matrix computations on 64 nodes, using 6<span>40 MPI tasks.<span class="Apple-converted-space"> </span><br></span></div><div style="margin-top: 0px; margin-bottom: 0px;"><span>And I wanted to check the memory usage with the<span class="Apple-converted-space"> </span><span>-memory_view</span><span class="Apple-converted-space"> </span>flag.</span></div><div style="margin-top: 0px; margin-bottom: 0px;">I get the following output:<span class="Apple-converted-space"> </span><br></div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><p style="margin-top: 0px; margin-bottom: 0px;"></p><div>Summary of Memory Usage in PETSc<br>Maximum (over computational time) process memory: total 3.1056e+11 max 5.9918e+08 min 4.2213e+08<br>Current process memory: total 1.9194e+11 max 3.9960e+08 min 2.2761e+08</div><br><p style="margin-top: 0px; margin-bottom: 0px;"></p><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div style="margin-top: 0px; margin-bottom: 0px;">What is the difference between maximum process memory and current process memory?<span class="Apple-converted-space"> </span><br></div></div></div></blockquote><div><br></div> Over computational time means the maximum it ever was (the high water mark) and current means what it is right now. For memory usage obtained from the OS (what we call "process memory" in the output as opposed to PetscMalloc()ed memory) often the current does not ever go below the maximum it ever was because the "extra now unneeded memory" is not returned to the OS. </div><div><br></div><div><br><blockquote type="cite"><div><div id="divtagdefaultwrapper" dir="ltr" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;"><div style="margin-top: 0px; margin-bottom: 0px;">What does total mean here? (in each node or total of all the nodes)<br></div></div></div></blockquote><div><br></div> Total is sum over all MPI ranks, max is maximum over all ranks, min is minimum over all ranks</div><div><br><blockquote type="cite"><div><div id="divtagdefaultwrapper" dir="ltr" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;"><div style="margin-top: 0px; margin-bottom: 0px;">Also, if the job fails because of memory shortage, will this -memory_view still output some information?<br></div></div></div></blockquote><div><br></div> Generally not if the job fails before the memory information is printed. Usually one runs with smaller memory usage increasing the problem size several times to see how the memory usage scales with the problem size (linearly, quadratically, etc) and this guides understanding the memory usage and if it can be improved. Just running with a large memory usage alone is not that useful in providing information.</div><div><br></div><div><br><blockquote type="cite"><div><div id="divtagdefaultwrapper" dir="ltr" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;"><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div style="margin-top: 0px; margin-bottom: 0px;">Thank you.<br><br>Zakariae</div></div></div></blockquote></div></div></div></blockquote></div><br></div></body></html>