<div dir="ltr"><div dir="ltr">On Fri, Nov 22, 2024 at 12:57 PM David Scott <<a href="mailto:d.scott@epcc.ed.ac.uk">d.scott@epcc.ed.ac.uk</a>> wrote:</div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><u></u>
<div>
<font face="monospace">Matt,<br>
<br>
Thanks for the quick response.<br>
<br>
Yes 1) is trivially true.<br>
<br>
With regard to 2), from the SLURM output:<br>
[0] Maximum memory PetscMalloc()ed 29552 maximum size of entire
process 4312375296<br>
[1] Maximum memory PetscMalloc()ed 29552 maximum size of entire
process 4311990272<br>
Yes only 29KB was malloced but the total figure was 4GB per
process.<br>
<br>
Looking at<br>
mem0 = 16420864.000000000 <br>
mem0 = 16117760.000000000 <br>
mem1 = 4311490560.0000000 <br>
mem1 = 4311826432.0000000 <br>
mem2 = 4311490560.0000000 <br>
mem2 = 4311826432.0000000 <br>
mem0 is written after PetscInitialize.<br>
mem1 is written roughly half way through the options being read.<br>
mem2 is written on completion of the options being read.<br>
<br>
The code does very little other than read configuration options.
Why is so much memory used?<br></font></div></blockquote><div><br></div><div>This is not due to options processing, as that would fall under Petsc malloc allocations. I believe we are measuring this</div><div>using RSS which includes the binary, all shared libraries which are paged in, and stack/heap allocations. I think you are</div><div>seeing the shared libraries come in. You might be able to see all the libraries that come in using strace.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><font face="monospace">
I do not understand what is going on and I may have expressed
myself badly but I do have a problem as I certainly cannot use
anywhere near 128 processes on a node with 128GB of RAM before I
get an OOM error. (The code runs successfully on 32 processes but
not 64.)<br>
<br>
Regards,<br>
<br>
David<br>
</font><br>
<div>On 22/11/2024 16:53, Matthew Knepley
wrote:<br>
</div>
<blockquote type="cite">
<div style="background-color:rgb(255,242,230);border:2px dotted rgb(255,136,77)"><span style="font-size:12pt;font-family:sans-serif;color:black;font-weight:bold;padding:0.2em">This
email was sent to you by someone outside the University.</span>
<div style="font-size:10pt;font-family:sans-serif;font-style:normal;padding:0.2em">
You should only click on links or attachments if you are
certain that the email is genuine and the content is safe.</div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">On Fri, Nov 22, 2024 at 11:36 AM David Scott
<<a href="mailto:d.scott@epcc.ed.ac.uk" target="_blank">d.scott@epcc.ed.ac.uk</a>>
wrote:</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Hello,<br>
<br>
I am using the options mechanism of PETSc to configure my
CFD code. I<br>
have introduced options describing the size of the domain
etc. I have<br>
noticed that this consumes a lot of memory. I have found
that the amount<br>
of memory used scales linearly with the number of MPI
processes used.<br>
This restricts the number of MPI processes that I can use.<br>
</blockquote>
<div><br>
</div>
<div>There are two statements:</div>
<div><br>
</div>
<div>1) The memory scales linearly with P</div>
<div><br>
</div>
<div>2) This uses a lot of memory</div>
<div><br>
</div>
<div>Let's deal with 1) first. This seems to be trivially
true. If I want every process to have</div>
<div>access to a given option value, that option value must
be in the memory of every process.</div>
<div>The only alternative would be to communicate with some
process in order to get values.</div>
<div>Few codes seem to be willing to make this tradeoff, and
we do not offer it.</div>
<div><br>
</div>
<div>Now 2). Looking at the source, for each option we store
a PetscOptionItem, which I count</div>
<div>as having size 37 bytes (12 pointers/ints and a char).
However, there is data behind every</div>
<div>pointer, like the name, help text, available values
(sometimes), I could see it being as large</div>
<div>as 4K. Suppose it is. If I had 256 options, that would
be 1M. Is this a large amount of memory?</div>
<div><br>
</div>
<div>The way I read the SLURM output, 29K was malloced. Is
this a large amount of memory?</div>
<div><br>
</div>
<div>I am trying to get an idea of the scale.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Is there anything that I can do about this or do I need to
configure my<br>
code in a different way?<br>
<br>
I have attached some code extracted from my application
which<br>
demonstrates this along with the output from a running it
on 2 MPI<br>
processes.<br>
<br>
Best wishes,<br>
<br>
David Scott<br>
The University of Edinburgh is a charitable body,
registered in Scotland, with registration number SC005336.
Is e buidheann carthannais a th’ ann an Oilthigh Dhùn
Èideann, clàraichte an Alba, àireamh clàraidh SC005336.<br>
</blockquote>
</div>
<div><br clear="all">
</div>
<div><br>
</div>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted
before they begin their experiments is
infinitely more interesting than any results to
which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZLozvsSwhcAXnBdZ4m-ImNu-aXxMX8SpsHB7SM320hlG3hZEq3hw7UvNnb4c2hzs2_t_rAlLN5oIdCKCLif4$" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote></div><div><br clear="all"></div><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZLozvsSwhcAXnBdZ4m-ImNu-aXxMX8SpsHB7SM320hlG3hZEq3hw7UvNnb4c2hzs2_t_rAlLN5oIdCKCLif4$" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>