<!DOCTYPE html><html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<font face="monospace">Matt,<br>
<br>
Thanks for the quick response.<br>
<br>
Yes 1) is trivially true.<br>
<br>
With regard to 2), from the SLURM output:<br>
[0] Maximum memory PetscMalloc()ed 29552 maximum size of entire
process 4312375296<br>
[1] Maximum memory PetscMalloc()ed 29552 maximum size of entire
process 4311990272<br>
Yes only 29KB was malloced but the total figure was 4GB per
process.<br>
<br>
Looking at<br>
mem0 = 16420864.000000000 <br>
mem0 = 16117760.000000000 <br>
mem1 = 4311490560.0000000 <br>
mem1 = 4311826432.0000000 <br>
mem2 = 4311490560.0000000 <br>
mem2 = 4311826432.0000000 <br>
mem0 is written after PetscInitialize.<br>
mem1 is written roughly half way through the options being read.<br>
mem2 is written on completion of the options being read.<br>
<br>
The code does very little other than read configuration options.
Why is so much memory used?<br>
<br>
I do not understand what is going on and I may have expressed
myself badly but I do have a problem as I certainly cannot use
anywhere near 128 processes on a node with 128GB of RAM before I
get an OOM error. (The code runs successfully on 32 processes but
not 64.)<br>
<br>
Regards,<br>
<br>
David<br>
</font><br>
<div class="moz-cite-prefix">On 22/11/2024 16:53, Matthew Knepley
wrote:<br>
</div>
<blockquote type="cite" cite="mid:CAMYG4GkR-NLZstxPz_NM+uD20v9KRKHZZTvc-EaVyp98i11+CQ@mail.gmail.com">
<div style="background-color:#fff2e6; border:2px dotted #ff884d"><span style="font-size:12pt; font-family: sans-serif; color:black; font-weight:bold; padding:.2em">This
email was sent to you by someone outside the University.</span>
<div style="font-size:10pt; font-family: sans-serif; font-style:normal; padding:.2em">
You should only click on links or attachments if you are
certain that the email is genuine and the content is safe.</div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">On Fri, Nov 22, 2024 at 11:36 AM David Scott
<<a href="mailto:d.scott@epcc.ed.ac.uk" moz-do-not-send="true" class="moz-txt-link-freetext">d.scott@epcc.ed.ac.uk</a>>
wrote:</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Hello,<br>
<br>
I am using the options mechanism of PETSc to configure my
CFD code. I<br>
have introduced options describing the size of the domain
etc. I have<br>
noticed that this consumes a lot of memory. I have found
that the amount<br>
of memory used scales linearly with the number of MPI
processes used.<br>
This restricts the number of MPI processes that I can use.<br>
</blockquote>
<div><br>
</div>
<div>There are two statements:</div>
<div><br>
</div>
<div>1) The memory scales linearly with P</div>
<div><br>
</div>
<div>2) This uses a lot of memory</div>
<div><br>
</div>
<div>Let's deal with 1) first. This seems to be trivially
true. If I want every process to have</div>
<div>access to a given option value, that option value must
be in the memory of every process.</div>
<div>The only alternative would be to communicate with some
process in order to get values.</div>
<div>Few codes seem to be willing to make this tradeoff, and
we do not offer it.</div>
<div><br>
</div>
<div>Now 2). Looking at the source, for each option we store
a PetscOptionItem, which I count</div>
<div>as having size 37 bytes (12 pointers/ints and a char).
However, there is data behind every</div>
<div>pointer, like the name, help text, available values
(sometimes), I could see it being as large</div>
<div>as 4K. Suppose it is. If I had 256 options, that would
be 1M. Is this a large amount of memory?</div>
<div><br>
</div>
<div>The way I read the SLURM output, 29K was malloced. Is
this a large amount of memory?</div>
<div><br>
</div>
<div>I am trying to get an idea of the scale.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Is there anything that I can do about this or do I need to
configure my<br>
code in a different way?<br>
<br>
I have attached some code extracted from my application
which<br>
demonstrates this along with the output from a running it
on 2 MPI<br>
processes.<br>
<br>
Best wishes,<br>
<br>
David Scott<br>
The University of Edinburgh is a charitable body,
registered in Scotland, with registration number SC005336.
Is e buidheann carthannais a th’ ann an Oilthigh Dhùn
Èideann, clàraichte an Alba, àireamh clàraidh SC005336.<br>
</blockquote>
</div>
<div><br clear="all">
</div>
<div><br>
</div>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted
before they begin their experiments is
infinitely more interesting than any results to
which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZSsmcyvmT1HYNbUdssH9wNf_bUXn64WJkcv6TgscRIX6mEcDPKI4LxvsUWu9JcgeYQjCchlmOm8y7thpEupDq7BKajE$" originalsrc="http://www.cse.buffalo.edu/~knepley/" shash="d1JA7MIazqoJZzsAASHNWHm5qU5/Laj0ceqo2Ku9BXWaBxH3VmGlCTwc2LAi0LtPXM2J98xDJViLHQcT1CD70DGuZ+Fgx+UuBzvaUw01WQvTI36FShcAK392YZfX4uqtCbZDB9pn98fSXSJFL8m/iS0YZmtOcSrmHMvH+CNuxRs=" target="_blank" moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>