<!DOCTYPE html><html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<font face="monospace">OK.<br>
<br>
I had started to wonder if that was the case. I'll do some further
investigation.<br>
<br>
Thanks,<br>
<br>
David<br>
</font><br>
<div class="moz-cite-prefix">On 22/11/2024 22:10, Matthew Knepley
wrote:<br>
</div>
<blockquote type="cite" cite="mid:CAMYG4Gm-cO3pySoBHujKTMwfCO9ECfT1zbQhzZyc27vxwG4X5A@mail.gmail.com">
<div style="background-color:#fff2e6; border:2px dotted #ff884d"><span style="font-size:12pt; font-family: sans-serif; color:black; font-weight:bold; padding:.2em">This
email was sent to you by someone outside the University.</span>
<div style="font-size:10pt; font-family: sans-serif; font-style:normal; padding:.2em">
You should only click on links or attachments if you are
certain that the email is genuine and the content is safe.</div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">On Fri, Nov 22, 2024 at 12:57 PM David Scott
<<a href="mailto:d.scott@epcc.ed.ac.uk" moz-do-not-send="true" class="moz-txt-link-freetext">d.scott@epcc.ed.ac.uk</a>>
wrote:</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div><font face="monospace">Matt,<br>
<br>
Thanks for the quick response.<br>
<br>
Yes 1) is trivially true.<br>
<br>
With regard to 2), from the SLURM output:<br>
[0] Maximum memory PetscMalloc()ed 29552 maximum size
of entire process 4312375296<br>
[1] Maximum memory PetscMalloc()ed 29552 maximum size
of entire process 4311990272<br>
Yes only 29KB was malloced but the total figure was
4GB per process.<br>
<br>
Looking at<br>
mem0 = 16420864.000000000 <br>
mem0 = 16117760.000000000 <br>
mem1 = 4311490560.0000000 <br>
mem1 = 4311826432.0000000 <br>
mem2 = 4311490560.0000000 <br>
mem2 = 4311826432.0000000 <br>
mem0 is written after PetscInitialize.<br>
mem1 is written roughly half way through the options
being read.<br>
mem2 is written on completion of the options being
read.<br>
<br>
The code does very little other than read
configuration options. Why is so much memory used?<br>
</font></div>
</blockquote>
<div><br>
</div>
<div>This is not due to options processing, as that would
fall under Petsc malloc allocations. I believe we are
measuring this</div>
<div>using RSS which includes the binary, all shared
libraries which are paged in, and stack/heap allocations.
I think you are</div>
<div>seeing the shared libraries come in. You might be able
to see all the libraries that come in using strace.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div><font face="monospace">I do not understand what is
going on and I may have expressed myself badly but I
do have a problem as I certainly cannot use anywhere
near 128 processes on a node with 128GB of RAM before
I get an OOM error. (The code runs successfully on 32
processes but not 64.)<br>
<br>
Regards,<br>
<br>
David<br>
</font><br>
<div>On 22/11/2024 16:53, Matthew Knepley wrote:<br>
</div>
<blockquote type="cite">
<div style="background-color:rgb(255,242,230);border:2px dotted rgb(255,136,77)">
<span style="font-size:12pt;font-family:sans-serif;color:black;font-weight:bold;padding:0.2em">This
email was sent to you by someone outside the
University.</span>
<div style="font-size:10pt;font-family:sans-serif;font-style:normal;padding:0.2em">
You should only click on links or attachments if
you are certain that the email is genuine and the
content is safe.</div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">On Fri, Nov 22, 2024 at 11:36 AM
David Scott <<a href="mailto:d.scott@epcc.ed.ac.uk" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">d.scott@epcc.ed.ac.uk</a>>
wrote:</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Hello,<br>
<br>
I am using the options mechanism of PETSc to
configure my CFD code. I<br>
have introduced options describing the size of
the domain etc. I have<br>
noticed that this consumes a lot of memory. I
have found that the amount<br>
of memory used scales linearly with the number
of MPI processes used.<br>
This restricts the number of MPI processes
that I can use.<br>
</blockquote>
<div><br>
</div>
<div>There are two statements:</div>
<div><br>
</div>
<div>1) The memory scales linearly with P</div>
<div><br>
</div>
<div>2) This uses a lot of memory</div>
<div><br>
</div>
<div>Let's deal with 1) first. This seems to be
trivially true. If I want every process to
have</div>
<div>access to a given option value, that option
value must be in the memory of every process.</div>
<div>The only alternative would be to
communicate with some process in order to get
values.</div>
<div>Few codes seem to be willing to make this
tradeoff, and we do not offer it.</div>
<div><br>
</div>
<div>Now 2). Looking at the source, for each
option we store a PetscOptionItem, which I
count</div>
<div>as having size 37 bytes (12 pointers/ints
and a char). However, there is data behind
every</div>
<div>pointer, like the name, help text,
available values (sometimes), I could see it
being as large</div>
<div>as 4K. Suppose it is. If I had 256 options,
that would be 1M. Is this a large amount of
memory?</div>
<div><br>
</div>
<div>The way I read the SLURM output, 29K was
malloced. Is this a large amount of memory?</div>
<div><br>
</div>
<div>I am trying to get an idea of the scale.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Is there anything that I can do about this or
do I need to configure my<br>
code in a different way?<br>
<br>
I have attached some code extracted from my
application which<br>
demonstrates this along with the output from a
running it on 2 MPI<br>
processes.<br>
<br>
Best wishes,<br>
<br>
David Scott<br>
The University of Edinburgh is a charitable
body, registered in Scotland, with
registration number SC005336. Is e buidheann
carthannais a th’ ann an Oilthigh Dhùn
Èideann, clàraichte an Alba, àireamh clàraidh
SC005336.<br>
</blockquote>
</div>
<div><br clear="all">
</div>
<div><br>
</div>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for
granted before they begin their
experiments is infinitely more
interesting than any results to
which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cH8SjJvsuVEK1zv8noUjNUJC0VnHFqems68PjB2E94pqxc3q55YprX1q2JXFvPAzXJkh40J1-erXPWdIvc-xGybRwKU$" originalsrc="http://www.cse.buffalo.edu/~knepley/" shash="NVupmNUxZcIsasSZGApd8kCinAYB6BI4ySSX6c3FbmA0ISFUQweOjvA7dB1QTDvxWXmCP2PllkF57VSGxQlbc5rStuZ7lTr8VcpiMf04tjzu3214no4zUQNJMcrreKJTG1k050mrtMJWJ1lVHZRwDyyjiZkxTaHmImfrpOOIyC8=" target="_blank" moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<div><br clear="all">
</div>
<div><br>
</div>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted
before they begin their experiments is
infinitely more interesting than any results to
which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cH8SjJvsuVEK1zv8noUjNUJC0VnHFqems68PjB2E94pqxc3q55YprX1q2JXFvPAzXJkh40J1-erXPWdIvc-xGybRwKU$" originalsrc="http://www.cse.buffalo.edu/~knepley/" shash="pdDgvIkg6vbfvA3gAe1ialANtMsTkA3mIfLgDwyyJ+v3QE4dpoE4XXllLDm8Cfa7VeEJQWqaVbhK9RW9Iu0pHxnR/H5pyri57sOjvETyz5f7ew4AUSCKGynOZyHt0rKl408rf7SXKWKqooVHTNtcjQIzlLP+ULWOoURhzUKcdRE=" target="_blank" moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>