[petsc-users] Memory Used When Reading petscrc

Fabian.Jakub Fabian.Jakub at physik.uni-muenchen.de
Mon Nov 25 02:45:30 CST 2024 

test_configuration_options.F90:l.55
max_msg_length is quite large.... I guess the pow() is a typo.
Cheers,
Fabian


On 11/25/24 09:32, David Scott wrote:
> I'll have a look at heaptrack.
> 
> The code that I am looking at the moment does not create a mesh. All it 
> does is read a petscrc file.
> 
> Thanks,
> 
> David
> 
> On 25/11/2024 05:27, Jed Brown wrote:
>> This email was sent to you by someone outside the University.
>> You should only click on links or attachments if you are certain that 
>> the email is genuine and the content is safe.
>>
>> You're clearly doing almost all your allocation *not* using 
>> PetscMalloc (so not in a Vec or Mat). If you're managing your own mesh 
>> yourself, you might be allocating a global amount on each rank, 
>> instead of strictly using scalable data structures (i.e., always 
>> partitioned).
>>
>> My favorite tool for understanding memory use is heaptrack.
>>
>> https://urldefense.us/v3/__https://github.com/KDE/heaptrack__;!!G_uCfscf7eWS!bM8Vs5Ljq0ZJOl_Zl88PpU1JJWw39UMiu50wgyt0zhG4ax6DxOvabmaDYbKrrCATTeWrKDmDR5C-3bDziLRcXp30NMQ$
>> David Scott <d.scott at epcc.ed.ac.uk> writes:
>>
>>> OK.
>>>
>>> I had started to wonder if that was the case. I'll do some further
>>> investigation.
>>>
>>> Thanks,
>>>
>>> David
>>>
>>> On 22/11/2024 22:10, Matthew Knepley wrote:
>>>> This email was sent to you by someone outside the University.
>>>> You should only click on links or attachments if you are certain that
>>>> the email is genuine and the content is safe.
>>>> On Fri, Nov 22, 2024 at 12:57 PM David Scott <d.scott at epcc.ed.ac.uk>
>>>> wrote:
>>>>
>>>>      Matt,
>>>>
>>>>      Thanks for the quick response.
>>>>
>>>>      Yes 1) is trivially true.
>>>>
>>>>      With regard to 2), from the SLURM output:
>>>>      [0] Maximum memory PetscMalloc()ed 29552 maximum size of entire
>>>>      process 4312375296
>>>>      [1] Maximum memory PetscMalloc()ed 29552 maximum size of entire
>>>>      process 4311990272
>>>>      Yes only 29KB was malloced but the total figure was 4GB per 
>>>> process.
>>>>
>>>>      Looking at
>>>>       mem0 =    16420864.000000000
>>>>       mem0 =    16117760.000000000
>>>>       mem1 =    4311490560.0000000
>>>>       mem1 =    4311826432.0000000
>>>>       mem2 =    4311490560.0000000
>>>>       mem2 =    4311826432.0000000
>>>>      mem0 is written after PetscInitialize.
>>>>      mem1 is written roughly half way through the options being read.
>>>>      mem2 is written on completion of the options being read.
>>>>
>>>>      The code does very little other than read configuration options.
>>>>      Why is so much memory used?
>>>>
>>>>
>>>> This is not due to options processing, as that would fall under Petsc
>>>> malloc allocations. I believe we are measuring this
>>>> using RSS which includes the binary, all shared libraries which are
>>>> paged in, and stack/heap allocations. I think you are
>>>> seeing the shared libraries come in. You might be able to see all the
>>>> libraries that come in using strace.
>>>>
>>>>    Thanks,
>>>>
>>>>       Matt
>>>>
>>>>      I do not understand what is going on and I may have expressed
>>>>      myself badly but I do have a problem as I certainly cannot use
>>>>      anywhere near 128 processes on a node with 128GB of RAM before I
>>>>      get an OOM error. (The code runs successfully on 32 processes but
>>>>      not 64.)
>>>>
>>>>      Regards,
>>>>
>>>>      David
>>>>
>>>>      On 22/11/2024 16:53, Matthew Knepley wrote:
>>>>>      This email was sent to you by someone outside the University.
>>>>>      You should only click on links or attachments if you are certain
>>>>>      that the email is genuine and the content is safe.
>>>>>      On Fri, Nov 22, 2024 at 11:36 AM David Scott
>>>>>      <d.scott at epcc.ed.ac.uk> wrote:
>>>>>
>>>>>          Hello,
>>>>>
>>>>>          I am using the options mechanism of PETSc to configure my CFD
>>>>>          code. I
>>>>>          have introduced options describing the size of the domain
>>>>>          etc. I have
>>>>>          noticed that this consumes a lot of memory. I have found that
>>>>>          the amount
>>>>>          of memory used scales linearly with the number of MPI
>>>>>          processes used.
>>>>>          This restricts the number of MPI processes that I can use.
>>>>>
>>>>>
>>>>>      There are two statements:
>>>>>
>>>>>      1) The memory scales linearly with P
>>>>>
>>>>>      2) This uses a lot of memory
>>>>>
>>>>>      Let's deal with 1) first. This seems to be trivially true. If I
>>>>>      want every process to have
>>>>>      access to a given option value, that option value must be in the
>>>>>      memory of every process.
>>>>>      The only alternative would be to communicate with some process in
>>>>>      order to get values.
>>>>>      Few codes seem to be willing to make this tradeoff, and we do not
>>>>>      offer it.
>>>>>
>>>>>      Now 2). Looking at the source, for each option we store
>>>>>      a PetscOptionItem, which I count
>>>>>      as having size 37 bytes (12 pointers/ints and a char). However,
>>>>>      there is data behind every
>>>>>      pointer, like the name, help text, available values (sometimes),
>>>>>      I could see it being as large
>>>>>      as 4K. Suppose it is. If I had 256 options, that would be 1M. Is
>>>>>      this a large amount of memory?
>>>>>
>>>>>      The way I read the SLURM output, 29K was malloced. Is this a
>>>>>      large amount of memory?
>>>>>
>>>>>      I am trying to get an idea of the scale.
>>>>>
>>>>>        Thanks,
>>>>>
>>>>>            Matt
>>>>>
>>>>>          Is there anything that I can do about this or do I need to
>>>>>          configure my
>>>>>          code in a different way?
>>>>>
>>>>>          I have attached some code extracted from my application which
>>>>>          demonstrates this along with the output from a running it on
>>>>>          2 MPI
>>>>>          processes.
>>>>>
>>>>>          Best wishes,
>>>>>
>>>>>          David Scott
>>>>>          The University of Edinburgh is a charitable body, registered
>>>>>          in Scotland, with registration number SC005336. Is e
>>>>>          buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann,
>>>>>          clàraichte an Alba, àireamh clàraidh SC005336.
>>>>>
>>>>>
>>>>>
>>>>>      --
>>>>>      What most experimenters take for granted before they begin their
>>>>>      experiments is infinitely more interesting than any results to
>>>>>      which their experiments lead.
>>>>>      -- Norbert Wiener
>>>>>
>>>>>      
>>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cH8SjJvsuVEK1zv8noUjNUJC0VnHFqems68PjB2E94pqxc3q55YprX1q2JXFvPAzXJkh40J1-erXPWdIvc-xrLkRIgg$
>>>>>      
>>>>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cH8SjJvsuVEK1zv8noUjNUJC0VnHFqems68PjB2E94pqxc3q55YprX1q2JXFvPAzXJkh40J1-erXPWdIvc-xGybRwKU$ >
>>>>
>>>>
>>>> -- 
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which
>>>> their experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cH8SjJvsuVEK1zv8noUjNUJC0VnHFqems68PjB2E94pqxc3q55YprX1q2JXFvPAzXJkh40J1-erXPWdIvc-xrLkRIgg$
>>>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cH8SjJvsuVEK1zv8noUjNUJC0VnHFqems68PjB2E94pqxc3q55YprX1q2JXFvPAzXJkh40J1-erXPWdIvc-xGybRwKU$ >
>

More information about the petsc-users mailing list