[petsc-users] Memory Used When Reading petscrc

Fri Nov 22 19:16:22 CST 2024

OK.

I had started to wonder if that was the case. I'll do some further 
investigation.

Thanks,

David

On 22/11/2024 22:10, Matthew Knepley wrote:
> This email was sent to you by someone outside the University.
> You should only click on links or attachments if you are certain that 
> the email is genuine and the content is safe.
> On Fri, Nov 22, 2024 at 12:57 PM David Scott <d.scott at epcc.ed.ac.uk> 
> wrote:
>
>     Matt,
>
>     Thanks for the quick response.
>
>     Yes 1) is trivially true.
>
>     With regard to 2), from the SLURM output:
>     [0] Maximum memory PetscMalloc()ed 29552 maximum size of entire
>     process 4312375296
>     [1] Maximum memory PetscMalloc()ed 29552 maximum size of entire
>     process 4311990272
>     Yes only 29KB was malloced but the total figure was 4GB per process.
>
>     Looking at
>      mem0 =    16420864.000000000
>      mem0 =    16117760.000000000
>      mem1 =    4311490560.0000000
>      mem1 =    4311826432.0000000
>      mem2 =    4311490560.0000000
>      mem2 =    4311826432.0000000
>     mem0 is written after PetscInitialize.
>     mem1 is written roughly half way through the options being read.
>     mem2 is written on completion of the options being read.
>
>     The code does very little other than read configuration options.
>     Why is so much memory used?
>
>
> This is not due to options processing, as that would fall under Petsc 
> malloc allocations. I believe we are measuring this
> using RSS which includes the binary, all shared libraries which are 
> paged in, and stack/heap allocations. I think you are
> seeing the shared libraries come in. You might be able to see all the 
> libraries that come in using strace.
>
>   Thanks,
>
>      Matt
>
>     I do not understand what is going on and I may have expressed
>     myself badly but I do have a problem as I certainly cannot use
>     anywhere near 128 processes on a node with 128GB of RAM before I
>     get an OOM error. (The code runs successfully on 32 processes but
>     not 64.)
>
>     Regards,
>
>     David
>
>     On 22/11/2024 16:53, Matthew Knepley wrote:
>>     This email was sent to you by someone outside the University.
>>     You should only click on links or attachments if you are certain
>>     that the email is genuine and the content is safe.
>>     On Fri, Nov 22, 2024 at 11:36 AM David Scott
>>     <d.scott at epcc.ed.ac.uk> wrote:
>>
>>         Hello,
>>
>>         I am using the options mechanism of PETSc to configure my CFD
>>         code. I
>>         have introduced options describing the size of the domain
>>         etc. I have
>>         noticed that this consumes a lot of memory. I have found that
>>         the amount
>>         of memory used scales linearly with the number of MPI
>>         processes used.
>>         This restricts the number of MPI processes that I can use.
>>
>>
>>     There are two statements:
>>
>>     1) The memory scales linearly with P
>>
>>     2) This uses a lot of memory
>>
>>     Let's deal with 1) first. This seems to be trivially true. If I
>>     want every process to have
>>     access to a given option value, that option value must be in the
>>     memory of every process.
>>     The only alternative would be to communicate with some process in
>>     order to get values.
>>     Few codes seem to be willing to make this tradeoff, and we do not
>>     offer it.
>>
>>     Now 2). Looking at the source, for each option we store
>>     a PetscOptionItem, which I count
>>     as having size 37 bytes (12 pointers/ints and a char). However,
>>     there is data behind every
>>     pointer, like the name, help text, available values (sometimes),
>>     I could see it being as large
>>     as 4K. Suppose it is. If I had 256 options, that would be 1M. Is
>>     this a large amount of memory?
>>
>>     The way I read the SLURM output, 29K was malloced. Is this a
>>     large amount of memory?
>>
>>     I am trying to get an idea of the scale.
>>
>>       Thanks,
>>
>>           Matt
>>
>>         Is there anything that I can do about this or do I need to
>>         configure my
>>         code in a different way?
>>
>>         I have attached some code extracted from my application which
>>         demonstrates this along with the output from a running it on
>>         2 MPI
>>         processes.
>>
>>         Best wishes,
>>
>>         David Scott
>>         The University of Edinburgh is a charitable body, registered
>>         in Scotland, with registration number SC005336. Is e
>>         buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann,
>>         clàraichte an Alba, àireamh clàraidh SC005336.
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>>
>>     https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cH8SjJvsuVEK1zv8noUjNUJC0VnHFqems68PjB2E94pqxc3q55YprX1q2JXFvPAzXJkh40J1-erXPWdIvc-xrLkRIgg$ 
>>     <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cH8SjJvsuVEK1zv8noUjNUJC0VnHFqems68PjB2E94pqxc3q55YprX1q2JXFvPAzXJkh40J1-erXPWdIvc-xGybRwKU$ >
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
>
> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cH8SjJvsuVEK1zv8noUjNUJC0VnHFqems68PjB2E94pqxc3q55YprX1q2JXFvPAzXJkh40J1-erXPWdIvc-xrLkRIgg$  
> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cH8SjJvsuVEK1zv8noUjNUJC0VnHFqems68PjB2E94pqxc3q55YprX1q2JXFvPAzXJkh40J1-erXPWdIvc-xGybRwKU$ >
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20241123/f16f2058/attachment-0001.html>