[petsc-users] About recent changes in GAMG

[Ashish Patel]

Attaching the heaptrack files and some screenshots from one of the cores. Its most likely coming from MatStashSortCompress_Private, so if I understand correctly, it does some temporary allocations for mpi communication, but that memory gets released before it could be registered here https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/sys/objects/inherit.c*L111__;Iw!!G_uCfscf7eWS!fEVnmR9ofeaju0KXzcErICtGBrtw9qPbE9Dw1Y7vq1h4BzpT0-V2ZDldrxNysEDfCAOCKcYITz5WyYyQkUyJkkhCFYc$ 
[https://urldefense.us/v3/__https://gitlab.com/uploads/-/system/project/avatar/13882401/PETSc_RBG-logo.png__;!!G_uCfscf7eWS!fEVnmR9ofeaju0KXzcErICtGBrtw9qPbE9Dw1Y7vq1h4BzpT0-V2ZDldrxNysEDfCAOCKcYITz5WyYyQkUyJXeDf_RI$ ]<https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/sys/objects/inherit.c*L111__;Iw!!G_uCfscf7eWS!fEVnmR9ofeaju0KXzcErICtGBrtw9qPbE9Dw1Y7vq1h4BzpT0-V2ZDldrxNysEDfCAOCKcYITz5WyYyQkUyJkkhCFYc$ >
src/sys/objects/inherit.c · main · PETSc / petsc · GitLab<https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/sys/objects/inherit.c*L111__;Iw!!G_uCfscf7eWS!fEVnmR9ofeaju0KXzcErICtGBrtw9qPbE9Dw1Y7vq1h4BzpT0-V2ZDldrxNysEDfCAOCKcYITz5WyYyQkUyJkkhCFYc$ >
PETSc, pronounced PET-see (the S is silent), is a suite of data structures and routines for the scalable (parallel) solution of scientific applications modeled by partial differential equations.
gitlab.com
Ashish
________________________________
From: Jed Brown <jed at jedbrown.org>
Sent: Wednesday, April 24, 2024 10:25 AM
To: Ashish Patel <ashish.patel at ansys.com>; Mark Adams <mfadams at lbl.gov>; PETSc users list <petsc-users at mcs.anl.gov>
Cc: Scott McClennan <scott.mcclennan at ansys.com>; Jeremy Theler (External) <jeremy.theler-ext at ansys.com>
Subject: Re: [petsc-users] About recent changes in GAMG

[External Sender]

Ashish Patel <ashish.patel at ansys.com> writes:

> Hi Jed,
> VmRss is on a higher side and seems to match what PetscMallocGetMaximumUsage is reporting. HugetlbPages was 0 for me.
>
> Mark, running without the near nullspace also gives similar results. I have attached the malloc_view and gamg info for serial and 2 core runs. Some of the standout functions on rank 0 for parallel run seems to be
> 5.3 GB MatSeqAIJSetPreallocation_SeqAIJ
> 7.7 GB MatStashSortCompress_Private
> 10.1 GB PetscMatStashSpaceGet
> 7.7 GB  PetscSegBufferAlloc_Private
>
> malloc_view also says the following
> [0] Maximum memory PetscMalloc()ed 32387548912 maximum size of entire process 8270635008
> which fits the PetscMallocGetMaximumUsage > PetscMemoryGetMaximumUsage output.

This would occur if there was a large PetscMalloc'd block that did not get used (so only a portion of it is faulted and thus becomes resident).

Can you run a heap profiler like heaptrack?

https://urldefense.us/v3/__https://github.com/KDE/heaptrack__;!!G_uCfscf7eWS!fEVnmR9ofeaju0KXzcErICtGBrtw9qPbE9Dw1Y7vq1h4BzpT0-V2ZDldrxNysEDfCAOCKcYITz5WyYyQkUyJ9RdSDEA$ 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240425/12668a2e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: consumption.png
Type: image/png
Size: 171407 bytes
Desc: consumption.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240425/12668a2e/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: topdown.png
Type: image/png
Size: 549151 bytes
Desc: topdown.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240425/12668a2e/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: flamegraph.png
Type: image/png
Size: 283147 bytes
Desc: flamegraph.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240425/12668a2e/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex1_outputlog_3.21.0.1115_debug.log
Type: text/x-log
Size: 29505 bytes
Desc: ex1_outputlog_3.21.0.1115_debug.log
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240425/12668a2e/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: heaptrack.ex1.1128301.zst
Type: application/zstd
Size: 855213 bytes
Desc: heaptrack.ex1.1128301.zst
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240425/12668a2e/attachment-0004.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: heaptrack.ex1.1128300.zst
Type: application/zstd
Size: 980714 bytes
Desc: heaptrack.ex1.1128300.zst
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240425/12668a2e/attachment-0005.bin>