<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.gmailsignatureprefix
{mso-style-name:gmail_signature_prefix;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Thank you, that makes testing so much easier. So far, I’ve been able to shrink the matrix (now only 64x64) and see that it still has growing memory usage over time. Unfortunately, I’ve no access to a linux machine right now, so running
through valgrind like Barry suggested has to wait.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Lars<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Matthew Knepley <knepley@gmail.com><br>
<b>Date: </b>Thursday, 5 September 2024 at 19:56<br>
<b>To: </b>"Corbijn van Willenswaard, Lars (UT)" <l.j.corbijnvanwillenswaard@utwente.nl><br>
<b>Cc: </b>"petsc-users@mcs.anl.gov" <petsc-users@mcs.anl.gov><br>
<b>Subject: </b>Re: [petsc-users] KSPSolve + MUMPS memory growth issues<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">On Thu, Sep 5, 2024 at 1:40 PM Corbijn van Willenswaard, Lars (UT) via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<o:p></o:p></p>
</div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">Dear PETSc,<br>
<br>
For the last months I’ve struggled with a solver that I wrote for a FEM eigenvalue problem running out of memory. I’ve traced it to KSPSolve + MUMPS being the issue, but I'm getting stuck on digging deeper.<br>
<br>
The reason I suspect the KSPSolve/MUMPS is that when commenting out the KSPSolve the memory stays constant while running the rest of the algorithm. Of course, the algorithm also converges to a different result in this setup. When changing the KSP statement
to <br>
for(int i = 0; i < 100000000; i++) KSPSolve(A_, vec1_, vec2_);<br>
the memory grows faster than when running the algorithm. Logging shows that the program never the terminating i=100M. Measuring the memory growth using ps (started debugging before I knew of PETSc's features) I see a growth in the RSS on a single compute node
of up to 300MB/min for this artificial case. Real cases grow more like 60MB/min/node, which causes a kill due to memory exhaustion after about 2-3 days.<br>
<br>
Locally (Mac) I've been able to reproduce this both with 6 MPI processes and with a single one. Instrumenting the code to show differences in PetscMemoryGetCurrentUsage (full code below), shows that the memory increases every step at the start, but also does
at later iterations (small excerpt from the output):<br>
rank step memory (increase since prev step)<br>
0 6544 current 39469056( 8192)<br>
0 7086 current 39477248( 8192)<br>
0 7735 current 39497728( 20480)<br>
0 9029 current 39501824( 4096)<br>
A similar output is visible in a run with 6 ranks, where there does not seem to be a pattern as to which of the ranks increases at which step. (Note I've checked PetscMallocGetCurrentUsage, but that is constant)<br>
<br>
Switching the solver to petsc's own solver on a single rank does not show a memory increase after the first solve. Changing the solve to overwrite the vector will result in a few increases after the first solve, but these do not seem to repeat. So, changes
like VecCopy(vec2, vec1_); KSPSolve(A_, vec1_, vec1_);.<br>
<br>
Does anyone have an idea on how to further dig into this problem?<o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I think the best way is to construct the simplest code that reproduces your problem. For example, we could save your matrix in a binary file<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> -ksp_view_mat binary:mat.bin<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">and then use a very simple code:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">#include <petsc.h><br>
<br>
int main(int argc, char **argv)<br>
{<br>
PetscViewer viewer;<br>
Mat A;<br>
Vec b, x;<br>
<br>
PetscCall(PetscInitialize(&argc, &argv, NULL, NULL));<br>
PetscCall(PetscViewerBinaryOpen(PETSC_COMM_WORLD, "mat.bin", PETSC_MODE_READ, &viewer));<br>
PetscCall(MatLoad(A, viewer));<br>
PetscCall(PetscViewerDestroy(&viewer));<br>
PetscCall(MatCreateVecs(A, &x, &b));<br>
PetscCall(VecSet(b, 1.));<br>
<br>
PetscCall(KSPCreate(PETSC_COMM_WORLD, &ksp));<br>
PetscCall(KSPSetOperators(ksp, A, A));<br>
PetscCall(KSPSetFromOptions(ksp));<br>
for (PetscInt i = 0; i < 100000; ++i) PetscCall(KSPSolve(ksp, b, x));<br>
PetscCall(KSPDestroy(&ksp));<br>
<br>
PetscCall(MatDestroy(&A));<br>
PetscCall(VecDestroy(&b));<br>
PetscCall(VecDestroy(&x));<br>
PetscCall(PetscFinalize());<br>
return(0);<br>
}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">and see if you get memory increase.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> Matt<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal" style="margin-bottom:12.0pt">Kind regards,<br>
Lars Corbijn<br>
<br>
<br>
Instrumentation:<br>
<br>
PetscLogDouble lastCurrent, current;<br>
int rank;<br>
MPI_Comm_rank(PETSC_COMM_WORLD, &rank);<br>
for(int i = 0; i < 100000000; ++i) {<br>
PetscMemoryGetCurrentUsage(&lastCurrent);<br>
KSPSolve(A_, vec1_, vec2_);<br>
PetscMemoryGetCurrentUsage(¤t);<br>
if(current != lastCurrent) {<br>
std::cout << std::setw(2) << rank << " " << std::setw(6) << i<br>
<< " current " << std::setw(8) << (int) current << std::right<br>
<< "(" << std::setw(6) << (int)(current - lastCurrent) << ")"<br>
<< std::endl;<br>
}<br>
lastCurrent = current;<br>
}<br>
<br>
<br>
Matrix details<br>
The matrix A in question is created from a complex valued matrix C_ (type mataij) using the following code (modulo renames). Theoretically it should be a Laplacian with phase-shift periodic boundary conditions<br>
MatHermitianTranspose(C_, MAT_INITIAL_MATRIX, &Y_);<br>
MatProductCreate(C_, Y_, NULL, & A_);<br>
MatProductSetType(A_, MATPRODUCT_AB);<br>
MatProductSetFromOptions(A_);<br>
MatProductSymbolic(A_);<br>
MatProductNumeric(A_);<br>
<br>
Petsc arguments: -log_view_memory -log_view :petsc.log -ksp_type preonly -pc_type lu -pc_factor_mat_solver_type mumps -bv_matmult vecs -memory_view<o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><span class="gmailsignatureprefix">-- </span><o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bKTdR_AbB6A1b2l-FZyzbS1zaRY0f2Hc1eIWUu_y4SfPUGOHmJ-ik6LjzXWVOBLBSWuHNjKRZnpPG5HOaak5fTmPe4-kJ7A_qFmk$" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>