<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi everyone,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am using petsc-3.6.2 and I get an Out-of-memory error, from what I believe to be a leak from petsc. I am doing successive calls to MatAXPY during an optimization loop. I use MatAXPY to blend 2 matrices together; a linear combination of
the my operator and its first-order PC <o:p></o:p></p>
<p class="MsoNormal">B = w*A + (1-w)*B<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The matrix operator and PC does not have the same non-zeros patterns, hence the DIFFERENT_NONZERO_PATTERN option. In fact, the non-zeros of the PC are a subset of the non-zeros of A, but I found out about the SUBSET_NONZERO_PATTERN too
late…<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My matrices are pretty big (about 90 million unknown, with 33 non-zeros per row). So after the second call to MatAXPY, I already blow up the memory (on 256 cores). My code snippet would look like:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">PETSC_Mat_create(&( IMP_Mat_dR2dW) , 5, 0, 1 ,33,22);<o:p></o:p></p>
<p class="MsoNormal">PETSC_Mat_create(&( IMP_Mat_dR1dW_PC), 5, 0, 1, 19,5); // notice the different non-zeros patterns<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">while(1)
<o:p></o:p></p>
<p class="MsoNormal">{<o:p></o:p></p>
<p class="MsoNormal"> MatZeroEntries( IMP_Mat_dR2dW);<o:p></o:p></p>
<p class="MsoNormal"> MatZeroEntries( IMP_Mat_dR1dW_PC);<o:p></o:p></p>
<p class="MsoNormal"> // fill the dR2dW matrix …. <o:p></o:p></p>
<p class="MsoNormal"> // …<o:p></o:p></p>
<p class="MsoNormal"> ierr = MatSetValuesBlocked( IMP_Mat_dR2dW, nnz_W, dRdW_AD_blocked_cols, 1, &dRdW_AD_blocked_row, dRdW_AD_values, ADD_VALUES); // the dRdW_AD_values already contain the w factor<o:p></o:p></p>
<p class="MsoNormal"> // ….<o:p></o:p></p>
<p class="MsoNormal"> MatAssemblyBegin( IMP_Mat_dR2dW, MAT_FINAL_ASSEMBLY);<o:p></o:p></p>
<p class="MsoNormal"> MatAssemblyEnd( IMP_Mat_dR2dW, MAT_FINAL_ASSEMBLY);<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> // fill the dR1dW_PC matrix …. <o:p></o:p></p>
<p class="MsoNormal"> // …<o:p></o:p></p>
<p class="MsoNormal"> ierr = MatSetValuesBlocked( IMP_Mat_dR1dW_PC, nnz_W, dRdW_AD_blocked_cols, 1, &dRdW_AD_blocked_row, dRdW_AD_values, ADD_VALUES);<o:p></o:p></p>
<p class="MsoNormal"> // …<o:p></o:p></p>
<p class="MsoNormal"> MatAssemblyBegin( IMP_Mat_dR1dW_PC, MAT_FINAL_ASSEMBLY);<o:p></o:p></p>
<p class="MsoNormal">MatAssemblyEnd( IMP_Mat_dR1dW_PC, MAT_FINAL_ASSEMBLY);<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">// blend the matrices<o:p></o:p></p>
<p class="MsoNormal">// Out-of-memory appears here<o:p></o:p></p>
<p class="MsoNormal">MatAXPY( IMP_Mat_dR1dW_PC,<o:p></o:p></p>
<p class="MsoNormal"> 1-w,<o:p></o:p></p>
<p class="MsoNormal"> IMP_Mat_dR2dW,<o:p></o:p></p>
<p class="MsoNormal"> SAME_NONZERO_PATTERN);<o:p></o:p></p>
<p class="MsoNormal"> }<o:p></o:p></p>
<p class="MsoNormal">// KSPSetOperators<o:p></o:p></p>
<p class="MsoNormal">// KSPsolve <o:p></o:p></p>
<p class="MsoNormal">}<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I looked up the …/petsc-3.6.2/src/mat/impls/baij/mpi/mpibaij.c at line 2120. It appears that the temporary matrix B is allocated, but never freed. Note that I have not run Valgrind to confirm this leak.
<o:p></o:p></p>
<p class="MsoNormal">For now, I will make the non-zeros of the PC the same as the operator and use SAME_NONZERO_PATTERN.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you for your time and efforts,<o:p></o:p></p>
<p class="MsoNormal">Antoine<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">PETSC error below:<o:p></o:p></p>
<p class="MsoNormal">[117]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<o:p></o:p></p>
<p class="MsoNormal">[117]PETSC ERROR: Out of memory. This could be due to allocating<o:p></o:p></p>
<p class="MsoNormal">[117]PETSC ERROR: too large an object or bleeding by not properly<o:p></o:p></p>
<p class="MsoNormal">[117]PETSC ERROR: destroying unneeded objects.<o:p></o:p></p>
<p class="MsoNormal">[117]PETSC ERROR: Memory allocated 0 Memory used by process 8423399424<o:p></o:p></p>
<p class="MsoNormal">[117]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.<o:p></o:p></p>
<p class="MsoNormal">[117]PETSC ERROR: Memory requested 704476168<o:p></o:p></p>
<p class="MsoNormal">[117]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.<o:p></o:p></p>
<p class="MsoNormal">[117]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015
<o:p></o:p></p>
<p class="MsoNormal">[117]PETSC ERROR: /home/ad007804/CODES/FANSC_REPO_after_merge/fansc13.2.1_argus on a ARGUS_impi_opt named node1064 by ad007804 Tue Apr 26 09:34:52 2016<o:p></o:p></p>
<p class="MsoNormal">[117]PETSC ERROR: Configure options --CFLAGS=-axAVX --CXXFLAGS="-std=c++11 -DMPICH_IGNORE_CXX_SEEK -DMPICH_IGNORE_CXX_SEEK" --FC=mpiifort --FFLAGS=-axAVX --download-hypre=/home/b0528796/libs/petsc_extpacks/hypre-2.10.1.tar.gz --FC=mpiifort
--FFLAGS=-axAVX --download-metis=/home/b0528796/libs/petsc_extpacks/metis-5.1.0-p1.tar.gz --download-ml=/home/b0528796/libs/petsc_extpacks/ml-6.2-win.tar.gz --download-parmetis=/home/b0528796/libs/petsc_extpacks/parmetis-4.0.3-p1.tar.gz --download-suitesparse=/home/b0528796/libs/petsc_extpacks/SuiteSparse-4.4.3.tar.gz
--download-superlu_dist --with-blas-lapack-lib=-mkl --with-cc=mpiicc --with-cxx=mpiicpc --with-debugging=no --download-mumps=/home/b0528796/libs/petsc_extpacks/MUMPS_5.0.0-p1.tar.gz --with-scalapack-lib="-L/gpfs/fs1/intel/mkl/lib/intel64 -lmkl_scalapack_lp64
-lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/gpfs/fs1/intel/mkl/include<o:p></o:p></p>
<p class="MsoNormal">[117]PETSC ERROR: #1 MatAXPY_BasicWithPreallocation() line 117 in /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.6.2/src/mat/utils/axpy.c<o:p></o:p></p>
<p class="MsoNormal">[117]PETSC ERROR: #2 MatAXPY_BasicWithPreallocation() line 117 in /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.6.2/src/mat/utils/axpy.c<o:p></o:p></p>
<p class="MsoNormal">[117]PETSC ERROR: #3 MatAXPY_MPIBAIJ() line 2120 in /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.6.2/src/mat/impls/baij/mpi/mpibaij.c<o:p></o:p></p>
<p class="MsoNormal">[117]PETSC ERROR: #4 MatAXPY() line 39 in /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.6.2/src/mat/utils/axpy.c<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F">Antoine DeBlois</span></b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F"><br>
Spécialiste ingénierie, MDO lead / Engineering Specialist, MDO lead<br>
Aéronautique / Aerospace</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#131E29"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F">514-855-5001, x 50862<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#131E29"><a href="mailto:antoine.deblois@aero.bombardier.com"><span style="color:blue">antoine.deblois@aero.bombardier.com</span></a><br>
<br>
</span><i><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F">2351 Blvd Alfred-Nobel</span></i><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F"><br>
<i>Montreal, Qc<o:p></o:p></i></span></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F">H4S 1A9</span></i><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F"><br>
</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#131E29"><br>
<img border="0" width="162" height="34" id="Image_x0020_1" src="cid:image001.jpg@01D19FCA.80B0DDE0" alt="Description : http://signatures.ca.aero.bombardier.net/eom_logo_164x39_fr.jpg"><o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:8.5pt;font-family:"Arial",sans-serif;color:#555555">CONFIDENTIALITY NOTICE</span></b><span style="font-size:8.5pt;font-family:"Arial",sans-serif;color:#666666"> - This communication may contain privileged or confidential
information.<br>
If you are not the intended recipient or received this communication by error, please notify the sender<br>
and delete the message without copying</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>