<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<!-- Template generated by Exclaimer Mail Disclaimers on 09:55:37 Tuesday, 4 May 2021 -->
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:"Yu Gothic";
panose-1:2 11 4 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@Yu Gothic";
panose-1:2 11 4 0 0 0 0 0 0 0;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.MailQuote, li.MailQuote, div.MailQuote
{mso-style-name:"Mail Quote";
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
border:none;
padding:0in;
font-size:12.0pt;
font-family:"Calibri",sans-serif;
color:black;
mso-fareast-language:EN-GB;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<p></p>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Barry,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thank you for this message about finalisation. I have checked that PetscFinalize is called after the problematic call to MatDestroy, and that is indeed the case.
Furthermore, the module does not use "final".<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Menno<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div></div>
</div>
<br>
<font style="font-family:Verdana;font-size:8pt;color:#7D7D7D;font-weight:bold;">dr. ir.</font><font style="font-family:Verdana;font-size:8pt;color:#7D7D7D;font-weight:bold;">
</font><font style="font-family: Verdana;font-size:8pt;color:#7D7D7D;font-weight:bold;">Menno A.</font><font style="font-family: Verdana;font-size:8pt;color:#7D7D7D;font-weight:bold;">
</font><font style="font-family:Verdana;font-size:8pt;color:#7D7D7D;font-weight:bold;">Deij-van Rijswijk</font><font size="3" face="Segoe UI"></font><font style="font-family:Verdana;font-size:8pt;color:#7D7D7D;font-weight:bold;"> |
</font><font style="font-family:Verdana;font-size:8pt;color:#7D7D7D;font-weight:bold;">Researcher</font><font style="font-family:Verdana;font-size:8pt;color:#7D7D7D;font-weight:bold;"> |
</font><font style="font-family:Verdana;font-size:8pt;color:#7D7D7D;font-weight:bold;">Research & Development</font><br>
<font style="font-family:Verdana;font-size:9pt;color:#7D7D7D;font-weight:bold;">MARIN</font><font style="font-family:Verdana;font-size:8pt;color:#7D7D7D;"> | T
</font><font style="font-family:Verdana;font-size:8pt;color:#7D7D7D;">+31 317 49 35 06</font><font color="#7d7d7d" size="2" face="Segoe UI"></font><font style="font-family:Verdana;font-size:8pt;"> |
</font><span style="font-family:Verdana;font-size:8pt;"><a href="mailto:M.Deij@marin.nl" title="" target="" style="font-family:Verdana;font-size:8pt;"><span style="font-family:Verdana; font-size:8pt;">M.Deij@marin.nl</span></a></span><font color="#7d7d7d" size="2" face="Segoe UI"></font><font style="font-family:Verdana;font-size:8pt;"> |
</font><span style="font-family:Verdana;font-size:8pt;"><a href="http://www.marin.nl" title="" target="" style="font-family:Verdana;font-size:8pt;"><span style="font-family:Verdana; font-size:8pt;">www.marin.nl</span></a></span><br>
<br>
<a href="https://www.linkedin.com/company/marin" target=""><img width="16" height="16" style="border: 0px Solid ; " src="cid:image021333.PNG@58263797.48ab6e99" alt="LinkedIn"></a> <a href="http://www.youtube.com/marinmultimedia" target=""><img width="16" height="16" style="border: 0px Solid ; " src="cid:image6d0c90.PNG@7008c216.4b81e1e4" alt="YouTube"></a> <a href="https://twitter.com/MARIN_nieuws" target=""><img width="16" height="16" style="border: 0px Solid ; " src="cid:image8d0af3.PNG@da7bf37a.47b69f5e" alt="Twitter"></a> <a href="https://www.facebook.com/marin.wageningen" target=""><img width="16" height="16" style="border: 0px Solid ; " src="cid:image92bfe7.PNG@9e24a3e5.46a6f32a" alt="Facebook"></a><br>
<font style="FONT-FAMILY: Verdana; FONT-SIZE: 8pt" color="#7d7d7d" size="2" face="Verdana">MARIN news:
<a href="https://www.marin.nl/news/working-paper-on-the-design-of-the-wageningen-f-series" title="" target="" style="">
Working paper on the Design of the Wageningen F-series</a></font>
<p></p>
<p id="undefined"></p>
<p></p>
<div class="WordSection1">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Barry Smith <bsmith@petsc.dev>
<br>
<b>Sent:</b> Sunday, May 2, 2021 6:30 PM<br>
<b>To:</b> Deij-van Rijswijk, Menno <M.Deij@marin.nl><br>
<b>Cc:</b> petsc-users@mcs.anl.gov<br>
<b>Subject:</b> Re: [petsc-users] MatDestroy problem with multiple matrices and SUPERLU_DIST<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x5317899: MatDestroy (in /home/mdeij/install-gnu/extLibs/lib/libpetsc.so.3.14.5)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x5336E58: matdestroy_ (in /home/mdeij/install-gnu/extLibs/lib/libpetsc.so.3.14.5)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x1528710: __fsi_MOD_fem_constructmatricespetscexit (fsi.F90:2297)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== Address 0x2ce67398 is 11,112 bytes inside an unallocated block of size 11,232 in arena "client"</span><o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"> Is it possible that this <span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">__fsi_MOD_fem_constructmatricespetscexit</span> is being called AFTER PetscFinalize()? Perhaps it is defined with a "final" and the compiler/linker
schedule it to be called after the program has "completed".<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> This would explain the crash, the valgrind stack frames and why it even does not crash with MPICH. This can happen with C++ destructors in code such as<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> MyC++Class my; <-- has a destructor that destroys PETSc objects<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> PetscInitialize()<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> ....<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> PetscFinalize()<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <-- the destructor gets called here and messes with MPI data that no longer exists.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> return 0;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> } <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">The fix is to force the destructor to be called before PETSc finalize and this can be done with <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> PetscInitialize()<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> {<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> MyC++Class my; <-- has a destructor that destroys PETSc objects<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> ....<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <-- the destructor gets called here and everything is fine<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> }<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> PetscFinalize()<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> return 0;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> } <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I don't know the details of how Fortran's final is implemented but this is my current guess as to what is happening in your code and you need to somehow arrange for the module final to be called before PetscFinalize().<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> Barry<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Apr 28, 2021, at 7:22 AM, Deij-van Rijswijk, Menno <<a href="mailto:M.Deij@marin.nl">M.Deij@marin.nl</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The modules have automatic freeing in as much as that when a variable that is local to a subroutine is ALLOCATE'd, it is automatically freed when the subroutine
returns. I don't think that is problematic, as MatDestroy is used a lot in the code and normally executes just fine.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">As far as I can see, no specific new communicators are created; MatCreateAIJ or MatCreateSeqAIJ are called with PETSC_COMM_WORLD, resp. PETSC_COMM_SELF as first
argument.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">We also run this with the Intel MPI library, which is based on MPICH. There this problem does not occur.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The Valgrind run did not produce any new insights (at least not for me), I have pasted the relevant bits at the end of this message. I did a run on debug versions
of PETSc (v3.14.5) and OpenMPI (v 3.1.2) and I find the following stack trace with line numbers for each frame. Maybe that helps in further pinpointing the problem.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">0x0000155540d11719 in ompi_comm_free (comm=0x483f4e0) at /home/mdeij/build-libs-gnu/superbuild/openmpi/src/ompi/communicator/comm.c:1470</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">1470 if ( ! OMPI_COMM_IS_INTRINSIC((*comm)->c_local_comm)) {</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">Missing separate debuginfos, use: yum debuginfo-install libgcc-8.3.1-5.el8.0.2.x86_64 libgfortran-8.3.1-5.el8.0.2.x86_64 libibumad-47mlnx1-1.47329.x86_64 libibverbs-47mlnx1-1.47329.x86_64
libnl3-3.5.0-1.el8.x86_64 libquadmath-8.3.1-5.el8.0.2.x86_64 librdmacm-47mlnx1-1.47329.x86_64 libstdc++-8.3.1-5.el8.0.2.x86_64 libxml2-2.9.7-7.el8.x86_64 numactl-libs-2.0.12-9.el8.x86_64 opensm-libs-5.5.1.MLNX20191120.0c8dde0-0.1.47329.x86_64 openssl-libs-1.1.1c-15.el8.x86_64
python3-libs-3.6.8-23.el8.x86_64 sssd-client-2.2.3-20.el8.x86_64 ucx-cma-1.7.0-1.47329.x86_64 ucx-ib-1.7.0-1.47329.x86_64 xz-libs-5.2.4-3.el8.x86_64 zlib-1.2.11-16.el8_2.x86_64</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">(gdb) bt</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">#0 0x0000155540d11719 in ompi_comm_free (comm=0x483f4e0) at /home/mdeij/build-libs-gnu/superbuild/openmpi/src/ompi/communicator/comm.c:1470</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">#1 0x0000155540d4f1af in PMPI_Comm_free (comm=0x483f4e0) at pcomm_free.c:62</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">#2 0x000015555346329a in superlu_gridexit (grid=0x483f4e0) at /home/mdeij/install-gnu/extLibs/Linux-x86_64-Intel/superlu_dist-6.3.0/SRC/superlu_grid.c:174</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">#3 0x0000155553ca2ff1 in Petsc_Superlu_dist_keyval_Delete_Fn (comm=0x3921b10, keyval=16, attr_val=0x483f4d0, extra_state=0x0) at /home/mdeij/build-libs-gnu/superbuild/petsc/src/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:97</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">#4 0x0000155540d0baa1 in ompi_attr_delete_impl (type=COMM_ATTR, object=0x3921b10, attr_hash=0x377efe0, key=16, predefined=true) at /home/mdeij/build-libs-gnu/superbuild/openmpi/src/ompi/attribute/attribute.c:1062</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">#5 0x0000155540d0c039 in ompi_attr_delete_all (type=COMM_ATTR, object=0x3921b10, attr_hash=0x377efe0) at /home/mdeij/build-libs-gnu/superbuild/openmpi/src/ompi/attribute/attribute.c:1166</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">#6 0x0000155540d11676 in ompi_comm_free (comm=0x7fffffffc5c0) at /home/mdeij/build-libs-gnu/superbuild/openmpi/src/ompi/communicator/comm.c:1462</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">#7 0x0000155540d4f1af in PMPI_Comm_free (comm=0x7fffffffc5c0) at pcomm_free.c:62</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">#8 0x000015555393fb68 in PetscCommDestroy (comm=0x3943a60) at /home/mdeij/build-libs-gnu/superbuild/petsc/src/src/sys/objects/tagm.c:217</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">#9 0x0000155553941e07 in PetscHeaderDestroy_Private (h=0x3943a20) at /home/mdeij/build-libs-gnu/superbuild/petsc/src/src/sys/objects/inherit.c:121</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">#10 0x000015555408edfe in MatDestroy (A=0x3558c18) at /home/mdeij/build-libs-gnu/superbuild/petsc/src/src/mat/interface/matrix.c:1306</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">#11 0x00001555540cb5fa in matdestroy_ (A=0x3558c18, __ierr=0x7fffffffc73c) at /home/mdeij/build-libs-gnu/superbuild/petsc/src/src/mat/interface/ftn-auto/matrixf.c:770</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">Valgrind output:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== Invalid read of size 1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== at 0x19184538: PMPI_Comm_free (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x6943B61: superlu_gridexit (in /home/mdeij/install-gnu/extLibs/lib/libsuperlu_dist.so.6.3.0)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x56F398E: Petsc_Superlu_dist_keyval_Delete_Fn (in /home/mdeij/install-gnu/extLibs/lib/libpetsc.so.3.14.5)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x1912447B: ompi_attr_delete_impl (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x19126FFE: ompi_attr_delete_all (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x1912ACC6: ompi_comm_free (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x19184555: PMPI_Comm_free (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x4FEE49D: PetscCommDestroy (in /home/mdeij/install-gnu/extLibs/lib/libpetsc.so.3.14.5)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x4FF0EE1: PetscHeaderDestroy_Private (in /home/mdeij/install-gnu/extLibs/lib/libpetsc.so.3.14.5)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x5317899: MatDestroy (in /home/mdeij/install-gnu/extLibs/lib/libpetsc.so.3.14.5)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x5336E58: matdestroy_ (in /home/mdeij/install-gnu/extLibs/lib/libpetsc.so.3.14.5)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x1528710: __fsi_MOD_fem_constructmatricespetscexit (fsi.F90:2297)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== Address 0x2ce67398 is 11,112 bytes inside an unallocated block of size 11,232 in arena "client"</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905==</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== Invalid read of size 8</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== at 0x1912AC9A: ompi_comm_free (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x19184555: PMPI_Comm_free (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x6943B61: superlu_gridexit (in /home/mdeij/install-gnu/extLibs/lib/libsuperlu_dist.so.6.3.0)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x56F398E: Petsc_Superlu_dist_keyval_Delete_Fn (in /home/mdeij/install-gnu/extLibs/lib/libpetsc.so.3.14.5)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x1912447B: ompi_attr_delete_impl (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x19126FFE: ompi_attr_delete_all (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x1912ACC6: ompi_comm_free (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x19184555: PMPI_Comm_free (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x4FEE49D: PetscCommDestroy (in /home/mdeij/install-gnu/extLibs/lib/libpetsc.so.3.14.5)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x4FF0EE1: PetscHeaderDestroy_Private (in /home/mdeij/install-gnu/extLibs/lib/libpetsc.so.3.14.5)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x5317899: MatDestroy (in /home/mdeij/install-gnu/extLibs/lib/libpetsc.so.3.14.5)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x5336E58: matdestroy_ (in /home/mdeij/install-gnu/extLibs/lib/libpetsc.so.3.14.5)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== Address 0x2ce673c0 is 11,152 bytes inside an unallocated block of size 11,232 in arena "client"</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905==</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== Invalid read of size 8</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== at 0x19126E5B: ompi_attr_delete_all (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x1912ACC6: ompi_comm_free (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x19184555: PMPI_Comm_free (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x6943B61: superlu_gridexit (in /home/mdeij/install-gnu/extLibs/lib/libsuperlu_dist.so.6.3.0)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x56F398E: Petsc_Superlu_dist_keyval_Delete_Fn (in /home/mdeij/install-gnu/extLibs/lib/libpetsc.so.3.14.5)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x1912447B: ompi_attr_delete_impl (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x19126FFE: ompi_attr_delete_all (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x1912ACC6: ompi_comm_free (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x19184555: PMPI_Comm_free (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x4FEE49D: PetscCommDestroy (in /home/mdeij/install-gnu/extLibs/lib/libpetsc.so.3.14.5)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x4FF0EE1: PetscHeaderDestroy_Private (in /home/mdeij/install-gnu/extLibs/lib/libpetsc.so.3.14.5)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x5317899: MatDestroy (in /home/mdeij/install-gnu/extLibs/lib/libpetsc.so.3.14.5)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== Address 0x91 is not stack'd, malloc'd or (recently) free'd</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905==</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905==</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== Process terminating with default action of signal 11 (SIGSEGV)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== Access not within mapped region at address 0x91</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== at 0x19126E5B: ompi_attr_delete_all (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x1912ACC6: ompi_comm_free (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x19184555: PMPI_Comm_free (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x6943B61: superlu_gridexit (in /home/mdeij/install-gnu/extLibs/lib/libsuperlu_dist.so.6.3.0)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x56F398E: Petsc_Superlu_dist_keyval_Delete_Fn (in /home/mdeij/install-gnu/extLibs/lib/libpetsc.so.3.14.5)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x1912447B: ompi_attr_delete_impl (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x19126FFE: ompi_attr_delete_all (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x1912ACC6: ompi_comm_free (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x19184555: PMPI_Comm_free (in /home/mdeij/install-gnu/extLibs/lib/libmpi.so.40.10.2)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x4FEE49D: PetscCommDestroy (in /home/mdeij/install-gnu/extLibs/lib/libpetsc.so.3.14.5)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x4FF0EE1: PetscHeaderDestroy_Private (in /home/mdeij/install-gnu/extLibs/lib/libpetsc.so.3.14.5)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== by 0x5317899: MatDestroy (in /home/mdeij/install-gnu/extLibs/lib/libpetsc.so.3.14.5)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== If you believe this happened as a result of a stack</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== overflow in your program's main thread (unlikely but</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== possible), you can try to increase the size of the</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== main thread stack using the --main-stacksize= flag.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">==1026905== The main thread stack size used in this run was 16777216.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><br>
</span><b><span style="font-size:8.0pt;font-family:"Verdana",sans-serif;color:#7D7D7D">dr. ir.<span class="apple-converted-space"> </span>Menno A.<span class="apple-converted-space"> </span>Deij-van Rijswijk<span class="apple-converted-space"> </span>|<span class="apple-converted-space"> </span>Researcher<span class="apple-converted-space"> </span>|<span class="apple-converted-space"> </span>Research
& Development</span></b><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><br>
</span><b><span style="font-size:9.0pt;font-family:"Verdana",sans-serif;color:#7D7D7D">MARIN</span></b><span class="apple-converted-space"><span style="font-size:8.0pt;font-family:"Verdana",sans-serif;color:#7D7D7D"> </span></span><span style="font-size:8.0pt;font-family:"Verdana",sans-serif;color:#7D7D7D">|
T<span class="apple-converted-space"> </span>+31 317 49 35 06</span><span class="apple-converted-space"><span style="font-size:8.0pt;font-family:"Verdana",sans-serif"> </span></span><span style="font-size:8.0pt;font-family:"Verdana",sans-serif">|<span class="apple-converted-space"> </span><a href="mailto:M.Deij@marin.nl"><span style="color:purple">M.Deij@marin.nl</span></a> |<span class="apple-converted-space"> </span><a href="http://www.marin.nl/"><span style="color:purple">www.marin.nl</span></a></span><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><br>
<br>
</span><a href="https://www.linkedin.com/company/marin"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif;color:purple"><imagebf865c.PNG></span></a><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"> </span><a href="http://www.youtube.com/marinmultimedia"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif;color:purple"><image1edec1.PNG></span></a><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"> </span><a href="https://twitter.com/MARIN_nieuws"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif;color:purple"><imagedbdbd7.PNG></span></a><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"> </span><a href="https://www.facebook.com/marin.wageningen"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif;color:purple"><image4abcc0.PNG></span></a><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><br>
</span><span style="font-size:8.0pt;font-family:"Verdana",sans-serif;color:#7D7D7D">MARIN news:<span class="apple-converted-space"> </span><a href="https://www.marin.nl/news/wasp-webinar-wisp-workshop-april-22"><span style="color:purple">WASP webinar & WiSP
workshop</span></a></span><o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span class="apple-converted-space"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Barry
Smith <<a href="mailto:bsmith@petsc.dev"><span style="color:purple">bsmith@petsc.dev</span></a>><span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Friday, April 23, 2021 7:09 PM<br>
<b>To:</b><span class="apple-converted-space"> </span>Deij-van Rijswijk, Menno <<a href="mailto:M.Deij@marin.nl"><span style="color:purple">M.Deij@marin.nl</span></a>><br>
<b>Cc:</b><span class="apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov"><span style="color:purple">petsc-users@mcs.anl.gov</span></a><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [petsc-users] MatDestroy problem with multiple matrices and SUPERLU_DIST</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> Thanks for looking. Do these modules have any "automatic freeing" when variables go out of scope (like C++ classes do)? <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> Do you make specific new MPI communicators to use create the matrices? <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> Have you tried MPICH or a different version of OpenMPI. <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> Maybe run the program with valgrind. The stack frames you sent look "funny", that is I would not normally expect them to be in such an order.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> Barry<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="background:white"><o:p> </o:p></span></p>
<p align="center" style="text-align:center"><b><span style="background:white">Help us improve the spam filter. If this message contains SPAM, click
<a href="https://www.mailcontrol.com/sr/lqKC67CZnPPGX2PQPOmvUhkLFoJbzkFEyBNkQNATPXFrmmQ3cY8Q4d5cDBrY7_s6LHWuLmbsjXSzbAWAmKJQAw==">
here</a> to report. Thank you, MARIN Digital Services</span></b><span style="background:white"><o:p></o:p></span></p>
</div>
</EXCLAIMER:ORIGINAL_BODY_AFTER>
<p></p>
</EXCLAIMER:TEXT_ONLY></EXCLAIMER:AD></EXCLAIMER:AD>
</body>
</html>