<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.m-8063839492804910327m2433361434654699384m-1889958021554448809p1, li.m-8063839492804910327m2433361434654699384m-1889958021554448809p1, div.m-8063839492804910327m2433361434654699384m-1889958021554448809p1
{mso-style-name:m_-8063839492804910327m2433361434654699384m-1889958021554448809p1;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.m-8063839492804910327m2433361434654699384m-1889958021554448809s1
{mso-style-name:m_-8063839492804910327m2433361434654699384m-1889958021554448809s1;}
span.m-8063839492804910327m2433361434654699384m-1889958021554448809apple-converted-space
{mso-style-name:m_-8063839492804910327m2433361434654699384m-1889958021554448809apple-converted-space;}
span.m-8063839492804910327m2433361434654699384gmailsignatureprefix
{mso-style-name:m_-8063839492804910327m2433361434654699384gmailsignatureprefix;}
span.m-8063839492804910327gmailsignatureprefix
{mso-style-name:m_-8063839492804910327gmailsignatureprefix;}
span.gmailsignatureprefix
{mso-style-name:gmail_signature_prefix;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="FR-BE" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">Hello,
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US">I have used
</span><span lang="EN-US" style="font-size:11.0pt">Address Sanitizer to check any memory errors. On my computer, no errors are found. Unfortunately, on the supercomputer that I am using, I get lots of errors… I attach my log files (running on 1 and 70 procs).
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Do you have any idea of what I could do?
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Thanks a lot for your help.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Best regards, <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Joauma</span><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">De :
</span></b><span style="font-size:12.0pt;color:black">Matthew Knepley <knepley@gmail.com><br>
<b>Date : </b>lundi, 18 décembre 2023 à 12:00<br>
<b>À : </b>Joauma Marichal <joauma.marichal@uclouvain.be><br>
<b>Cc : </b>petsc-maint@mcs.anl.gov <petsc-maint@mcs.anl.gov>, petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Objet : </b>Re: [petsc-maint] DMSwarm on multiple processors<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">On Mon, Dec 18, 2023 at 5:09 AM Joauma Marichal <<a href="mailto:joauma.marichal@uclouvain.be">joauma.marichal@uclouvain.be</a>> wrote:<o:p></o:p></span></p>
</div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt">Hello,
</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt">Sorry for the delay. I attach the file that I obtain when running the code with the debug mode.</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Okay, we can now see where this is happening:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">malloc_consolidate(): invalid chunk size<br>
[cns263:3265170] *** Process received signal ***<br>
[cns263:3265170] Signal: Aborted (6)<br>
[cns263:3265170] Signal code: (-6)<br>
[cns263:3265170] [ 0] /lib64/libc.so.6(+0x4eb20)[0x7f3bd9148b20]<br>
[cns263:3265170] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x7f3bd9148a9f]<br>
[cns263:3265170] [ 2] /lib64/libc.so.6(abort+0x127)[0x7f3bd911be05]<br>
[cns263:3265170] [ 3] /lib64/libc.so.6(+0x91037)[0x7f3bd918b037]<br>
[cns263:3265170] [ 4] /lib64/libc.so.6(+0x9819c)[0x7f3bd919219c]<br>
[cns263:3265170] [ 5] /lib64/libc.so.6(+0x98b68)[0x7f3bd9192b68]<br>
[cns263:3265170] [ 6] /lib64/libc.so.6(+0x9af18)[0x7f3bd9194f18]<br>
[cns263:3265170] [ 7] /lib64/libc.so.6(__libc_malloc+0x1e2)[0x7f3bd9196822]<br>
[cns263:3265170] [ 8] /lib64/libc.so.6(posix_memalign+0x3c)[0x7f3bd91980fc]<br>
[cns263:3265170] [ 9] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(PetscMallocAlign+0x45)[0x7f3bda5f1625]<br>
[cns263:3265170] [10] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(PetscMallocA+0x297)[0x7f3bda5f1b07]<br>
[cns263:3265170] [11] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMCreate+0x5b)[0x7f3bdaa73c1b]<br>
[cns263:3265170] [12] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMDACreate+0x9)[0x7f3bdab0a2f9]<br>
[cns263:3265170] [13] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMDACreate3d+0x9a)[0x7f3bdab07dea]<br>
[cns263:3265170] [14] ./cobpor[0x402de8]<br>
[cns263:3265170] [15] /lib64/libc.so.6(__libc_start_main+0xf3)[0x7f3bd9134cf3]<br>
[cns263:3265170] [16] ./cobpor[0x40304e]<br>
[cns263:3265170] *** End of error message ***<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">However, this is not great. First, the amount of memory being allocated is quite small, and this does not appear to be an Out of Memory error. Second, the error occurs in libc:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> malloc_consolidate(): invalid chunk size<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">which means something is wrong internally. I agree with this analysis (<a href="https://stackoverflow.com/questions/18760999/sample-example-program-to-get-the-malloc-consolidate-error">https://stackoverflow.com/questions/18760999/sample-example-program-to-get-the-malloc-consolidate-error</a>)
that says you have probably overwritten memory somewhere in your code. I recommend running under valgrind, or using Address Sanitizer from clang.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> Thanks,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> Matt<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt">Thanks for your help.
</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt">Best regards,
</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt">Joauma</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">De :
</span></b><span style="font-size:12.0pt;color:black">Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Date : </b>jeudi, 23 novembre 2023 à 15:32<br>
<b>À : </b>Joauma Marichal <<a href="mailto:joauma.marichal@uclouvain.be" target="_blank">joauma.marichal@uclouvain.be</a>><br>
<b>Cc : </b><a href="mailto:petsc-maint@mcs.anl.gov" target="_blank">petsc-maint@mcs.anl.gov</a> <<a href="mailto:petsc-maint@mcs.anl.gov" target="_blank">petsc-maint@mcs.anl.gov</a>>,
<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Objet : </b>Re: [petsc-maint] DMSwarm on multiple processors</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">On Thu, Nov 23, 2023 at 9:01 AM Joauma Marichal <<a href="mailto:joauma.marichal@uclouvain.be" target="_blank">joauma.marichal@uclouvain.be</a>> wrote:<o:p></o:p></span></p>
</div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt">Hello,
</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt">My problem persists… Is there anything I could try?</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Yes. It appears to be failing from a call inside PetscSFSetUpRanks(). It does allocation, and the failure<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">is in libc, and it only happens on larger examples, so I suspect some allocation problem. Can you rebuild with debugging and run this example? Then
we can see if the allocation fails.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> Thanks,<br>
<br>
Matt<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt">Thanks a lot.
</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt">Best regards,
</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt">Joauma</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">De :
</span></b><span style="font-size:12.0pt;color:black">Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Date : </b>mercredi, 25 octobre 2023 à 14:45<br>
<b>À : </b>Joauma Marichal <<a href="mailto:joauma.marichal@uclouvain.be" target="_blank">joauma.marichal@uclouvain.be</a>><br>
<b>Cc : </b><a href="mailto:petsc-maint@mcs.anl.gov" target="_blank">petsc-maint@mcs.anl.gov</a> <<a href="mailto:petsc-maint@mcs.anl.gov" target="_blank">petsc-maint@mcs.anl.gov</a>>,
<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Objet : </b>Re: [petsc-maint] DMSwarm on multiple processors</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">On Wed, Oct 25, 2023 at 8:32 AM Joauma Marichal via petsc-maint <<a href="mailto:petsc-maint@mcs.anl.gov" target="_blank">petsc-maint@mcs.anl.gov</a>>
wrote:<o:p></o:p></span></p>
</div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;color:#212121">Hello, </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;color:#212121"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;color:#212121">I am using the DMSwarm library in some Eulerian-Lagrangian approach to have vapor bubbles in water. </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;color:#212121">I have obtained nice results recently and wanted to perform bigger simulations. Unfortunately, when I increase the number
of processors used to run the simulation, I get the following error:</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;color:#212121"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">free(): invalid size</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] *** Process received signal ***</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] Signal: Aborted (6)</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] Signal code:</span></span><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809apple-converted-space"><span lang="EN-US">
</span></span><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">(-6)</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [ 0] /lib64/libc.so.6(+0x4eb20)[0x7f56cd4c9b20]</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x7f56cd4c9a9f]</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [ 2] /lib64/libc.so.6(abort+0x127)[0x7f56cd49ce05]</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [ 3] /lib64/libc.so.6(+0x91037)[0x7f56cd50c037]</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [ 4] /lib64/libc.so.6(+0x9819c)[0x7f56cd51319c]</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [ 5] /lib64/libc.so.6(+0x99aac)[0x7f56cd514aac]</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [ 6] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(PetscSFSetUpRanks+0x4c4)[0x7f56cea71e64]</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [ 7] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(+0x841642)[0x7f56cea83642]</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [ 8] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(PetscSFSetUp+0x9e)[0x7f56cea7043e]</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1">[cns136:590327] [ 9] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(VecScatterCreate+0x164e)[0x7f56cea7bbde]</span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1">[cns136:590327] [10] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMSetUp_DA_3D+0x3e38)[0x7f56cee84dd8]</span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1">[cns136:590327] [11] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMSetUp_DA+0xd8)[0x7f56cee9b448]</span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1">[cns136:590327] [12] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMSetUp+0x20)[0x7f56cededa20]</span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [13] ./cobpor[0x4418dc]</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [14] ./cobpor[0x408b63]</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [15] /lib64/libc.so.6(__libc_start_main+0xf3)[0x7f56cd4b5cf3]</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [16] ./cobpor[0x40bdee]</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] *** End of error message ***</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">--------------------------------------------------------------------------</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">Primary job</span></span><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809apple-converted-space"><span lang="EN-US">
</span></span><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">terminated normally, but 1 process returned</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1">a non-zero exit code.
</span><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">Per user-direction, the job has been aborted.</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">--------------------------------------------------------------------------</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">--------------------------------------------------------------------------</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">mpiexec noticed that process rank 84 with PID 590327 on node cns136 exited on signal
6 (Aborted).</span></span><o:p></o:p></p>
<p class="m-8063839492804910327m2433361434654699384m-1889958021554448809p1"><span class="m-8063839492804910327m2433361434654699384m-1889958021554448809s1"><span lang="EN-US">--------------------------------------------------------------------------</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;color:#212121"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;color:#212121">When I reduce the number of processors the error disappears and when I run my code without the vapor bubbles it also works.
</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;color:#212121">The problem seems to take place at this moment:</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;color:#212121"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:13.5pt;background:#1E1E1E">
<span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:#DCDCAA">DMCreate</span><span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:#D4D4D4">(PETSC_COMM_WORLD,swarm);</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:13.5pt;background:#1E1E1E">
<span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:#D4D4D4"> </span>
<span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:#DCDCAA">DMSetType</span><span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:#D4D4D4">(*swarm,DMSWARM);</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:13.5pt;background:#1E1E1E">
<span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:#D4D4D4"> </span>
<span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:#DCDCAA">DMSetDimension</span><span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:#D4D4D4">(*swarm,</span><span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:#B5CEA8">3</span><span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:#D4D4D4">);</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:13.5pt;background:#1E1E1E">
<span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:#D4D4D4"> </span>
<span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:#DCDCAA">DMSwarmSetType</span><span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:#D4D4D4">(*swarm,DMSWARM_PIC);</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:13.5pt;background:#1E1E1E">
<span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:#D4D4D4"> </span>
<span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:#DCDCAA">DMSwarmSetCellDM</span><span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:#D4D4D4">(*swarm,*dmcell);</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;color:#212121"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;color:#212121"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;color:#212121">Thanks a lot for your help. </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Things that would help us track this down:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">1) The smallest example where it fails<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">2) The smallest number of processes where it fails<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">3) A stack trace of the failure<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">4) A simple example that we can run that also fails<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> Thanks,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> Matt<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;color:#212121">Best regards, </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;color:#212121"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;color:#212121">Joauma</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"><br clear="all">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="m-8063839492804910327m2433361434654699384gmailsignatureprefix"><span style="font-size:11.0pt">--
</span></span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments
lead.<br>
-- Norbert Wiener<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"><br clear="all">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="m-8063839492804910327gmailsignatureprefix"><span style="font-size:11.0pt">--
</span></span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments
lead.<br>
-- Norbert Wiener<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"><br clear="all">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span class="gmailsignatureprefix"><span style="font-size:11.0pt">--
</span></span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>