<div dir="ltr">Okay, there were a few problems:<div><br></div><div>1) You overwrote the bounds on string loc_grid_gen[]</div><div><br></div><div>2) You destroyed the coordinate DA</div><div><br></div><div>I fixed these and it runs for me fine on several processes. I am including my revised source since I check a lot more error values. I converted it to C because that is easier for me, although C has a problem with your sqrt() in a compile-time constant.</div><div><br></div><div>  Thanks,</div><div><br></div><div>     Matt</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Oct 26, 2023 at 10:59 AM Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><br></div>   Please run with -malloc_debug option or even better run under Valgrind <a href="https://petsc.org/release/faq/" target="_blank">https://petsc.org/release/faq/</a><div><br></div><div><br id="m_-382879391909209636lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Oct 26, 2023, at 10:35 AM, Joauma Marichal via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:</div><br><div><div style="font-family:Helvetica;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">Hello,<span> </span><u></u><u></u></span></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US"><u></u> <u></u></span></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">Here is a very simple version where I have issues.</span></span><span lang="EN-US"><u></u><u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US"><u></u> <u></u></span></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">Which I run as follows:<u></u><u></u></span></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US"><u></u> <u></u></span></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">cd Grid_generation</span></span><span><span lang="EN-US"> </span></span><span lang="EN-US"><u></u><u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">make clean</span></span><span><span lang="EN-US"> </span></span><span lang="EN-US"><u></u><u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">make all</span></span><span lang="EN-US"><u></u><u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">./grid_generation</span></span><span><span lang="EN-US"> <u></u><u></u></span></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">cd ..</span></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">make clean</span></span><span><span lang="EN-US"> </span></span><span lang="EN-US"><u></u><u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">make all</span></span><span lang="EN-US"><u></u><u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">./cobpor</span></span><span><span lang="EN-US"> # on 1 proc<u></u><u></u></span></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US"># OR<u></u><u></u></span></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span lang="EN-US">mpiexec ./cobpor<span> </span></span><span lang="EN-US" style="color:rgb(200,20,201)">-ksp_type</span><span lang="EN-US"><span> </span>cg<span> </span></span><span lang="EN-US" style="color:rgb(200,20,201)">-pc_type</span><span lang="EN-US"><span> </span>pfmg<span> </span></span><span lang="EN-US" style="color:rgb(200,20,201)">-dm_mat_type</span><span lang="EN-US"><span> </span>hyprestruct<span> </span></span><span lang="EN-US" style="color:rgb(200,20,201)">-pc_pfmg_skip_relax</span><span lang="EN-US"><span> </span></span><span lang="EN-US" style="color:rgb(180,36,25)">1</span><span lang="EN-US"><span> </span></span><span lang="EN-US" style="color:rgb(200,20,201)">-pc_pfmg_rap_time</span><span lang="EN-US"><span> </span>non-Galerkin<span> </span><span># on multiple procs<u></u><u></u></span></span></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:10.5pt;font-family:Menlo;color:rgb(200,20,201)"><u></u> <u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span lang="EN-US">The error that I get is the following:<u></u><u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">munmap_chunk(): invalid pointer</span></span><span lang="EN-US"><u></u><u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">[cns266:2552391] *** Process received signal ***</span></span><span lang="EN-US"><u></u><u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">[cns266:2552391] Signal: Aborted (6)</span></span><span lang="EN-US"><u></u><u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">[cns266:2552391] Signal code:</span></span><span><span lang="EN-US"> <span> </span></span></span><span><span lang="EN-US">(-6)</span></span><span lang="EN-US"><u></u><u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">[cns266:2552391] [ 0] /lib64/<a href="http://libc.so/" style="color:blue;text-decoration:underline" target="_blank">libc.so</a>.6(+0x4eb20)[0x7fd7fd194b20]</span></span><span lang="EN-US"><u></u><u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">[cns266:2552391] [ 1] /lib64/<a href="http://libc.so/" style="color:blue;text-decoration:underline" target="_blank">libc.so</a>.6(gsignal+0x10f)[0x7fd7fd194a9f]</span></span><span lang="EN-US"><u></u><u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">[cns266:2552391] [ 2] /lib64/<a href="http://libc.so/" style="color:blue;text-decoration:underline" target="_blank">libc.so</a>.6(abort+0x127)[0x7fd7fd167e05]</span></span><span lang="EN-US"><u></u><u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">[cns266:2552391] [ 3] /lib64/<a href="http://libc.so/" style="color:blue;text-decoration:underline" target="_blank">libc.so</a>.6(+0x91037)[0x7fd7fd1d7037]</span></span><span lang="EN-US"><u></u><u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span>[cns266:2552391] [ 4] /lib64/<a href="http://libc.so/" style="color:blue;text-decoration:underline" target="_blank">libc.so</a>.6(+0x9819c)[0x7fd7fd1de19c]</span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span>[cns266:2552391] [ 5] /lib64/<a href="http://libc.so/" style="color:blue;text-decoration:underline" target="_blank">libc.so</a>.6(+0x9844c)[0x7fd7fd1de44c]</span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span>[cns266:2552391] [ 6] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/<a href="http://libpetsc.so/" style="color:blue;text-decoration:underline" target="_blank">libpetsc.so</a>.3.019(PetscFreeAlign+0xe)[0x7fd7fe63d50e]</span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span>[cns266:2552391] [ 7] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/<a href="http://libpetsc.so/" style="color:blue;text-decoration:underline" target="_blank">libpetsc.so</a>.3.019(DMSetMatType+0x3d)[0x7fd7feab87ad]</span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span>[cns266:2552391] [ 8] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/<a href="http://libpetsc.so/" style="color:blue;text-decoration:underline" target="_blank">libpetsc.so</a>.3.019(DMSetFromOptions+0x109)[0x7fd7feab8b59]</span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span>[cns266:2552391] [ 9] ./cobpor[0x402df9]</span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span>[cns266:2552391] [10] /lib64/<a href="http://libc.so/" style="color:blue;text-decoration:underline" target="_blank">libc.so</a>.6(__libc_start_main+0xf3)[0x7fd7fd180cf3]</span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">[cns266:2552391] [11] ./cobpor[0x40304e]</span></span><span lang="EN-US"><u></u><u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span><span lang="EN-US">[cns266:2552391] *** End of error message ***</span></span><span lang="EN-US"><u></u><u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span lang="EN-US"><u></u> <u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span lang="EN-US"><u></u> <u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span lang="EN-US">Thanks a lot for your help.<span> </span><u></u><u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span lang="EN-US"><u></u> <u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span lang="EN-US">Best regards,<span> </span><u></u><u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span lang="EN-US"><u></u> <u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span lang="EN-US">Joauma<u></u><u></u></span></div><div style="margin:0cm;font-size:10.5pt;font-family:Menlo"><span lang="EN-US"><u></u> <u></u></span></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt"><u></u> <u></u></span></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt"><u></u> <u></u></span></div><div style="border-width:1pt medium medium;border-style:solid none none;border-color:rgb(181,196,223) currentcolor currentcolor;padding:3pt 0cm 0cm"><p class="MsoNormal" style="margin:0cm 0cm 12pt;font-size:10pt;font-family:Calibri,sans-serif"><b><span lang="EN-US" style="font-size:12pt">De :<span> </span></span></b><span lang="EN-US" style="font-size:12pt">Matthew Knepley <<a href="mailto:knepley@gmail.com" style="color:blue;text-decoration:underline" target="_blank">knepley@gmail.com</a>><br><b>Date :<span> </span></b>mercredi, 25 octobre 2023 à 14:45<br><b>À :<span> </span></b>Joauma Marichal <<a href="mailto:joauma.marichal@uclouvain.be" style="color:blue;text-decoration:underline" target="_blank">joauma.marichal@uclouvain.be</a>><br><b>Cc :<span> </span></b><a href="mailto:petsc-maint@mcs.anl.gov" style="color:blue;text-decoration:underline" target="_blank">petsc-maint@mcs.anl.gov</a><span> </span><<a href="mailto:petsc-maint@mcs.anl.gov" style="color:blue;text-decoration:underline" target="_blank">petsc-maint@mcs.anl.gov</a>>,<span> </span><a href="mailto:petsc-users@mcs.anl.gov" style="color:blue;text-decoration:underline" target="_blank">petsc-users@mcs.anl.gov</a><span> </span><<a href="mailto:petsc-users@mcs.anl.gov" style="color:blue;text-decoration:underline" target="_blank">petsc-users@mcs.anl.gov</a>><br><b>Objet :<span> </span></b>Re: [petsc-maint] DMSwarm on multiple processors<u></u><u></u></span></p></div><div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt">On Wed, Oct 25, 2023 at 8:32 AM Joauma Marichal via petsc-maint <</span><a href="mailto:petsc-maint@mcs.anl.gov" style="color:blue;text-decoration:underline" target="_blank"><span lang="EN-US" style="font-size:11pt">petsc-maint@mcs.anl.gov</span></a><span lang="EN-US" style="font-size:11pt">> wrote:<u></u><u></u></span></div></div><div><blockquote style="border-width:medium medium medium 1pt;border-style:none none none solid;border-color:currentcolor currentcolor currentcolor rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt"><div><div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt;color:rgb(33,33,33)">Hello, </span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt;color:rgb(33,33,33)"> </span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt;color:rgb(33,33,33)">I am using the DMSwarm library in some Eulerian-Lagrangian approach to have vapor bubbles in water. </span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt;color:rgb(33,33,33)">I have obtained nice results recently and wanted to perform bigger simulations. Unfortunately, when I increase the number of processors used to run the simulation, I get the following error:</span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt;color:rgb(33,33,33)"> </span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">free(): invalid size</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] *** Process received signal ***</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] Signal: Aborted (6)</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] Signal code:</span></span><span><span lang="EN-US"> <span> </span></span></span><span><span lang="EN-US">(-6)</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] [ 0] /lib64/<a href="http://libc.so/" style="color:blue;text-decoration:underline" target="_blank">libc.so</a>.6(+0x4eb20)[0x7f56cd4c9b20]</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] [ 1] /lib64/<a href="http://libc.so/" style="color:blue;text-decoration:underline" target="_blank">libc.so</a>.6(gsignal+0x10f)[0x7f56cd4c9a9f]</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] [ 2] /lib64/<a href="http://libc.so/" style="color:blue;text-decoration:underline" target="_blank">libc.so</a>.6(abort+0x127)[0x7f56cd49ce05]</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] [ 3] /lib64/<a href="http://libc.so/" style="color:blue;text-decoration:underline" target="_blank">libc.so</a>.6(+0x91037)[0x7f56cd50c037]</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] [ 4] /lib64/<a href="http://libc.so/" style="color:blue;text-decoration:underline" target="_blank">libc.so</a>.6(+0x9819c)[0x7f56cd51319c]</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] [ 5] /lib64/<a href="http://libc.so/" style="color:blue;text-decoration:underline" target="_blank">libc.so</a>.6(+0x99aac)[0x7f56cd514aac]</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] [ 6] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/<a href="http://libpetsc.so/" style="color:blue;text-decoration:underline" target="_blank">libpetsc.so</a>.3.019(PetscSFSetUpRanks+0x4c4)[0x7f56cea71e64]</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] [ 7] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/<a href="http://libpetsc.so/" style="color:blue;text-decoration:underline" target="_blank">libpetsc.so</a>.3.019(+0x841642)[0x7f56cea83642]</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] [ 8] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/<a href="http://libpetsc.so/" style="color:blue;text-decoration:underline" target="_blank">libpetsc.so</a>.3.019(PetscSFSetUp+0x9e)[0x7f56cea7043e]</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] [ 9] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/<a href="http://libpetsc.so/" style="color:blue;text-decoration:underline" target="_blank">libpetsc.so</a>.3.019(VecScatterCreate+0x164e)[0x7f56cea7bbde]</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] [10] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/<a href="http://libpetsc.so/" style="color:blue;text-decoration:underline" target="_blank">libpetsc.so</a>.3.019(DMSetUp_DA_3D+0x3e38)[0x7f56cee84dd8]</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] [11] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/<a href="http://libpetsc.so/" style="color:blue;text-decoration:underline" target="_blank">libpetsc.so</a>.3.019(DMSetUp_DA+0xd8)[0x7f56cee9b448]</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] [12] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/<a href="http://libpetsc.so/" style="color:blue;text-decoration:underline" target="_blank">libpetsc.so</a>.3.019(DMSetUp+0x20)[0x7f56cededa20]</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] [13] ./cobpor[0x4418dc]</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] [14] ./cobpor[0x408b63]</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] [15] /lib64/<a href="http://libc.so/" style="color:blue;text-decoration:underline" target="_blank">libc.so</a>.6(__libc_start_main+0xf3)[0x7f56cd4b5cf3]</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] [16] ./cobpor[0x40bdee]</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">[cns136:590327] *** End of error message ***</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">--------------------------------------------------------------------------</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">Primary job</span></span><span><span lang="EN-US"> <span> </span></span></span><span><span lang="EN-US">terminated normally, but 1 process returned</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span>a non-zero exit code.<span> </span></span><span><span lang="EN-US">Per user-direction, the job has been aborted.</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">--------------------------------------------------------------------------</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">--------------------------------------------------------------------------</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">mpiexec noticed that process rank 84 with PID 590327 on node cns136 exited on signal 6 (Aborted).</span></span><span lang="EN-US"><u></u><u></u></span></p><p style="margin-right:0cm;margin-left:0cm;font-size:11pt;font-family:Calibri,sans-serif"><span><span lang="EN-US">--------------------------------------------------------------------------</span></span><span lang="EN-US"><u></u><u></u></span></p><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt;color:rgb(33,33,33)"> </span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt;color:rgb(33,33,33)">When I reduce the number of processors the error disappears and when I run my code without the vapor bubbles it also works.</span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt;color:rgb(33,33,33)">The problem seems to take place at this moment:</span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt;color:rgb(33,33,33)"> </span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif;line-height:13.5pt;background:rgb(30,30,30)"><span lang="EN-US" style="font-size:9pt;font-family:Menlo;color:rgb(220,220,170)">DMCreate</span><span lang="EN-US" style="font-size:9pt;font-family:Menlo;color:rgb(212,212,212)">(PETSC_COMM_WORLD,swarm);</span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif;line-height:13.5pt;background:rgb(30,30,30)"><span lang="EN-US" style="font-size:9pt;font-family:Menlo;color:rgb(212,212,212)">   <span> </span></span><span lang="EN-US" style="font-size:9pt;font-family:Menlo;color:rgb(220,220,170)">DMSetType</span><span lang="EN-US" style="font-size:9pt;font-family:Menlo;color:rgb(212,212,212)">(*swarm,DMSWARM);</span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif;line-height:13.5pt;background:rgb(30,30,30)"><span lang="EN-US" style="font-size:9pt;font-family:Menlo;color:rgb(212,212,212)">   <span> </span></span><span lang="EN-US" style="font-size:9pt;font-family:Menlo;color:rgb(220,220,170)">DMSetDimension</span><span lang="EN-US" style="font-size:9pt;font-family:Menlo;color:rgb(212,212,212)">(*swarm,</span><span lang="EN-US" style="font-size:9pt;font-family:Menlo;color:rgb(181,206,168)">3</span><span lang="EN-US" style="font-size:9pt;font-family:Menlo;color:rgb(212,212,212)">);</span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif;line-height:13.5pt;background:rgb(30,30,30)"><span lang="EN-US" style="font-size:9pt;font-family:Menlo;color:rgb(212,212,212)">   <span> </span></span><span lang="EN-US" style="font-size:9pt;font-family:Menlo;color:rgb(220,220,170)">DMSwarmSetType</span><span lang="EN-US" style="font-size:9pt;font-family:Menlo;color:rgb(212,212,212)">(*swarm,DMSWARM_PIC);</span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif;line-height:13.5pt;background:rgb(30,30,30)"><span lang="EN-US" style="font-size:9pt;font-family:Menlo;color:rgb(212,212,212)">   <span> </span></span><span lang="EN-US" style="font-size:9pt;font-family:Menlo;color:rgb(220,220,170)">DMSwarmSetCellDM</span><span lang="EN-US" style="font-size:9pt;font-family:Menlo;color:rgb(212,212,212)">(*swarm,*dmcell);</span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt;color:rgb(33,33,33)"> </span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt;color:rgb(33,33,33)"> </span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt;color:rgb(33,33,33)">Thanks a lot for your help. </span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div></div></div></div></blockquote><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt"><u></u> <u></u></span></div></div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt">Things that would help us track this down:<u></u><u></u></span></div></div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt"><u></u> <u></u></span></div></div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt">1) The smallest example where it fails<u></u><u></u></span></div></div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt"><u></u> <u></u></span></div></div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt">2) The smallest number of processes where it fails<u></u><u></u></span></div></div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt"><u></u> <u></u></span></div></div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt">3) A stack trace of the failure<u></u><u></u></span></div></div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt"><u></u> <u></u></span></div></div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt">4) A simple example that we can run that also fails<u></u><u></u></span></div></div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt"><u></u> <u></u></span></div></div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt">  Thanks,<u></u><u></u></span></div></div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt"><u></u> <u></u></span></div></div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt">     Matt<u></u><u></u></span></div></div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt"> <u></u><u></u></span></div></div><blockquote style="border-width:medium medium medium 1pt;border-style:none none none solid;border-color:currentcolor currentcolor currentcolor rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt"><div><div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt;color:rgb(33,33,33)">Best regards, </span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt;color:rgb(33,33,33)"> </span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt;color:rgb(33,33,33)">Joauma</span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div></div></div></div></blockquote></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt"><br clear="all"><u></u><u></u></span></div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt"><u></u> <u></u></span></div></div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span><span lang="EN-US" style="font-size:11pt">--</span></span><span lang="EN-US" style="font-size:11pt"><u></u><u></u></span></div><div><div><div><div><div><div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<u></u><u></u></span></div></div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><span lang="EN-US" style="font-size:11pt"><u></u> <u></u></span></div></div><div><div style="margin:0cm;font-size:10pt;font-family:Calibri,sans-serif"><a href="http://www.cse.buffalo.edu/~knepley/" style="color:blue;text-decoration:underline" target="_blank"><span lang="EN-US" style="font-size:11pt">https://www.cse.buffalo.edu/~knepley/</span></a></div></div></div></div></div></div></div></div></div></div></div></blockquote></div><br></div></div></blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div>