<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> Please run with -malloc_debug option or even better run under Valgrind <a href="https://petsc.org/release/faq/">https://petsc.org/release/faq/</a><div><br></div><div><br id="lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Oct 26, 2023, at 10:35 AM, Joauma Marichal via petsc-users <petsc-users@mcs.anl.gov> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">Hello,<span class="Apple-converted-space"> </span><o:p></o:p></span></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US"><o:p> </o:p></span></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">Here is a very simple version where I have issues.</span></span><span lang="EN-US"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US"><o:p> </o:p></span></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">Which I run as follows:<o:p></o:p></span></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US"><o:p> </o:p></span></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">cd Grid_generation</span></span><span class="apple-converted-space"><span lang="EN-US"> </span></span><span lang="EN-US"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">make clean</span></span><span class="apple-converted-space"><span lang="EN-US"> </span></span><span lang="EN-US"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">make all</span></span><span lang="EN-US"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">./grid_generation</span></span><span class="apple-converted-space"><span lang="EN-US"> <o:p></o:p></span></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">cd ..</span></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">make clean</span></span><span class="apple-converted-space"><span lang="EN-US"> </span></span><span lang="EN-US"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">make all</span></span><span lang="EN-US"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">./cobpor</span></span><span class="apple-converted-space"><span lang="EN-US"> # on 1 proc<o:p></o:p></span></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="apple-converted-space"><span lang="EN-US"># OR<o:p></o:p></span></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span lang="EN-US">mpiexec ./cobpor<span class="Apple-converted-space"> </span></span><span lang="EN-US" style="color: rgb(200, 20, 201);">-ksp_type</span><span lang="EN-US"><span class="Apple-converted-space"> </span>cg<span class="Apple-converted-space"> </span></span><span lang="EN-US" style="color: rgb(200, 20, 201);">-pc_type</span><span lang="EN-US"><span class="Apple-converted-space"> </span>pfmg<span class="Apple-converted-space"> </span></span><span lang="EN-US" style="color: rgb(200, 20, 201);">-dm_mat_type</span><span lang="EN-US"><span class="Apple-converted-space"> </span>hyprestruct<span class="Apple-converted-space"> </span></span><span lang="EN-US" style="color: rgb(200, 20, 201);">-pc_pfmg_skip_relax</span><span lang="EN-US"><span class="Apple-converted-space"> </span></span><span lang="EN-US" style="color: rgb(180, 36, 25);">1</span><span lang="EN-US"><span class="Apple-converted-space"> </span></span><span lang="EN-US" style="color: rgb(200, 20, 201);">-pc_pfmg_rap_time</span><span lang="EN-US"><span class="Apple-converted-space"> </span>non-Galerkin<span class="Apple-converted-space"> </span><span class="apple-converted-space"># on multiple procs<o:p></o:p></span></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 10.5pt; font-family: Menlo; color: rgb(200, 20, 201);"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span lang="EN-US">The error that I get is the following:<o:p></o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">munmap_chunk(): invalid pointer</span></span><span lang="EN-US"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">[cns266:2552391] *** Process received signal ***</span></span><span lang="EN-US"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">[cns266:2552391] Signal: Aborted (6)</span></span><span lang="EN-US"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">[cns266:2552391] Signal code:</span></span><span class="apple-converted-space"><span lang="EN-US"> <span class="Apple-converted-space"> </span></span></span><span class="s1"><span lang="EN-US">(-6)</span></span><span lang="EN-US"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">[cns266:2552391] [ 0] /lib64/<a href="http://libc.so/" style="color: blue; text-decoration: underline;">libc.so</a>.6(+0x4eb20)[0x7fd7fd194b20]</span></span><span lang="EN-US"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">[cns266:2552391] [ 1] /lib64/<a href="http://libc.so/" style="color: blue; text-decoration: underline;">libc.so</a>.6(gsignal+0x10f)[0x7fd7fd194a9f]</span></span><span lang="EN-US"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">[cns266:2552391] [ 2] /lib64/<a href="http://libc.so/" style="color: blue; text-decoration: underline;">libc.so</a>.6(abort+0x127)[0x7fd7fd167e05]</span></span><span lang="EN-US"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">[cns266:2552391] [ 3] /lib64/<a href="http://libc.so/" style="color: blue; text-decoration: underline;">libc.so</a>.6(+0x91037)[0x7fd7fd1d7037]</span></span><span lang="EN-US"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1">[cns266:2552391] [ 4] /lib64/<a href="http://libc.so/" style="color: blue; text-decoration: underline;">libc.so</a>.6(+0x9819c)[0x7fd7fd1de19c]</span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1">[cns266:2552391] [ 5] /lib64/<a href="http://libc.so/" style="color: blue; text-decoration: underline;">libc.so</a>.6(+0x9844c)[0x7fd7fd1de44c]</span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1">[cns266:2552391] [ 6] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/<a href="http://libpetsc.so/" style="color: blue; text-decoration: underline;">libpetsc.so</a>.3.019(PetscFreeAlign+0xe)[0x7fd7fe63d50e]</span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1">[cns266:2552391] [ 7] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/<a href="http://libpetsc.so/" style="color: blue; text-decoration: underline;">libpetsc.so</a>.3.019(DMSetMatType+0x3d)[0x7fd7feab87ad]</span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1">[cns266:2552391] [ 8] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/<a href="http://libpetsc.so/" style="color: blue; text-decoration: underline;">libpetsc.so</a>.3.019(DMSetFromOptions+0x109)[0x7fd7feab8b59]</span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1">[cns266:2552391] [ 9] ./cobpor[0x402df9]</span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1">[cns266:2552391] [10] /lib64/<a href="http://libc.so/" style="color: blue; text-decoration: underline;">libc.so</a>.6(__libc_start_main+0xf3)[0x7fd7fd180cf3]</span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">[cns266:2552391] [11] ./cobpor[0x40304e]</span></span><span lang="EN-US"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span class="s1"><span lang="EN-US">[cns266:2552391] *** End of error message ***</span></span><span lang="EN-US"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span lang="EN-US"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span lang="EN-US"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span lang="EN-US">Thanks a lot for your help.<span class="Apple-converted-space"> </span><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span lang="EN-US"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span lang="EN-US">Best regards,<span class="Apple-converted-space"> </span><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span lang="EN-US"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span lang="EN-US">Joauma<o:p></o:p></span></div><div style="margin: 0cm; font-size: 10.5pt; font-family: Menlo;"><span lang="EN-US"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"><o:p> </o:p></span></div><div style="border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(181, 196, 223) currentcolor currentcolor; border-image: none; padding: 3pt 0cm 0cm;"><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 10pt; font-family: Calibri, sans-serif;"><b><span lang="EN-US" style="font-size: 12pt;">De :<span class="Apple-converted-space"> </span></span></b><span lang="EN-US" style="font-size: 12pt;">Matthew Knepley <<a href="mailto:knepley@gmail.com" style="color: blue; text-decoration: underline;">knepley@gmail.com</a>><br><b>Date :<span class="Apple-converted-space"> </span></b>mercredi, 25 octobre 2023 à 14:45<br><b>À :<span class="Apple-converted-space"> </span></b>Joauma Marichal <<a href="mailto:joauma.marichal@uclouvain.be" style="color: blue; text-decoration: underline;">joauma.marichal@uclouvain.be</a>><br><b>Cc :<span class="Apple-converted-space"> </span></b><a href="mailto:petsc-maint@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-maint@mcs.anl.gov</a><span class="Apple-converted-space"> </span><<a href="mailto:petsc-maint@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-maint@mcs.anl.gov</a>>,<span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a><span class="Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a>><br><b>Objet :<span class="Apple-converted-space"> </span></b>Re: [petsc-maint] DMSwarm on multiple processors<o:p></o:p></span></p></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">On Wed, Oct 25, 2023 at 8:32 AM Joauma Marichal via petsc-maint <</span><a href="mailto:petsc-maint@mcs.anl.gov" style="color: blue; text-decoration: underline;"><span lang="EN-US" style="font-size: 11pt;">petsc-maint@mcs.anl.gov</span></a><span lang="EN-US" style="font-size: 11pt;">> wrote:<o:p></o:p></span></div></div><div><blockquote style="border-width: medium medium medium 1pt; border-style: none none none solid; border-color: currentcolor currentcolor currentcolor rgb(204, 204, 204); border-image: none; padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;"><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt; color: rgb(33, 33, 33);">Hello, </span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt; color: rgb(33, 33, 33);"> </span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt; color: rgb(33, 33, 33);">I am using the DMSwarm library in some Eulerian-Lagrangian approach to have vapor bubbles in water. </span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt; color: rgb(33, 33, 33);">I have obtained nice results recently and wanted to perform bigger simulations. Unfortunately, when I increase the number of processors used to run the simulation, I get the following error:</span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt; color: rgb(33, 33, 33);"> </span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">free(): invalid size</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] *** Process received signal ***</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] Signal: Aborted (6)</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] Signal code:</span></span><span class="m-1889958021554448809apple-converted-space"><span lang="EN-US"> <span class="Apple-converted-space"> </span></span></span><span class="m-1889958021554448809s1"><span lang="EN-US">(-6)</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [ 0] /lib64/<a href="http://libc.so/" style="color: blue; text-decoration: underline;">libc.so</a>.6(+0x4eb20)[0x7f56cd4c9b20]</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [ 1] /lib64/<a href="http://libc.so/" style="color: blue; text-decoration: underline;">libc.so</a>.6(gsignal+0x10f)[0x7f56cd4c9a9f]</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [ 2] /lib64/<a href="http://libc.so/" style="color: blue; text-decoration: underline;">libc.so</a>.6(abort+0x127)[0x7f56cd49ce05]</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [ 3] /lib64/<a href="http://libc.so/" style="color: blue; text-decoration: underline;">libc.so</a>.6(+0x91037)[0x7f56cd50c037]</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [ 4] /lib64/<a href="http://libc.so/" style="color: blue; text-decoration: underline;">libc.so</a>.6(+0x9819c)[0x7f56cd51319c]</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [ 5] /lib64/<a href="http://libc.so/" style="color: blue; text-decoration: underline;">libc.so</a>.6(+0x99aac)[0x7f56cd514aac]</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [ 6] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/<a href="http://libpetsc.so/" style="color: blue; text-decoration: underline;">libpetsc.so</a>.3.019(PetscSFSetUpRanks+0x4c4)[0x7f56cea71e64]</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [ 7] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/<a href="http://libpetsc.so/" style="color: blue; text-decoration: underline;">libpetsc.so</a>.3.019(+0x841642)[0x7f56cea83642]</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [ 8] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/<a href="http://libpetsc.so/" style="color: blue; text-decoration: underline;">libpetsc.so</a>.3.019(PetscSFSetUp+0x9e)[0x7f56cea7043e]</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [ 9] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/<a href="http://libpetsc.so/" style="color: blue; text-decoration: underline;">libpetsc.so</a>.3.019(VecScatterCreate+0x164e)[0x7f56cea7bbde]</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [10] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/<a href="http://libpetsc.so/" style="color: blue; text-decoration: underline;">libpetsc.so</a>.3.019(DMSetUp_DA_3D+0x3e38)[0x7f56cee84dd8]</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [11] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/<a href="http://libpetsc.so/" style="color: blue; text-decoration: underline;">libpetsc.so</a>.3.019(DMSetUp_DA+0xd8)[0x7f56cee9b448]</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [12] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/<a href="http://libpetsc.so/" style="color: blue; text-decoration: underline;">libpetsc.so</a>.3.019(DMSetUp+0x20)[0x7f56cededa20]</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [13] ./cobpor[0x4418dc]</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [14] ./cobpor[0x408b63]</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [15] /lib64/<a href="http://libc.so/" style="color: blue; text-decoration: underline;">libc.so</a>.6(__libc_start_main+0xf3)[0x7f56cd4b5cf3]</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] [16] ./cobpor[0x40bdee]</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">[cns136:590327] *** End of error message ***</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">--------------------------------------------------------------------------</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">Primary job</span></span><span class="m-1889958021554448809apple-converted-space"><span lang="EN-US"> <span class="Apple-converted-space"> </span></span></span><span class="m-1889958021554448809s1"><span lang="EN-US">terminated normally, but 1 process returned</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1">a non-zero exit code.<span class="Apple-converted-space"> </span></span><span class="m-1889958021554448809s1"><span lang="EN-US">Per user-direction, the job has been aborted.</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">--------------------------------------------------------------------------</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">--------------------------------------------------------------------------</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">mpiexec noticed that process rank 84 with PID 590327 on node cns136 exited on signal 6 (Aborted).</span></span><span lang="EN-US"><o:p></o:p></span></p><p class="m-1889958021554448809p1" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="m-1889958021554448809s1"><span lang="EN-US">--------------------------------------------------------------------------</span></span><span lang="EN-US"><o:p></o:p></span></p><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt; color: rgb(33, 33, 33);"> </span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt; color: rgb(33, 33, 33);">When I reduce the number of processors the error disappears and when I run my code without the vapor bubbles it also works.</span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt; color: rgb(33, 33, 33);">The problem seems to take place at this moment:</span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt; color: rgb(33, 33, 33);"> </span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif; line-height: 13.5pt; background: rgb(30, 30, 30);"><span lang="EN-US" style="font-size: 9pt; font-family: Menlo; color: rgb(220, 220, 170);">DMCreate</span><span lang="EN-US" style="font-size: 9pt; font-family: Menlo; color: rgb(212, 212, 212);">(PETSC_COMM_WORLD,swarm);</span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif; line-height: 13.5pt; background: rgb(30, 30, 30);"><span lang="EN-US" style="font-size: 9pt; font-family: Menlo; color: rgb(212, 212, 212);"> <span class="Apple-converted-space"> </span></span><span lang="EN-US" style="font-size: 9pt; font-family: Menlo; color: rgb(220, 220, 170);">DMSetType</span><span lang="EN-US" style="font-size: 9pt; font-family: Menlo; color: rgb(212, 212, 212);">(*swarm,DMSWARM);</span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif; line-height: 13.5pt; background: rgb(30, 30, 30);"><span lang="EN-US" style="font-size: 9pt; font-family: Menlo; color: rgb(212, 212, 212);"> <span class="Apple-converted-space"> </span></span><span lang="EN-US" style="font-size: 9pt; font-family: Menlo; color: rgb(220, 220, 170);">DMSetDimension</span><span lang="EN-US" style="font-size: 9pt; font-family: Menlo; color: rgb(212, 212, 212);">(*swarm,</span><span lang="EN-US" style="font-size: 9pt; font-family: Menlo; color: rgb(181, 206, 168);">3</span><span lang="EN-US" style="font-size: 9pt; font-family: Menlo; color: rgb(212, 212, 212);">);</span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif; line-height: 13.5pt; background: rgb(30, 30, 30);"><span lang="EN-US" style="font-size: 9pt; font-family: Menlo; color: rgb(212, 212, 212);"> <span class="Apple-converted-space"> </span></span><span lang="EN-US" style="font-size: 9pt; font-family: Menlo; color: rgb(220, 220, 170);">DMSwarmSetType</span><span lang="EN-US" style="font-size: 9pt; font-family: Menlo; color: rgb(212, 212, 212);">(*swarm,DMSWARM_PIC);</span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif; line-height: 13.5pt; background: rgb(30, 30, 30);"><span lang="EN-US" style="font-size: 9pt; font-family: Menlo; color: rgb(212, 212, 212);"> <span class="Apple-converted-space"> </span></span><span lang="EN-US" style="font-size: 9pt; font-family: Menlo; color: rgb(220, 220, 170);">DMSwarmSetCellDM</span><span lang="EN-US" style="font-size: 9pt; font-family: Menlo; color: rgb(212, 212, 212);">(*swarm,*dmcell);</span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt; color: rgb(33, 33, 33);"> </span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt; color: rgb(33, 33, 33);"> </span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt; color: rgb(33, 33, 33);">Thanks a lot for your help. </span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div></div></div></div></blockquote><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"><o:p> </o:p></span></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">Things that would help us track this down:<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"><o:p> </o:p></span></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">1) The smallest example where it fails<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"><o:p> </o:p></span></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">2) The smallest number of processes where it fails<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"><o:p> </o:p></span></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">3) A stack trace of the failure<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"><o:p> </o:p></span></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">4) A simple example that we can run that also fails<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"><o:p> </o:p></span></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> Thanks,<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"><o:p> </o:p></span></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> Matt<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"> <o:p></o:p></span></div></div><blockquote style="border-width: medium medium medium 1pt; border-style: none none none solid; border-color: currentcolor currentcolor currentcolor rgb(204, 204, 204); border-image: none; padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;"><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt; color: rgb(33, 33, 33);">Best regards, </span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt; color: rgb(33, 33, 33);"> </span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt; color: rgb(33, 33, 33);">Joauma</span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div></div></div></div></blockquote></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"><br clear="all"><o:p></o:p></span></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"><o:p> </o:p></span></div></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span class="gmailsignatureprefix"><span lang="EN-US" style="font-size: 11pt;">--</span></span><span lang="EN-US" style="font-size: 11pt;"><o:p></o:p></span></div><div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span lang="EN-US" style="font-size: 11pt;"><o:p> </o:p></span></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" style="color: blue; text-decoration: underline;"><span lang="EN-US" style="font-size: 11pt;">https://www.cse.buffalo.edu/~knepley/</span></a></div></div></div></div></div></div></div></div></div></div></div></blockquote></div><br></div></body></html>