<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><br class=""></div><div class=""><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">Hi,</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> </span></div></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> I am running a code a nonlinear simulation using mesh-refinement on libMesh. The code runs without issues on a Mac (can run for days without issues), but crashes on Linux (Centos 6). I am using version 3.11 on Linux with openmpi 3.1.3 and gcc8.2. </span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> I tried to use the -on_error_attach_debugger, but it only gave me this message. Does this message imply something to the more experienced eyes? </span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> I am going to try to build a debug version of petsc to figure out what is going wrong. I will get and share more detailed logs in a bit. </span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">Regards,</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">Manav</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">[8]PETSC ERROR: ------------------------------------------------------------------------</span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">[8]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range</span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">[8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">[8]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" class="">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">[8]PETSC ERROR: or try <a href="http://valgrind.org" class="">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to find memory corruption errors</span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">[8]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run </span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">[8]PETSC ERROR: to get more information on the crash.</span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">[8]PETSC ERROR: User provided function() line 0 in unknown file </span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2108 on display localhost:10.0 on machine <a href="http://Warhawk1.HPC.MsState.Edu" class="">Warhawk1.HPC.MsState.Edu</a></span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2112 on display localhost:10.0 on machine <a href="http://Warhawk1.HPC.MsState.Edu" class="">Warhawk1.HPC.MsState.Edu</a></span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> 0 :INTERNAL Error: recvd root arrowhead </span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> 0 :not belonging to me. IARR,JARR= 67525 67525</span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> 0 :IROW_GRID,JCOL_GRID= 0 4</span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> 0 :MYROW, MYCOL= 0 0</span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> 0 :IPOSROOT,JPOSROOT= 92264688 92264688</span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">--------------------------------------------------------------------------</span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD</span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">with errorcode -99.</span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098); min-height: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""></span><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.</span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">You may or may not see output from other processes, depending on</span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">exactly when Open MPI kills them.</span></div><div style="margin: 0px; font-stretch: normal; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(242, 242, 242); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">--------------------------------------------------------------------------</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class=""></span></div></span></div></body></html>