<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 8, 2019 at 2:23 PM Manav Bhatia <<a href="mailto:bhatiamanav@gmail.com">bhatiamanav@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">Thanks for identifying this, Mark. <div><br></div><div>If I compile the debug version of Petsc, will it also build a debug version of Mumps? <br></div></div></blockquote><div><br></div><div>The debug compiler flags will get passed down to MUMPS if you are downloading MUMPS in PETSc. Otherwise, yes build a debug version.</div><div><br></div><div>Are you able to run the exact same job on your Mac? ie, same number of processes, etc.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div><div><br><blockquote type="cite"><div>On Apr 8, 2019, at 12:58 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:</div><br class="gmail-m_-6318203121302555419Apple-interchange-newline"><div><div dir="ltr">This looks like an error in MUMPS:<br><div><br></div><div><pre style="white-space:pre-wrap"> IF ( IROW_GRID .NE. root%MYROW .OR.
& JCOL_GRID .NE. root%MYCOL ) THEN
WRITE(*,*) MYID,':INTERNAL Error: recvd root arrowhead '</pre></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 8, 2019 at 1:37 PM Smith, Barry F. via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Difficult to tell what is going on. <br>
<br>
The message User provided function() line 0 in unknown file indicates the crash took place OUTSIDE of PETSc code and error message INTERNAL Error: recvd root arrowhead is definitely not coming from PETSc. <br>
<br>
Yes, debug with the debug version and also try valgrind.<br>
<br>
Barry<br>
<br>
<br>
> On Apr 8, 2019, at 12:12 PM, Manav Bhatia via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
> <br>
> <br>
> Hi,<br>
> <br>
> I am running a code a nonlinear simulation using mesh-refinement on libMesh. The code runs without issues on a Mac (can run for days without issues), but crashes on Linux (Centos 6). I am using version 3.11 on Linux with openmpi 3.1.3 and gcc8.2. <br>
> <br>
> I tried to use the -on_error_attach_debugger, but it only gave me this message. Does this message imply something to the more experienced eyes? <br>
> <br>
> I am going to try to build a debug version of petsc to figure out what is going wrong. I will get and share more detailed logs in a bit. <br>
> <br>
> Regards,<br>
> Manav<br>
> <br>
> [8]PETSC ERROR: ------------------------------------------------------------------------<br>
> [8]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range<br>
> [8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
> [8]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a><br>
> [8]PETSC ERROR: or try <a href="http://valgrind.org/" rel="noreferrer" target="_blank">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to find memory corruption errors<br>
> [8]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run <br>
> [8]PETSC ERROR: to get more information on the crash.<br>
> [8]PETSC ERROR: User provided function() line 0 in unknown file <br>
> PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2108 on display localhost:10.0 on machine <a href="http://warhawk1.hpc.msstate.edu/" rel="noreferrer" target="_blank">Warhawk1.HPC.MsState.Edu</a><br>
> PETSC: Attaching gdb to /cavs/projects/brg_codes/users/bhatia/mast/mast_topology/opt/examples/structural/example_5/structural_example_5 of pid 2112 on display localhost:10.0 on machine <a href="http://warhawk1.hpc.msstate.edu/" rel="noreferrer" target="_blank">Warhawk1.HPC.MsState.Edu</a><br>
> 0 :INTERNAL Error: recvd root arrowhead <br>
> 0 :not belonging to me. IARR,JARR= 67525 67525<br>
> 0 :IROW_GRID,JCOL_GRID= 0 4<br>
> 0 :MYROW, MYCOL= 0 0<br>
> 0 :IPOSROOT,JPOSROOT= 92264688 92264688<br>
> --------------------------------------------------------------------------<br>
> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD<br>
> with errorcode -99.<br>
> <br>
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.<br>
> You may or may not see output from other processes, depending on<br>
> exactly when Open MPI kills them.<br>
> --------------------------------------------------------------------------<br>
> <br>
<br>
</blockquote></div>
</div></blockquote></div><br></div></div></blockquote></div></div>