<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">Gideon :<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">running with -mat_mumps_icntl_7 4 got it to run on problems that it couldn’t do before, thanks. How should I understand how this choice of flag is impacting whether or not it stalls? </div></blockquote><div>Good to know that your code starts running :-)</div><div>I've never encountered hang with mumps. It turns that the hang occurred in metis, which I rarely use. </div><div><br></div><div>Debugging what causes hang, I usually run code with a debugger, e.g. use opiton '-start_in_debugger', when code hangs, hit control^C to see where it hangs. </div><div><br></div><div>Hong</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div class="h5">
<br><div><blockquote type="cite"><div>On Aug 25, 2015, at 4:35 PM, Hong <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>> wrote:</div><br><div><div dir="ltr" style="font-family:Helvetica;font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div class="gmail_extra"><div class="gmail_quote">Gideon :<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div><br></div><div>I ran with that flag because, while solving a SNES with MUMPS, the code would just sit there as though it had died, and never seem to recover. I tried using that flag just to determine where it had stalled, which was at the "ordering based on METIS” bit.</div></div></blockquote><div> </div><div>If you suspect METIS/ParMetis hangs,</div><div>then turn to other sequential matrix orderings, e.g., </div><div>' -mat_mumps_icntl_29 0 -mat_mumps_icntl_7 2', which I found the most robust ordering.</div><div>Run your code with '-help |grep mumps', it will display mumps options.</div><div><br></div><div>Hong</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div><span><font color="#888888"><br><div><span style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">-gideon</span></div></font></span><div><div><br><div><blockquote type="cite"><div>On Aug 25, 2015, at 12:24 PM, Hong <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>> wrote:</div><br><div><div dir="ltr"><div>Gideon:<br></div>-mat_mumps_icntl_4 <0>: ICNTL(4): level of printing (0 to 4) (None)<div>This is for algorithmic diagnosis, not for regular runs. Use default '0' for it.</div><div><br></div><div>Hong<br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 25, 2015 at 9:06 AM, Gideon Simpson<span> </span><span dir="ltr"><<a href="mailto:gideon.simpson@gmail.com" target="_blank">gideon.simpson@gmail.com</a>></span><span> </span>wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Regarding the MUMPS issue, I’m not sure if this is useful, but when I run with the mumps flags -mat_mumps_icntl_4 4, to see the progress, it hangs at this point:<br><br><br> ... Structural symmetry (in percent)= 75<br> Density: NBdense, Average, Median = 2 9 7<br> Ordering based on METIS<br><span><br>-gideon<br><br>> On Aug 22, 2015, at 5:12 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>><br>><br></span><div><div>>> On Aug 22, 2015, at 4:04 PM, Gideon Simpson <<a href="mailto:gideon.simpson@gmail.com" target="_blank">gideon.simpson@gmail.com</a>> wrote:<br>>><br>>> I’m having issues with both SuperLU dist and MUMPS, as compiled by PETsc, in the following sense:<br>>><br>>> 1. For large enough systems, which seems to vary depending on which computer I’m on, MUMPS seems to just die and never start, when it’s used as the linear solver within a SNES. There’s no error message, it just sits there and doesn’t do anything.<br>><br>> You will need to use a debugger to figure out where it is "hanging"; we haven't heard reports about this.<br>>><br>>> 2. When running with SuperLU dist, I got the following error, with no further information:<br>><br>> The last release of SuperLU_DIST had some pretty nasty bugs, memory corruption that caused crashes etc. We think they are now fixed if you use the maint branch of the PETSc repository and --download-superlu_dist If you stick with the PETSc release and SuperLU_Dist you are using you will keep seeing these crashes<br>><br>> Barry<br>><br>><br>>><br>>> [3]PETSC ERROR: ------------------------------------------------------------------------<br>>> [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range<br>>> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>>> [3]PETSC ERROR: or see<span> </span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a><br>>> [3]PETSC ERROR: or try<span> </span><a href="http://valgrind.org/" rel="noreferrer" target="_blank">http://valgrind.org</a><span> </span>on GNU/linux and Apple Mac OS X to find memory corruption errors<br>>> [3]PETSC ERROR: likely location of problem given in stack below<br>>> [3]PETSC ERROR: --------------------- Stack Frames ------------------------------------<br>>> [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>>> [3]PETSC ERROR: INSTEAD the line number of the start of the function<br>>> [3]PETSC ERROR: is given.<br>>> [3]PETSC ERROR: [3] SuperLU_DIST:pdgssvx line 161 /home/simpson/software/petsc-3.5.4/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c<br>>> [3]PETSC ERROR: [3] MatSolve_SuperLU_DIST line 121 /home/simpson/software/petsc-3.5.4/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c<br>>> [3]PETSC ERROR: [3] MatSolve line 3104 /home/simpson/software/petsc-3.5.4/src/mat/interface/matrix.c<br>>> [3]PETSC ERROR: [3] PCApply_LU line 194 /home/simpson/software/petsc-3.5.4/src/ksp/pc/impls/factor/lu/lu.c<br>>> [3]PETSC ERROR: [3] KSP_PCApplyBAorAB line 258 /home/simpson/software/petsc-3.5.4/include/petsc-private/kspimpl.h<br>>> [3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>>> [3]PETSC ERROR: Signal received<br>>> [3]PETSC ERROR: See<span> </span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><span> </span>for trouble shooting.<br>>> [3]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015<br>>> [3]PETSC ERROR: ./blowup_batch2 on a arch-linux2-c-debug named proteusi01 by simpson Sat Aug 22 17:01:41 2015<br>>> [3]PETSC ERROR: Configure options --with-mpi-dir=/mnt/HA/opt/openmpi/intel/64/1.8.1-mlnx-ofed --with-blas-lib=/mnt/HA/opt/blas/gcc/64/20110419/libblas.a --with-lapack-lib=/liblapack.a --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-metis=yes --download-parmetis=yes --download-scalapack=yes<br>>> [3]PETSC ERROR: #1 User provided function() line 0 in unknown file<br>>> --------------------------------------------------------------------------<br>>> MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD<br>>> with errorcode 59.<br>>><br>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.<br>>> You may or may not see output from other processes, depending on<br>>> exactly when Open MPI kills them.<br>>> --------------------------------------------------------------------------<br>>> [proteusi01:14037] 1 more process has sent help message help-mpi-api.txt / mpi-abort<br>>> [proteusi01:14037] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages<br>>> [6]PETSC ERROR: ------------------------------------------------------------------------<br>>> [6]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>>> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>>> [6]PETSC ERROR: or see<span> </span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a><br>>> [6]PETSC ERROR: or try<span> </span><a href="http://valgrind.org/" rel="noreferrer" target="_blank">http://valgrind.org</a><span> </span>on GNU/linux and Apple Mac OS X to find memory corruption errors<br>>> [6]PETSC ERROR: likely location of problem given in stack below<br>>> [6]PETSC ERROR: --------------------- Stack Frames ------------------------------------<br>>> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>>> [6]PETSC ERROR: INSTEAD the line number of the start of the function<br>>> [6]PETSC ERROR: is given.<br>>> [6]PETSC ERROR: [6] SuperLU_DIST:pdgssvx line 161 /home/simpson/software/petsc-3.5.4/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c<br>>> [6]PETSC ERROR: [6] MatSolve_SuperLU_DIST line 121 /home/simpson/software/petsc-3.5.4/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c<br>>> [6]PETSC ERROR: [6] MatSolve line 3104 /home/simpson/software/petsc-3.5.4/src/mat/interface/matrix.c<br>>> [6]PETSC ERROR: [6] PCApply_LU line 194 /home/simpson/software/petsc-3.5.4/src/ksp/pc/impls/factor/lu/lu.c<br>>> [6]PETSC ERROR: [6] KSP_PCApplyBAorAB line 258 /home/simpson/software/petsc-3.5.4/include/petsc-private/kspimpl.h<br>>> [6]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>>> [6]PETSC ERROR: Signal received<br>>> [6]PETSC ERROR: See<span> </span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><span> </span>for trouble shooting.<br>>> [6]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015<br>>> [6]PETSC ERROR: ./blowup_batch2 on a arch-linux2-c-debug named proteusi01 by simpson Sat Aug 22 17:01:41 2015<br>>> [6]PETSC ERROR: Configure options --with-mpi-dir=/mnt/HA/opt/openmpi/intel/64/1.8.1-mlnx-ofed --with-blas-lib=/mnt/HA/opt/blas/gcc/64/20110419/libblas.a --with-lapack-lib=/liblapack.a --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-metis=yes --download-parmetis=yes --download-scalapack=yes<br>>> [6]PETSC ERROR: #1 User provided function() line 0 in unknown file<br>>> [7]PETSC ERROR: ------------------------------------------------------------------------<br>>> [7]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>>> [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>>> [7]PETSC ERROR: or see<span> </span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a><br>>> [7]PETSC ERROR: or try<span> </span><a href="http://valgrind.org/" rel="noreferrer" target="_blank">http://valgrind.org</a><span> </span>on GNU/linux and Apple Mac OS X to find memory corruption errors<br>>> [7]PETSC ERROR: likely location of problem given in stack below<br>>> [7]PETSC ERROR: --------------------- Stack Frames ------------------------------------<br>>> [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>>> [7]PETSC ERROR: INSTEAD the line number of the start of the function<br>>> [7]PETSC ERROR: is given.<br>>> [7]PETSC ERROR: [7] SuperLU_DIST:pdgssvx line 161 /home/simpson/software/petsc-3.5.4/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c<br>>> [7]PETSC ERROR: [7] MatSolve_SuperLU_DIST line 121 /home/simpson/software/petsc-3.5.4/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c<br>>> [7]PETSC ERROR: [7] MatSolve line 3104 /home/simpson/software/petsc-3.5.4/src/mat/interface/matrix.c<br>>> [7]PETSC ERROR: [7] PCApply_LU line 194 /home/simpson/software/petsc-3.5.4/src/ksp/pc/impls/factor/lu/lu.c<br>>> [7]PETSC ERROR: [7] KSP_PCApplyBAorAB line 258 /home/simpson/software/petsc-3.5.4/include/petsc-private/kspimpl.h<br>>> [7]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>>> [7]PETSC ERROR: Signal received<br>>> [7]PETSC ERROR: See<span> </span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><span> </span>for trouble shooting.<br>>> [7]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015<br>>> [7]PETSC ERROR: ./blowup_batch2 on a arch-linux2-c-debug named proteusi01 by simpson Sat Aug 22 17:01:41 2015<br>>> [7]PETSC ERROR: Configure options --with-mpi-dir=/mnt/HA/opt/openmpi/intel/64/1.8.1-mlnx-ofed --with-blas-lib=/mnt/HA/opt/blas/gcc/64/20110419/libblas.a --with-lapack-lib=/liblapack.a --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-metis=yes --download-parmetis=yes --download-scalapack=yes<br>>> [7]PETSC ERROR: #1 User provided function() line 0 in unknown file<br>>> [0]PETSC ERROR: ------------------------------------------------------------------------<br>>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>>> [0]PETSC ERROR: or see<span> </span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a><br>>> [0]PETSC ERROR: or try<span> </span><a href="http://valgrind.org/" rel="noreferrer" target="_blank">http://valgrind.org</a><span> </span>on GNU/linux and Apple Mac OS X to find memory corruption errors<br>>> [0]PETSC ERROR: likely location of problem given in stack below<br>>> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------<br>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function<br>>> [0]PETSC ERROR: is given.<br>>> [0]PETSC ERROR: [0] SuperLU_DIST:pdgssvx line 161 /home/simpson/software/petsc-3.5.4/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c<br>>> [0]PETSC ERROR: [0] MatSolve_SuperLU_DIST line 121 /home/simpson/software/petsc-3.5.4/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c<br>>> [0]PETSC ERROR: [0] MatSolve line 3104 /home/simpson/software/petsc-3.5.4/src/mat/interface/matrix.c<br>>> [0]PETSC ERROR: [0] PCApply_LU line 194 /home/simpson/software/petsc-3.5.4/src/ksp/pc/impls/factor/lu/lu.c<br>>> [0]PETSC ERROR: [0] KSP_PCApplyBAorAB line 258 /home/simpson/software/petsc-3.5.4/include/petsc-private/kspimpl.h<br>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>>> [0]PETSC ERROR: Signal received<br>>> [0]PETSC ERROR: See<span> </span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><span> </span>for trouble shooting.<br>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015<br>>> [0]PETSC ERROR: ./blowup_batch2 on a arch-linux2-c-debug named proteusi01 by simpson Sat Aug 22 17:01:41 2015<br>>> [0]PETSC ERROR: Configure options --with-mpi-dir=/mnt/HA/opt/openmpi/intel/64/1.8.1-mlnx-ofed --with-blas-lib=/mnt/HA/opt/blas/gcc/64/20110419/libblas.a --with-lapack-lib=/liblapack.a --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-metis=yes --download-parmetis=yes --download-scalapack=yes<br>>> [0]PETSC ERROR: #1 User provided function() line 0 in unknown file<br>>> [1]PETSC ERROR: ------------------------------------------------------------------------<br>>> [1]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>>> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>>> [1]PETSC ERROR: or see<span> </span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a><br>>> [1]PETSC ERROR: or try<span> </span><a href="http://valgrind.org/" rel="noreferrer" target="_blank">http://valgrind.org</a><span> </span>on GNU/linux and Apple Mac OS X to find memory corruption errors<br>>> [1]PETSC ERROR: likely location of problem given in stack below<br>>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------<br>>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>>> [1]PETSC ERROR: INSTEAD the line number of the start of the function<br>>> [1]PETSC ERROR: is given.<br>>> [1]PETSC ERROR: [1] SuperLU_DIST:pdgssvx line 161 /home/simpson/software/petsc-3.5.4/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c<br>>> [1]PETSC ERROR: [1] MatSolve_SuperLU_DIST line 121 /home/simpson/software/petsc-3.5.4/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c<br>>> [1]PETSC ERROR: [1] MatSolve line 3104 /home/simpson/software/petsc-3.5.4/src/mat/interface/matrix.c<br>>> [1]PETSC ERROR: [1] PCApply_LU line 194 /home/simpson/software/petsc-3.5.4/src/ksp/pc/impls/factor/lu/lu.c<br>>> [1]PETSC ERROR: [1] KSP_PCApplyBAorAB line 258 /home/simpson/software/petsc-3.5.4/include/petsc-private/kspimpl.h<br>>> [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>>> [1]PETSC ERROR: Signal received<br>>> [1]PETSC ERROR: See<span> </span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><span> </span>for trouble shooting.<br>>> [1]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015<br>>> [1]PETSC ERROR: ./blowup_batch2 on a arch-linux2-c-debug named proteusi01 by simpson Sat Aug 22 17:01:41 2015<br>>> [1]PETSC ERROR: Configure options --with-mpi-dir=/mnt/HA/opt/openmpi/intel/64/1.8.1-mlnx-ofed --with-blas-lib=/mnt/HA/opt/blas/gcc/64/20110419/libblas.a --with-lapack-lib=/liblapack.a --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-metis=yes --download-parmetis=yes --download-scalapack=yes<br>>> [1]PETSC ERROR: #1 User provided function() line 0 in unknown file<br>>> [2]PETSC ERROR: ------------------------------------------------------------------------<br>>> [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>>> [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>>> [2]PETSC ERROR: or see<span> </span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a><br>>> [2]PETSC ERROR: or try<span> </span><a href="http://valgrind.org/" rel="noreferrer" target="_blank">http://valgrind.org</a><span> </span>on GNU/linux and Apple Mac OS X to find memory corruption errors<br>>> [2]PETSC ERROR: likely location of problem given in stack below<br>>> [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------<br>>> [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>>> [2]PETSC ERROR: INSTEAD the line number of the start of the function<br>>> [2]PETSC ERROR: is given.<br>>> [2]PETSC ERROR: [2] SuperLU_DIST:pdgssvx line 161 /home/simpson/software/petsc-3.5.4/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c<br>>> [2]PETSC ERROR: [2] MatSolve_SuperLU_DIST line 121 /home/simpson/software/petsc-3.5.4/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c<br>>> [2]PETSC ERROR: [2] MatSolve line 3104 /home/simpson/software/petsc-3.5.4/src/mat/interface/matrix.c<br>>> [2]PETSC ERROR: [2] PCApply_LU line 194 /home/simpson/software/petsc-3.5.4/src/ksp/pc/impls/factor/lu/lu.c<br>>> [2]PETSC ERROR: [2] KSP_PCApplyBAorAB line 258 /home/simpson/software/petsc-3.5.4/include/petsc-private/kspimpl.h<br>>> [2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>>> [2]PETSC ERROR: Signal received<br>>> [2]PETSC ERROR: See<span> </span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><span> </span>for trouble shooting.<br>>> [2]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015<br>>> [2]PETSC ERROR: ./blowup_batch2 on a arch-linux2-c-debug named proteusi01 by simpson Sat Aug 22 17:01:41 2015<br>>> [2]PETSC ERROR: Configure options --with-mpi-dir=/mnt/HA/opt/openmpi/intel/64/1.8.1-mlnx-ofed --with-blas-lib=/mnt/HA/opt/blas/gcc/64/20110419/libblas.a --with-lapack-lib=/liblapack.a --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-metis=yes --download-parmetis=yes --download-scalapack=yes<br>>> [2]PETSC ERROR: #1 User provided function() line 0 in unknown file<br>>> [4]PETSC ERROR: ------------------------------------------------------------------------<br>>> [4]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>>> [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>>> [4]PETSC ERROR: or see<span> </span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a><br>>> [4]PETSC ERROR: or try<span> </span><a href="http://valgrind.org/" rel="noreferrer" target="_blank">http://valgrind.org</a><span> </span>on GNU/linux and Apple Mac OS X to find memory corruption errors<br>>> [4]PETSC ERROR: likely location of problem given in stack below<br>>> [4]PETSC ERROR: --------------------- Stack Frames ------------------------------------<br>>> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>>> [4]PETSC ERROR: INSTEAD the line number of the start of the function<br>>> [4]PETSC ERROR: is given.<br>>> [4]PETSC ERROR: [4] SuperLU_DIST:pdgssvx line 161 /home/simpson/software/petsc-3.5.4/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c<br>>> [4]PETSC ERROR: [4] MatSolve_SuperLU_DIST line 121 /home/simpson/software/petsc-3.5.4/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c<br>>> [4]PETSC ERROR: [4] MatSolve line 3104 /home/simpson/software/petsc-3.5.4/src/mat/interface/matrix.c<br>>> [4]PETSC ERROR: [4] PCApply_LU line 194 /home/simpson/software/petsc-3.5.4/src/ksp/pc/impls/factor/lu/lu.c<br>>> [4]PETSC ERROR: [4] KSP_PCApplyBAorAB line 258 /home/simpson/software/petsc-3.5.4/include/petsc-private/kspimpl.h<br>>> [4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>>> [4]PETSC ERROR: Signal received<br>>> [4]PETSC ERROR: See<span> </span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><span> </span>for trouble shooting.<br>>> [4]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015<br>>> [4]PETSC ERROR: ./blowup_batch2 on a arch-linux2-c-debug named proteusi01 by simpson Sat Aug 22 17:01:41 2015<br>>> [4]PETSC ERROR: Configure options --with-mpi-dir=/mnt/HA/opt/openmpi/intel/64/1.8.1-mlnx-ofed --with-blas-lib=/mnt/HA/opt/blas/gcc/64/20110419/libblas.a --with-lapack-lib=/liblapack.a --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-metis=yes --download-parmetis=yes --download-scalapack=yes<br>>> [4]PETSC ERROR: #1 User provided function() line 0 in unknown file<br>>> [5]PETSC ERROR: ------------------------------------------------------------------------<br>>> [5]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>>> [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>>> [5]PETSC ERROR: or see<span> </span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a><br>>> [5]PETSC ERROR: or try<span> </span><a href="http://valgrind.org/" rel="noreferrer" target="_blank">http://valgrind.org</a><span> </span>on GNU/linux and Apple Mac OS X to find memory corruption errors<br>>> [5]PETSC ERROR: likely location of problem given in stack below<br>>> [5]PETSC ERROR: --------------------- Stack Frames ------------------------------------<br>>> [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>>> [5]PETSC ERROR: INSTEAD the line number of the start of the function<br>>> [5]PETSC ERROR: is given.<br>>> [5]PETSC ERROR: [5] SuperLU_DIST:pdgssvx line 161 /home/simpson/software/petsc-3.5.4/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c<br>>> [5]PETSC ERROR: [5] MatSolve_SuperLU_DIST line 121 /home/simpson/software/petsc-3.5.4/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c<br>>> [5]PETSC ERROR: [5] MatSolve line 3104 /home/simpson/software/petsc-3.5.4/src/mat/interface/matrix.c<br>>> [5]PETSC ERROR: [5] PCApply_LU line 194 /home/simpson/software/petsc-3.5.4/src/ksp/pc/impls/factor/lu/lu.c<br>>> [5]PETSC ERROR: [5] KSP_PCApplyBAorAB line 258 /home/simpson/software/petsc-3.5.4/include/petsc-private/kspimpl.h<br>>> [5]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>>> [5]PETSC ERROR: Signal received<br>>> [5]PETSC ERROR: See<span> </span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><span> </span>for trouble shooting.<br>>> [5]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015<br>>> [5]PETSC ERROR: ./blowup_batch2 on a arch-linux2-c-debug named proteusi01 by simpson Sat Aug 22 17:01:41 2015<br>>> [5]PETSC ERROR: Configure options --with-mpi-dir=/mnt/HA/opt/openmpi/intel/64/1.8.1-mlnx-ofed --with-blas-lib=/mnt/HA/opt/blas/gcc/64/20110419/libblas.a --with-lapack-lib=/liblapack.a --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-metis=yes --download-parmetis=yes --download-scalapack=yes<br>>> [5]PETSC ERROR: #1 User provided function() line 0 in unknown file<br>>><br>>> -gideon<br>>><br>>><br>></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></div></blockquote></div><br></div></div></div></blockquote></div><br></div></div>