-------------------------------------------------------------------------- MPI_ABORT was invoked on rank 14 in communicator MPI_COMM_WORLD with errorcode 59. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- [15]PETSC ERROR: ------------------------------------------------------------------------ [15]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [15]PETSC ERROR: [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] MatSetValues_MPIAIJ line 549 /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpiaij.c [0]PETSC ERROR: [0] MatSetValues line 1316 /lustre/home/vef002/petsc/src/mat/interface/matrix.c [0]PETSC ERROR: [0] PCGAMGFilterGraph line 212 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/util.c [0]PETSC ERROR: [0] PCGAMGGraph_AGG line 828 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c [0]PETSC ERROR: [0] PCSetUp_GAMG line 428 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c [0]PETSC ERROR: [0] PCSetUp line 894 /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: [0] KSPSetUp line 304 /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.10.2, unknown [0]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r03n01 by vef002 Thu Jan 17 02:51:58 2019 [0]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx --download-parmetis --download-metis --download-ptscotch --download-superlu_dist --with-64-bit-indices --with-scalar-type=complex --with-debugging=yes --download-scalapack --download-fblaslapack=1 --download-cmake [0]PETSC ERROR: #1 User provided function() line 0 in unknown file [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [2]PETSC ERROR: likely location of problem given in stack below [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [2]PETSC ERROR: INSTEAD the line number of the start of the function [2]PETSC ERROR: is given. [2]PETSC ERROR: [2] MatSetValues line 1316 /lustre/home/vef002/petsc/src/mat/interface/matrix.c [2]PETSC ERROR: [2] PCGAMGFilterGraph line 212 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/util.c [2]PETSC ERROR: [2] PCGAMGGraph_AGG line 828 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c [2]PETSC ERROR: [2] PCSetUp_GAMG line 428 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c [2]PETSC ERROR: [2] PCSetUp line 894 /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c [2]PETSC ERROR: [2] KSPSetUp line 304 /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [2]PETSC ERROR: Signal received [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [2]PETSC ERROR: Petsc Release Version 3.10.2, unknown [2]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r03n01 by vef002 Thu Jan 17 02:51:58 2019 [2]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx --download-parmetis --download-metis --download-ptscotch --download-superlu_dist --with-64-bit-indices --with-scalar-type=complex --with-debugging=yes --download-scalapack --download-fblaslapack=1 --download-cmake [2]PETSC ERROR: #1 User provided function() line 0 in unknown file [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [4]PETSC ERROR: ------------------------------------------------------------------------ [4]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [4]PETSC ERROR: likely location of problem given in stack below [4]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [4]PETSC ERROR: INSTEAD the line number of the start of the function [4]PETSC ERROR: is given. [4]PETSC ERROR: [4] MatGetRow line 565 /lustre/home/vef002/petsc/src/mat/interface/matrix.c [4]PETSC ERROR: [4] PCGAMGFilterGraph line 212 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/util.c [4]PETSC ERROR: [4] PCGAMGGraph_AGG line 828 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c [4]PETSC ERROR: [4] PCSetUp_GAMG line 428 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c [4]PETSC ERROR: [4] PCSetUp line 894 /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c [4]PETSC ERROR: [4] KSPSetUp line 304 /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c [4]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [4]PETSC ERROR: Signal received [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [4]PETSC ERROR: Petsc Release Version 3.10.2, unknown [4]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r03n01 by vef002 Thu Jan 17 02:51:58 2019 [4]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx --download-parmetis --download-metis --download-ptscotch --download-superlu_dist --with-64-bit-indices --with-scalar-type=complex --with-debugging=yes --download-scalapack --download-fblaslapack=1 --download-cmake [4]PETSC ERROR: #1 User provided function() line 0 in unknown file [5]PETSC ERROR: ------------------------------------------------------------------------ [5]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [6]PETSC ERROR: ------------------------------------------------------------------------ [6]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [6]PETSC ERROR: likely location of problem given in stack below [6]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [6]PETSC ERROR: INSTEAD the line number of the start of the function [6]PETSC ERROR: is given. [6]PETSC ERROR: [6] PCGAMGFilterGraph line 212 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/util.c [6]PETSC ERROR: [6] PCGAMGGraph_AGG line 828 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c [6]PETSC ERROR: [6] PCSetUp_GAMG line 428 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c [6]PETSC ERROR: [6] PCSetUp line 894 /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c [6]PETSC ERROR: [6] KSPSetUp line 304 /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c [6]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [6]PETSC ERROR: Signal received [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [6]PETSC ERROR: Petsc Release Version 3.10.2, unknown [6]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r03n01 by vef002 Thu Jan 17 02:51:58 2019 [6]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx --download-parmetis --download-metis --download-ptscotch --download-superlu_dist --with-64-bit-indices --with-scalar-type=complex --with-debugging=yes --download-scalapack --download-fblaslapack=1 --download-cmake [6]PETSC ERROR: #1 User provided function() line 0 in unknown file [8]PETSC ERROR: ------------------------------------------------------------------------ [8]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [8]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [8]PETSC ERROR: likely location of problem given in stack below [8]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [8]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [8]PETSC ERROR: INSTEAD the line number of the start of the function [8]PETSC ERROR: is given. [8]PETSC ERROR: [8] PetscFPrintf line 610 /lustre/home/vef002/petsc/src/sys/fileio/mprint.c [8]PETSC ERROR: [8] PCGAMGFilterGraph line 212 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/util.c [8]PETSC ERROR: [8] PCGAMGGraph_AGG line 828 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c [8]PETSC ERROR: [8] PCSetUp_GAMG line 428 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c [8]PETSC ERROR: [8] PCSetUp line 894 /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c [8]PETSC ERROR: [8] KSPSetUp line 304 /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c [8]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [8]PETSC ERROR: Signal received [8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [8]PETSC ERROR: Petsc Release Version 3.10.2, unknown [8]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r03n01 by vef002 Thu Jan 17 02:51:58 2019 [8]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx --download-parmetis --download-metis --download-ptscotch --download-superlu_dist --with-64-bit-indices --with-scalar-type=complex --with-debugging=yes --download-scalapack --download-fblaslapack=1 --download-cmake [8]PETSC ERROR: #1 User provided function() line 0 in unknown file [9]PETSC ERROR: ------------------------------------------------------------------------ [9]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [10]PETSC ERROR: ------------------------------------------------------------------------ [10]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [10]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [10]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [10]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [10]PETSC ERROR: likely location of problem given in stack below [10]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [10]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [10]PETSC ERROR: INSTEAD the line number of the start of the function [10]PETSC ERROR: is given. [10]PETSC ERROR: [10] MatSetValues line 1316 /lustre/home/vef002/petsc/src/mat/interface/matrix.c [10]PETSC ERROR: [10] PCGAMGFilterGraph line 212 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/util.c [10]PETSC ERROR: [10] PCGAMGGraph_AGG line 828 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c [10]PETSC ERROR: [10] PCSetUp_GAMG line 428 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c [10]PETSC ERROR: [10] PCSetUp line 894 /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c [10]PETSC ERROR: [10] KSPSetUp line 304 /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c [10]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [10]PETSC ERROR: Signal received [10]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [10]PETSC ERROR: Petsc Release Version 3.10.2, unknown [10]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r03n01 by vef002 Thu Jan 17 02:51:58 2019 [10]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx --download-parmetis --download-metis --download-ptscotch --download-superlu_dist --with-64-bit-indices --with-scalar-type=complex --with-debugging=yes --download-scalapack --download-fblaslapack=1 --download-cmake [10]PETSC ERROR: #1 User provided function() line 0 in unknown file [11]PETSC ERROR: ------------------------------------------------------------------------ [11]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [12]PETSC ERROR: ------------------------------------------------------------------------ [12]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [12]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [12]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [12]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [12]PETSC ERROR: likely location of problem given in stack below [12]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [12]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [12]PETSC ERROR: INSTEAD the line number of the start of the function [12]PETSC ERROR: is given. [12]PETSC ERROR: [12] MatSetValues_MPIAIJ line 549 /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mpiaij.c [12]PETSC ERROR: [12] MatSetValues line 1316 /lustre/home/vef002/petsc/src/mat/interface/matrix.c [12]PETSC ERROR: [12] PCGAMGFilterGraph line 212 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/util.c [12]PETSC ERROR: [12] PCGAMGGraph_AGG line 828 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c [12]PETSC ERROR: [12] PCSetUp_GAMG line 428 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c [12]PETSC ERROR: [12] PCSetUp line 894 /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c [12]PETSC ERROR: [12] KSPSetUp line 304 /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c [12]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [12]PETSC ERROR: Signal received [12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [12]PETSC ERROR: Petsc Release Version 3.10.2, unknown [12]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r03n01 by vef002 Thu Jan 17 02:51:58 2019 [12]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx --download-parmetis --download-metis --download-ptscotch --download-superlu_dist --with-64-bit-indices --with-scalar-type=complex --with-debugging=yes --download-scalapack --download-fblaslapack=1 --download-cmake [12]PETSC ERROR: #1 User provided function() line 0 in unknown file [13]PETSC ERROR: ------------------------------------------------------------------------ [13]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [13]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [13]PETSC ERROR: [14]PETSC ERROR: ------------------------------------------------------------------------ [14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [14]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [14]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [14]PETSC ERROR: likely location of problem given in stack below [14]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [14]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [14]PETSC ERROR: INSTEAD the line number of the start of the function [14]PETSC ERROR: is given. [14]PETSC ERROR: [14] MatSetValues line 1316 /lustre/home/vef002/petsc/src/mat/interface/matrix.c [14]PETSC ERROR: [14] PCGAMGFilterGraph line 212 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/util.c [14]PETSC ERROR: [14] PCGAMGGraph_AGG line 828 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/agg.c [14]PETSC ERROR: [14] PCSetUp_GAMG line 428 /lustre/home/vef002/petsc/src/ksp/pc/impls/gamg/gamg.c [14]PETSC ERROR: [14] PCSetUp line 894 /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c [14]PETSC ERROR: [14] KSPSetUp line 304 /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c [14]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [14]PETSC ERROR: Signal received [14]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [14]PETSC ERROR: Petsc Release Version 3.10.2, unknown [14]PETSC ERROR: ./solveCSys on a linux-cumulus-debug named r03n01 by vef002 Thu Jan 17 02:51:58 2019 [14]PETSC ERROR: Configure options PETSC_ARCH=linux-cumulus-debug --with-cc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicc --with-fc=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpifort --with-cxx=/usr/local/depot/openmpi-3.1.1-gcc-7.3.0/bin/mpicxx --download-parmetis --download-metis --download-ptscotch --download-superlu_dist --with-64-bit-indices --with-scalar-type=complex --with-debugging=yes --download-scalapack --download-fblaslapack=1 --download-cmake [14]PETSC ERROR: #1 User provided function() line 0 in unknown file -------------------------------------------------------------------------- mpiexec noticed that process rank 7 with PID 0 on node r03n01 exited on signal 9 (Killed). -------------------------------------------------------------------------- [r03n01:36099] 6 more processes have sent help message help-mpi-api.txt / mpi-abort [r03n01:36099] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages