[petsc-users] Pastix with Petsc and Slepc

venkatesh g venkateshgk.j at gmail.com
Tue May 26 07:01:29 CDT 2015


Hi I am trying to solve my eigenproblem Ax=lambdaBx with a direct linear
solver Pastix

However I get this error (attached). It works without the pastix option.

My matrices are in binary format.

I gave the command : aprun -n 24 -N 24 ./ex7 -f1 a20t -f2 b20t -st_type
sinvert -eps_nev 3 -log_summary -st_ksp_type preonly -st_pc_type lu
-st_pc_factor_mat_solver_package pastix -ksp_monitor

Kindly help me out.

Venkatesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150526/6c43b10c/attachment-0001.html>
-------------- next part --------------

Generalized eigenproblem stored in file.

 Reading COMPLEX matrices from binary files...

ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: The matrix is not in the correct formatThe matrix is not in the correct formatThe matrix is not in the correct format

The matrix is not in the correct formatThe matrix is not in the correct format

The matrix is not in the correct format



The matrix is not in the correct format





The matrix is not in the correct formatThe matrix is not in the correct formatThe matrix is not in the correct formatThe matrix is not in the correct format

The matrix is not in the correct formatThe matrix is not in the correct formatThe matrix is not in the correct format







The matrix is not in the correct format

The matrix is not in the correct formatThe matrix is not in the correct format

The matrix is not in the correct formatThe matrix is not in the correct formatThe matrix is not in the correct formatThe matrix is not in the correct formatThe matrix is not in the correct format









The matrix is not in the correct format







The matrix is not in the correct format

[9]PETSC ERROR: ------------------------------------------------------------------------
[9]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[9]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[9]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[15]PETSC ERROR: [14]PETSC ERROR: [4]PETSC ERROR: ------------------------------------------------------------------------
------------------------------------------------------------------------
[15]PETSC ERROR: [9]PETSC ERROR: [14]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[5]PETSC ERROR: ------------------------------------------------------------------------
[15]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[21]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[14]PETSC ERROR: [4]PETSC ERROR: ------------------------------------------------------------------------
[6]PETSC ERROR: [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[15]PETSC ERROR: [14]PETSC ERROR: [1]PETSC ERROR: [13]PETSC ERROR: [8]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
------------------------------------------------------------------------
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[5]PETSC ERROR: [13]PETSC ERROR: [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
Try option -start_in_debugger or -on_error_attach_debugger
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
------------------------------------------------------------------------
[18]PETSC ERROR: [19]PETSC ERROR: [14]PETSC ERROR: ------------------------------------------------------------------------
------------------------------------------------------------------------
------------------------------------------------------------------------
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[15]PETSC ERROR: [21]PETSC ERROR: [19]PETSC ERROR: [18]PETSC ERROR: [8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[13]PETSC ERROR: [21]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[18]PETSC ERROR: [19]PETSC ERROR: [21]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
Try option -start_in_debugger or -on_error_attach_debugger
Try option -start_in_debugger or -on_error_attach_debugger
[21]PETSC ERROR: [13]PETSC ERROR: [19]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[13]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[18]PETSC ERROR: [8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[19]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[18]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[14]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
likely location of problem given in stack below
[15]PETSC ERROR: [14]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
---------------------  Stack Frames ------------------------------------
likely location of problem given in stack below
[14]PETSC ERROR: [15]PETSC ERROR: ------------------------------------------------------------------------
Note: The EXACT line numbers in the stack are not available,
---------------------  Stack Frames ------------------------------------
[7]PETSC ERROR: [21]PETSC ERROR: [14]PETSC ERROR: [15]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
      INSTEAD the line number of the start of the function
Note: The EXACT line numbers in the stack are not available,
[7]PETSC ERROR: [13]PETSC ERROR: [15]PETSC ERROR: [14]PETSC ERROR: likely location of problem given in stack below
      is given.
      INSTEAD the line number of the start of the function
[14]PETSC ERROR: [13]PETSC ERROR: [15]PETSC ERROR: [14] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
      is given.
---------------------  Stack Frames ------------------------------------
[8]PETSC ERROR: [14]PETSC ERROR: [15]PETSC ERROR: [13]PETSC ERROR: [15] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[14] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[15]PETSC ERROR: likely location of problem given in stack below
Note: The EXACT line numbers in the stack are not available,
[18]PETSC ERROR: [19]PETSC ERROR: [13]PETSC ERROR: [8]PETSC ERROR: [9]PETSC ERROR: [15] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[14]PETSC ERROR: likely location of problem given in stack below
likely location of problem given in stack below
      INSTEAD the line number of the start of the function
---------------------  Stack Frames ------------------------------------
likely location of problem given in stack below
[19]PETSC ERROR: [18]PETSC ERROR: [13]PETSC ERROR: [15]PETSC ERROR: [14] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[15] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[9]PETSC ERROR: [8]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
---------------------  Stack Frames ------------------------------------
[15]PETSC ERROR: [14]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
Note: The EXACT line numbers in the stack are not available,
[15] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[9]PETSC ERROR: [8]PETSC ERROR: [15]PETSC ERROR: [14] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[18]PETSC ERROR: [19]PETSC ERROR: [14]PETSC ERROR: [15] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
      INSTEAD the line number of the start of the function
Note: The EXACT line numbers in the stack are not available,
Note: The EXACT line numbers in the stack are not available,
Note: The EXACT line numbers in the stack are not available,
[15]PETSC ERROR: [14] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[18]PETSC ERROR: [19]PETSC ERROR: [14]PETSC ERROR: [15] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[9]PETSC ERROR: [3]PETSC ERROR: [10]PETSC ERROR: [12]PETSC ERROR: [14] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
      INSTEAD the line number of the start of the function
[8]PETSC ERROR: [22]PETSC ERROR: [14]PETSC ERROR: [15]PETSC ERROR: [9]PETSC ERROR:       is given.
      is given.
[15] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[14] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[8]PETSC ERROR: [9]PETSC ERROR: [14]PETSC ERROR: [15]PETSC ERROR: [8] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[9] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
      INSTEAD the line number of the start of the function
      INSTEAD the line number of the start of the function
------------------------------------------------------------------------
[20]PETSC ERROR: [9]PETSC ERROR: [18]PETSC ERROR: [19]PETSC ERROR:       is given.
[15] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[14] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
      is given.
[18]PETSC ERROR: [3]PETSC ERROR: ------------------------------------------------------------------------
[15]PETSC ERROR: [14]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[18] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[19]PETSC ERROR: [14] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[15] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[9] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[19] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[18]PETSC ERROR: [3]PETSC ERROR: [18] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[19]PETSC ERROR: [9]PETSC ERROR: [20]PETSC ERROR: [15]PETSC ERROR: [14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[18]PETSC ERROR: [19] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[9] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[14] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[15] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[9]PETSC ERROR: [14]PETSC ERROR: [15]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[14] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[15] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[9] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[3]PETSC ERROR: [19]PETSC ERROR: [18] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[19] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[18]PETSC ERROR: [3]PETSC ERROR: [9]PETSC ERROR: [18] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[19]PETSC ERROR: [20]PETSC ERROR: [9] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
Try option -start_in_debugger or -on_error_attach_debugger
[19] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[18]PETSC ERROR: [19]PETSC ERROR: [9]PETSC ERROR: [19] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[18] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[9] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[20]PETSC ERROR: [19]PETSC ERROR: [18]PETSC ERROR: [9]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[19] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[18] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[20]PETSC ERROR: [15]PETSC ERROR: [14]PETSC ERROR: [9] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[19]PETSC ERROR: [18]PETSC ERROR: [9]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
--------------------- Error Message --------------------------------------------------------------
[18] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[15]PETSC ERROR: [14]PETSC ERROR: [9] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[18]PETSC ERROR: [19] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
Signal received
Signal received
[19]PETSC ERROR: [18] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[9]PETSC ERROR: ------------------------------------------------------------------------
[9] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[14]PETSC ERROR: [15]PETSC ERROR: [18]PETSC ERROR: ------------------------------------------------------------------------
[9]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[3]PETSC ERROR: [6]PETSC ERROR: [1]PETSC ERROR: [9] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
likely location of problem given in stack below
[14]PETSC ERROR: [15]PETSC ERROR: [9]PETSC ERROR: [20]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
Petsc Release Version 3.5.3, Jan, 31, 2015 
[9] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[3]PETSC ERROR: [14]PETSC ERROR: [15]PETSC ERROR: likely location of problem given in stack below
---------------------  Stack Frames ------------------------------------
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[3]PETSC ERROR: [20]PETSC ERROR: [14]PETSC ERROR: [15]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
Note: The EXACT line numbers in the stack are not available,
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[14]PETSC ERROR: [20]PETSC ERROR: [15]PETSC ERROR: #1 User provided function() line 0 in  unknown file
[9]PETSC ERROR: #1 User provided function() line 0 in  unknown file
Note: The EXACT line numbers in the stack are not available,
--------------------- Error Message --------------------------------------------------------------
[20]PETSC ERROR: [5]PETSC ERROR:       INSTEAD the line number of the start of the function
[9]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
Signal received
[20]PETSC ERROR: [5]PETSC ERROR: [9]PETSC ERROR:       is given.
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[20]PETSC ERROR: [5]PETSC ERROR: [9]PETSC ERROR: [20] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
Petsc Release Version 3.5.3, Jan, 31, 2015 
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[20]PETSC ERROR: [9]PETSC ERROR: [4]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[20] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
Try option -start_in_debugger or -on_error_attach_debugger
[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[7]PETSC ERROR: [20]PETSC ERROR: [9]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
------------------------------------------------------------------------
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[2]PETSC ERROR: [9]PETSC ERROR: [7]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
#1 User provided function() line 0 in  unknown file
[2]PETSC ERROR: likely location of problem given in stack below
Try option -start_in_debugger or -on_error_attach_debugger
      is given.
[2]PETSC ERROR: [21]PETSC ERROR: [13]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[13] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[5]PETSC ERROR: [21]PETSC ERROR: [2]PETSC ERROR: likely location of problem given in stack below
Note: The EXACT line numbers in the stack are not available,
Rank 14 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 14
Rank 15 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 15
[5]PETSC ERROR: [13]PETSC ERROR: [16]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[21]PETSC ERROR: [13] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[7]PETSC ERROR: [4]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
likely location of problem given in stack below
[13]PETSC ERROR: likely location of problem given in stack below
      INSTEAD the line number of the start of the function
[5]PETSC ERROR: [4]PETSC ERROR: [21]PETSC ERROR: [7]PETSC ERROR: ------------------------------------------------------------------------
---------------------  Stack Frames ------------------------------------
---------------------  Stack Frames ------------------------------------
      is given.
[4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[7]PETSC ERROR: Rank 9 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 9
[13] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
Note: The EXACT line numbers in the stack are not available,
[17]PETSC ERROR: [16]PETSC ERROR: [7]PETSC ERROR: [13]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[5]PETSC ERROR: [13] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
      INSTEAD the line number of the start of the function
[13]PETSC ERROR:       INSTEAD the line number of the start of the function
[4]PETSC ERROR: ------------------------------------------------------------------------
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[7]PETSC ERROR: [17]PETSC ERROR: [16]PETSC ERROR: [2]PETSC ERROR:       INSTEAD the line number of the start of the function
[5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[16]PETSC ERROR: [21]PETSC ERROR: [13] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[21] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
      is given.
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[17]PETSC ERROR: [13]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[23]PETSC ERROR: [17]PETSC ERROR: [16]PETSC ERROR: [7]PETSC ERROR: [13] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
------------------------------------------------------------------------
[17]PETSC ERROR: [7] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[23]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[7]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[13]PETSC ERROR: [7] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[23]PETSC ERROR: [13] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[7] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[13]PETSC ERROR: [23]PETSC ERROR: [7]PETSC ERROR: [13] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[13]PETSC ERROR: [7] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[23]PETSC ERROR: [7]PETSC ERROR: [13] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[7] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[13]PETSC ERROR: [7]PETSC ERROR: [13] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[7] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
------------------------------------------------------------------------
[7]PETSC ERROR: [13]PETSC ERROR: [7] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[22]PETSC ERROR: [16]PETSC ERROR: [13] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
likely location of problem given in stack below
[7]PETSC ERROR: [22]PETSC ERROR: [7] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
------------------------------------------------------------------------
Try option -start_in_debugger or -on_error_attach_debugger
[16]PETSC ERROR: [17]PETSC ERROR: [22]PETSC ERROR: likely location of problem given in stack below
---------------------  Stack Frames ------------------------------------
[12]PETSC ERROR: [17]PETSC ERROR: [16]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[7]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
---------------------  Stack Frames ------------------------------------
Note: The EXACT line numbers in the stack are not available,
[22]PETSC ERROR: [23]PETSC ERROR: [16]PETSC ERROR: [17]PETSC ERROR: [12]PETSC ERROR: [13]PETSC ERROR: [7] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
      INSTEAD the line number of the start of the function
--------------------- Error Message --------------------------------------------------------------
Try option -start_in_debugger or -on_error_attach_debugger
likely location of problem given in stack below
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[23]PETSC ERROR: [12]PETSC ERROR: [13]PETSC ERROR: [11]PETSC ERROR: Signal received
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[7]PETSC ERROR: ------------------------------------------------------------------------
[7] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[16]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
      is given.
---------------------  Stack Frames ------------------------------------
[16]PETSC ERROR: [17]PETSC ERROR: ------------------------------------------------------------------------
[23]PETSC ERROR: [10]PETSC ERROR:       INSTEAD the line number of the start of the function
[16] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
Note: The EXACT line numbers in the stack are not available,
[17]PETSC ERROR: [16]PETSC ERROR: [23]PETSC ERROR: [10]PETSC ERROR:       is given.
[16] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[17]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[17] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[16]PETSC ERROR: [10]PETSC ERROR: [22]PETSC ERROR: [16] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[17]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[16]PETSC ERROR: [17] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[10]PETSC ERROR: [16] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[17]PETSC ERROR: likely location of problem given in stack below
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[22]PETSC ERROR: [16]PETSC ERROR: [17] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
---------------------  Stack Frames ------------------------------------
[16] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[17]PETSC ERROR: [22]PETSC ERROR: [16]PETSC ERROR: [17] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
Note: The EXACT line numbers in the stack are not available,
[16] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[17]PETSC ERROR: [16]PETSC ERROR: [22]PETSC ERROR: [17] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[16] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
      INSTEAD the line number of the start of the function
[17]PETSC ERROR: [16]PETSC ERROR: [22]PETSC ERROR: [16] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[17] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[16]PETSC ERROR:       is given.
[16] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[17]PETSC ERROR: [22]PETSC ERROR: [16]PETSC ERROR: [17] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[22] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[17]PETSC ERROR: [16] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[17] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[10]PETSC ERROR: [17]PETSC ERROR: [16]PETSC ERROR: [22]PETSC ERROR: [17] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[22] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
likely location of problem given in stack below
[17]PETSC ERROR: [16] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[17] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[22]PETSC ERROR: [17]PETSC ERROR: [22] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[17] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[22]PETSC ERROR: [8]PETSC ERROR: [22] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[22]PETSC ERROR: [8] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[22] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[8]PETSC ERROR: [22]PETSC ERROR: [0]PETSC ERROR: [22] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[8] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[22]PETSC ERROR: [1]PETSC ERROR: [8]PETSC ERROR: [22] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
Try option -start_in_debugger or -on_error_attach_debugger
[8] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[1]PETSC ERROR: [8]PETSC ERROR: [17]PETSC ERROR: [16]PETSC ERROR: [8] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[22]PETSC ERROR: [8]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[22] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[8] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
--------------------- Error Message --------------------------------------------------------------
--------------------- Error Message --------------------------------------------------------------
[22]PETSC ERROR: [1]PETSC ERROR: [16]PETSC ERROR: [17]PETSC ERROR: [22] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
Signal received
Signal received
[8]PETSC ERROR: [17]PETSC ERROR: [16]PETSC ERROR: [8] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[22]PETSC ERROR: [8]PETSC ERROR: [17]PETSC ERROR: [16]PETSC ERROR: [8] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
Petsc Release Version 3.5.3, Jan, 31, 2015 
Petsc Release Version 3.5.3, Jan, 31, 2015 
[19] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[8]PETSC ERROR: [22] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[8] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[16]PETSC ERROR: [17]PETSC ERROR: [22]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[8]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[16]PETSC ERROR: [8] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[17]PETSC ERROR: [8]PETSC ERROR: [19]PETSC ERROR: [8] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[17]PETSC ERROR: [19] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
#1 User provided function() line 0 in  unknown file
[16]PETSC ERROR: [19]PETSC ERROR: [1]PETSC ERROR: [19] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
#1 User provided function() line 0 in  unknown file
likely location of problem given in stack below
[19]PETSC ERROR: [1]PETSC ERROR: [19] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
---------------------  Stack Frames ------------------------------------
[18] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[1]PETSC ERROR: [6]PETSC ERROR: [8]PETSC ERROR: [18]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[18] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
Try option -start_in_debugger or -on_error_attach_debugger
--------------------- Error Message --------------------------------------------------------------
[18]PETSC ERROR: [6]PETSC ERROR: [8]PETSC ERROR: [18] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
Signal received
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[1]PETSC ERROR: [8]PETSC ERROR: [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
      INSTEAD the line number of the start of the function
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[8]PETSC ERROR: [1]PETSC ERROR: [3]PETSC ERROR:       is given.
Petsc Release Version 3.5.3, Jan, 31, 2015 
[19]PETSC ERROR: [1]PETSC ERROR:       INSTEAD the line number of the start of the function
[8]PETSC ERROR: [1] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
--------------------- Error Message --------------------------------------------------------------
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[1]PETSC ERROR: [3]PETSC ERROR: [18]PETSC ERROR: [19]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
Signal received
      is given.
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[18]PETSC ERROR: [19]PETSC ERROR: [8]PETSC ERROR: Signal received
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
#1 User provided function() line 0 in  unknown file
[19]PETSC ERROR: [18]PETSC ERROR: [6]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[19]PETSC ERROR: likely location of problem given in stack below
[18]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[6]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
---------------------  Stack Frames ------------------------------------
[19]PETSC ERROR: [18]PETSC ERROR: [6]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
Note: The EXACT line numbers in the stack are not available,
[19]PETSC ERROR: [18]PETSC ERROR: [6]PETSC ERROR: #1 User provided function() line 0 in  unknown file
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
      INSTEAD the line number of the start of the function
[20] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[6]PETSC ERROR: [18]PETSC ERROR: [20]PETSC ERROR:       is given.
#1 User provided function() line 0 in  unknown file
[20] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[6]PETSC ERROR: [20]PETSC ERROR: [6] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[20] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[6]PETSC ERROR: [20]PETSC ERROR: [6] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
Rank 8 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 8
[6]PETSC ERROR: [20] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[6] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[20]PETSC ERROR: [6]PETSC ERROR: [20] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[6] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[20]PETSC ERROR: [6]PETSC ERROR: [20] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[6] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[20]PETSC ERROR: likely location of problem given in stack below
[6]PETSC ERROR: [20] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[6] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[20]PETSC ERROR: [2]PETSC ERROR: [20] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[6]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[6] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[20]PETSC ERROR: [6]PETSC ERROR: [2]PETSC ERROR: [6] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[20] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
Note: The EXACT line numbers in the stack are not available,
[6]PETSC ERROR:       is given.
[2]PETSC ERROR: [6] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[5]PETSC ERROR:       INSTEAD the line number of the start of the function
[4]PETSC ERROR: [6]PETSC ERROR: [2]PETSC ERROR: [6] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
      is given.
[5] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[6]PETSC ERROR:       is given.
[5]PETSC ERROR: [4]PETSC ERROR: [2]PETSC ERROR: [6] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[5] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[20]PETSC ERROR: [2] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[5]PETSC ERROR: [4] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
Rank 19 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 19
Rank 18 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 18
[2]PETSC ERROR: [4]PETSC ERROR: [5] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[4] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[2] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[4]PETSC ERROR: [5]PETSC ERROR: [4] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[6]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
--------------------- Error Message --------------------------------------------------------------
[20]PETSC ERROR: [21]PETSC ERROR: Signal received
[4]PETSC ERROR: [5] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[2]PETSC ERROR: [4] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[5]PETSC ERROR: [20]PETSC ERROR: [21] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[11]PETSC ERROR: [5] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[4]PETSC ERROR: [2] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[4] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[5]PETSC ERROR: [2]PETSC ERROR: [5] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[4]PETSC ERROR: [21]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[4] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[5]PETSC ERROR: [20]PETSC ERROR: [21] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
Petsc Release Version 3.5.3, Jan, 31, 2015 
[2] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[5] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[2]PETSC ERROR: [11]PETSC ERROR: [20]PETSC ERROR: [2] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[4] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[21]PETSC ERROR: [20]PETSC ERROR: [2]PETSC ERROR: [5] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[4]PETSC ERROR: [2] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[11]PETSC ERROR: [2]PETSC ERROR: [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[2] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[21] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[5] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[20]PETSC ERROR: [21]PETSC ERROR: [4] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[5]PETSC ERROR: [4]PETSC ERROR: [11]PETSC ERROR: [4] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[5] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[21] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
#1 User provided function() line 0 in  unknown file
[2]PETSC ERROR: [21]PETSC ERROR: [5]PETSC ERROR: [4]PETSC ERROR: [21] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[5] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[4] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[2] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[4]PETSC ERROR: [2]PETSC ERROR: [21]PETSC ERROR: [4] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[2] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[21] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[2]PETSC ERROR: [21]PETSC ERROR: [2] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[21] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[2]PETSC ERROR: [21]PETSC ERROR: [2] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[11]PETSC ERROR: [21] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
likely location of problem given in stack below
[7]PETSC ERROR: [11]PETSC ERROR: [5]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[7] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[21]PETSC ERROR: [11]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[21] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
Note: The EXACT line numbers in the stack are not available,
[5]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[21]PETSC ERROR: [4]PETSC ERROR: Signal received
[21] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
Rank 20 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 20
[11]PETSC ERROR: [5]PETSC ERROR: Signal received
      INSTEAD the line number of the start of the function
[4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[5]PETSC ERROR: [11]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[2]PETSC ERROR:       is given.
[7]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
Signal received
Petsc Release Version 3.5.3, Jan, 31, 2015 
[2]PETSC ERROR: [11]PETSC ERROR: [5]PETSC ERROR: [4]PETSC ERROR: [7]PETSC ERROR: [11] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
Signal received
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[11]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
Petsc Release Version 3.5.3, Jan, 31, 2015 
[5]PETSC ERROR: [21]PETSC ERROR: [2]PETSC ERROR: [7]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[4]PETSC ERROR: [11] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
--------------------- Error Message --------------------------------------------------------------
Petsc Release Version 3.5.3, Jan, 31, 2015 
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[5]PETSC ERROR: [4]PETSC ERROR: [7]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
#1 User provided function() line 0 in  unknown file
[21]PETSC ERROR: [4]PETSC ERROR: Signal received
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[21]PETSC ERROR: [11]PETSC ERROR: [7]PETSC ERROR: #1 User provided function() line 0 in  unknown file
[11] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[11]PETSC ERROR: [21]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
Petsc Release Version 3.5.3, Jan, 31, 2015 
[11] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[7]PETSC ERROR: [21]PETSC ERROR: [11]PETSC ERROR: #1 User provided function() line 0 in  unknown file
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[11] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[21]PETSC ERROR: [12]PETSC ERROR: [11]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[11] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[21]PETSC ERROR: [11]PETSC ERROR: [13]PETSC ERROR: #1 User provided function() line 0 in  unknown file
[11] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
      INSTEAD the line number of the start of the function
[11]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[23]PETSC ERROR: [11] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
      is given.
[11]PETSC ERROR: [23]PETSC ERROR: [13]PETSC ERROR: [11] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[23] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
Petsc Release Version 3.5.3, Jan, 31, 2015 
[11]PETSC ERROR: [23]PETSC ERROR: Rank 4 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4
Rank 5 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 5
[13]PETSC ERROR: [23] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[11] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
Rank 7 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 7
[23]PETSC ERROR: [11]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[11] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[23] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[13]PETSC ERROR: [12]PETSC ERROR: [23]PETSC ERROR: likely location of problem given in stack below
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[12]PETSC ERROR: [23] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
---------------------  Stack Frames ------------------------------------
[13]PETSC ERROR: [12]PETSC ERROR: Rank 21 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 21
Note: The EXACT line numbers in the stack are not available,
#1 User provided function() line 0 in  unknown file
[10]PETSC ERROR: [12]PETSC ERROR: ------------------------------------------------------------------------
      INSTEAD the line number of the start of the function
---------------------  Stack Frames ------------------------------------
[12]PETSC ERROR:       is given.
[22] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[10]PETSC ERROR: [11]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[12]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[10]PETSC ERROR: [11]PETSC ERROR: [12] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
Signal received
[1] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[11]PETSC ERROR: [12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[1]PETSC ERROR: [11]PETSC ERROR: [12] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
Petsc Release Version 3.5.3, Jan, 31, 2015 
[22]PETSC ERROR: [12]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[12] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[22]PETSC ERROR: [12]PETSC ERROR: Signal received
[12] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[22]PETSC ERROR: Rank 17 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 17
[12]PETSC ERROR: Rank 13 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 13
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[12] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[22]PETSC ERROR: Rank 16 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 16
[12]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[12] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[22]PETSC ERROR: [12]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[12] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[22]PETSC ERROR: [12]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[12] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[22]PETSC ERROR: [12]PETSC ERROR: #1 User provided function() line 0 in  unknown file
[12] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[12]PETSC ERROR: [12] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[12]PETSC ERROR: [12] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[12]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[12]PETSC ERROR: Signal received
[12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[12]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[12]PETSC ERROR: [3]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
Rank 22 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 22
[12]PETSC ERROR: [3] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[3]PETSC ERROR: [12]PETSC ERROR: [6]PETSC ERROR: [3] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
#1 User provided function() line 0 in  unknown file
[3]PETSC ERROR: Signal received
[3] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[6]PETSC ERROR: [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[3] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[6]PETSC ERROR: [3]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[6]PETSC ERROR: [3] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[3]PETSC ERROR: [6]PETSC ERROR: [3] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[3]PETSC ERROR: [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[3] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
#1 User provided function() line 0 in  unknown file
[3]PETSC ERROR: [2]PETSC ERROR: [23]PETSC ERROR: [3] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
Petsc Release Version 3.5.3, Jan, 31, 2015 
[3]PETSC ERROR: [23] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[2]PETSC ERROR: [3] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[23]PETSC ERROR: [3]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[23] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[2]PETSC ERROR: [3] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[3]PETSC ERROR: [23]PETSC ERROR: [2]PETSC ERROR: [3] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[23] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
#1 User provided function() line 0 in  unknown file
[23]PETSC ERROR: [0]PETSC ERROR: [23] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
      INSTEAD the line number of the start of the function
[23]PETSC ERROR: [1] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[23] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[10]PETSC ERROR: [1]PETSC ERROR:       is given.
Rank 6 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 6
[23]PETSC ERROR: [1] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[10]PETSC ERROR: [23] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[1]PETSC ERROR: [10] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[23]PETSC ERROR: [1] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[23] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[3]PETSC ERROR: [1]PETSC ERROR: [10]PETSC ERROR: [11]PETSC ERROR: [10] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[1] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
--------------------- Error Message --------------------------------------------------------------
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[10]PETSC ERROR: [11]PETSC ERROR: [3]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[10] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[1]PETSC ERROR: [10]PETSC ERROR: [11]PETSC ERROR: [1] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[10] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
#1 User provided function() line 0 in  unknown file
[10]PETSC ERROR: [1]PETSC ERROR: Rank 2 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2
[10] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[1] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[23]PETSC ERROR: [10]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: [10] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[23]PETSC ERROR: [1] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[10]PETSC ERROR: Signal received
[1]PETSC ERROR: [23]PETSC ERROR: [10] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[1] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[10]PETSC ERROR: [23]PETSC ERROR: [1]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
Rank 12 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 12
[23]PETSC ERROR: [10] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[1] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[10]PETSC ERROR: [23]PETSC ERROR: [10] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[10]PETSC ERROR: [23]PETSC ERROR: [10] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
#1 User provided function() line 0 in  unknown file
[10]PETSC ERROR: [10] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
Rank 11 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 11
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
Signal received
[1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[3]PETSC ERROR: [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
Signal received
[1]PETSC ERROR: [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
Try option -start_in_debugger or -on_error_attach_debugger
[1]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
Petsc Release Version 3.5.3, Jan, 31, 2015 
[10]PETSC ERROR: [3]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[10]PETSC ERROR: [1]PETSC ERROR: [3]PETSC ERROR: Signal received
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[10]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[1]PETSC ERROR: [3]PETSC ERROR: [10]PETSC ERROR: #1 User provided function() line 0 in  unknown file
#1 User provided function() line 0 in  unknown file
Petsc Release Version 3.5.3, Jan, 31, 2015 
[0]PETSC ERROR: [10]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[10]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
Rank 23 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 23
[10]PETSC ERROR: #1 User provided function() line 0 in  unknown file
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
Rank 3 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3
Rank 10 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 10
Rank 1 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[0]PETSC ERROR: [0] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[0]PETSC ERROR: [0] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[0]PETSC ERROR: [0] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[0]PETSC ERROR: [0] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[0]PETSC ERROR: [0] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: [0] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[0]PETSC ERROR: [0] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[0]PETSC ERROR: [0] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[0]PETSC ERROR: [0] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[0]PETSC ERROR: [0] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[0]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[0]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
Rank 0 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
_pmiu_daemon(SIGCHLD): [NID 00828] [c4-0c0s15n0] [Tue May 26 06:39:10 2015] PE RANK 15 exit signal Aborted
Application 406154 exit codes: 134
Application 406154 resources: utime ~1s, stime ~2s, Rss ~57484, inblocks ~1676, outblocks ~288


More information about the petsc-users mailing list