[petsc-users] "Petsc has generated inconsistent data" during EPSSolve

Mon May 9 11:03:42 CDT 2016

Hi all. I am using the EPSCISS solver in SLEPc to calculate eigenvalues of a Hermitian matrix close to the origin. (Actually, it is only very nearly Hermitian, but this happens also for NHEP-routines). I am running with 4 cores, and the matrix prints fine with MatView() just before the EPSSolve. Anyhow, I get the following error:
(I am new to both PETSc and this mailing list. Should error messages be attached or pasted into the body?)

[2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[2]PETSC ERROR: Petsc has generated inconsistent data
[2]PETSC ERROR: ith 3844 block entry 7688 not owned by any process, upper bound 7688
[2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[2]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 
[2]PETSC ERROR: hello on a complex named st-d13411.nfit.au.dk by jonat4n Mon May  9 17:47:26 2016
[2]PETSC ERROR: Configure options CC=/usr/local/bin/mpicc CXX=/usr/local/bin/mpicxx F77=/usr/local/bin/mpif77 FC=/usr/local/bin/mpif90 --with-shared-libraries=1 --with-pthread=0 --with-openmp=0 --with-debugging=0 --with-ssl=0 --with-superlu_dist-include=/usr/local/opt/superlu_dist/include/superlu_dist --with-superlu_dist-lib="-L/usr/local/opt/superlu_dist/lib -lsuperlu_dist" --with-superlu-include=/usr/local/Cellar/superlu43/4.3_1/include/superlu --with-superlu-lib="-L/usr/local/Cellar/superlu43/4.3_1/lib -lsuperlu" --with-fftw-dir=/usr/local/opt/fftw --with-netcdf-dir=/usr/local/opt/netcdf --with-suitesparse-dir=/usr/local/opt/suite-sparse --with-hdf5-dir=/usr/local/opt/hdf5 --with-metis-dir=/usr/local/opt/metis --with-parmetis-dir=/usr/local/opt/parmetis --with-scalapack-dir=/usr/local/opt/scalapack --with-mumps-dir=/usr/local/opt/mumps/libexec --with-x=0 --prefix=/usr/local/Cellar/petsc/3.6.3_4/complex --with-scalar-type=complex
[2]PETSC ERROR: #1 VecScatterCreate_PtoS() line 2339 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
[2]PETSC ERROR: #2 VecScatterCreate_StoP() line 2795 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
[2]PETSC ERROR: #3 VecScatterCreate_PtoP() line 2984 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
[2]PETSC ERROR: #4 VecScatterCreate() line 1570 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vscat.c
[2]PETSC ERROR: #5 PCSetUp_Redundant() line 133 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/pc/impls/redundant/redundant.c
[2]PETSC ERROR: #6 PCSetUp() line 983 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/pc/interface/precon.c
[3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[3]PETSC ERROR: Petsc has generated inconsistent data
[3]PETSC ERROR: ith 1922 block entry 7688 not owned by any process, upper bound 7688
[3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[3]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 
[3]PETSC ERROR: hello on a complex named st-d13411.nfit.au.dk by jonat4n Mon May  9 17:47:26 2016
[3]PETSC ERROR: Configure options CC=/usr/local/bin/mpicc CXX=/usr/local/bin/mpicxx F77=/usr/local/bin/mpif77 FC=/usr/local/bin/mpif90 --with-shared-libraries=1 --with-pthread=0 --with-openmp=0 --with-debugging=0 --with-ssl=0 --with-superlu_dist-include=/usr/local/opt/superlu_dist/include/superlu_dist --with-superlu_dist-lib="-L/usr/local/opt/superlu_dist/lib -lsuperlu_dist" --with-superlu-include=/usr/local/Cellar/superlu43/4.3_1/include/superlu --with-superlu-lib="-L/usr/local/Cellar/superlu43/4.3_1/lib -lsuperlu" --with-fftw-dir=/usr/local/opt/fftw --with-netcdf-dir=/usr/local/opt/netcdf --with-suitesparse-dir=/usr/local/opt/suite-sparse --with-hdf5-dir=/usr/local/opt/hdf5 --with-metis-dir=/usr/local/opt/metis --with-parmetis-dir=/usr/local/opt/parmetis --with-scalapack-dir=/usr/local/opt/scalapack --with-mumps-dir=/usr/local/opt/mumps/libexec --with-x=0 --prefix=/usr/local/Cellar/petsc/3.6.3_4/complex --with-scalar-type=complex
[3]PETSC ERROR: #1 VecScatterCreate_PtoS() line 2339 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
[3]PETSC ERROR: #2 VecScatterCreate_StoP() line 2795 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
[3]PETSC ERROR: #3 VecScatterCreate_PtoP() line 2984 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
[3]PETSC ERROR: #4 VecScatterCreate() line 1570 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vscat.c
[3]PETSC ERROR: #5 PCSetUp_Redundant() line 133 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/pc/impls/redundant/redundant.c
[3]PETSC ERROR: #6 PCSetUp() line 983 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/pc/interface/precon.c
[3]PETSC ERROR: #7 KSPSetUp() line 332 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[3]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: Petsc has generated inconsistent data
[1]PETSC ERROR: ith 5766 block entry 7688 not owned by any process, upper bound 7688
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 
[1]PETSC ERROR: hello on a complex named st-d13411.nfit.au.dk by jonat4n Mon May  9 17:47:26 2016
[1]PETSC ERROR: Configure options CC=/usr/local/bin/mpicc CXX=/usr/local/bin/mpicxx F77=/usr/local/bin/mpif77 FC=/usr/local/bin/mpif90 --with-shared-libraries=1 --with-pthread=0 --with-openmp=0 --with-debugging=0 --with-ssl=0 --with-superlu_dist-include=/usr/local/opt/superlu_dist/include/superlu_dist --with-superlu_dist-lib="-L/usr/local/opt/superlu_dist/lib -lsuperlu_dist" --with-superlu-include=/usr/local/Cellar/superlu43/4.3_1/include/superlu --with-superlu-lib="-L/usr/local/Cellar/superlu43/4.3_1/lib -lsuperlu" --with-fftw-dir=/usr/local/opt/fftw --with-netcdf-dir=/usr/local/opt/netcdf --with-suitesparse-dir=/usr/local/opt/suite-sparse --with-hdf5-dir=/usr/local/opt/hdf5 --with-metis-dir=/usr/local/opt/metis --with-parmetis-dir=/usr/local/opt/parmetis --with-scalapack-dir=/usr/local/opt/scalapack --with-mumps-dir=/usr/local/opt/mumps/libexec --with-x=0 --prefix=/usr/local/Cellar/petsc/3.6.3_4/complex --with-scalar-type=complex
[1]PETSC ERROR: #1 VecScatterCreate_PtoS() line 2339 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
[1]PETSC ERROR: #2 VecScatterCreate_StoP() line 2795 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
[1]PETSC ERROR: #3 VecScatterCreate_PtoP() line 2984 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
[1]PETSC ERROR: #4 VecScatterCreate() line 1570 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vscat.c
[1]PETSC ERROR: #5 PCSetUp_Redundant() line 133 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/pc/impls/redundant/redundant.c
[1]PETSC ERROR: #6 PCSetUp() line 983 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/pc/interface/precon.c
[1]PETSC ERROR: #7 KSPSetUp() line 332 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR: #8 KSPSolve() line 546 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR: [2]PETSC ERROR: #7 KSPSetUp() line 332 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[2]PETSC ERROR: #8 KSPSolve() line 546 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[2]PETSC ERROR: #9 SolveLinearSystem() line 337 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
[2]PETSC ERROR: #10 EPSSolve_CISS() line 868 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
[2]PETSC ERROR: #11 EPSSolve() line 101 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/interface/epssolve.c
[2]PETSC ERROR: #12 main() line 795 in hello.c
[2]PETSC ERROR: PETSc Option Table entries:
[2]PETSC ERROR: -n 31
[2]PETSC ERROR: #8 KSPSolve() line 546 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[3]PETSC ERROR: #9 SolveLinearSystem() line 337 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
[3]PETSC ERROR: #10 EPSSolve_CISS() line 868 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
[3]PETSC ERROR: #11 EPSSolve() line 101 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/interface/epssolve.c
[3]PETSC ERROR: #12 main() line 795 in hello.c
[3]PETSC ERROR: PETSc Option Table entries:
[3]PETSC ERROR: -n 31
#9 SolveLinearSystem() line 337 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
[1]PETSC ERROR: #10 EPSSolve_CISS() line 868 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
[1]PETSC ERROR: #11 EPSSolve() line 101 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/interface/epssolve.c
[1]PETSC ERROR: #12 main() line 795 in hello.c
[1]PETSC ERROR: PETSc Option Table entries:
[1]PETSC ERROR: -n 31
----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[3]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD 
with errorcode 77.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 
[0]PETSC ERROR: hello on a complex named st-d13411.nfit.au.dk by jonat4n Mon May  9 17:47:26 2016
[0]PETSC ERROR: Configure options CC=/usr/local/bin/mpicc CXX=/usr/local/bin/mpicxx F77=/usr/local/bin/mpif77 FC=/usr/local/bin/mpif90 --with-shared-libraries=1 --with-pthread=0 --with-openmp=0 --with-debugging=0 --with-ssl=0 --with-superlu_dist-include=/usr/local/opt/superlu_dist/include/superlu_dist --with-superlu_dist-lib="-L/usr/local/opt/superlu_dist/lib -lsuperlu_dist" --with-superlu-include=/usr/local/Cellar/superlu43/4.3_1/include/superlu --with-superlu-lib="-L/usr/local/Cellar/superlu43/4.3_1/lib -lsuperlu" --with-fftw-dir=/usr/local/opt/fftw --with-netcdf-dir=/usr/local/opt/netcdf --with-suitesparse-dir=/usr/local/opt/suite-sparse --with-hdf5-dir=/usr/local/opt/hdf5 --with-metis-dir=/usr/local/opt/metis --with-parmetis-dir=/usr/local/opt/parmetis --with-scalapack-dir=/usr/local/opt/scalapack --with-mumps-dir=/usr/local/opt/mumps/libexec --with-x=0 --prefix=/usr/local/Cellar/petsc/3.6.3_4/complex --with-scalar-type=complex
[0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
[st-d13411.nfit.au.dk:90621] 3 more processes have sent help message help-mpi-api.txt / mpi-abort
[st-d13411.nfit.au.dk:90621] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages  

———————————
When running with 1 core, I get the error message:

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Error in external library
[0]PETSC ERROR: Error in Lapack xGGES 154
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 
[0]PETSC ERROR: hello on a complex named st-d13411.nfit.au.dk by jonat4n Mon May  9 17:59:03 2016
[0]PETSC ERROR: Configure options CC=/usr/local/bin/mpicc CXX=/usr/local/bin/mpicxx F77=/usr/local/bin/mpif77 FC=/usr/local/bin/mpif90 --with-shared-libraries=1 --with-pthread=0 --with-openmp=0 --with-debugging=0 --with-ssl=0 --with-superlu_dist-include=/usr/local/opt/superlu_dist/include/superlu_dist --with-superlu_dist-lib="-L/usr/local/opt/superlu_dist/lib -lsuperlu_dist" --with-superlu-include=/usr/local/Cellar/superlu43/4.3_1/include/superlu --with-superlu-lib="-L/usr/local/Cellar/superlu43/4.3_1/lib -lsuperlu" --with-fftw-dir=/usr/local/opt/fftw --with-netcdf-dir=/usr/local/opt/netcdf --with-suitesparse-dir=/usr/local/opt/suite-sparse --with-hdf5-dir=/usr/local/opt/hdf5 --with-metis-dir=/usr/local/opt/metis --with-parmetis-dir=/usr/local/opt/parmetis --with-scalapack-dir=/usr/local/opt/scalapack --with-mumps-dir=/usr/local/opt/mumps/libexec --with-x=0 --prefix=/usr/local/Cellar/petsc/3.6.3_4/complex --with-scalar-type=complex
[0]PETSC ERROR: #1 DSSolve_GNHEP() line 581 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/sys/classes/ds/impls/gnhep/dsgnhep.c
[0]PETSC ERROR: #2 DSSolve() line 543 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/sys/classes/ds/interface/dsops.c
[0]PETSC ERROR: #3 EPSSolve_CISS() line 943 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
[0]PETSC ERROR: #4 EPSSolve() line 101 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/interface/epssolve.c
[0]PETSC ERROR: #5 main() line 795 in hello.c
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -n 31
[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160509/e8de8fe6/attachment-0001.html>