[petsc-users] "Petsc has generated inconsistent data" during EPSSolve

Mon May 9 13:16:09 CDT 2016

   To eliminate the possibility that the problem is due to memory corruption due to some programming error you should run under valgrind:
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

   Barry


> On May 9, 2016, at 11:03 AM, Jonatan Midtgaard <jonatan.midtgaard at gmail.com> wrote:
> 
> Hi all. I am using the EPSCISS solver in SLEPc to calculate eigenvalues of a Hermitian matrix close to the origin. (Actually, it is only very nearly Hermitian, but this happens also for NHEP-routines). I am running with 4 cores, and the matrix prints fine with MatView() just before the EPSSolve. Anyhow, I get the following error:
> (I am new to both PETSc and this mailing list. Should error messages be attached or pasted into the body?)
> 
> [2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [2]PETSC ERROR: Petsc has generated inconsistent data
> [2]PETSC ERROR: ith 3844 block entry 7688 not owned by any process, upper bound 7688
> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [2]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 
> [2]PETSC ERROR: hello on a complex named st-d13411.nfit.au.dk by jonat4n Mon May  9 17:47:26 2016
> [2]PETSC ERROR: Configure options CC=/usr/local/bin/mpicc CXX=/usr/local/bin/mpicxx F77=/usr/local/bin/mpif77 FC=/usr/local/bin/mpif90 --with-shared-libraries=1 --with-pthread=0 --with-openmp=0 --with-debugging=0 --with-ssl=0 --with-superlu_dist-include=/usr/local/opt/superlu_dist/include/superlu_dist --with-superlu_dist-lib="-L/usr/local/opt/superlu_dist/lib -lsuperlu_dist" --with-superlu-include=/usr/local/Cellar/superlu43/4.3_1/include/superlu --with-superlu-lib="-L/usr/local/Cellar/superlu43/4.3_1/lib -lsuperlu" --with-fftw-dir=/usr/local/opt/fftw --with-netcdf-dir=/usr/local/opt/netcdf --with-suitesparse-dir=/usr/local/opt/suite-sparse --with-hdf5-dir=/usr/local/opt/hdf5 --with-metis-dir=/usr/local/opt/metis --with-parmetis-dir=/usr/local/opt/parmetis --with-scalapack-dir=/usr/local/opt/scalapack --with-mumps-dir=/usr/local/opt/mumps/libexec --with-x=0 --prefix=/usr/local/Cellar/petsc/3.6.3_4/complex --with-scalar-type=complex
> [2]PETSC ERROR: #1 VecScatterCreate_PtoS() line 2339 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
> [2]PETSC ERROR: #2 VecScatterCreate_StoP() line 2795 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
> [2]PETSC ERROR: #3 VecScatterCreate_PtoP() line 2984 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
> [2]PETSC ERROR: #4 VecScatterCreate() line 1570 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vscat.c
> [2]PETSC ERROR: #5 PCSetUp_Redundant() line 133 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/pc/impls/redundant/redundant.c
> [2]PETSC ERROR: #6 PCSetUp() line 983 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/pc/interface/precon.c
> [3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [3]PETSC ERROR: Petsc has generated inconsistent data
> [3]PETSC ERROR: ith 1922 block entry 7688 not owned by any process, upper bound 7688
> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [3]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 
> [3]PETSC ERROR: hello on a complex named st-d13411.nfit.au.dk by jonat4n Mon May  9 17:47:26 2016
> [3]PETSC ERROR: Configure options CC=/usr/local/bin/mpicc CXX=/usr/local/bin/mpicxx F77=/usr/local/bin/mpif77 FC=/usr/local/bin/mpif90 --with-shared-libraries=1 --with-pthread=0 --with-openmp=0 --with-debugging=0 --with-ssl=0 --with-superlu_dist-include=/usr/local/opt/superlu_dist/include/superlu_dist --with-superlu_dist-lib="-L/usr/local/opt/superlu_dist/lib -lsuperlu_dist" --with-superlu-include=/usr/local/Cellar/superlu43/4.3_1/include/superlu --with-superlu-lib="-L/usr/local/Cellar/superlu43/4.3_1/lib -lsuperlu" --with-fftw-dir=/usr/local/opt/fftw --with-netcdf-dir=/usr/local/opt/netcdf --with-suitesparse-dir=/usr/local/opt/suite-sparse --with-hdf5-dir=/usr/local/opt/hdf5 --with-metis-dir=/usr/local/opt/metis --with-parmetis-dir=/usr/local/opt/parmetis --with-scalapack-dir=/usr/local/opt/scalapack --with-mumps-dir=/usr/local/opt/mumps/libexec --with-x=0 --prefix=/usr/local/Cellar/petsc/3.6.3_4/complex --with-scalar-type=complex
> [3]PETSC ERROR: #1 VecScatterCreate_PtoS() line 2339 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
> [3]PETSC ERROR: #2 VecScatterCreate_StoP() line 2795 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
> [3]PETSC ERROR: #3 VecScatterCreate_PtoP() line 2984 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
> [3]PETSC ERROR: #4 VecScatterCreate() line 1570 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vscat.c
> [3]PETSC ERROR: #5 PCSetUp_Redundant() line 133 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/pc/impls/redundant/redundant.c
> [3]PETSC ERROR: #6 PCSetUp() line 983 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/pc/interface/precon.c
> [3]PETSC ERROR: #7 KSPSetUp() line 332 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
> [3]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [1]PETSC ERROR: Petsc has generated inconsistent data
> [1]PETSC ERROR: ith 5766 block entry 7688 not owned by any process, upper bound 7688
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 
> [1]PETSC ERROR: hello on a complex named st-d13411.nfit.au.dk by jonat4n Mon May  9 17:47:26 2016
> [1]PETSC ERROR: Configure options CC=/usr/local/bin/mpicc CXX=/usr/local/bin/mpicxx F77=/usr/local/bin/mpif77 FC=/usr/local/bin/mpif90 --with-shared-libraries=1 --with-pthread=0 --with-openmp=0 --with-debugging=0 --with-ssl=0 --with-superlu_dist-include=/usr/local/opt/superlu_dist/include/superlu_dist --with-superlu_dist-lib="-L/usr/local/opt/superlu_dist/lib -lsuperlu_dist" --with-superlu-include=/usr/local/Cellar/superlu43/4.3_1/include/superlu --with-superlu-lib="-L/usr/local/Cellar/superlu43/4.3_1/lib -lsuperlu" --with-fftw-dir=/usr/local/opt/fftw --with-netcdf-dir=/usr/local/opt/netcdf --with-suitesparse-dir=/usr/local/opt/suite-sparse --with-hdf5-dir=/usr/local/opt/hdf5 --with-metis-dir=/usr/local/opt/metis --with-parmetis-dir=/usr/local/opt/parmetis --with-scalapack-dir=/usr/local/opt/scalapack --with-mumps-dir=/usr/local/opt/mumps/libexec --with-x=0 --prefix=/usr/local/Cellar/petsc/3.6.3_4/complex --with-scalar-type=complex
> [1]PETSC ERROR: #1 VecScatterCreate_PtoS() line 2339 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
> [1]PETSC ERROR: #2 VecScatterCreate_StoP() line 2795 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
> [1]PETSC ERROR: #3 VecScatterCreate_PtoP() line 2984 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
> [1]PETSC ERROR: #4 VecScatterCreate() line 1570 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vscat.c
> [1]PETSC ERROR: #5 PCSetUp_Redundant() line 133 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/pc/impls/redundant/redundant.c
> [1]PETSC ERROR: #6 PCSetUp() line 983 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/pc/interface/precon.c
> [1]PETSC ERROR: #7 KSPSetUp() line 332 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: #8 KSPSolve() line 546 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: [2]PETSC ERROR: #7 KSPSetUp() line 332 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
> [2]PETSC ERROR: #8 KSPSolve() line 546 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
> [2]PETSC ERROR: #9 SolveLinearSystem() line 337 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
> [2]PETSC ERROR: #10 EPSSolve_CISS() line 868 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
> [2]PETSC ERROR: #11 EPSSolve() line 101 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/interface/epssolve.c
> [2]PETSC ERROR: #12 main() line 795 in hello.c
> [2]PETSC ERROR: PETSc Option Table entries:
> [2]PETSC ERROR: -n 31
> [2]PETSC ERROR: #8 KSPSolve() line 546 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
> [3]PETSC ERROR: #9 SolveLinearSystem() line 337 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
> [3]PETSC ERROR: #10 EPSSolve_CISS() line 868 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
> [3]PETSC ERROR: #11 EPSSolve() line 101 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/interface/epssolve.c
> [3]PETSC ERROR: #12 main() line 795 in hello.c
> [3]PETSC ERROR: PETSc Option Table entries:
> [3]PETSC ERROR: -n 31
> #9 SolveLinearSystem() line 337 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
> [1]PETSC ERROR: #10 EPSSolve_CISS() line 868 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
> [1]PETSC ERROR: #11 EPSSolve() line 101 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/interface/epssolve.c
> [1]PETSC ERROR: #12 main() line 795 in hello.c
> [1]PETSC ERROR: PETSc Option Table entries:
> [1]PETSC ERROR: -n 31
> ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> [3]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD 
> with errorcode 77.
> 
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
> --------------------------------------------------------------------------
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 
> [0]PETSC ERROR: hello on a complex named st-d13411.nfit.au.dk by jonat4n Mon May  9 17:47:26 2016
> [0]PETSC ERROR: Configure options CC=/usr/local/bin/mpicc CXX=/usr/local/bin/mpicxx F77=/usr/local/bin/mpif77 FC=/usr/local/bin/mpif90 --with-shared-libraries=1 --with-pthread=0 --with-openmp=0 --with-debugging=0 --with-ssl=0 --with-superlu_dist-include=/usr/local/opt/superlu_dist/include/superlu_dist --with-superlu_dist-lib="-L/usr/local/opt/superlu_dist/lib -lsuperlu_dist" --with-superlu-include=/usr/local/Cellar/superlu43/4.3_1/include/superlu --with-superlu-lib="-L/usr/local/Cellar/superlu43/4.3_1/lib -lsuperlu" --with-fftw-dir=/usr/local/opt/fftw --with-netcdf-dir=/usr/local/opt/netcdf --with-suitesparse-dir=/usr/local/opt/suite-sparse --with-hdf5-dir=/usr/local/opt/hdf5 --with-metis-dir=/usr/local/opt/metis --with-parmetis-dir=/usr/local/opt/parmetis --with-scalapack-dir=/usr/local/opt/scalapack --with-mumps-dir=/usr/local/opt/mumps/libexec --with-x=0 --prefix=/usr/local/Cellar/petsc/3.6.3_4/complex --with-scalar-type=complex
> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> [st-d13411.nfit.au.dk:90621] 3 more processes have sent help message help-mpi-api.txt / mpi-abort
> [st-d13411.nfit.au.dk:90621] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages  
> 
> ———————————
> When running with 1 core, I get the error message:
> 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Error in external library
> [0]PETSC ERROR: Error in Lapack xGGES 154
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 
> [0]PETSC ERROR: hello on a complex named st-d13411.nfit.au.dk by jonat4n Mon May  9 17:59:03 2016
> [0]PETSC ERROR: Configure options CC=/usr/local/bin/mpicc CXX=/usr/local/bin/mpicxx F77=/usr/local/bin/mpif77 FC=/usr/local/bin/mpif90 --with-shared-libraries=1 --with-pthread=0 --with-openmp=0 --with-debugging=0 --with-ssl=0 --with-superlu_dist-include=/usr/local/opt/superlu_dist/include/superlu_dist --with-superlu_dist-lib="-L/usr/local/opt/superlu_dist/lib -lsuperlu_dist" --with-superlu-include=/usr/local/Cellar/superlu43/4.3_1/include/superlu --with-superlu-lib="-L/usr/local/Cellar/superlu43/4.3_1/lib -lsuperlu" --with-fftw-dir=/usr/local/opt/fftw --with-netcdf-dir=/usr/local/opt/netcdf --with-suitesparse-dir=/usr/local/opt/suite-sparse --with-hdf5-dir=/usr/local/opt/hdf5 --with-metis-dir=/usr/local/opt/metis --with-parmetis-dir=/usr/local/opt/parmetis --with-scalapack-dir=/usr/local/opt/scalapack --with-mumps-dir=/usr/local/opt/mumps/libexec --with-x=0 --prefix=/usr/local/Cellar/petsc/3.6.3_4/complex --with-scalar-type=complex
> [0]PETSC ERROR: #1 DSSolve_GNHEP() line 581 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/sys/classes/ds/impls/gnhep/dsgnhep.c
> [0]PETSC ERROR: #2 DSSolve() line 543 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/sys/classes/ds/interface/dsops.c
> [0]PETSC ERROR: #3 EPSSolve_CISS() line 943 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
> [0]PETSC ERROR: #4 EPSSolve() line 101 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/interface/epssolve.c
> [0]PETSC ERROR: #5 main() line 795 in hello.c
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -n 31
> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------