[petsc-users] "Petsc has generated inconsistent data" during EPSSolve

Mon May 9 11:27:26 CDT 2016

I am on Mac OS X, so I am using the Homebrew version, SLEPc 3.6.2. PETSc is 3.6.3. 
The options for the CISS have worked before, with simpler matrices. As for the options, I am using an ‘ellipse’ region (around 0, with radius=0.16 and vscale=0.5), the problem type is EPS_HEP. For the EPS settings, I set the tolerance to 1e-10 and maxit=200.

- Jonatan

> On 09 May 2016, at 18:16, Jose E. Roman <jroman at dsic.upv.es> wrote:
> 
> Is this with the master branch of SLEPc?
> Which options are setting for the CISS solver?
> 
> Jose
> 
> 
>> El 9 may 2016, a las 18:03, Jonatan Midtgaard <jonatan.midtgaard at gmail.com> escribió:
>> 
>> Hi all. I am using the EPSCISS solver in SLEPc to calculate eigenvalues of a Hermitian matrix close to the origin. (Actually, it is only very nearly Hermitian, but this happens also for NHEP-routines). I am running with 4 cores, and the matrix prints fine with MatView() just before the EPSSolve. Anyhow, I get the following error:
>> (I am new to both PETSc and this mailing list. Should error messages be attached or pasted into the body?)
>> 
>> [2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [2]PETSC ERROR: Petsc has generated inconsistent data
>> [2]PETSC ERROR: ith 3844 block entry 7688 not owned by any process, upper bound 7688
>> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [2]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 
>> [2]PETSC ERROR: hello on a complex named st-d13411.nfit.au.dk by jonat4n Mon May  9 17:47:26 2016
>> [2]PETSC ERROR: Configure options CC=/usr/local/bin/mpicc CXX=/usr/local/bin/mpicxx F77=/usr/local/bin/mpif77 FC=/usr/local/bin/mpif90 --with-shared-libraries=1 --with-pthread=0 --with-openmp=0 --with-debugging=0 --with-ssl=0 --with-superlu_dist-include=/usr/local/opt/superlu_dist/include/superlu_dist --with-superlu_dist-lib="-L/usr/local/opt/superlu_dist/lib -lsuperlu_dist" --with-superlu-include=/usr/local/Cellar/superlu43/4.3_1/include/superlu --with-superlu-lib="-L/usr/local/Cellar/superlu43/4.3_1/lib -lsuperlu" --with-fftw-dir=/usr/local/opt/fftw --with-netcdf-dir=/usr/local/opt/netcdf --with-suitesparse-dir=/usr/local/opt/suite-sparse --with-hdf5-dir=/usr/local/opt/hdf5 --with-metis-dir=/usr/local/opt/metis --with-parmetis-dir=/usr/local/opt/parmetis --with-scalapack-dir=/usr/local/opt/scalapack --with-mumps-dir=/usr/local/opt/mumps/libexec --with-x=0 --prefix=/usr/local/Cellar/petsc/3.6.3_4/complex --with-scalar-type=complex
>> [2]PETSC ERROR: #1 VecScatterCreate_PtoS() line 2339 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
>> [2]PETSC ERROR: #2 VecScatterCreate_StoP() line 2795 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
>> [2]PETSC ERROR: #3 VecScatterCreate_PtoP() line 2984 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
>> [2]PETSC ERROR: #4 VecScatterCreate() line 1570 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vscat.c
>> [2]PETSC ERROR: #5 PCSetUp_Redundant() line 133 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/pc/impls/redundant/redundant.c
>> [2]PETSC ERROR: #6 PCSetUp() line 983 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/pc/interface/precon.c
>> [3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [3]PETSC ERROR: Petsc has generated inconsistent data
>> [3]PETSC ERROR: ith 1922 block entry 7688 not owned by any process, upper bound 7688
>> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [3]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 
>> [3]PETSC ERROR: hello on a complex named st-d13411.nfit.au.dk by jonat4n Mon May  9 17:47:26 2016
>> [3]PETSC ERROR: Configure options CC=/usr/local/bin/mpicc CXX=/usr/local/bin/mpicxx F77=/usr/local/bin/mpif77 FC=/usr/local/bin/mpif90 --with-shared-libraries=1 --with-pthread=0 --with-openmp=0 --with-debugging=0 --with-ssl=0 --with-superlu_dist-include=/usr/local/opt/superlu_dist/include/superlu_dist --with-superlu_dist-lib="-L/usr/local/opt/superlu_dist/lib -lsuperlu_dist" --with-superlu-include=/usr/local/Cellar/superlu43/4.3_1/include/superlu --with-superlu-lib="-L/usr/local/Cellar/superlu43/4.3_1/lib -lsuperlu" --with-fftw-dir=/usr/local/opt/fftw --with-netcdf-dir=/usr/local/opt/netcdf --with-suitesparse-dir=/usr/local/opt/suite-sparse --with-hdf5-dir=/usr/local/opt/hdf5 --with-metis-dir=/usr/local/opt/metis --with-parmetis-dir=/usr/local/opt/parmetis --with-scalapack-dir=/usr/local/opt/scalapack --with-mumps-dir=/usr/local/opt/mumps/libexec --with-x=0 --prefix=/usr/local/Cellar/petsc/3.6.3_4/complex --with-scalar-type=complex
>> [3]PETSC ERROR: #1 VecScatterCreate_PtoS() line 2339 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
>> [3]PETSC ERROR: #2 VecScatterCreate_StoP() line 2795 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
>> [3]PETSC ERROR: #3 VecScatterCreate_PtoP() line 2984 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
>> [3]PETSC ERROR: #4 VecScatterCreate() line 1570 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vscat.c
>> [3]PETSC ERROR: #5 PCSetUp_Redundant() line 133 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/pc/impls/redundant/redundant.c
>> [3]PETSC ERROR: #6 PCSetUp() line 983 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/pc/interface/precon.c
>> [3]PETSC ERROR: #7 KSPSetUp() line 332 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
>> [3]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [1]PETSC ERROR: Petsc has generated inconsistent data
>> [1]PETSC ERROR: ith 5766 block entry 7688 not owned by any process, upper bound 7688
>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [1]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 
>> [1]PETSC ERROR: hello on a complex named st-d13411.nfit.au.dk by jonat4n Mon May  9 17:47:26 2016
>> [1]PETSC ERROR: Configure options CC=/usr/local/bin/mpicc CXX=/usr/local/bin/mpicxx F77=/usr/local/bin/mpif77 FC=/usr/local/bin/mpif90 --with-shared-libraries=1 --with-pthread=0 --with-openmp=0 --with-debugging=0 --with-ssl=0 --with-superlu_dist-include=/usr/local/opt/superlu_dist/include/superlu_dist --with-superlu_dist-lib="-L/usr/local/opt/superlu_dist/lib -lsuperlu_dist" --with-superlu-include=/usr/local/Cellar/superlu43/4.3_1/include/superlu --with-superlu-lib="-L/usr/local/Cellar/superlu43/4.3_1/lib -lsuperlu" --with-fftw-dir=/usr/local/opt/fftw --with-netcdf-dir=/usr/local/opt/netcdf --with-suitesparse-dir=/usr/local/opt/suite-sparse --with-hdf5-dir=/usr/local/opt/hdf5 --with-metis-dir=/usr/local/opt/metis --with-parmetis-dir=/usr/local/opt/parmetis --with-scalapack-dir=/usr/local/opt/scalapack --with-mumps-dir=/usr/local/opt/mumps/libexec --with-x=0 --prefix=/usr/local/Cellar/petsc/3.6.3_4/complex --with-scalar-type=complex
>> [1]PETSC ERROR: #1 VecScatterCreate_PtoS() line 2339 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
>> [1]PETSC ERROR: #2 VecScatterCreate_StoP() line 2795 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
>> [1]PETSC ERROR: #3 VecScatterCreate_PtoP() line 2984 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vpscat.c
>> [1]PETSC ERROR: #4 VecScatterCreate() line 1570 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/vec/vec/utils/vscat.c
>> [1]PETSC ERROR: #5 PCSetUp_Redundant() line 133 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/pc/impls/redundant/redundant.c
>> [1]PETSC ERROR: #6 PCSetUp() line 983 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/pc/interface/precon.c
>> [1]PETSC ERROR: #7 KSPSetUp() line 332 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
>> [1]PETSC ERROR: #8 KSPSolve() line 546 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
>> [1]PETSC ERROR: [2]PETSC ERROR: #7 KSPSetUp() line 332 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
>> [2]PETSC ERROR: #8 KSPSolve() line 546 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
>> [2]PETSC ERROR: #9 SolveLinearSystem() line 337 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
>> [2]PETSC ERROR: #10 EPSSolve_CISS() line 868 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
>> [2]PETSC ERROR: #11 EPSSolve() line 101 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/interface/epssolve.c
>> [2]PETSC ERROR: #12 main() line 795 in hello.c
>> [2]PETSC ERROR: PETSc Option Table entries:
>> [2]PETSC ERROR: -n 31
>> [2]PETSC ERROR: #8 KSPSolve() line 546 in /private/tmp/petsc-20160503-43754-184nsu/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
>> [3]PETSC ERROR: #9 SolveLinearSystem() line 337 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
>> [3]PETSC ERROR: #10 EPSSolve_CISS() line 868 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
>> [3]PETSC ERROR: #11 EPSSolve() line 101 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/interface/epssolve.c
>> [3]PETSC ERROR: #12 main() line 795 in hello.c
>> [3]PETSC ERROR: PETSc Option Table entries:
>> [3]PETSC ERROR: -n 31
>> #9 SolveLinearSystem() line 337 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
>> [1]PETSC ERROR: #10 EPSSolve_CISS() line 868 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
>> [1]PETSC ERROR: #11 EPSSolve() line 101 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/interface/epssolve.c
>> [1]PETSC ERROR: #12 main() line 795 in hello.c
>> [1]PETSC ERROR: PETSc Option Table entries:
>> [1]PETSC ERROR: -n 31
>> ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
>> [3]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
>> [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
>> --------------------------------------------------------------------------
>> MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD 
>> with errorcode 77.
>> 
>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>> You may or may not see output from other processes, depending on
>> exactly when Open MPI kills them.
>> --------------------------------------------------------------------------
>> [0]PETSC ERROR: ------------------------------------------------------------------------
>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
>> [0]PETSC ERROR: to get more information on the crash.
>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [0]PETSC ERROR: Signal received
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 
>> [0]PETSC ERROR: hello on a complex named st-d13411.nfit.au.dk by jonat4n Mon May  9 17:47:26 2016
>> [0]PETSC ERROR: Configure options CC=/usr/local/bin/mpicc CXX=/usr/local/bin/mpicxx F77=/usr/local/bin/mpif77 FC=/usr/local/bin/mpif90 --with-shared-libraries=1 --with-pthread=0 --with-openmp=0 --with-debugging=0 --with-ssl=0 --with-superlu_dist-include=/usr/local/opt/superlu_dist/include/superlu_dist --with-superlu_dist-lib="-L/usr/local/opt/superlu_dist/lib -lsuperlu_dist" --with-superlu-include=/usr/local/Cellar/superlu43/4.3_1/include/superlu --with-superlu-lib="-L/usr/local/Cellar/superlu43/4.3_1/lib -lsuperlu" --with-fftw-dir=/usr/local/opt/fftw --with-netcdf-dir=/usr/local/opt/netcdf --with-suitesparse-dir=/usr/local/opt/suite-sparse --with-hdf5-dir=/usr/local/opt/hdf5 --with-metis-dir=/usr/local/opt/metis --with-parmetis-dir=/usr/local/opt/parmetis --with-scalapack-dir=/usr/local/opt/scalapack --with-mumps-dir=/usr/local/opt/mumps/libexec --with-x=0 --prefix=/usr/local/Cellar/petsc/3.6.3_4/complex --with-scalar-type=complex
>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>> [st-d13411.nfit.au.dk:90621] 3 more processes have sent help message help-mpi-api.txt / mpi-abort
>> [st-d13411.nfit.au.dk:90621] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages  
>> 
>> ———————————
>> When running with 1 core, I get the error message:
>> 
>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [0]PETSC ERROR: Error in external library
>> [0]PETSC ERROR: Error in Lapack xGGES 154
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 
>> [0]PETSC ERROR: hello on a complex named st-d13411.nfit.au.dk by jonat4n Mon May  9 17:59:03 2016
>> [0]PETSC ERROR: Configure options CC=/usr/local/bin/mpicc CXX=/usr/local/bin/mpicxx F77=/usr/local/bin/mpif77 FC=/usr/local/bin/mpif90 --with-shared-libraries=1 --with-pthread=0 --with-openmp=0 --with-debugging=0 --with-ssl=0 --with-superlu_dist-include=/usr/local/opt/superlu_dist/include/superlu_dist --with-superlu_dist-lib="-L/usr/local/opt/superlu_dist/lib -lsuperlu_dist" --with-superlu-include=/usr/local/Cellar/superlu43/4.3_1/include/superlu --with-superlu-lib="-L/usr/local/Cellar/superlu43/4.3_1/lib -lsuperlu" --with-fftw-dir=/usr/local/opt/fftw --with-netcdf-dir=/usr/local/opt/netcdf --with-suitesparse-dir=/usr/local/opt/suite-sparse --with-hdf5-dir=/usr/local/opt/hdf5 --with-metis-dir=/usr/local/opt/metis --with-parmetis-dir=/usr/local/opt/parmetis --with-scalapack-dir=/usr/local/opt/scalapack --with-mumps-dir=/usr/local/opt/mumps/libexec --with-x=0 --prefix=/usr/local/Cellar/petsc/3.6.3_4/complex --with-scalar-type=complex
>> [0]PETSC ERROR: #1 DSSolve_GNHEP() line 581 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/sys/classes/ds/impls/gnhep/dsgnhep.c
>> [0]PETSC ERROR: #2 DSSolve() line 543 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/sys/classes/ds/interface/dsops.c
>> [0]PETSC ERROR: #3 EPSSolve_CISS() line 943 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/impls/ciss/ciss.c
>> [0]PETSC ERROR: #4 EPSSolve() line 101 in /private/tmp/slepc-20160503-86919-187yrgp/slepc-3.6.2/src/eps/interface/epssolve.c
>> [0]PETSC ERROR: #5 main() line 795 in hello.c
>> [0]PETSC ERROR: PETSc Option Table entries:
>> [0]PETSC ERROR: -n 31
>> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> 