[petsc-users] MUMPS Error

Tue Jan 8 10:13:39 CST 2019

Sal:
How large is the matrix, its dimension and number of non-zeros?
Can you run it sequentially or with less number of processors?
The matrix is 139603x139603 and with 15975113 of non-zero elements.
This is quite large for direct solver, but it is very sparse. Have you tried different matrix orderings to reduce fillings?
You may try to use superu_dist which might consume less memory than mumps.

I tried using the iterative solver and started on one node then increased to two there was no performance increase in using 2 nodes with 32 cores on each node vs using 1 node with 32 cores (both took ~12min). Increasing the number of nodes seems to crash it though.
What  iterative solver do you use? Increasing the number of nodes likely changes preconditioner.

 Can you run your case on different machine?
Running it on the work machine, it seems to work using the iterative solver but will give me similar error I first posted if I use MUMPS (INFOG(1)=-9, INFO(2)=0)
We need know what   iterative solver is being used. Run your code with '-ksp_view'.

Hong
.

On Mon, Jan 7, 2019 at 4:47 PM Zhang, Hong <hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov>> wrote:
Sal:
Recent versions of MUMPS have bugs when using SCALAPACK. You may also try to disable parallelism on the root node by using
tried that, does not seem to help... but it is odd. this is not a big system, it is small and made for testing.

How large is the matrix, its dimension and number of non-zeros?
Can you run it sequentially or with less number of processors?

perhaps there is something wrong with the MPI....If anyone is using the PBSPro for queueing system, could it be the incompatibility between the MPICH and the PBS? Should one use OpenMPI?
 Can you run your case on different machine?
Hong

On Thu, Jan 3, 2019 at 3:55 PM Stefano Zampini <stefano.zampini at gmail.com<mailto:stefano.zampini at gmail.com>> wrote:
Recent versions of MUMPS have bugs when using SCALAPACK. You may also try to disable parallelism on the root node by using

 -mat_mumps_icntl_13 1

On Jan 3, 2019, at 4:33 PM, Zhang, Hong via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:

You may try different matrix orderings, or try superlu_dist.
Hong

On Thu, Jan 3, 2019 at 7:39 AM Matthew Knepley via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
On Thu, Jan 3, 2019 at 5:58 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Error in external library
[0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-22193

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[0]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[0]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[0]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[4]PETSC ERROR: Error in external library
[4]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-24197

[4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[4]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[4]PETSC ERROR: [16]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[16]PETSC ERROR: Error in external library
[16]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[16]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[16]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[16]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[16]PETSC ERROR: [20]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[20]PETSC ERROR: Error in external library
[20]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[20]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[20]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[20]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[20]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[20]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[24]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[24]PETSC ERROR: Error in external library
[24]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[24]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[24]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[24]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[24]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[24]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[24]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[24]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[24]PETSC ERROR: #4 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[24]PETSC ERROR: #5 KSPSetUp() line 391 in /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[24]PETSC ERROR: [32]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[32]PETSC ERROR: Error in external library
[32]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[32]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[32]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[32]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[32]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[32]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[32]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[32]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[32]PETSC ERROR: #4 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[32]PETSC ERROR: #5 KSPSetUp() line 391 in /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[32]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[8]PETSC ERROR: Error in external library
[8]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-35665

[8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[8]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[8]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[8]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[8]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[8]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[8]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[8]PETSC ERROR: #4 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[8]PETSC ERROR: [12]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[12]PETSC ERROR: Error in external library
[12]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-16661

[12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[12]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[12]PETSC ERROR: ./solveCSys on a linux-opt named r03n13 by vef002 Thu Jan  3 04:32:43 2019
[12]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[12]PETSC ERROR: #6 main() line 68 in /lustre/home/vef002/tests/solveCmplxLinearSys.cpp
[24]PETSC ERROR: PETSc Option Table entries:
[24]PETSC ERROR: -ksp_error_if_not_converged
[24]PETSC ERROR: -ksp_max_it 1
[24]PETSC ERROR: -ksp_monitor_true_residual
[24]PETSC ERROR: -ksp_type richardson
[24]PETSC ERROR: -log_view
[24]PETSC ERROR: -mat_mumps_icntl_14 100
[24]PETSC ERROR: -pc_factor_mat_solver_type mumps
[24]PETSC ERROR: -pc_type lu
[24]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov<mailto:petsc-maint at mcs.anl.gov>----------

Sorry to bother again, but increasing ICNTL(14) does not seem to have helped... I am not sure what I am doing wrong exactly. As you can see I have increased it by 100%, prior, I tried to increase it with 40-70% but none worked.

I think this is for the MUMPS list, but it appears you have a lot of fill-in for this matrix and it might be too much for a direct solver.

  Thanks,

    Matt

On Wed, Dec 12, 2018 at 3:38 PM Zhang, Junchao <jczhang at mcs.anl.gov<mailto:jczhang at mcs.anl.gov>> wrote:

On Wed, Dec 12, 2018 at 7:14 AM Sal Am via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hi I am getting an error using MUMPS.
How I run it:
bash-4.2$ mpiexec -n 16 ./solveCSys -ksp_type richardson -pc_type lu -pc_factor_mat_solver_type mumps -ksp_max_it 1 -ksp_monitor_true_residual -log_view -ksp_error_if_not_converged

The error output:
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Error in external library
[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: Error in external library
[1]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[6]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[6]PETSC ERROR: Error in external library
[6]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[6]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[8]PETSC ERROR: Error in external library
[8]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[8]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[8]PETSC ERROR: [9]PETSC ERROR: [15]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[15]PETSC ERROR: Error in external library
[15]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-36536

[15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[15]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[15]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[15]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[15]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-81813

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[1]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[1]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[1]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[2]PETSC ERROR: Error in external library
[2]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[2]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[2]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[2]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[3]PETSC ERROR: Error in external library
[3]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-28194

[3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[3]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[3]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[3]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[3]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[4]PETSC ERROR: Error in external library
[4]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[4]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[4]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[4]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[5]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[5]PETSC ERROR: Error in external library
[5]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-27685

[5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[5]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[5]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[5]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[6]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[6]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[6]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[6]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[6]PETSC ERROR: [7]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[7]PETSC ERROR: Error in external library
[7]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[7]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[7]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[7]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[7]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[7]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[8]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[8]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[8]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[8]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[10]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[10]PETSC ERROR: Error in external library
[10]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[10]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[10]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[10]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[10]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[10]PETSC ERROR: [11]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[11]PETSC ERROR: Error in external library
[11]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-30519

[11]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[11]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[11]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[11]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[11]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[11]PETSC ERROR: [12]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[12]PETSC ERROR: Error in external library
[12]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-28435

[12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[12]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[12]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[12]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[12]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[12]PETSC ERROR: [13]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[13]PETSC ERROR: Error in external library
[13]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=-30110

[13]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[13]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[13]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[13]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[13]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[13]PETSC ERROR: [14]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[14]PETSC ERROR: Error in external library
[14]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-13, INFO(2)=0

[14]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[14]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[14]PETSC ERROR: ./solveCSys on a linux-opt named r02g03 by vef002 Wed Dec 12 10:44:13 2018
[14]PETSC ERROR: Configure options PETSC_ARCH=linux-opt --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-clanguage=cxx --download-superlu_dist --download-mumps --with-scalar-type=complex --with-debugging=no --download-scalapack --download-superlu --download-mpich --download-fblaslapack=1 --download-cmake
[14]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[14]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
#1 MatFactorNumeric_MUMPS() line 1230 in /lustre/home/vef002/petsc/src/mat/impls/aij/mpi/mumps/mumps.c
[15]PETSC ERROR: #2 MatLUFactorNumeric() line 3065 in /lustre/home/vef002/petsc/src/mat/interface/matrix.c
[15]PETSC ERROR: #3 PCSetUp_LU() line 131 in /lustre/home/vef002/petsc/src/ksp/pc/impls/factor/lu/lu.c
[15]PETSC ERROR: #4 PCSetUp() line 932 in /lustre/home/vef002/petsc/src/ksp/pc/interface/precon.c
[15]PETSC ERROR: #5 KSPSetUp() line 391 in /lustre/home/vef002/petsc/src/ksp/ksp/interface/itfunc.c
[15]PETSC ERROR: #6 main() line 68 in /lustre/home/vef002/tests/solveCmplxLinearSys.cpp
[15]PETSC ERROR: PETSc Option Table entries:
[15]PETSC ERROR: -ksp_error_if_not_converged
[15]PETSC ERROR: -ksp_max_it 1
[15]PETSC ERROR: -ksp_monitor_true_residual
[15]PETSC ERROR: -ksp_type richardson
[15]PETSC ERROR: -log_view
[15]PETSC ERROR: -pc_factor_mat_solver_type mumps
[15]PETSC ERROR: -pc_type lu
[15]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov<mailto:petsc-maint at mcs.anl.gov>----------

From the MUMPS manual it seems INFOG(1)=-13, INFO(2)=0 :
Problem of workspace allocation of size INFO(2) during the factorization or solve steps. The size that the package tried to allocate with a Fortran ALLOCATE statement is available in INFO(2). If INFO(2) is negative, then the size that the package requested is obtained by multiplying the absolute value of INFO(2) by 1 million. In general, the unit for INFO(2) is the number of scalar entries of  the type of the input matrix (real, complex, single or double precision).

Not quite sure what to make of it as the first error indicates size INFO(2)= 0?. Would I need to make use of -mat_mumps_icntl_14?
But you can see there are processes with negative INFO(2). According to MUMPS manual, you can try to increase ICNTL(14).

Thanks

--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190108/a5a8cba3/attachment-0001.html>