[petsc-users] SuperLU_dist bug with parallel symbolic factorisation

[Smith, Barry F.]

0x00007f96a2148e52 in libmetis__FM_2WayCutRefine (ctrl=0x2784d20, graph=0x2784940, ntpwgts=0x7ffdfa323060, niter=4)
at /home/mefpp_ericc/petsc-3.9.2-debug/arch-linux2-c-debug/externalpackages/git.metis/libmetis/fm.c:60

It appears the crash is in metis, not SuperLU_Dist.

  So either a bug in Metis or a bug in our Metis is called by ParMetis or SuperLU_Dist.

   Barry




> On May 22, 2018, at 10:37 AM, Hong <hzhang at mcs.anl.gov> wrote:
> 
> Eric:
> Likely, you encounter a zero pivot. Run your code with '-ksp_error_if_not_converged' would show it.
> Adding option '-mat_superlu_dist_replacetinypivot' might help.
> Hong
> 
> Hi,
> 
> The given matrix+vector is bogus with SuperLU_Dist on some of our nighlty validation tests since I activated the parallel symbolic factorisation. (with -mat_superlu_dist_colperm PARMETIS -mat_superlu_dist_parsymbfact 1 )
> 
> I extracted an example system and reproduced the bug with src/ksp/ksp/examples/tests/ex6.c that I can run it with 2 or 3 processes, but with 4 it gives a FPE on process #1:
> 
> mpirun -n 4 ./ex6 -f AssembleurGD_resolution_no_0_0 -ksp_view -ksp_type preonly -pc_type lu -pc_factor_mat_solver_type superlu_dist -mat_superlu_dist_colperm PARMETIS -mat_superlu_dist_parsymbfact 1
> 
> ...
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero
> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [1]PETSC ERROR: likely location of problem given in stack below
> [1]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [1]PETSC ERROR:       INSTEAD the line number of the start of the function
> [1]PETSC ERROR:       is given.
> [1]PETSC ERROR: [1] SuperLU_DIST:pdgssvx line 467 /home/mefpp_ericc/petsc-3.9.2-debug/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [1]PETSC ERROR: [1] MatLUFactorNumeric_SuperLU_DIST line 314 /home/mefpp_ericc/petsc-3.9.2-debug/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [1]PETSC ERROR: [1] MatLUFactorNumeric line 3014 /home/mefpp_ericc/petsc-3.9.2-debug/src/mat/interface/matrix.c
> [1]PETSC ERROR: [1] PCSetUp_LU line 59 /home/mefpp_ericc/petsc-3.9.2-debug/src/ksp/pc/impls/factor/lu/lu.c
> [1]PETSC ERROR: [1] PCSetUp line 885 /home/mefpp_ericc/petsc-3.9.2-debug/src/ksp/pc/interface/precon.c
> [1]PETSC ERROR: [1] KSPSetUp line 294 /home/mefpp_ericc/petsc-3.9.2-debug/src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [1]PETSC ERROR: Signal received
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.9.2, May, 20, 2018
> [1]PETSC ERROR: ./ex6 on a  named lorien by eric Tue May 22 10:39:15 2018
> [1]PETSC ERROR: Configure options --prefix=/opt/petsc-3.9.2_debug_openmpi-1.10.2 --with-mpi-compilers=1 --with-mpi-dir=/opt/openmpi-1.10.2 --with-make-np=12 --with-shared-libraries=1 --with-debugging=yes --with-memalign=64 --with-visibility=0 --with-64-bit-indices=0 --download-ml=yes --download-mumps=yes --download-superlu=yes --download-superlu_dist=yes --download-parmetis=yes --download-ptscotch=yes --download-metis=yes --download-suitesparse=yes --download-hypre=yes --with-blaslapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 --with-mkl_pardiso-dir=/opt/intel/composer_xe_2015.2.164/mkl --with-mkl_cpardiso-dir=/opt/intel/composer_xe_2015.2.164/mkl --with-scalapack=1 --with-scalapack-include=/opt/intel/composer_xe_2015.2.164/mkl/include --with-scalapack-lib="-L/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64"
> [1]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> ...
> 
> The given Matrix+Vector are available here:
> 
> http://www.giref.ulaval.ca/~ericc/bug_superlu_dist_parallel_factorisation/AssembleurGD_resolution_no_0_0
> 
> http://www.giref.ulaval.ca/~ericc/bug_superlu_dist_parallel_factorisation/AssembleurGD_resolution_no_0_0.info
> 
> If I run with -on_error_attach_debugger, I can see a division by zero here:
> 
> #8  <signal handler called>
> (gdb)
> #9  0x00007f96a2148e52 in libmetis__FM_2WayCutRefine (ctrl=0x2784d20, graph=0x2784940, ntpwgts=0x7ffdfa323060, niter=4)
>     at /home/mefpp_ericc/petsc-3.9.2-debug/arch-linux2-c-debug/externalpackages/git.metis/libmetis/fm.c:60
> 60        avgvwgt = gk_min((pwgts[0]+pwgts[1])/20, 2*(pwgts[0]+pwgts[1])/nvtxs);
> 
> and nvtxs value is "0"...
> 
> Thanks!
> 
> Eric
>