[petsc-users] SuperLU_dist bug with parallel symbolic factorisation

[Eric Chamberland]

Hi,

The given matrix+vector is bogus with SuperLU_Dist on some of our 
nighlty validation tests since I activated the parallel symbolic 
factorisation. (with -mat_superlu_dist_colperm PARMETIS 
-mat_superlu_dist_parsymbfact 1 )

I extracted an example system and reproduced the bug with 
src/ksp/ksp/examples/tests/ex6.c that I can run it with 2 or 3 
processes, but with 4 it gives a FPE on process #1:

mpirun -n 4 ./ex6 -f AssembleurGD_resolution_no_0_0 -ksp_view -ksp_type 
preonly -pc_type lu -pc_factor_mat_solver_type superlu_dist 
-mat_superlu_dist_colperm PARMETIS -mat_superlu_dist_parsymbfact 1

...
[1]PETSC ERROR: 
------------------------------------------------------------------------
[1]PETSC ERROR: Caught signal number 8 FPE: Floating Point 
Exception,probably divide by zero
[1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[1]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS 
X to find memory corruption errors
[1]PETSC ERROR: likely location of problem given in stack below
[1]PETSC ERROR: ---------------------  Stack Frames 
------------------------------------
[1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[1]PETSC ERROR:       INSTEAD the line number of the start of the function
[1]PETSC ERROR:       is given.
[1]PETSC ERROR: [1] SuperLU_DIST:pdgssvx line 467 
/home/mefpp_ericc/petsc-3.9.2-debug/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[1]PETSC ERROR: [1] MatLUFactorNumeric_SuperLU_DIST line 314 
/home/mefpp_ericc/petsc-3.9.2-debug/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[1]PETSC ERROR: [1] MatLUFactorNumeric line 3014 
/home/mefpp_ericc/petsc-3.9.2-debug/src/mat/interface/matrix.c
[1]PETSC ERROR: [1] PCSetUp_LU line 59 
/home/mefpp_ericc/petsc-3.9.2-debug/src/ksp/pc/impls/factor/lu/lu.c
[1]PETSC ERROR: [1] PCSetUp line 885 
/home/mefpp_ericc/petsc-3.9.2-debug/src/ksp/pc/interface/precon.c
[1]PETSC ERROR: [1] KSPSetUp line 294 
/home/mefpp_ericc/petsc-3.9.2-debug/src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR: --------------------- Error Message 
--------------------------------------------------------------
[1]PETSC ERROR: Signal received
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html 
for trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.9.2, May, 20, 2018
[1]PETSC ERROR: ./ex6 on a  named lorien by eric Tue May 22 10:39:15 2018
[1]PETSC ERROR: Configure options 
--prefix=/opt/petsc-3.9.2_debug_openmpi-1.10.2 --with-mpi-compilers=1 
--with-mpi-dir=/opt/openmpi-1.10.2 --with-make-np=12 
--with-shared-libraries=1 --with-debugging=yes --with-memalign=64 
--with-visibility=0 --with-64-bit-indices=0 --download-ml=yes 
--download-mumps=yes --download-superlu=yes --download-superlu_dist=yes 
--download-parmetis=yes --download-ptscotch=yes --download-metis=yes 
--download-suitesparse=yes --download-hypre=yes 
--with-blaslapack-dir=/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 
--with-mkl_pardiso-dir=/opt/intel/composer_xe_2015.2.164/mkl 
--with-mkl_cpardiso-dir=/opt/intel/composer_xe_2015.2.164/mkl 
--with-scalapack=1 
--with-scalapack-include=/opt/intel/composer_xe_2015.2.164/mkl/include 
--with-scalapack-lib="-L/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 
-lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64"
[1]PETSC ERROR: #1 User provided function() line 0 in  unknown file
...

The given Matrix+Vector are available here:

http://www.giref.ulaval.ca/~ericc/bug_superlu_dist_parallel_factorisation/AssembleurGD_resolution_no_0_0

http://www.giref.ulaval.ca/~ericc/bug_superlu_dist_parallel_factorisation/AssembleurGD_resolution_no_0_0.info

If I run with -on_error_attach_debugger, I can see a division by zero here:

#8  <signal handler called>
(gdb)
#9  0x00007f96a2148e52 in libmetis__FM_2WayCutRefine (ctrl=0x2784d20, 
graph=0x2784940, ntpwgts=0x7ffdfa323060, niter=4)
     at 
/home/mefpp_ericc/petsc-3.9.2-debug/arch-linux2-c-debug/externalpackages/git.metis/libmetis/fm.c:60
60        avgvwgt = gk_min((pwgts[0]+pwgts[1])/20, 
2*(pwgts[0]+pwgts[1])/nvtxs);

and nvtxs value is "0"...

Thanks!

Eric