<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">Eric:</div><div class="gmail_quote">Likely, you encounter a zero pivot. Run your code with '-ksp_error_if_not_converged' would show it.</div><div class="gmail_quote">Adding option '-mat_superlu_dist_replacetinypivot' might help.</div><div class="gmail_quote">Hong</div><div class="gmail_quote"><br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
The given matrix+vector is bogus with SuperLU_Dist on some of our nighlty validation tests since I activated the parallel symbolic factorisation. (with -mat_superlu_dist_colperm PARMETIS -mat_superlu_dist_parsymbfact 1 )<br>
<br>
I extracted an example system and reproduced the bug with src/ksp/ksp/examples/tests/ex6<wbr>.c that I can run it with 2 or 3 processes, but with 4 it gives a FPE on process #1:<br>
<br>
mpirun -n 4 ./ex6 -f AssembleurGD_resolution_no_0_0 -ksp_view -ksp_type preonly -pc_type lu -pc_factor_mat_solver_type superlu_dist -mat_superlu_dist_colperm PARMETIS -mat_superlu_dist_parsymbfact 1<br>
<br>
...<br>
[1]PETSC ERROR: ------------------------------<wbr>------------------------------<wbr>------------<br>
[1]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero<br>
[1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
[1]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html#valgrind</a><br>
[1]PETSC ERROR: or try <a href="http://valgrind.org" rel="noreferrer" target="_blank">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to find memory corruption errors<br>
[1]PETSC ERROR: likely location of problem given in stack below<br>
[1]PETSC ERROR: --------------------- Stack Frames ------------------------------<wbr>------<br>
[1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>
[1]PETSC ERROR: INSTEAD the line number of the start of the function<br>
[1]PETSC ERROR: is given.<br>
[1]PETSC ERROR: [1] SuperLU_DIST:pdgssvx line 467 /home/mefpp_ericc/petsc-3.9.2-<wbr>debug/src/mat/impls/aij/mpi/su<wbr>perlu_dist/superlu_dist.c<br>
[1]PETSC ERROR: [1] MatLUFactorNumeric_SuperLU_DIS<wbr>T line 314 /home/mefpp_ericc/petsc-3.9.2-<wbr>debug/src/mat/impls/aij/mpi/su<wbr>perlu_dist/superlu_dist.c<br>
[1]PETSC ERROR: [1] MatLUFactorNumeric line 3014 /home/mefpp_ericc/petsc-3.9.2-<wbr>debug/src/mat/interface/matrix<wbr>.c<br>
[1]PETSC ERROR: [1] PCSetUp_LU line 59 /home/mefpp_ericc/petsc-3.9.2-<wbr>debug/src/ksp/pc/impls/factor/<wbr>lu/lu.c<br>
[1]PETSC ERROR: [1] PCSetUp line 885 /home/mefpp_ericc/petsc-3.9.2-<wbr>debug/src/ksp/pc/interface/pre<wbr>con.c<br>
[1]PETSC ERROR: [1] KSPSetUp line 294 /home/mefpp_ericc/petsc-3.9.2-<wbr>debug/src/ksp/ksp/interface/it<wbr>func.c<br>
[1]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
[1]PETSC ERROR: Signal received<br>
[1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.<br>
[1]PETSC ERROR: Petsc Release Version 3.9.2, May, 20, 2018<br>
[1]PETSC ERROR: ./ex6 on a named lorien by eric Tue May 22 10:39:15 2018<br>
[1]PETSC ERROR: Configure options --prefix=/opt/petsc-3.9.2_debu<wbr>g_openmpi-1.10.2 --with-mpi-compilers=1 --with-mpi-dir=/opt/openmpi-1.<wbr>10.2 --with-make-np=12 --with-shared-libraries=1 --with-debugging=yes --with-memalign=64 --with-visibility=0 --with-64-bit-indices=0 --download-ml=yes --download-mumps=yes --download-superlu=yes --download-superlu_dist=yes --download-parmetis=yes --download-ptscotch=yes --download-metis=yes --download-suitesparse=yes --download-hypre=yes --with-blaslapack-dir=/opt/int<wbr>el/composer_xe_2015.2.164/mkl/<wbr>lib/intel64 --with-mkl_pardiso-dir=/opt/in<wbr>tel/composer_xe_2015.2.164/mkl --with-mkl_cpardiso-dir=/opt/i<wbr>ntel/composer_xe_2015.2.164/mk<wbr>l --with-scalapack=1 --with-scalapack-include=/opt/<wbr>intel/composer_xe_2015.2.164/m<wbr>kl/include --with-scalapack-lib="-L/opt/i<wbr>ntel/composer_xe_2015.2.164/mk<wbr>l/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64"<br>
[1]PETSC ERROR: #1 User provided function() line 0 in unknown file<br>
...<br>
<br>
The given Matrix+Vector are available here:<br>
<br>
<a href="http://www.giref.ulaval.ca/~ericc/bug_superlu_dist_parallel_factorisation/AssembleurGD_resolution_no_0_0" rel="noreferrer" target="_blank">http://www.giref.ulaval.ca/~er<wbr>icc/bug_superlu_dist_parallel_<wbr>factorisation/AssembleurGD_<wbr>resolution_no_0_0</a><br>
<br>
<a href="http://www.giref.ulaval.ca/~ericc/bug_superlu_dist_parallel_factorisation/AssembleurGD_resolution_no_0_0.info" rel="noreferrer" target="_blank">http://www.giref.ulaval.ca/~er<wbr>icc/bug_superlu_dist_parallel_<wbr>factorisation/AssembleurGD_<wbr>resolution_no_0_0.info</a><br>
<br>
If I run with -on_error_attach_debugger, I can see a division by zero here:<br>
<br>
#8 <signal handler called><br>
(gdb)<br>
#9 0x00007f96a2148e52 in libmetis__FM_2WayCutRefine (ctrl=0x2784d20, graph=0x2784940, ntpwgts=0x7ffdfa323060, niter=4)<br>
at /home/mefpp_ericc/petsc-3.9.2-<wbr>debug/arch-linux2-c-debug/exte<wbr>rnalpackages/git.metis/libmeti<wbr>s/fm.c:60<br>
60 avgvwgt = gk_min((pwgts[0]+pwgts[1])/20, 2*(pwgts[0]+pwgts[1])/nvtxs);<br>
<br>
and nvtxs value is "0"...<br>
<br>
Thanks!<span class="gmail-HOEnZb"><font color="#888888"><br>
<br>
Eric<br>
</font></span></blockquote></div><br></div></div>