<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/4.6.6">
</HEAD>
<BODY>
Hong,<BR>
<BR>
here if the output of grep -info:<BR>
<BR>
using diagonal shift on blocks to prevent zero pivot [INBLOCKS]<BR>
Replace tiny pivots FALSE<BR>
tolerance for zero pivot 2.22045e-14<BR>
<BR>
It seems it is not replacing small pivots: could this be the problem?<BR>
I will also try Barry's suggestion to diagnose the problem.<BR>
<BR>
Thanks,<BR>
Michele<BR>
<BR>
<BR>
On Wed, 2016-02-10 at 21:22 -0600, Barry Smith wrote:
<BLOCKQUOTE TYPE=CITE>
<PRE>
<FONT COLOR="#737373">> On Feb 10, 2016, at 9:00 PM, Hong <<A HREF="mailto:hzhang@mcs.anl.gov">hzhang@mcs.anl.gov</A>> wrote:</FONT>
<FONT COLOR="#737373">> </FONT>
<FONT COLOR="#737373">> Michele :</FONT>
<FONT COLOR="#737373">> Superlu_dist LU is used for coarse grid PC, which likely produces a zero-pivot.</FONT>
<FONT COLOR="#737373">> Run your code with '-info |grep pivot' to verify.</FONT>
Michele
You can also run with -ksp_error_if_not_converged in or not in the debugger and it will stop immediately when the problem is detected and hopefully provide additional useful information about what has happened.
Barry
<FONT COLOR="#737373">> </FONT>
<FONT COLOR="#737373">> Hong</FONT>
<FONT COLOR="#737373">> </FONT>
<FONT COLOR="#737373">> Hi Matt,</FONT>
<FONT COLOR="#737373">> </FONT>
<FONT COLOR="#737373">> the ksp_view output was an attachment to my previous email.</FONT>
<FONT COLOR="#737373">> Here it is:</FONT>
<FONT COLOR="#737373">> </FONT>
<FONT COLOR="#737373">> KSP Object: 1 MPI processes</FONT>
<FONT COLOR="#737373">> type: cg</FONT>
<FONT COLOR="#737373">> maximum iterations=10000</FONT>
<FONT COLOR="#737373">> tolerances: relative=1e-08, absolute=1e-50, divergence=10000.</FONT>
<FONT COLOR="#737373">> left preconditioning</FONT>
<FONT COLOR="#737373">> using nonzero initial guess</FONT>
<FONT COLOR="#737373">> using UNPRECONDITIONED norm type for convergence test</FONT>
<FONT COLOR="#737373">> PC Object: 1 MPI processes</FONT>
<FONT COLOR="#737373">> type: mg</FONT>
<FONT COLOR="#737373">> MG: type is MULTIPLICATIVE, levels=4 cycles=v</FONT>
<FONT COLOR="#737373">> Cycles per PCApply=1</FONT>
<FONT COLOR="#737373">> Using Galerkin computed coarse grid matrices</FONT>
<FONT COLOR="#737373">> Coarse grid solver -- level -------------------------------</FONT>
<FONT COLOR="#737373">> KSP Object: (mg_coarse_) 1 MPI processes</FONT>
<FONT COLOR="#737373">> type: preonly</FONT>
<FONT COLOR="#737373">> maximum iterations=1, initial guess is zero</FONT>
<FONT COLOR="#737373">> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</FONT>
<FONT COLOR="#737373">> left preconditioning</FONT>
<FONT COLOR="#737373">> using NONE norm type for convergence test</FONT>
<FONT COLOR="#737373">> PC Object: (mg_coarse_) 1 MPI processes</FONT>
<FONT COLOR="#737373">> type: lu</FONT>
<FONT COLOR="#737373">> LU: out-of-place factorization</FONT>
<FONT COLOR="#737373">> tolerance for zero pivot 2.22045e-14</FONT>
<FONT COLOR="#737373">> using diagonal shift on blocks to prevent zero pivot [INBLOCKS]</FONT>
<FONT COLOR="#737373">> matrix ordering: nd</FONT>
<FONT COLOR="#737373">> factor fill ratio given 0., needed 0.</FONT>
<FONT COLOR="#737373">> Factored matrix follows:</FONT>
<FONT COLOR="#737373">> Mat Object: 1 MPI processes</FONT>
<FONT COLOR="#737373">> type: seqaij</FONT>
<FONT COLOR="#737373">> rows=16, cols=16</FONT>
<FONT COLOR="#737373">> package used to perform factorization: superlu_dist</FONT>
<FONT COLOR="#737373">> total: nonzeros=0, allocated nonzeros=0</FONT>
<FONT COLOR="#737373">> total number of mallocs used during MatSetValues calls =0</FONT>
<FONT COLOR="#737373">> SuperLU_DIST run parameters:</FONT>
<FONT COLOR="#737373">> Process grid nprow 1 x npcol 1 </FONT>
<FONT COLOR="#737373">> Equilibrate matrix TRUE </FONT>
<FONT COLOR="#737373">> Matrix input mode 0 </FONT>
<FONT COLOR="#737373">> Replace tiny pivots FALSE </FONT>
<FONT COLOR="#737373">> Use iterative refinement FALSE </FONT>
<FONT COLOR="#737373">> Processors in row 1 col partition 1 </FONT>
<FONT COLOR="#737373">> Row permutation LargeDiag </FONT>
<FONT COLOR="#737373">> Column permutation METIS_AT_PLUS_A</FONT>
<FONT COLOR="#737373">> Parallel symbolic factorization FALSE </FONT>
<FONT COLOR="#737373">> Repeated factorization SamePattern</FONT>
<FONT COLOR="#737373">> linear system matrix = precond matrix:</FONT>
<FONT COLOR="#737373">> Mat Object: 1 MPI processes</FONT>
<FONT COLOR="#737373">> type: seqaij</FONT>
<FONT COLOR="#737373">> rows=16, cols=16</FONT>
<FONT COLOR="#737373">> total: nonzeros=72, allocated nonzeros=72</FONT>
<FONT COLOR="#737373">> total number of mallocs used during MatSetValues calls =0</FONT>
<FONT COLOR="#737373">> not using I-node routines</FONT>
<FONT COLOR="#737373">> Down solver (pre-smoother) on level 1 -------------------------------</FONT>
<FONT COLOR="#737373">> KSP Object: (mg_levels_1_) 1 MPI processes</FONT>
<FONT COLOR="#737373">> type: richardson</FONT>
<FONT COLOR="#737373">> Richardson: damping factor=1.</FONT>
<FONT COLOR="#737373">> maximum iterations=2</FONT>
<FONT COLOR="#737373">> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</FONT>
<FONT COLOR="#737373">> left preconditioning</FONT>
<FONT COLOR="#737373">> using nonzero initial guess</FONT>
<FONT COLOR="#737373">> using NONE norm type for convergence test</FONT>
<FONT COLOR="#737373">> PC Object: (mg_levels_1_) 1 MPI processes</FONT>
<FONT COLOR="#737373">> type: sor</FONT>
<FONT COLOR="#737373">> SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.</FONT>
<FONT COLOR="#737373">> linear system matrix = precond matrix:</FONT>
<FONT COLOR="#737373">> Mat Object: 1 MPI processes</FONT>
<FONT COLOR="#737373">> type: seqaij</FONT>
<FONT COLOR="#737373">> rows=64, cols=64</FONT>
<FONT COLOR="#737373">> total: nonzeros=304, allocated nonzeros=304</FONT>
<FONT COLOR="#737373">> total number of mallocs used during MatSetValues calls =0</FONT>
<FONT COLOR="#737373">> not using I-node routines</FONT>
<FONT COLOR="#737373">> Up solver (post-smoother) same as down solver (pre-smoother)</FONT>
<FONT COLOR="#737373">> Down solver (pre-smoother) on level 2 -------------------------------</FONT>
<FONT COLOR="#737373">> KSP Object: (mg_levels_2_) 1 MPI processes</FONT>
<FONT COLOR="#737373">> type: richardson</FONT>
<FONT COLOR="#737373">> Richardson: damping factor=1.</FONT>
<FONT COLOR="#737373">> maximum iterations=2</FONT>
<FONT COLOR="#737373">> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</FONT>
<FONT COLOR="#737373">> left preconditioning</FONT>
<FONT COLOR="#737373">> using nonzero initial guess</FONT>
<FONT COLOR="#737373">> using NONE norm type for convergence test</FONT>
<FONT COLOR="#737373">> PC Object: (mg_levels_2_) 1 MPI processes</FONT>
<FONT COLOR="#737373">> type: sor</FONT>
<FONT COLOR="#737373">> SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.</FONT>
<FONT COLOR="#737373">> linear system matrix = precond matrix:</FONT>
<FONT COLOR="#737373">> Mat Object: 1 MPI processes</FONT>
<FONT COLOR="#737373">> type: seqaij</FONT>
<FONT COLOR="#737373">> rows=256, cols=256</FONT>
<FONT COLOR="#737373">> total: nonzeros=1248, allocated nonzeros=1248</FONT>
<FONT COLOR="#737373">> total number of mallocs used during MatSetValues calls =0</FONT>
<FONT COLOR="#737373">> not using I-node routines</FONT>
<FONT COLOR="#737373">> Up solver (post-smoother) same as down solver (pre-smoother)</FONT>
<FONT COLOR="#737373">> Down solver (pre-smoother) on level 3 -------------------------------</FONT>
<FONT COLOR="#737373">> KSP Object: (mg_levels_3_) 1 MPI processes</FONT>
<FONT COLOR="#737373">> type: richardson</FONT>
<FONT COLOR="#737373">> Richardson: damping factor=1.</FONT>
<FONT COLOR="#737373">> maximum iterations=2</FONT>
<FONT COLOR="#737373">> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</FONT>
<FONT COLOR="#737373">> left preconditioning</FONT>
<FONT COLOR="#737373">> using nonzero initial guess</FONT>
<FONT COLOR="#737373">> using NONE norm type for convergence test</FONT>
<FONT COLOR="#737373">> PC Object: (mg_levels_3_) 1 MPI processes</FONT>
<FONT COLOR="#737373">> type: sor</FONT>
<FONT COLOR="#737373">> SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.</FONT>
<FONT COLOR="#737373">> linear system matrix = precond matrix:</FONT>
<FONT COLOR="#737373">> Mat Object: 1 MPI processes</FONT>
<FONT COLOR="#737373">> type: seqaij</FONT>
<FONT COLOR="#737373">> rows=1024, cols=1024</FONT>
<FONT COLOR="#737373">> total: nonzeros=5056, allocated nonzeros=5056</FONT>
<FONT COLOR="#737373">> total number of mallocs used during MatSetValues calls =0</FONT>
<FONT COLOR="#737373">> has attached null space</FONT>
<FONT COLOR="#737373">> not using I-node routines</FONT>
<FONT COLOR="#737373">> Up solver (post-smoother) same as down solver (pre-smoother)</FONT>
<FONT COLOR="#737373">> linear system matrix = precond matrix:</FONT>
<FONT COLOR="#737373">> Mat Object: 1 MPI processes</FONT>
<FONT COLOR="#737373">> type: seqaij</FONT>
<FONT COLOR="#737373">> rows=1024, cols=1024</FONT>
<FONT COLOR="#737373">> total: nonzeros=5056, allocated nonzeros=5056</FONT>
<FONT COLOR="#737373">> total number of mallocs used during MatSetValues calls =0</FONT>
<FONT COLOR="#737373">> has attached null space</FONT>
<FONT COLOR="#737373">> not using I-node routines</FONT>
<FONT COLOR="#737373">> </FONT>
<FONT COLOR="#737373">> </FONT>
<FONT COLOR="#737373">> Michele</FONT>
<FONT COLOR="#737373">> </FONT>
<FONT COLOR="#737373">> </FONT>
<FONT COLOR="#737373">> </FONT>
<FONT COLOR="#737373">> </FONT>
<FONT COLOR="#737373">> On Wed, 2016-02-10 at 19:37 -0600, Matthew Knepley wrote:</FONT>
<FONT COLOR="#737373">>> On Wed, Feb 10, 2016 at 7:33 PM, Michele Rosso <<A HREF="mailto:mrosso@uci.edu">mrosso@uci.edu</A>> wrote:</FONT>
<FONT COLOR="#737373">>> Hi,</FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> I encountered the following error while solving a symmetric positive defined system:</FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0</FONT>
<FONT COLOR="#737373">>> PCSETUP_FAILED due to SUBPC_ERROR </FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> This error appears only if I use the optimized version of both petsc and my code ( compiler: gfortran, flags: -O3 ).</FONT>
<FONT COLOR="#737373">>> It is weird since I am solving a time-dependent problem and everything, i.e. results and convergence rate, are as expected until the above error shows up. If I run both petsc and my code in debug mode, everything goes smooth till the end of the simulation.</FONT>
<FONT COLOR="#737373">>> However, if I reduce the ksp_rtol, even the debug run fails, after running as expected for a while, because of a KSP_DIVERGED_INDEFINITE_PC . </FONT>
<FONT COLOR="#737373">>> The options I am using are:</FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> -ksp_type cg</FONT>
<FONT COLOR="#737373">>> -ksp_norm_type unpreconditioned</FONT>
<FONT COLOR="#737373">>> -ksp_rtol 1e-8</FONT>
<FONT COLOR="#737373">>> -ksp_lag_norm</FONT>
<FONT COLOR="#737373">>> -ksp_initial_guess_nonzero yes</FONT>
<FONT COLOR="#737373">>> -pc_type mg</FONT>
<FONT COLOR="#737373">>> -pc_mg_galerkin</FONT>
<FONT COLOR="#737373">>> -pc_mg_levels 4</FONT>
<FONT COLOR="#737373">>> -mg_levels_ksp_type richardson</FONT>
<FONT COLOR="#737373">>> -mg_coarse_ksp_constant_null_space</FONT>
<FONT COLOR="#737373">>> -mg_coarse_pc_type lu</FONT>
<FONT COLOR="#737373">>> -mg_coarse_pc_factor_mat_solver_package superlu_dist</FONT>
<FONT COLOR="#737373">>> -options_left</FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> I attached a copy of ksp_view. I am currently using petsc-master (last updated yesterday).</FONT>
<FONT COLOR="#737373">>> I would appreciate any suggestion on this matter.</FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> I suspect you have a nonlinear PC. Can you send the output of -ksp_view?</FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> Matt</FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> Thanks,</FONT>
<FONT COLOR="#737373">>> Michele</FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> </FONT>
<FONT COLOR="#737373">>> --</FONT>
<FONT COLOR="#737373">>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.</FONT>
<FONT COLOR="#737373">>> -- Norbert Wiener</FONT>
<FONT COLOR="#737373">> </FONT>
<FONT COLOR="#737373">> </FONT>
</PRE>
</BLOCKQUOTE>
<BR>
</BODY>
</HTML>