<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> When you get a huge change at restart this means something is seriously wrong with either the linear operator or the linear preconditioner. <div class=""><br class=""></div><div class=""> How are you doing the matrix vector product? Note both the operator and preconditioner must be linear operators for GMRES.</div><div class=""><br class=""></div><div class=""> FGMRES allows the preconditioner to be nonlinear. You can try</div><div class=""><br class=""></div><div class=""> -ksp_type fgmres -ksp_monitor_true_residual</div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Oct 8, 2020, at 2:43 AM, Yang Juntao <<a href="mailto:Y.Juntao@hotmail.com" class="">Y.Juntao@hotmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Hello,<span class="Apple-converted-space"> </span><o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I’m working on a nonlinear solver with SNES with handcoded jacobian and function. Each linear solver is solved with KSP solver.<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">But sometimes I got issues with ksp solver convergence. I tried with finite difference approximated jacobian, but get the same error.<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">From the iterations, the convergence seems ok at the beginning but suddenly diverged in the last iteration.<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Hope anyone with experience on ksp solvers could direct me to a direction I can debug the problem.<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iter = 0, SNES Function norm 2.94934e-06<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 0 KSP Residual norm 1.094600281831e-06<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 1 KSP Residual norm 1.264284474186e-08<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 2 KSP Residual norm 6.593269221816e-09<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 3 KSP Residual norm 1.689570779457e-09<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 4 KSP Residual norm 1.040661505932e-09<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 5 KSP Residual norm 5.422761817348e-10<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 6 KSP Residual norm 2.492867371369e-10<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 7 KSP Residual norm 8.261522376775e-11<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 8 KSP Residual norm 4.246401544245e-11<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 9 KSP Residual norm 2.514366787388e-11<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 10 KSP Residual norm 1.982940267051e-11<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 11 KSP Residual norm 1.586470414676e-11<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 12 KSP Residual norm 9.866392216207e-12<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 13 KSP Residual norm 4.951342176999e-12<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 14 KSP Residual norm 2.418292660318e-12<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 15 KSP Residual norm 1.747418526086e-12<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 16 KSP Residual norm 1.094150535809e-12<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 17 KSP Residual norm 4.464287492066e-13<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 18 KSP Residual norm 3.530090494462e-13<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 19 KSP Residual norm 2.825698091454e-13<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 20 KSP Residual norm 1.950568425807e-13<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 21 KSP Residual norm 1.227898091813e-13<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 22 KSP Residual norm 5.411106347374e-14<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 23 KSP Residual norm 4.511115848564e-14<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 24 KSP Residual norm 4.063546606691e-14<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 25 KSP Residual norm 3.677694771949e-14<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 26 KSP Residual norm 3.459244943466e-14<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 27 KSP Residual norm 3.263954971093e-14<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 28 KSP Residual norm 3.087344619079e-14<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 29 KSP Residual norm 2.809426925625e-14<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">iteration 30 KSP Residual norm 4.366149884754e-01<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Linear solve did not converge due to DIVERGED_DTOL iterations 30<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class=""><o:p class=""> </o:p></b></div><div style="border-style: none none solid; border-bottom-width: 1pt; border-bottom-color: windowtext; padding: 0in 0in 1pt;" class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; border: none; padding: 0in;" class=""><o:p class=""> </o:p></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">SNES Object: 1 MPI processes<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> type: newtonls<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> SNES has not been set up so information may be incomplete<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> maximum iterations=50, maximum function evaluations=10000<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> total number of linear solver iterations=0<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> total number of function evaluations=0<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> norm schedule ALWAYS<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> SNESLineSearch Object: 1 MPI processes<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> type: bt<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> interpolation: cubic<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> alpha=1.000000e-04<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> maxstep=1.000000e+08, minlambda=1.000000e-12<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> maximum iterations=40<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> KSP Object: 1 MPI processes<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> type: gmres<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> happy breakdown tolerance 1e-30<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> maximum iterations=10000, initial guess is zero<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> tolerances: relative=1e-08, absolute=1e-50, divergence=10000.<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> left preconditioning<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> using DEFAULT norm type for convergence test<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> PC Object: 1 MPI processes<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> type: fieldsplit<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> PC has not been set up so information may be incomplete<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> FieldSplit with Schur preconditioner, factorization FULL<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Preconditioner for the Schur complement formed from S itself<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Split info:<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> KSP solver for A00 block<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> not yet available<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> KSP solver for S = A11 - A10 inv(A00) A01<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> not yet available<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> linear system matrix = precond matrix:<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Mat Object: 1 MPI processes<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> type: seqaij<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> rows=659, cols=659<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> total: nonzeros=659, allocated nonzeros=7908<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> total number of mallocs used during MatSetValues calls=0<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> not using I-node routines<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Regards<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Juntao</div></div></div></blockquote></div><br class=""></div></body></html>