<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Apr 30, 2019, at 6:22 PM, Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" class="">bsmith@mcs.anl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><br class=""> Boyce,<br class=""><br class=""> I noticed that in the KSPView you sent the solver inside GASM was fgmres, I don't know why!</div></div></blockquote><div><br class=""></div>Thanks for checking! I don’t know why either. I often use FGMRES for outer KSPs, and probably used it for the inner ones without thinking. However, switching to preonly does not appear to help here:</div><div><br class=""></div><div><div class="">==========</div><div class=""><br class=""></div></div><div>Using GMRES as the outer KSP with ILU for the subdomain PC:</div><div><br class=""></div><div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class="">./main -d 3 -n 10 -ksp_type gmres -ksp_max_it 20 -ksp_monitor_true_residual -ksp_view</div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><br class=""></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 0 KSP preconditioned resid norm 7.956172961039e-01 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 1 KSP preconditioned resid norm 1.790924766864e-01 true resid norm 6.415624322140e+01 ||r(i)||/||b|| 1.222031831626e-01</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 2 KSP preconditioned resid norm 1.021248567184e-02 true resid norm 1.531971188768e+00 ||r(i)||/||b|| 2.918059823653e-03</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 3 KSP preconditioned resid norm 1.245290521533e-03 true resid norm 1.344228387002e-01 ||r(i)||/||b|| 2.560452101634e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 4 KSP preconditioned resid norm 1.090553693795e-04 true resid norm 1.055832655005e-01 ||r(i)||/||b|| 2.011123233687e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 5 KSP preconditioned resid norm 5.774209753120e-06 true resid norm 1.045883881261e-01 ||r(i)||/||b|| 1.992173062059e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 6 KSP preconditioned resid norm 4.856074176119e-07 true resid norm 1.047048288204e-01 ||r(i)||/||b|| 1.994390994840e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 7 KSP preconditioned resid norm 3.569834574103e-08 true resid norm 1.046981572164e-01 ||r(i)||/||b|| 1.994263915821e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 8 KSP preconditioned resid norm 1.914112077836e-09 true resid norm 1.046980716404e-01 ||r(i)||/||b|| 1.994262285791e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 9 KSP preconditioned resid norm 9.380275080687e-11 true resid norm 1.046980942278e-01 ||r(i)||/||b|| 1.994262716030e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 10 KSP preconditioned resid norm 3.490998066884e-12 true resid norm 1.046980929565e-01 ||r(i)||/||b|| 1.994262691815e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 11 KSP preconditioned resid norm 2.275544655754e-13 true resid norm 1.046980929905e-01 ||r(i)||/||b|| 1.994262692463e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">KSP Object: 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: gmres</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> happy breakdown tolerance 1e-30</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> maximum iterations=20, nonzero initial guess</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerances: relative=1e-12, absolute=1e-50, divergence=10000.</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> left preconditioning</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> using PRECONDITIONED norm type for convergence test</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">PC Object: 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: gasm</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Restriction/interpolation type: RESTRICT</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> requested amount of overlap = 0</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of subdomains = 2</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> max number of local subdomains = 2</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> [0|1] number of locally-supported subdomains = 2</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Subdomain solver info is as follows:</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> - - - - - - - - - - - - - - - - - -</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> KSP Object: (sub_) 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: preonly</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> maximum iterations=10000, initial guess is zero</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> left preconditioning</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> using NONE norm type for convergence test</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> PC Object: (sub_) 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: ilu</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> out-of-place factorization</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 0 levels of fill</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerance for zero pivot 2.22045e-14</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> matrix ordering: natural</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> factor fill ratio given 1., needed 1.</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Factored matrix follows:</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Mat Object: 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: seqaij</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rows=484, cols=484</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> package used to perform factorization: petsc</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total: nonzeros=15376, allocated nonzeros=15376</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of mallocs used during MatSetValues calls =0</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> not using I-node routines</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> linear system matrix = precond matrix:</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Mat Object: () 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: seqaij</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rows=484, cols=484</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total: nonzeros=15376, allocated nonzeros=15376</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of mallocs used during MatSetValues calls =0</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> not using I-node routines</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> - - - - - - - - - - - - - - - - - -</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> KSP Object: (sub_) 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: preonly</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> maximum iterations=10000, initial guess is zero</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> left preconditioning</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> using NONE norm type for convergence test</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> PC Object: (sub_) 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: ilu</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> out-of-place factorization</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 0 levels of fill</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerance for zero pivot 2.22045e-14</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> matrix ordering: natural</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> factor fill ratio given 1., needed 1.</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Factored matrix follows:</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Mat Object: 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: seqaij</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rows=121, cols=121</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> package used to perform factorization: petsc</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total: nonzeros=961, allocated nonzeros=961</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of mallocs used during MatSetValues calls =0</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> not using I-node routines</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> linear system matrix = precond matrix:</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Mat Object: () 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: seqaij</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rows=121, cols=121</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total: nonzeros=961, allocated nonzeros=961</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of mallocs used during MatSetValues calls =0</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> not using I-node routines</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> - - - - - - - - - - - - - - - - - -</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div></span></div><div class="">This shows convergence in the preconditioned norm but not in the true norm, and the solution appears nonphysical.</div><div class=""><br class=""></div><div class="">==========</div><div class=""><br class=""></div><div class="">Using GMRES for the outer KSP with LU for the subdomain PC:</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">./main -d 3 -n 10 -ksp_type gmres -sub_pc_type lu -ksp_monitor_true_residual -ksp_view</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941); min-height: 16px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 0 KSP preconditioned resid norm 7.587523490344e-01 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 1 KSP preconditioned resid norm 1.452294080360e-15 true resid norm 1.026159563147e-01 ||r(i)||/||b|| 1.954602681716e-04</span></div></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">KSP Object: 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: gmres</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> happy breakdown tolerance 1e-30</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> maximum iterations=5000, nonzero initial guess</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerances: relative=1e-12, absolute=1e-50, divergence=10000.</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> left preconditioning</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> using PRECONDITIONED norm type for convergence test</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">PC Object: 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: gasm</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Restriction/interpolation type: RESTRICT</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> requested amount of overlap = 0</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of subdomains = 2</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> max number of local subdomains = 2</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> [0|1] number of locally-supported subdomains = 2</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Subdomain solver info is as follows:</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> - - - - - - - - - - - - - - - - - -</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> KSP Object: (sub_) 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: preonly</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> maximum iterations=10000, initial guess is zero</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> left preconditioning</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> using NONE norm type for convergence test</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> PC Object: (sub_) 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: lu</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> out-of-place factorization</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerance for zero pivot 2.22045e-14</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> matrix ordering: nd</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> factor fill ratio given 5., needed 2.69979</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Factored matrix follows:</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Mat Object: 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: seqaij</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rows=484, cols=484</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> package used to perform factorization: petsc</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total: nonzeros=41512, allocated nonzeros=41512</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of mallocs used during MatSetValues calls =0</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> using I-node routines: found 194 nodes, limit used is 5</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> linear system matrix = precond matrix:</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Mat Object: () 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: seqaij</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rows=484, cols=484</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total: nonzeros=15376, allocated nonzeros=15376</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of mallocs used during MatSetValues calls =0</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> not using I-node routines</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> - - - - - - - - - - - - - - - - - -</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> KSP Object: (sub_) 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: preonly</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> maximum iterations=10000, initial guess is zero</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> left preconditioning</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> using NONE norm type for convergence test</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> PC Object: (sub_) 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: lu</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> out-of-place factorization</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerance for zero pivot 2.22045e-14</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> matrix ordering: nd</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> factor fill ratio given 5., needed 2.57128</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Factored matrix follows:</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Mat Object: 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: seqaij</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rows=121, cols=121</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> package used to perform factorization: petsc</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total: nonzeros=2471, allocated nonzeros=2471</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of mallocs used during MatSetValues calls =0</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> not using I-node routines</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> linear system matrix = precond matrix:</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Mat Object: () 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: seqaij</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rows=121, cols=121</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total: nonzeros=961, allocated nonzeros=961</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of mallocs used during MatSetValues calls =0</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> not using I-node routines</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> - - - - - - - - - - - - - - - - - -</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div><div class=""><br class=""></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class="">==========</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div></div><div>Using FGMRES as the outer KSP:</div><div><br class=""></div><div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">./main -d 3 -n 10 -ksp_type fgmres -ksp_max_it 20 -ksp_monitor_true_residual -ksp_view</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941); min-height: 16px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 0 KSP unpreconditioned resid norm 5.249964981356e+02 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 1 KSP unpreconditioned resid norm 9.908062627364e+00 true resid norm 9.908062627364e+00 ||r(i)||/||b|| 1.887262612713e-02</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 2 KSP unpreconditioned resid norm 1.198244246143e+00 true resid norm 1.198244246143e+00 ||r(i)||/||b|| 2.282385216660e-03</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 3 KSP unpreconditioned resid norm 1.011712741080e-01 true resid norm 1.011712741080e-01 ||r(i)||/||b|| 1.927084741847e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 4 KSP unpreconditioned resid norm 9.890789240724e-02 true resid norm 9.890789240713e-02 ||r(i)||/||b|| 1.883972421881e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 5 KSP unpreconditioned resid norm 9.790912262081e-02 true resid norm 9.790912262142e-02 ||r(i)||/||b|| 1.864948108590e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 6 KSP unpreconditioned resid norm 9.758733533827e-02 true resid norm 9.758733534329e-02 ||r(i)||/||b|| 1.858818786218e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 7 KSP unpreconditioned resid norm 9.754239526505e-02 true resid norm 9.754239528968e-02 ||r(i)||/||b|| 1.857962779487e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 8 KSP unpreconditioned resid norm 9.732735871034e-02 true resid norm 9.732735925876e-02 ||r(i)||/||b|| 1.853866827767e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 9 KSP unpreconditioned resid norm 9.681037606662e-02 true resid norm 9.681036910553e-02 ||r(i)||/||b|| 1.844019330592e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 10 KSP unpreconditioned resid norm 9.672261014893e-02 true resid norm 9.672255386871e-02 ||r(i)||/||b|| 1.842346648258e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 11 KSP unpreconditioned resid norm 9.636675313899e-02 true resid norm 9.636858548398e-02 ||r(i)||/||b|| 1.835604348338e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 12 KSP unpreconditioned resid norm 9.626809760271e-02 true resid norm 9.624997274413e-02 ||r(i)||/||b|| 1.833345042985e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 13 KSP unpreconditioned resid norm 9.538127889520e-02 true resid norm 9.646040937528e-02 ||r(i)||/||b|| 1.837353386505e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 14 KSP unpreconditioned resid norm 7.698037210447e-02 true resid norm 1.004972547621e-01 ||r(i)||/||b|| 1.914246192480e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 15 KSP unpreconditioned resid norm 6.963471194663e-02 true resid norm 1.021051124617e-01 ||r(i)||/||b|| 1.944872257707e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 16 KSP unpreconditioned resid norm 6.288795988348e-02 true resid norm 1.048077408458e-01 ||r(i)||/||b|| 1.996351236971e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 17 KSP unpreconditioned resid norm 5.576252453977e-02 true resid norm 1.071287580389e-01 ||r(i)||/||b|| 2.040561383159e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 18 KSP unpreconditioned resid norm 5.096717927451e-02 true resid norm 1.092140137471e-01 ||r(i)||/||b|| 2.080280804443e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 19 KSP unpreconditioned resid norm 4.698350635435e-02 true resid norm 1.104283980783e-01 ||r(i)||/||b|| 2.103412088852e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 20 KSP unpreconditioned resid norm 4.387914680239e-02 true resid norm 1.116698560164e-01 ||r(i)||/||b|| 2.127059064451e-04</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">KSP Object: 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: fgmres</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> happy breakdown tolerance 1e-30</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> maximum iterations=20, nonzero initial guess</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerances: relative=1e-12, absolute=1e-50, divergence=10000.</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> right preconditioning</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> using UNPRECONDITIONED norm type for convergence test</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">PC Object: 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: gasm</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Restriction/interpolation type: RESTRICT</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> requested amount of overlap = 0</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of subdomains = 2</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> max number of local subdomains = 2</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> [0|1] number of locally-supported subdomains = 2</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Subdomain solver info is as follows:</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> - - - - - - - - - - - - - - - - - -</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> KSP Object: (sub_) 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: preonly</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> maximum iterations=10000, initial guess is zero</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> left preconditioning</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> using NONE norm type for convergence test</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> PC Object: (sub_) 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: ilu</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> out-of-place factorization</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 0 levels of fill</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerance for zero pivot 2.22045e-14</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> matrix ordering: natural</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> factor fill ratio given 1., needed 1.</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Factored matrix follows:</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Mat Object: 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: seqaij</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rows=484, cols=484</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> package used to perform factorization: petsc</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total: nonzeros=15376, allocated nonzeros=15376</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of mallocs used during MatSetValues calls =0</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> not using I-node routines</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> linear system matrix = precond matrix:</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Mat Object: () 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: seqaij</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rows=484, cols=484</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total: nonzeros=15376, allocated nonzeros=15376</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of mallocs used during MatSetValues calls =0</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> not using I-node routines</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> - - - - - - - - - - - - - - - - - -</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> KSP Object: (sub_) 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: preonly</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> maximum iterations=10000, initial guess is zero</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> left preconditioning</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> using NONE norm type for convergence test</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> PC Object: (sub_) 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: ilu</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> out-of-place factorization</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 0 levels of fill</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerance for zero pivot 2.22045e-14</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> matrix ordering: natural</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> factor fill ratio given 1., needed 1.</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Factored matrix follows:</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Mat Object: 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: seqaij</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rows=121, cols=121</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> package used to perform factorization: petsc</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total: nonzeros=961, allocated nonzeros=961</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of mallocs used during MatSetValues calls =0</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> not using I-node routines</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> linear system matrix = precond matrix:</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Mat Object: () 1 MPI processes</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: seqaij</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rows=121, cols=121</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total: nonzeros=961, allocated nonzeros=961</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of mallocs used during MatSetValues calls =0</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> not using I-node routines</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Monaco; color: rgb(131, 148, 150); background-color: rgba(2, 43, 54, 0.952941);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> - - - - - - - - - - - - - - - - - -</span></div></div><div><br class=""></div><div>==========</div><div><br class=""></div><div>Again, note that using the same subdomains, ASM appears to work fine.</div><div><br class=""></div><div><blockquote type="cite" class=""><div class=""><div class="">This would explain why the outer GMRES had inconsistent residuals. If you switch the inner solver to preonly + LU for GASM what happens? <br class=""><br class=""><blockquote type="cite" class="">On Apr 30, 2019, at 11:36 AM, Boyce Griffith via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a>> wrote:<br class=""><br class=""><br class=""><br class=""><blockquote type="cite" class="">On Apr 30, 2019, at 12:31 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:<br class=""><br class="">When I said it was singular I was looking at "preconditioned residual norm to an rtol of 1e-12. If I look at the true residual norm, however, it stagnates around 1e-4."<br class=""><br class="">This is not what I am seeing in this output.<br class=""></blockquote><br class="">Whoops, I switched the outer KSP from GMRES to FGMRES.<br class=""><br class="">With GMRES, the preconditioned residual norm drops nicely (but, of course, the solution is no good):<br class=""><br class="">$ ./main -d 3 -n 10 -ksp_type gmres -ksp_monitor_true_residual<br class="">Running ./main -d 3 -n 10 -ksp_type gmres -ksp_monitor_true_residual<br class=""><br class=""> 0 KSP preconditioned resid norm 7.954859454640e-01 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00<br class=""> 1 KSP preconditioned resid norm 1.791745669837e-01 true resid norm 6.420515097608e+01 ||r(i)||/||b|| 1.222963414120e-01<br class=""> 2 KSP preconditioned resid norm 1.018932536518e-02 true resid norm 1.538149013353e+00 ||r(i)||/||b|| 2.929827187068e-03<br class=""> 3 KSP preconditioned resid norm 1.247250041620e-03 true resid norm 1.231074134137e-01 ||r(i)||/||b|| 2.344918753761e-04<br class=""> 4 KSP preconditioned resid norm 1.090687825399e-04 true resid norm 9.214204251786e-02 ||r(i)||/||b|| 1.755098231037e-04<br class=""> 5 KSP preconditioned resid norm 5.773017036638e-06 true resid norm 9.199655161085e-02 ||r(i)||/||b|| 1.752326957181e-04<br class=""> 6 KSP preconditioned resid norm 4.880868222010e-07 true resid norm 9.199488147685e-02 ||r(i)||/||b|| 1.752295144892e-04<br class=""> 7 KSP preconditioned resid norm 3.528569945380e-08 true resid norm 9.199485972669e-02 ||r(i)||/||b|| 1.752294730601e-04<br class=""> 8 KSP preconditioned resid norm 1.875782938387e-09 true resid norm 9.199486015879e-02 ||r(i)||/||b|| 1.752294738832e-04<br class=""> 9 KSP preconditioned resid norm 8.952213054230e-11 true resid norm 9.199486012037e-02 ||r(i)||/||b|| 1.752294738100e-04<br class=""> 10 KSP preconditioned resid norm 3.450175063457e-12 true resid norm 9.199486011997e-02 ||r(i)||/||b|| 1.752294738092e-04<br class=""> 11 KSP preconditioned resid norm 2.186653062508e-13 true resid norm 9.199486012016e-02 ||r(i)||/||b|| 1.752294738096e-04<br class=""><br class=""><blockquote type="cite" class="">It is just a poor PC. The big drop in the residual at the beginning is suspicious. When you solve this problem well have you checked that you are getting a good solution?<br class=""></blockquote><br class="">Yes indeed, the solution appears to be no good with this preconditioner.<br class=""><br class="">Note that all that the PC is doing is applying GASM to split the problem onto two subdomains.<br class=""><br class=""><blockquote type="cite" class="">That is, do you check that your model is not messed up, like a bad mesh.<br class=""></blockquote><br class="">This is a uniform HEX mesh automatically generated by libMesh.<br class=""><br class=""><blockquote type="cite" class="">On Tue, Apr 30, 2019 at 11:35 AM Boyce Griffith via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a>> wrote:<br class=""><br class=""><br class=""><blockquote type="cite" class="">On Apr 30, 2019, at 11:23 AM, Fande Kong <<a href="mailto:fdkong.jd@gmail.com" class="">fdkong.jd@gmail.com</a>> wrote:<br class=""><br class=""><br class=""><br class="">On Tue, Apr 30, 2019 at 7:40 AM Boyce Griffith via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a>> wrote:<br class=""><br class=""><br class=""><blockquote type="cite" class="">On Apr 30, 2019, at 9:06 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:<br class=""><br class=""><br class=""><br class=""><br class="">Allowing GASM to construct the "outer" subdomains from the non-overlapping "inner" subdomains, and using "exact" subdomain solvers (subdomain KSPs are using FGMRES+ILU with an rtol of 1e-12), I get convergence in ~2 iterations in the preconditioned residual norm to an rtol of 1e-12. If I look at the true residual norm, however, it stagnates around 1e-4.<br class=""><br class=""><br class="">That PC is singular. <br class=""></blockquote><br class="">Yes. I am confused about why GASM is giving a singular PC but ASM is not with the same subdomains.<br class=""><br class="">We could not tell much without the info of detailed solver setup. The overlapping function was implemented by me and Dmitry a couple of years ago, and it is trick when a subdomain is shared by multiple cores. Do you mind to start from our example? Or setup an example for us to demonstrate your issues? <br class=""><br class="">At least please use "-ksp_view" (linear solver) or "-snes_view" (nonlinear solver) to print more information that will help us a bit.<br class=""></blockquote><br class="">Here you go:<br class=""><br class="">$ ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres -sub_pc_type ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason -ksp_monitor_true_residual -ksp_max_it 10 -ksp_view<br class="">make: `main' is up to date.<br class="">Running ./main -d 3 -n 10 -ksp_type fgmres -sub_ksp_type fgmres -sub_pc_type ilu -sub_ksp_rtol 1.0e-12 -ksp_converged_reason -ksp_monitor_true_residual -ksp_max_it 10 -ksp_view<br class=""><br class=""> 0 KSP unpreconditioned resid norm 5.249964981356e+02 true resid norm 5.249964981356e+02 ||r(i)||/||b|| 1.000000000000e+00<br class=""> 1 KSP unpreconditioned resid norm 9.316296223724e-02 true resid norm 9.316296223724e-02 ||r(i)||/||b|| 1.774544450641e-04<br class=""> 2 KSP unpreconditioned resid norm 9.314881028141e-02 true resid norm 9.314881028141e-02 ||r(i)||/||b|| 1.774274887779e-04<br class=""> 3 KSP unpreconditioned resid norm 9.299990517422e-02 true resid norm 9.299918556770e-02 ||r(i)||/||b|| 1.771424874222e-04<br class=""> 4 KSP unpreconditioned resid norm 9.224468272306e-02 true resid norm 9.393543403858e-02 ||r(i)||/||b|| 1.789258297382e-04<br class=""> 5 KSP unpreconditioned resid norm 9.150828598034e-02 true resid norm 9.511673987375e-02 ||r(i)||/||b|| 1.811759510997e-04<br class=""> 6 KSP unpreconditioned resid norm 9.078924839691e-02 true resid norm 1.013093335976e-01 ||r(i)||/||b|| 1.929714463951e-04<br class=""> 7 KSP unpreconditioned resid norm 9.008689850931e-02 true resid norm 1.011099594157e-01 ||r(i)||/||b|| 1.925916835155e-04<br class=""> 8 KSP unpreconditioned resid norm 8.940060065590e-02 true resid norm 1.090779251949e-01 ||r(i)||/||b|| 2.077688624253e-04<br class=""> 9 KSP unpreconditioned resid norm 8.872975256529e-02 true resid norm 1.102873098599e-01 ||r(i)||/||b|| 2.100724676289e-04<br class=""> 10 KSP unpreconditioned resid norm 8.807378313465e-02 true resid norm 1.071996745064e-01 ||r(i)||/||b|| 2.041912182026e-04<br class="">Linear solve did not converge due to DIVERGED_ITS iterations 10<br class="">KSP Object: 1 MPI processes<br class=""> type: fgmres<br class=""> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement<br class=""> happy breakdown tolerance 1e-30<br class=""> maximum iterations=10, nonzero initial guess<br class=""> tolerances: relative=1e-12, absolute=1e-50, divergence=10000.<br class=""> right preconditioning<br class=""> using UNPRECONDITIONED norm type for convergence test<br class="">PC Object: 1 MPI processes<br class=""> type: gasm<br class=""> Restriction/interpolation type: RESTRICT<br class=""> requested amount of overlap = 0<br class=""> total number of subdomains = 2<br class=""> max number of local subdomains = 2<br class=""> [0|1] number of locally-supported subdomains = 2<br class=""> Subdomain solver info is as follows:<br class=""> - - - - - - - - - - - - - - - - - -<br class=""> [0|1] (subcomm [0|1]) local subdomain number 0, local size = 484<br class=""> KSP Object: (sub_) 1 MPI processes<br class=""> type: fgmres<br class=""> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement<br class=""> happy breakdown tolerance 1e-30<br class=""> maximum iterations=10000, initial guess is zero<br class=""> tolerances: relative=1e-12, absolute=1e-50, divergence=10000.<br class=""> right preconditioning<br class=""> using UNPRECONDITIONED norm type for convergence test<br class=""> PC Object: (sub_) 1 MPI processes<br class=""> type: ilu<br class=""> out-of-place factorization<br class=""> 0 levels of fill<br class=""> tolerance for zero pivot 2.22045e-14<br class=""> matrix ordering: natural<br class=""> factor fill ratio given 1., needed 1.<br class=""> Factored matrix follows:<br class=""> Mat Object: 1 MPI processes<br class=""> type: seqaij<br class=""> rows=484, cols=484<br class=""> package used to perform factorization: petsc<br class=""> total: nonzeros=15376, allocated nonzeros=15376<br class=""> total number of mallocs used during MatSetValues calls =0<br class=""> not using I-node routines<br class=""> linear system matrix = precond matrix:<br class=""> Mat Object: () 1 MPI processes<br class=""> type: seqaij<br class=""> rows=484, cols=484<br class=""> total: nonzeros=15376, allocated nonzeros=15376<br class=""> total number of mallocs used during MatSetValues calls =0<br class=""> not using I-node routines<br class=""> - - - - - - - - - - - - - - - - - -<br class=""> [0|1] (subcomm [0|1]) local subdomain number 1, local size = 121<br class=""> KSP Object: (sub_) 1 MPI processes<br class=""> type: fgmres<br class=""> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement<br class=""> happy breakdown tolerance 1e-30<br class=""> maximum iterations=10000, initial guess is zero<br class=""> tolerances: relative=1e-12, absolute=1e-50, divergence=10000.<br class=""> right preconditioning<br class=""> using UNPRECONDITIONED norm type for convergence test<br class=""> PC Object: (sub_) 1 MPI processes<br class=""> type: ilu<br class=""> out-of-place factorization<br class=""> 0 levels of fill<br class=""> tolerance for zero pivot 2.22045e-14<br class=""> matrix ordering: natural<br class=""> factor fill ratio given 1., needed 1.<br class=""> Factored matrix follows:<br class=""> Mat Object: 1 MPI processes<br class=""> type: seqaij<br class=""> rows=121, cols=121<br class=""> package used to perform factorization: petsc<br class=""> total: nonzeros=961, allocated nonzeros=961<br class=""> total number of mallocs used during MatSetValues calls =0<br class=""> not using I-node routines<br class=""> linear system matrix = precond matrix:<br class=""> Mat Object: () 1 MPI processes<br class=""> type: seqaij<br class=""> rows=121, cols=121<br class=""> total: nonzeros=961, allocated nonzeros=961<br class=""> total number of mallocs used during MatSetValues calls =0<br class=""> not using I-node routines<br class=""> - - - - - - - - - - - - - - - - - -<br class="">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br class="">[0]PETSC ERROR: Petsc has generated inconsistent data<br class="">[0]PETSC ERROR: Called more times than PetscViewerASCIIPushSynchronized()<br class="">[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" class="">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br class="">[0]PETSC ERROR: Petsc Release Version 3.11.1, unknown <br class="">[0]PETSC ERROR: ./main on a darwin-dbg named <a href="http://boycesimacwork.dhcp.unc.edu" class="">boycesimacwork.dhcp.unc.edu</a> by boyceg Tue Apr 30 11:29:02 2019<br class="">[0]PETSC ERROR: Configure options --CC=mpicc --CXX=mpicxx --FC=mpif90 --PETSC_ARCH=darwin-dbg --with-debugging=1 --with-c++-support=1 --with-hypre=1 --download-hypre=1 --with-hdf5=1 --with-hdf5-dir=/usr/local<br class="">[0]PETSC ERROR: #1 PetscViewerASCIIPopSynchronized() line 438 in /Users/boyceg/sfw/petsc/petsc-maint/src/sys/classes/viewer/impls/ascii/filev.c<br class="">[0]PETSC ERROR: #2 PCView_GASM() line 251 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/impls/gasm/gasm.c<br class="">[0]PETSC ERROR: #3 PCView() line 1651 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/pc/interface/precon.c<br class="">[0]PETSC ERROR: #4 KSPView() line 213 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itcreate.c<br class="">[0]PETSC ERROR: #5 PetscObjectView() line 100 in /Users/boyceg/sfw/petsc/petsc-maint/src/sys/objects/destroy.c<br class="">[0]PETSC ERROR: #6 ObjectView() line 14 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c<br class="">[0]PETSC ERROR: #7 KSPSolve() line 831 in /Users/boyceg/sfw/petsc/petsc-maint/src/ksp/ksp/interface/itfunc.c<br class=""><br class=""><blockquote type="cite" class="">One difference between them is that the default GASM overlap is 0 (<a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGASMSetOverlap.html" class="">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGASMSetOverlap.html</a>), but the default ASM overlap is 1 (<a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCASMSetOverlap.html" class="">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCASMSetOverlap.html</a>).<br class=""><br class="">For your particular application, not sure you need any overlap since there are two different PDEs on two different subdomains. It may be fine to run the PC without the overlapping domain. It is a definitely interesting application for GASM. <br class=""><br class="">Fande,<br class=""><br class=""><br class="">However, changing the GASM overlap does not make any difference in the convergence history.<br class=""><br class="">-- Boyce<br class=""></blockquote><br class=""></blockquote><br class=""></blockquote><br class=""></div></div></blockquote></div><br class=""></body></html>