[petsc-users] Nested solves and fgmres residual normalization

[Barry Smith]

   You are referring to 

 8 KSP Residual norm 8.464122393957e-07 
    Residual norms for fieldsplit_p_lsc_ solve.
    0 KSP Residual norm 1.000000000000e+00 
    1 KSP Residual norm 1.627170127450e-05 
    2 KSP Residual norm 1.448537328052e-05 
    3 KSP Residual norm 1.309526985856e-05 
    4 KSP Residual norm 1.308667010157e-05 
    5 KSP Residual norm 1.254021842629e-05 
    6 KSP Residual norm 9.490928295823e-06 

  where because the default rtol is 1.e-5 it takes 6 iterations, mostly to get the last little bit? 

  The thing I notice is that "Residual norms for fieldsplit_p_ solve.:" are converging pretty slowly, around 1 digit per iteration. That tells me that the inner inner solve is "oversolving", that is giving more accuracy than is likely needed. Of course, when it takes 1 iteration there is no way for you to back it off to take 1/2 an iteration :-).

  Some things to consider:

1) do not depend on the rtol at all but just set that inner inner solve to do a single iteration; clearly in this particular case this would work fine. (you could even turn off norm calculations in this inner solve, just use Richardson  in this inner inner solve, to save a tiny bit of time), of course, only experimentation would tell if that always works for your family of problems.

 2) or just use an rtol of say 1.e-3 in that inner inner solve. Likely that is ok given the relative slow convergence of the fieldsplit_p_ solve.

Now if the "Residual norms for fieldsplit_p_ solve.:" was dropping say 4 or more digits each iteration, then I would not generally back off the inner inner solve since likely in that case it would not be oversolving.

 I don't think the scaling factor as part of the FGMRES algorithm is relevant here.

  Barry





> On Jul 6, 2023, at 3:09 PM, Alexander Lindsay <alexlindsay239 at gmail.com> wrote:
> 
> Hi all, I've found that having fgmres as an outer solve and the initial residual scaling that comes with it can cause difficulties for inner solves as I'm approaching convergence, presumably because I'm running out of precision. This is the kind of thing I would normally set an absolute tolerance for, but it feels a little silly to set `-ksp_atol 1e-4` when at the beginning of solves there is no problem reaching that tolerance. Here is an example solve history. You can see that the fieldsplit_p_lsc_solve converges in a single iteration until the very end where it's clearly struggling to hit the rtol of 1e-5. Is the scaling of the fgmres residual important to the algorithm? git blame showed that the comment regarding "scale VEC_VV (the initial residual)" was last hit in 1999 (and it existed before that time), see line 125 of fgmres.c
> 
> 0 KSP Residual norm 2.586864431320e-06 
>   Residual norms for fieldsplit_p_ solve.
>   0 KSP Residual norm 4.692593309369e-02 
>     Residual norms for fieldsplit_p_lsc_ solve.
>     0 KSP Residual norm 1.000000000000e+00 
>     1 KSP Residual norm 4.048060575312e-06 
>     Residual norms for fieldsplit_p_lsc_ solve.
>     0 KSP Residual norm 4.951804839158e+01 
>     1 KSP Residual norm 9.672178594775e-05 
>   1 KSP Residual norm 2.076409133334e-02 
>     Residual norms for fieldsplit_p_lsc_ solve.
>     0 KSP Residual norm 1.000000000000e+00 
>     1 KSP Residual norm 1.346618231753e-06 
>     Residual norms for fieldsplit_p_lsc_ solve.
>     0 KSP Residual norm 8.119436471232e+01 
>     1 KSP Residual norm 1.233237431511e-04 
>   2 KSP Residual norm 8.753379232559e-03 
>     Residual norms for fieldsplit_p_lsc_ solve.
>     0 KSP Residual norm 1.000000000000e+00 
>     1 KSP Residual norm 1.023335686682e-06 
>     Residual norms for fieldsplit_p_lsc_ solve.
>     0 KSP Residual norm 1.095058035930e+02 
>     1 KSP Residual norm 8.452076098586e-05 
>   3 KSP Residual norm 1.589938296474e-03 
>     Residual norms for fieldsplit_p_lsc_ solve.
>     0 KSP Residual norm 1.000000000000e+00 
>     1 KSP Residual norm 7.680981295089e-06 
>     Residual norms for fieldsplit_p_lsc_ solve.
>     0 KSP Residual norm 7.768759831332e+01 
>     1 KSP Residual norm 5.120395415891e-04 
>   4 KSP Residual norm 4.191115394200e-04 
>     Residual norms for fieldsplit_p_lsc_ solve.
>     0 KSP Residual norm 1.000000000000e+00 
>     1 KSP Residual norm 8.815992679344e-07 
>     Residual norms for fieldsplit_p_lsc_ solve.
>     0 KSP Residual norm 5.245482238296e+01 
>     1 KSP Residual norm 8.672309412797e-05 
>   5 KSP Residual norm 1.053200310197e-04 
>     Residual norms for fieldsplit_p_lsc_ solve.
>     0 KSP Residual norm 1.000000000000e+00 
>     1 KSP Residual norm 1.450184043299e-06 
>     Residual norms for fieldsplit_p_lsc_ solve.
>     0 KSP Residual norm 9.058043325461e+01 
>     1 KSP Residual norm 1.643116692318e-04 
>   6 KSP Residual norm 2.315708746597e-05 
>     Residual norms for fieldsplit_p_lsc_ solve.
>     0 KSP Residual norm 1.000000000000e+00 
>     1 KSP Residual norm 1.406236994541e-06 
>     Residual norms for fieldsplit_p_lsc_ solve.
>     0 KSP Residual norm 5.158957906917e+01 
>     1 KSP Residual norm 8.910977751230e-05 
>   7 KSP Residual norm 2.378083073391e-06 
>     Residual norms for fieldsplit_p_lsc_ solve.
>     0 KSP Residual norm 1.000000000000e+00 
>     1 KSP Residual norm 7.981173164359e-06 
>     Residual norms for fieldsplit_p_lsc_ solve.
>     0 KSP Residual norm 7.271546763656e+01 
>     1 KSP Residual norm 2.211194524633e-04 
>   8 KSP Residual norm 8.464122393957e-07 
>     Residual norms for fieldsplit_p_lsc_ solve.
>     0 KSP Residual norm 1.000000000000e+00 
>     1 KSP Residual norm 1.627170127450e-05 
>     2 KSP Residual norm 1.448537328052e-05 
>     3 KSP Residual norm 1.309526985856e-05 
>     4 KSP Residual norm 1.308667010157e-05 
>     5 KSP Residual norm 1.254021842629e-05 
>     6 KSP Residual norm 9.490928295823e-06 
>     Residual norms for fieldsplit_p_lsc_ solve.
>     0 KSP Residual norm 7.084510697783e+01 
>     1 KSP Residual norm 1.444782053473e-04 
>   9 KSP Residual norm 3.334507715334e-07 
> Linear fieldsplit_p_ solve converged due to CONVERGED_RTOL iterations 9