[petsc-users] issues with VecSetValues in petsc 3.19

Fri May 5 09:51:42 CDT 2023

  To expand on what Matt said slightly. 

When you have a preconditioner based on (possibly nested) sub solves one generally "tunes" the solves to minimize time to solution. We recommend doing this by first using very accurate subsolves (when possible using direct solves inside); this tells us the "best reasonably expected" convergence rate of the outermost solve. One then "backs off" on the accuracy of the inner solves (making them less accurate), monitoring how this "backing off" affects the convergence rate of the outermost solve (one will often allow the convergence of the outer-most solve to get "a bit worse" in the hope that the since the cost of the inner solves will be much cheaper (with the less accuracy) the cost of the extra outer iterations will not make the entire solve time longer. Deciding how much to "back-off" on each inner solve is mostly determined by monitoring the time for convergence of the entire problem (but monitoring number of iterations also provides important information, for example if the number of outer iterations suddenly shots up you know you are probably undersolving  a subproblem.

While doing this process, it is important to work with "production" size problems since for PDEs the convergence rates are often very affected by the problem size. One should start with smaller problems to get a feeling for the tolerances needed but usually final tuning needs to be done for large problems. And, sadly, if one changes some parameters in the system a very different tuning may be optimal.

  Barry

> On May 5, 2023, at 5:13 AM, Edoardo alinovi <edoardo.alinovi at gmail.com> wrote:
> 
> Hi Matt, 
> 
> I have some more questions on the fieldsplit saga :)
> 
> I am running a 1M cell ahmed body case using  the following options:
> 
>     "solver": "fgmres",
>     "preconditioner": "fieldsplit",
>      "absTol": 1e-6,
>      "relTol": 0.0,
> 
>     "options":{
>        "pc_fieldsplit_type": "multiplicative",
>        "fieldsplit_u_pc_type": "ml",
>        "fieldsplit_p_pc_type": "ml",
>        "fieldsplit_u_ksp_type": "preonly",
>        "fieldsplit_p_ksp_type": "preonly",
>        "fieldsplit_u_ksp_rtol": 1e-2,
>        "fieldsplit_p_ksp_rtol": 1e-2,
> }
> 
> 
> I have run the case using -ksp_monitor_true_residual and this is the results I am getting:
>      
>   Residual norms for UPeqn_ solve.
>   0 KSP unpreconditioned resid norm 5.003190920461e+00 true resid norm 5.003190920461e+00 ||r(i)||/||b|| 4.993739374163e-03
>   1 KSP unpreconditioned resid norm 4.959389845148e+00 true resid norm 4.959389845148e+00 ||r(i)||/||b|| 4.950021043622e-03
>   2 KSP unpreconditioned resid norm 4.793097370730e+00 true resid norm 4.793097370730e+00 ||r(i)||/||b|| 4.784042712927e-03
>   3 KSP unpreconditioned resid norm 4.187770916162e+00 true resid norm 4.187770916162e+00 ||r(i)||/||b|| 4.179859782782e-03
>   4 KSP unpreconditioned resid norm 3.099045576565e+00 true resid norm 3.099045576565e+00 ||r(i)||/||b|| 3.093191158212e-03
>   5 KSP unpreconditioned resid norm 2.072551338956e+00 true resid norm 2.072551338956e+00 ||r(i)||/||b|| 2.068636074628e-03
>   6 KSP unpreconditioned resid norm 1.414678932482e+00 true resid norm 1.414678932482e+00 ||r(i)||/||b|| 1.412006457328e-03
>   7 KSP unpreconditioned resid norm 1.006854855789e+00 true resid norm 1.006854855789e+00 ||r(i)||/||b|| 1.004952802592e-03
>   8 KSP unpreconditioned resid norm 7.332800358083e-01 true resid norm 7.332800358084e-01 ||r(i)||/||b|| 7.318947938062e-04
>   9 KSP unpreconditioned resid norm 5.406076142092e-01 true resid norm 5.406076142093e-01 ||r(i)||/||b|| 5.395863503846e-04
>  10 KSP unpreconditioned resid norm 4.037336888099e-01 true resid norm 4.037336888100e-01 ||r(i)||/||b|| 4.029709940193e-04
>  11 KSP unpreconditioned resid norm 3.041388930530e-01 true resid norm 3.041388930530e-01 ||r(i)||/||b|| 3.035643431559e-04
>  12 KSP unpreconditioned resid norm 2.299815364065e-01 true resid norm 2.299815364066e-01 ||r(i)||/||b|| 2.295470774436e-04
>  13 KSP unpreconditioned resid norm 1.739866268817e-01 true resid norm 1.739866268817e-01 ||r(i)||/||b|| 1.736579481074e-04
>  14 KSP unpreconditioned resid norm 1.317133652074e-01 true resid norm 1.317133652074e-01 ||r(i)||/||b|| 1.314645450067e-04
>  15 KSP unpreconditioned resid norm 9.966247212017e-02 true resid norm 9.966247212019e-02 ||r(i)||/||b|| 9.947419937897e-05
>  16 KSP unpreconditioned resid norm 7.531138284402e-02 true resid norm 7.531138284404e-02 ||r(i)||/||b|| 7.516911183479e-05
>  17 KSP unpreconditioned resid norm 5.646770286889e-02 true resid norm 5.646770286889e-02 ||r(i)||/||b|| 5.636102952452e-05
>  18 KSP unpreconditioned resid norm 4.225114444838e-02 true resid norm 4.225114444838e-02 ||r(i)||/||b|| 4.217132765661e-05
>  19 KSP unpreconditioned resid norm 3.160393382046e-02 true resid norm 3.160393382046e-02 ||r(i)||/||b|| 3.154423071329e-05
>  20 KSP unpreconditioned resid norm 2.366499890335e-02 true resid norm 2.366499890334e-02 ||r(i)||/||b|| 2.362029326721e-05
>  21 KSP unpreconditioned resid norm 1.759504138840e-02 true resid norm 1.759504138839e-02 ||r(i)||/||b|| 1.756180253124e-05
>  22 KSP unpreconditioned resid norm 1.309326628086e-02 true resid norm 1.309326628085e-02 ||r(i)||/||b|| 1.306853174355e-05
>  23 KSP unpreconditioned resid norm 9.710513089816e-03 true resid norm 9.710513089808e-03 ||r(i)||/||b|| 9.692168923954e-06
>  24 KSP unpreconditioned resid norm 7.171039760236e-03 true resid norm 7.171039760227e-03 ||r(i)||/||b|| 7.157492922743e-06
>  25 KSP unpreconditioned resid norm 5.277221846963e-03 true resid norm 5.277221846949e-03 ||r(i)||/||b|| 5.267252627824e-06
>  26 KSP unpreconditioned resid norm 3.906960734588e-03 true resid norm 3.906960734575e-03 ||r(i)||/||b|| 3.899580080737e-06
>  27 KSP unpreconditioned resid norm 2.896843259283e-03 true resid norm 2.896843259273e-03 ||r(i)||/||b|| 2.891370822059e-06
>  28 KSP unpreconditioned resid norm 2.140269358580e-03 true resid norm 2.140269358568e-03 ||r(i)||/||b|| 2.136226167881e-06
>  29 KSP unpreconditioned resid norm 1.585513255966e-03 true resid norm 1.585513255956e-03 ||r(i)||/||b|| 1.582518057055e-06
>  30 KSP unpreconditioned resid norm 1.173839272299e-03 true resid norm 1.173839272299e-03 ||r(i)||/||b|| 1.171621768229e-06
>  31 KSP unpreconditioned resid norm 8.777233482545e-04 true resid norm 8.777233482544e-04 ||r(i)||/||b|| 8.760652378615e-07
>  32 KSP unpreconditioned resid norm 6.546689191353e-04 true resid norm 6.546689191370e-04 ||r(i)||/||b|| 6.534321816833e-07
>  33 KSP unpreconditioned resid norm 4.973281362004e-04 true resid norm 4.973281362015e-04 ||r(i)||/||b|| 4.963886317973e-07
>  34 KSP unpreconditioned resid norm 3.775325682448e-04 true resid norm 3.775325682480e-04 ||r(i)||/||b|| 3.768193700902e-07
>  35 KSP unpreconditioned resid norm 2.908052735383e-04 true resid norm 2.908052735387e-04 ||r(i)||/||b|| 2.902559122310e-07
>  36 KSP unpreconditioned resid norm 2.239556213185e-04 true resid norm 2.239556213218e-04 ||r(i)||/||b|| 2.235325459370e-07
>  37 KSP unpreconditioned resid norm 1.698373081988e-04 true resid norm 1.698373081996e-04 ||r(i)||/||b|| 1.695164679184e-07
>  38 KSP unpreconditioned resid norm 1.277467123301e-04 true resid norm 1.277467123333e-04 ||r(i)||/||b|| 1.275053855510e-07
>  39 KSP unpreconditioned resid norm 9.506326626848e-05 true resid norm 9.506326626983e-05 ||r(i)||/||b|| 9.488368190523e-08
>  40 KSP unpreconditioned resid norm 7.223958235163e-05 true resid norm 7.223958235336e-05 ||r(i)||/||b|| 7.210311429367e-08
>  41 KSP unpreconditioned resid norm 5.509671615415e-05 true resid norm 5.509671615512e-05 ||r(i)||/||b|| 5.499263274677e-08
>  42 KSP unpreconditioned resid norm 4.189229263778e-05 true resid norm 4.189229263744e-05 ||r(i)||/||b|| 4.181315375394e-08
>  43 KSP unpreconditioned resid norm 3.067856645608e-05 true resid norm 3.067856645894e-05 ||r(i)||/||b|| 3.062061146665e-08
>  44 KSP unpreconditioned resid norm 2.340298386078e-05 true resid norm 2.340298386081e-05 ||r(i)||/||b|| 2.335877319825e-08
>  45 KSP unpreconditioned resid norm 1.791143784234e-05 true resid norm 1.791143784276e-05 ||r(i)||/||b|| 1.787760127991e-08
>  46 KSP unpreconditioned resid norm 1.355654057227e-05 true resid norm 1.355654057087e-05 ||r(i)||/||b|| 1.353093086041e-08
>  47 KSP unpreconditioned resid norm 1.020861247518e-05 true resid norm 1.020861247732e-05 ||r(i)||/||b|| 1.018932735009e-08
>  48 KSP unpreconditioned resid norm 7.642335784085e-06 true resid norm 7.642335784452e-06 ||r(i)||/||b|| 7.627898619923e-09
>  49 KSP unpreconditioned resid norm 5.874756954976e-06 true resid norm 5.874756956850e-06 ||r(i)||/||b|| 5.863658931960e-09
>  50 KSP unpreconditioned resid norm 4.512356844825e-06 true resid norm 4.512356846552e-06 ||r(i)||/||b|| 4.503832536701e-09
>  51 KSP unpreconditioned resid norm 3.438985239280e-06 true resid norm 3.438985240743e-06 ||r(i)||/||b|| 3.432488641125e-09
>  52 KSP unpreconditioned resid norm 2.655998139390e-06 true resid norm 2.655998140374e-06 ||r(i)||/||b|| 2.650980684556e-09
>  53 KSP unpreconditioned resid norm 2.051081181832e-06 true resid norm 2.051081181929e-06 ||r(i)||/||b|| 2.047206476954e-09
>  54 KSP unpreconditioned resid norm 1.581756364000e-06 true resid norm 1.581756364725e-06 ||r(i)||/||b|| 1.578768262982e-09
>  55 KSP unpreconditioned resid norm 1.207420527415e-06 true resid norm 1.207420527996e-06 ||r(i)||/||b|| 1.205139585453e-09
>  56 KSP unpreconditioned resid norm 9.377914349033e-07 true resid norm 9.377914347157e-07 ||r(i)||/||b|| 9.360198494806e-10
> 
> I have the feeling I am self-inflicting some overhead not using relative tolerance (if I set 1e-3 I converge in 8 iters vs 56...), however I would like to ask the following question:
> 
> From your point of view, is this a reasonably convergence history?
> 
> Thank you!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230505/411a3d5a/attachment.html>