[petsc-users] KSP iterations increase rapidly after 8 processors

Mon Oct 5 13:14:30 CDT 2015

Hi All,

I have a complex flow, heat transport and reactive transport problem 
that is not scalable after 8 processors. When using less than 8 
processors, it has almost linear speedup. But after 8 processor, the 
number of KSP iteration steps increases to 2 times more than using 1 
processors or 4 processor (47 ksp steps vs 15 ksp steps, attached 
below). Do you have any suggestion to improve the performance?

convergence summary using 1 or 4 processors
  ------------------------------------------------------------------------
  timestep:       1 time: 1.000E-02 years   delt: 1.000E-02 years iter:  
1 max.sia: 0.000E+00 tol.sia: 1.000E-01
   0 KSP preconditioned resid norm 1.137005501778e+06 true resid norm 
9.637162652703e+06 ||r(i)||/||b|| 1.000000000000e+00
   1 KSP preconditioned resid norm 1.506085526342e+05 true resid norm 
9.597161643502e+05 ||r(i)||/||b|| 9.958492960384e-02
   2 KSP preconditioned resid norm 5.456879627980e+04 true resid norm 
1.406907438671e+05 ||r(i)||/||b|| 1.459877237079e-02
   3 KSP preconditioned resid norm 2.710407097622e+04 true resid norm 
1.291826672621e+05 ||r(i)||/||b|| 1.340463701999e-02
   4 KSP preconditioned resid norm 9.897001667954e+03 true resid norm 
1.994507102853e+04 ||r(i)||/||b|| 2.069599917247e-03
   5 KSP preconditioned resid norm 4.429538402026e+03 true resid norm 
1.048435199832e+04 ||r(i)||/||b|| 1.087908586391e-03
   6 KSP preconditioned resid norm 2.183344020397e+03 true resid norm 
5.418816689871e+03 ||r(i)||/||b|| 5.622834111190e-04
   7 KSP preconditioned resid norm 1.322492709422e+03 true resid norm 
3.186218374195e+03 ||r(i)||/||b|| 3.306178892084e-04
   8 KSP preconditioned resid norm 6.738694120116e+02 true resid norm 
1.461978701106e+03 ||r(i)||/||b|| 1.517021922107e-04
   9 KSP preconditioned resid norm 3.720835435101e+02 true resid norm 
4.069516281820e+02 ||r(i)||/||b|| 4.222732798516e-05
  10 KSP preconditioned resid norm 1.874689621103e+02 true resid norm 
5.563177554677e+02 ||r(i)||/||b|| 5.772630135195e-05
  11 KSP preconditioned resid norm 8.612569469037e+01 true resid norm 
1.054136865792e+02 ||r(i)||/||b|| 1.093824918994e-05
  12 KSP preconditioned resid norm 4.181701635083e+01 true resid norm 
1.125544728389e+02 ||r(i)||/||b|| 1.167921274083e-05
  13 KSP preconditioned resid norm 2.181470261073e+01 true resid norm 
2.314218289439e+01 ||r(i)||/||b|| 2.401348169412e-06
  14 KSP preconditioned resid norm 1.180501182018e+01 true resid norm 
2.818614875745e+01 ||r(i)||/||b|| 2.924735191592e-06
  15 KSP preconditioned resid norm 6.607882346065e+00 true resid norm 
7.039364165934e+00 ||r(i)||/||b|| 7.304394892577e-07
Linear solve converged due to CONVERGED_RTOL iterations 15

convergence summary using 8 or 16 processors
  ------------------------------------------------------------------------
  timestep:       1 time: 1.000E-02 years   delt: 1.000E-02 years iter:  
1 max.sia: 0.000E+00 tol.sia: 1.000E-01
   0 KSP preconditioned resid norm 1.172246902854e+06 true resid norm 
9.637162652703e+06 ||r(i)||/||b|| 1.000000000000e+00
   1 KSP preconditioned resid norm 2.111438753837e+05 true resid norm 
3.422907872797e+06 ||r(i)||/||b|| 3.551779705448e-01
   2 KSP preconditioned resid norm 9.189117316053e+04 true resid norm 
1.197217247701e+06 ||r(i)||/||b|| 1.242292250163e-01
   3 KSP preconditioned resid norm 3.567188384324e+04 true resid norm 
3.070624730376e+05 ||r(i)||/||b|| 3.186233169485e-02
   4 KSP preconditioned resid norm 1.629589856587e+04 true resid norm 
1.857713260583e+05 ||r(i)||/||b|| 1.927655812743e-02
   5 KSP preconditioned resid norm 8.006759563515e+03 true resid norm 
1.180288951151e+05 ||r(i)||/||b|| 1.224726606456e-02
   6 KSP preconditioned resid norm 5.358897276807e+03 true resid norm 
1.244255645533e+05 ||r(i)||/||b|| 1.291101634758e-02
   7 KSP preconditioned resid norm 4.389072625628e+03 true resid norm 
1.090628475655e+05 ||r(i)||/||b|| 1.131690431051e-02
   8 KSP preconditioned resid norm 3.744362402037e+03 true resid norm 
1.068327247800e+05 ||r(i)||/||b|| 1.108549566194e-02
   9 KSP preconditioned resid norm 3.329272298367e+03 true resid norm 
1.000736812587e+05 ||r(i)||/||b|| 1.038414363906e-02
  10 KSP preconditioned resid norm 2.973880694381e+03 true resid norm 
9.519213877340e+04 ||r(i)||/||b|| 9.877610475599e-03
  11 KSP preconditioned resid norm 2.672681833195e+03 true resid norm 
8.484262500566e+04 ||r(i)||/||b|| 8.803693375649e-03
  12 KSP preconditioned resid norm 2.174736539891e+03 true resid norm 
7.637960549691e+04 ||r(i)||/||b|| 7.925528316728e-03
  13 KSP preconditioned resid norm 1.920508771340e+03 true resid norm 
6.875465496734e+04 ||r(i)||/||b|| 7.134325469547e-03
  14 KSP preconditioned resid norm 1.748769817174e+03 true resid norm 
6.402474320438e+04 ||r(i)||/||b|| 6.643526265111e-03
  15 KSP preconditioned resid norm 1.668781376573e+03 true resid norm 
5.938074141272e+04 ||r(i)||/||b|| 6.161641507219e-03
  16 KSP preconditioned resid norm 1.507934446836e+03 true resid norm 
5.376897976816e+04 ||r(i)||/||b|| 5.579337166534e-03
  17 KSP preconditioned resid norm 1.341780677843e+03 true resid norm 
4.418909196385e+04 ||r(i)||/||b|| 4.585280290092e-03
  18 KSP preconditioned resid norm 1.096322610019e+03 true resid norm 
3.281273489858e+04 ||r(i)||/||b|| 3.404812814836e-03
  19 KSP preconditioned resid norm 8.572434892128e+02 true resid norm 
2.162598538334e+04 ||r(i)||/||b|| 2.244019963415e-03
  20 KSP preconditioned resid norm 6.507523424872e+02 true resid norm 
1.670057090484e+04 ||r(i)||/||b|| 1.732934423406e-03
  21 KSP preconditioned resid norm 5.390162236367e+02 true resid norm 
1.360673589185e+04 ||r(i)||/||b|| 1.411902691922e-03
  22 KSP preconditioned resid norm 4.424143347792e+02 true resid norm 
1.137879849375e+04 ||r(i)||/||b|| 1.180720810037e-03
  23 KSP preconditioned resid norm 3.720257830024e+02 true resid norm 
9.449031285412e+03 ||r(i)||/||b|| 9.804785522388e-04
  24 KSP preconditioned resid norm 3.030947404532e+02 true resid norm 
8.347069391513e+03 ||r(i)||/||b|| 8.661334972044e-04
  25 KSP preconditioned resid norm 2.667874861724e+02 true resid norm 
7.530125164047e+03 ||r(i)||/||b|| 7.813632949253e-04
  26 KSP preconditioned resid norm 2.367701662092e+02 true resid norm 
6.792074618833e+03 ||r(i)||/||b|| 7.047794940898e-04
  27 KSP preconditioned resid norm 2.083554010658e+02 true resid norm 
5.833539191175e+03 ||r(i)||/||b|| 6.053170836065e-04
  28 KSP preconditioned resid norm 1.735399579409e+02 true resid norm 
4.883873002196e+03 ||r(i)||/||b|| 5.067749895065e-04
  29 KSP preconditioned resid norm 1.467413383315e+02 true resid norm 
4.058954246436e+03 ||r(i)||/||b|| 4.211773104502e-04
  30 KSP preconditioned resid norm 1.244235694540e+02 true resid norm 
3.337590813452e+03 ||r(i)||/||b|| 3.463250474989e-04
  31 KSP preconditioned resid norm 1.198432875407e+02 true resid norm 
3.199014077607e+03 ||r(i)||/||b|| 3.319456351305e-04
  32 KSP preconditioned resid norm 1.027669478022e+02 true resid norm 
3.150305508056e+03 ||r(i)||/||b|| 3.268913913342e-04
  33 KSP preconditioned resid norm 9.663893782079e+01 true resid norm 
2.857716117586e+03 ||r(i)||/||b|| 2.965308587777e-04
  34 KSP preconditioned resid norm 8.531640537674e+01 true resid norm 
2.584725514230e+03 ||r(i)||/||b|| 2.682039939945e-04
  35 KSP preconditioned resid norm 7.610279175833e+01 true resid norm 
2.212227973785e+03 ||r(i)||/||b|| 2.295517937704e-04
  36 KSP preconditioned resid norm 6.331885100455e+01 true resid norm 
1.858910367331e+03 ||r(i)||/||b|| 1.928897990333e-04
  37 KSP preconditioned resid norm 5.635209736607e+01 true resid norm 
1.571087791205e+03 ||r(i)||/||b|| 1.630238948769e-04
  38 KSP preconditioned resid norm 4.815528377271e+01 true resid norm 
1.313627294747e+03 ||r(i)||/||b|| 1.363085113416e-04
  39 KSP preconditioned resid norm 4.314593066310e+01 true resid norm 
1.141179411417e+03 ||r(i)||/||b|| 1.184144599963e-04
  40 KSP preconditioned resid norm 3.700552222031e+01 true resid norm 
9.643354964443e+02 ||r(i)||/||b|| 1.000642545110e-04
  41 KSP preconditioned resid norm 3.059145971331e+01 true resid norm 
7.755572523134e+02 ||r(i)||/||b|| 8.047568358679e-05
  42 KSP preconditioned resid norm 2.553811607631e+01 true resid norm 
6.248805390867e+02 ||r(i)||/||b|| 6.484071729467e-05
  43 KSP preconditioned resid norm 2.016013790111e+01 true resid norm 
5.415908243972e+02 ||r(i)||/||b|| 5.619816162855e-05
  44 KSP preconditioned resid norm 1.757243660826e+01 true resid norm 
4.812748098035e+02 ||r(i)||/||b|| 4.993947151743e-05
  45 KSP preconditioned resid norm 1.498503590085e+01 true resid norm 
4.478420094581e+02 ||r(i)||/||b|| 4.647031762326e-05
  46 KSP preconditioned resid norm 1.348574164098e+01 true resid norm 
3.944236184677e+02 ||r(i)||/||b|| 4.092735929461e-05
  47 KSP preconditioned resid norm 1.156613274720e+01 true resid norm 
3.682508444462e+02 ||r(i)||/||b|| 3.821154189432e-05
Linear solve converged due to CONVERGED_RTOL iterations 47

Thanks and regards,

Danyang