[petsc-users] KSP iterations increase rapidly after 8 processors

Matthew Knepley knepley at gmail.com
Mon Oct 5 13:47:22 CDT 2015 

We cannot say anything without also having -ksp_view and knowing something
about the system

    Matt

On Mon, Oct 5, 2015 at 1:14 PM, Danyang Su <danyang.su at gmail.com> wrote:

> Hi All,
>
> I have a complex flow, heat transport and reactive transport problem that
> is not scalable after 8 processors. When using less than 8 processors, it
> has almost linear speedup. But after 8 processor, the number of KSP
> iteration steps increases to 2 times more than using 1 processors or 4
> processor (47 ksp steps vs 15 ksp steps, attached below). Do you have any
> suggestion to improve the performance?
>
> convergence summary using 1 or 4 processors
>  ------------------------------------------------------------------------
>  timestep:       1 time: 1.000E-02 years   delt: 1.000E-02 years iter:  1
> max.sia: 0.000E+00 tol.sia: 1.000E-01
>   0 KSP preconditioned resid norm 1.137005501778e+06 true resid norm
> 9.637162652703e+06 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.506085526342e+05 true resid norm
> 9.597161643502e+05 ||r(i)||/||b|| 9.958492960384e-02
>   2 KSP preconditioned resid norm 5.456879627980e+04 true resid norm
> 1.406907438671e+05 ||r(i)||/||b|| 1.459877237079e-02
>   3 KSP preconditioned resid norm 2.710407097622e+04 true resid norm
> 1.291826672621e+05 ||r(i)||/||b|| 1.340463701999e-02
>   4 KSP preconditioned resid norm 9.897001667954e+03 true resid norm
> 1.994507102853e+04 ||r(i)||/||b|| 2.069599917247e-03
>   5 KSP preconditioned resid norm 4.429538402026e+03 true resid norm
> 1.048435199832e+04 ||r(i)||/||b|| 1.087908586391e-03
>   6 KSP preconditioned resid norm 2.183344020397e+03 true resid norm
> 5.418816689871e+03 ||r(i)||/||b|| 5.622834111190e-04
>   7 KSP preconditioned resid norm 1.322492709422e+03 true resid norm
> 3.186218374195e+03 ||r(i)||/||b|| 3.306178892084e-04
>   8 KSP preconditioned resid norm 6.738694120116e+02 true resid norm
> 1.461978701106e+03 ||r(i)||/||b|| 1.517021922107e-04
>   9 KSP preconditioned resid norm 3.720835435101e+02 true resid norm
> 4.069516281820e+02 ||r(i)||/||b|| 4.222732798516e-05
>  10 KSP preconditioned resid norm 1.874689621103e+02 true resid norm
> 5.563177554677e+02 ||r(i)||/||b|| 5.772630135195e-05
>  11 KSP preconditioned resid norm 8.612569469037e+01 true resid norm
> 1.054136865792e+02 ||r(i)||/||b|| 1.093824918994e-05
>  12 KSP preconditioned resid norm 4.181701635083e+01 true resid norm
> 1.125544728389e+02 ||r(i)||/||b|| 1.167921274083e-05
>  13 KSP preconditioned resid norm 2.181470261073e+01 true resid norm
> 2.314218289439e+01 ||r(i)||/||b|| 2.401348169412e-06
>  14 KSP preconditioned resid norm 1.180501182018e+01 true resid norm
> 2.818614875745e+01 ||r(i)||/||b|| 2.924735191592e-06
>  15 KSP preconditioned resid norm 6.607882346065e+00 true resid norm
> 7.039364165934e+00 ||r(i)||/||b|| 7.304394892577e-07
> Linear solve converged due to CONVERGED_RTOL iterations 15
>
>
> convergence summary using 8 or 16 processors
>  ------------------------------------------------------------------------
>  timestep:       1 time: 1.000E-02 years   delt: 1.000E-02 years iter:  1
> max.sia: 0.000E+00 tol.sia: 1.000E-01
>   0 KSP preconditioned resid norm 1.172246902854e+06 true resid norm
> 9.637162652703e+06 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.111438753837e+05 true resid norm
> 3.422907872797e+06 ||r(i)||/||b|| 3.551779705448e-01
>   2 KSP preconditioned resid norm 9.189117316053e+04 true resid norm
> 1.197217247701e+06 ||r(i)||/||b|| 1.242292250163e-01
>   3 KSP preconditioned resid norm 3.567188384324e+04 true resid norm
> 3.070624730376e+05 ||r(i)||/||b|| 3.186233169485e-02
>   4 KSP preconditioned resid norm 1.629589856587e+04 true resid norm
> 1.857713260583e+05 ||r(i)||/||b|| 1.927655812743e-02
>   5 KSP preconditioned resid norm 8.006759563515e+03 true resid norm
> 1.180288951151e+05 ||r(i)||/||b|| 1.224726606456e-02
>   6 KSP preconditioned resid norm 5.358897276807e+03 true resid norm
> 1.244255645533e+05 ||r(i)||/||b|| 1.291101634758e-02
>   7 KSP preconditioned resid norm 4.389072625628e+03 true resid norm
> 1.090628475655e+05 ||r(i)||/||b|| 1.131690431051e-02
>   8 KSP preconditioned resid norm 3.744362402037e+03 true resid norm
> 1.068327247800e+05 ||r(i)||/||b|| 1.108549566194e-02
>   9 KSP preconditioned resid norm 3.329272298367e+03 true resid norm
> 1.000736812587e+05 ||r(i)||/||b|| 1.038414363906e-02
>  10 KSP preconditioned resid norm 2.973880694381e+03 true resid norm
> 9.519213877340e+04 ||r(i)||/||b|| 9.877610475599e-03
>  11 KSP preconditioned resid norm 2.672681833195e+03 true resid norm
> 8.484262500566e+04 ||r(i)||/||b|| 8.803693375649e-03
>  12 KSP preconditioned resid norm 2.174736539891e+03 true resid norm
> 7.637960549691e+04 ||r(i)||/||b|| 7.925528316728e-03
>  13 KSP preconditioned resid norm 1.920508771340e+03 true resid norm
> 6.875465496734e+04 ||r(i)||/||b|| 7.134325469547e-03
>  14 KSP preconditioned resid norm 1.748769817174e+03 true resid norm
> 6.402474320438e+04 ||r(i)||/||b|| 6.643526265111e-03
>  15 KSP preconditioned resid norm 1.668781376573e+03 true resid norm
> 5.938074141272e+04 ||r(i)||/||b|| 6.161641507219e-03
>  16 KSP preconditioned resid norm 1.507934446836e+03 true resid norm
> 5.376897976816e+04 ||r(i)||/||b|| 5.579337166534e-03
>  17 KSP preconditioned resid norm 1.341780677843e+03 true resid norm
> 4.418909196385e+04 ||r(i)||/||b|| 4.585280290092e-03
>  18 KSP preconditioned resid norm 1.096322610019e+03 true resid norm
> 3.281273489858e+04 ||r(i)||/||b|| 3.404812814836e-03
>  19 KSP preconditioned resid norm 8.572434892128e+02 true resid norm
> 2.162598538334e+04 ||r(i)||/||b|| 2.244019963415e-03
>  20 KSP preconditioned resid norm 6.507523424872e+02 true resid norm
> 1.670057090484e+04 ||r(i)||/||b|| 1.732934423406e-03
>  21 KSP preconditioned resid norm 5.390162236367e+02 true resid norm
> 1.360673589185e+04 ||r(i)||/||b|| 1.411902691922e-03
>  22 KSP preconditioned resid norm 4.424143347792e+02 true resid norm
> 1.137879849375e+04 ||r(i)||/||b|| 1.180720810037e-03
>  23 KSP preconditioned resid norm 3.720257830024e+02 true resid norm
> 9.449031285412e+03 ||r(i)||/||b|| 9.804785522388e-04
>  24 KSP preconditioned resid norm 3.030947404532e+02 true resid norm
> 8.347069391513e+03 ||r(i)||/||b|| 8.661334972044e-04
>  25 KSP preconditioned resid norm 2.667874861724e+02 true resid norm
> 7.530125164047e+03 ||r(i)||/||b|| 7.813632949253e-04
>  26 KSP preconditioned resid norm 2.367701662092e+02 true resid norm
> 6.792074618833e+03 ||r(i)||/||b|| 7.047794940898e-04
>  27 KSP preconditioned resid norm 2.083554010658e+02 true resid norm
> 5.833539191175e+03 ||r(i)||/||b|| 6.053170836065e-04
>  28 KSP preconditioned resid norm 1.735399579409e+02 true resid norm
> 4.883873002196e+03 ||r(i)||/||b|| 5.067749895065e-04
>  29 KSP preconditioned resid norm 1.467413383315e+02 true resid norm
> 4.058954246436e+03 ||r(i)||/||b|| 4.211773104502e-04
>  30 KSP preconditioned resid norm 1.244235694540e+02 true resid norm
> 3.337590813452e+03 ||r(i)||/||b|| 3.463250474989e-04
>  31 KSP preconditioned resid norm 1.198432875407e+02 true resid norm
> 3.199014077607e+03 ||r(i)||/||b|| 3.319456351305e-04
>  32 KSP preconditioned resid norm 1.027669478022e+02 true resid norm
> 3.150305508056e+03 ||r(i)||/||b|| 3.268913913342e-04
>  33 KSP preconditioned resid norm 9.663893782079e+01 true resid norm
> 2.857716117586e+03 ||r(i)||/||b|| 2.965308587777e-04
>  34 KSP preconditioned resid norm 8.531640537674e+01 true resid norm
> 2.584725514230e+03 ||r(i)||/||b|| 2.682039939945e-04
>  35 KSP preconditioned resid norm 7.610279175833e+01 true resid norm
> 2.212227973785e+03 ||r(i)||/||b|| 2.295517937704e-04
>  36 KSP preconditioned resid norm 6.331885100455e+01 true resid norm
> 1.858910367331e+03 ||r(i)||/||b|| 1.928897990333e-04
>  37 KSP preconditioned resid norm 5.635209736607e+01 true resid norm
> 1.571087791205e+03 ||r(i)||/||b|| 1.630238948769e-04
>  38 KSP preconditioned resid norm 4.815528377271e+01 true resid norm
> 1.313627294747e+03 ||r(i)||/||b|| 1.363085113416e-04
>  39 KSP preconditioned resid norm 4.314593066310e+01 true resid norm
> 1.141179411417e+03 ||r(i)||/||b|| 1.184144599963e-04
>  40 KSP preconditioned resid norm 3.700552222031e+01 true resid norm
> 9.643354964443e+02 ||r(i)||/||b|| 1.000642545110e-04
>  41 KSP preconditioned resid norm 3.059145971331e+01 true resid norm
> 7.755572523134e+02 ||r(i)||/||b|| 8.047568358679e-05
>  42 KSP preconditioned resid norm 2.553811607631e+01 true resid norm
> 6.248805390867e+02 ||r(i)||/||b|| 6.484071729467e-05
>  43 KSP preconditioned resid norm 2.016013790111e+01 true resid norm
> 5.415908243972e+02 ||r(i)||/||b|| 5.619816162855e-05
>  44 KSP preconditioned resid norm 1.757243660826e+01 true resid norm
> 4.812748098035e+02 ||r(i)||/||b|| 4.993947151743e-05
>  45 KSP preconditioned resid norm 1.498503590085e+01 true resid norm
> 4.478420094581e+02 ||r(i)||/||b|| 4.647031762326e-05
>  46 KSP preconditioned resid norm 1.348574164098e+01 true resid norm
> 3.944236184677e+02 ||r(i)||/||b|| 4.092735929461e-05
>  47 KSP preconditioned resid norm 1.156613274720e+01 true resid norm
> 3.682508444462e+02 ||r(i)||/||b|| 3.821154189432e-05
> Linear solve converged due to CONVERGED_RTOL iterations 47
>
> Thanks and regards,
>
> Danyang
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151005/f835110e/attachment.html>

More information about the petsc-users mailing list