<div dir="ltr">We cannot say anything without also having -ksp_view and knowing something about the system<div><br></div><div> Matt</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 5, 2015 at 1:14 PM, Danyang Su <span dir="ltr"><<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi All,<br>
<br>
I have a complex flow, heat transport and reactive transport problem that is not scalable after 8 processors. When using less than 8 processors, it has almost linear speedup. But after 8 processor, the number of KSP iteration steps increases to 2 times more than using 1 processors or 4 processor (47 ksp steps vs 15 ksp steps, attached below). Do you have any suggestion to improve the performance?<br>
<br>
convergence summary using 1 or 4 processors<br>
------------------------------------------------------------------------<br>
timestep: 1 time: 1.000E-02 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 1.000E-01<br>
0 KSP preconditioned resid norm 1.137005501778e+06 true resid norm 9.637162652703e+06 ||r(i)||/||b|| 1.000000000000e+00<br>
1 KSP preconditioned resid norm 1.506085526342e+05 true resid norm 9.597161643502e+05 ||r(i)||/||b|| 9.958492960384e-02<br>
2 KSP preconditioned resid norm 5.456879627980e+04 true resid norm 1.406907438671e+05 ||r(i)||/||b|| 1.459877237079e-02<br>
3 KSP preconditioned resid norm 2.710407097622e+04 true resid norm 1.291826672621e+05 ||r(i)||/||b|| 1.340463701999e-02<br>
4 KSP preconditioned resid norm 9.897001667954e+03 true resid norm 1.994507102853e+04 ||r(i)||/||b|| 2.069599917247e-03<br>
5 KSP preconditioned resid norm 4.429538402026e+03 true resid norm 1.048435199832e+04 ||r(i)||/||b|| 1.087908586391e-03<br>
6 KSP preconditioned resid norm 2.183344020397e+03 true resid norm 5.418816689871e+03 ||r(i)||/||b|| 5.622834111190e-04<br>
7 KSP preconditioned resid norm 1.322492709422e+03 true resid norm 3.186218374195e+03 ||r(i)||/||b|| 3.306178892084e-04<br>
8 KSP preconditioned resid norm 6.738694120116e+02 true resid norm 1.461978701106e+03 ||r(i)||/||b|| 1.517021922107e-04<br>
9 KSP preconditioned resid norm 3.720835435101e+02 true resid norm 4.069516281820e+02 ||r(i)||/||b|| 4.222732798516e-05<br>
10 KSP preconditioned resid norm 1.874689621103e+02 true resid norm 5.563177554677e+02 ||r(i)||/||b|| 5.772630135195e-05<br>
11 KSP preconditioned resid norm 8.612569469037e+01 true resid norm 1.054136865792e+02 ||r(i)||/||b|| 1.093824918994e-05<br>
12 KSP preconditioned resid norm 4.181701635083e+01 true resid norm 1.125544728389e+02 ||r(i)||/||b|| 1.167921274083e-05<br>
13 KSP preconditioned resid norm 2.181470261073e+01 true resid norm 2.314218289439e+01 ||r(i)||/||b|| 2.401348169412e-06<br>
14 KSP preconditioned resid norm 1.180501182018e+01 true resid norm 2.818614875745e+01 ||r(i)||/||b|| 2.924735191592e-06<br>
15 KSP preconditioned resid norm 6.607882346065e+00 true resid norm 7.039364165934e+00 ||r(i)||/||b|| 7.304394892577e-07<br>
Linear solve converged due to CONVERGED_RTOL iterations 15<br>
<br>
<br>
convergence summary using 8 or 16 processors<br>
------------------------------------------------------------------------<br>
timestep: 1 time: 1.000E-02 years delt: 1.000E-02 years iter: 1 max.sia: 0.000E+00 tol.sia: 1.000E-01<br>
0 KSP preconditioned resid norm 1.172246902854e+06 true resid norm 9.637162652703e+06 ||r(i)||/||b|| 1.000000000000e+00<br>
1 KSP preconditioned resid norm 2.111438753837e+05 true resid norm 3.422907872797e+06 ||r(i)||/||b|| 3.551779705448e-01<br>
2 KSP preconditioned resid norm 9.189117316053e+04 true resid norm 1.197217247701e+06 ||r(i)||/||b|| 1.242292250163e-01<br>
3 KSP preconditioned resid norm 3.567188384324e+04 true resid norm 3.070624730376e+05 ||r(i)||/||b|| 3.186233169485e-02<br>
4 KSP preconditioned resid norm 1.629589856587e+04 true resid norm 1.857713260583e+05 ||r(i)||/||b|| 1.927655812743e-02<br>
5 KSP preconditioned resid norm 8.006759563515e+03 true resid norm 1.180288951151e+05 ||r(i)||/||b|| 1.224726606456e-02<br>
6 KSP preconditioned resid norm 5.358897276807e+03 true resid norm 1.244255645533e+05 ||r(i)||/||b|| 1.291101634758e-02<br>
7 KSP preconditioned resid norm 4.389072625628e+03 true resid norm 1.090628475655e+05 ||r(i)||/||b|| 1.131690431051e-02<br>
8 KSP preconditioned resid norm 3.744362402037e+03 true resid norm 1.068327247800e+05 ||r(i)||/||b|| 1.108549566194e-02<br>
9 KSP preconditioned resid norm 3.329272298367e+03 true resid norm 1.000736812587e+05 ||r(i)||/||b|| 1.038414363906e-02<br>
10 KSP preconditioned resid norm 2.973880694381e+03 true resid norm 9.519213877340e+04 ||r(i)||/||b|| 9.877610475599e-03<br>
11 KSP preconditioned resid norm 2.672681833195e+03 true resid norm 8.484262500566e+04 ||r(i)||/||b|| 8.803693375649e-03<br>
12 KSP preconditioned resid norm 2.174736539891e+03 true resid norm 7.637960549691e+04 ||r(i)||/||b|| 7.925528316728e-03<br>
13 KSP preconditioned resid norm 1.920508771340e+03 true resid norm 6.875465496734e+04 ||r(i)||/||b|| 7.134325469547e-03<br>
14 KSP preconditioned resid norm 1.748769817174e+03 true resid norm 6.402474320438e+04 ||r(i)||/||b|| 6.643526265111e-03<br>
15 KSP preconditioned resid norm 1.668781376573e+03 true resid norm 5.938074141272e+04 ||r(i)||/||b|| 6.161641507219e-03<br>
16 KSP preconditioned resid norm 1.507934446836e+03 true resid norm 5.376897976816e+04 ||r(i)||/||b|| 5.579337166534e-03<br>
17 KSP preconditioned resid norm 1.341780677843e+03 true resid norm 4.418909196385e+04 ||r(i)||/||b|| 4.585280290092e-03<br>
18 KSP preconditioned resid norm 1.096322610019e+03 true resid norm 3.281273489858e+04 ||r(i)||/||b|| 3.404812814836e-03<br>
19 KSP preconditioned resid norm 8.572434892128e+02 true resid norm 2.162598538334e+04 ||r(i)||/||b|| 2.244019963415e-03<br>
20 KSP preconditioned resid norm 6.507523424872e+02 true resid norm 1.670057090484e+04 ||r(i)||/||b|| 1.732934423406e-03<br>
21 KSP preconditioned resid norm 5.390162236367e+02 true resid norm 1.360673589185e+04 ||r(i)||/||b|| 1.411902691922e-03<br>
22 KSP preconditioned resid norm 4.424143347792e+02 true resid norm 1.137879849375e+04 ||r(i)||/||b|| 1.180720810037e-03<br>
23 KSP preconditioned resid norm 3.720257830024e+02 true resid norm 9.449031285412e+03 ||r(i)||/||b|| 9.804785522388e-04<br>
24 KSP preconditioned resid norm 3.030947404532e+02 true resid norm 8.347069391513e+03 ||r(i)||/||b|| 8.661334972044e-04<br>
25 KSP preconditioned resid norm 2.667874861724e+02 true resid norm 7.530125164047e+03 ||r(i)||/||b|| 7.813632949253e-04<br>
26 KSP preconditioned resid norm 2.367701662092e+02 true resid norm 6.792074618833e+03 ||r(i)||/||b|| 7.047794940898e-04<br>
27 KSP preconditioned resid norm 2.083554010658e+02 true resid norm 5.833539191175e+03 ||r(i)||/||b|| 6.053170836065e-04<br>
28 KSP preconditioned resid norm 1.735399579409e+02 true resid norm 4.883873002196e+03 ||r(i)||/||b|| 5.067749895065e-04<br>
29 KSP preconditioned resid norm 1.467413383315e+02 true resid norm 4.058954246436e+03 ||r(i)||/||b|| 4.211773104502e-04<br>
30 KSP preconditioned resid norm 1.244235694540e+02 true resid norm 3.337590813452e+03 ||r(i)||/||b|| 3.463250474989e-04<br>
31 KSP preconditioned resid norm 1.198432875407e+02 true resid norm 3.199014077607e+03 ||r(i)||/||b|| 3.319456351305e-04<br>
32 KSP preconditioned resid norm 1.027669478022e+02 true resid norm 3.150305508056e+03 ||r(i)||/||b|| 3.268913913342e-04<br>
33 KSP preconditioned resid norm 9.663893782079e+01 true resid norm 2.857716117586e+03 ||r(i)||/||b|| 2.965308587777e-04<br>
34 KSP preconditioned resid norm 8.531640537674e+01 true resid norm 2.584725514230e+03 ||r(i)||/||b|| 2.682039939945e-04<br>
35 KSP preconditioned resid norm 7.610279175833e+01 true resid norm 2.212227973785e+03 ||r(i)||/||b|| 2.295517937704e-04<br>
36 KSP preconditioned resid norm 6.331885100455e+01 true resid norm 1.858910367331e+03 ||r(i)||/||b|| 1.928897990333e-04<br>
37 KSP preconditioned resid norm 5.635209736607e+01 true resid norm 1.571087791205e+03 ||r(i)||/||b|| 1.630238948769e-04<br>
38 KSP preconditioned resid norm 4.815528377271e+01 true resid norm 1.313627294747e+03 ||r(i)||/||b|| 1.363085113416e-04<br>
39 KSP preconditioned resid norm 4.314593066310e+01 true resid norm 1.141179411417e+03 ||r(i)||/||b|| 1.184144599963e-04<br>
40 KSP preconditioned resid norm 3.700552222031e+01 true resid norm 9.643354964443e+02 ||r(i)||/||b|| 1.000642545110e-04<br>
41 KSP preconditioned resid norm 3.059145971331e+01 true resid norm 7.755572523134e+02 ||r(i)||/||b|| 8.047568358679e-05<br>
42 KSP preconditioned resid norm 2.553811607631e+01 true resid norm 6.248805390867e+02 ||r(i)||/||b|| 6.484071729467e-05<br>
43 KSP preconditioned resid norm 2.016013790111e+01 true resid norm 5.415908243972e+02 ||r(i)||/||b|| 5.619816162855e-05<br>
44 KSP preconditioned resid norm 1.757243660826e+01 true resid norm 4.812748098035e+02 ||r(i)||/||b|| 4.993947151743e-05<br>
45 KSP preconditioned resid norm 1.498503590085e+01 true resid norm 4.478420094581e+02 ||r(i)||/||b|| 4.647031762326e-05<br>
46 KSP preconditioned resid norm 1.348574164098e+01 true resid norm 3.944236184677e+02 ||r(i)||/||b|| 4.092735929461e-05<br>
47 KSP preconditioned resid norm 1.156613274720e+01 true resid norm 3.682508444462e+02 ||r(i)||/||b|| 3.821154189432e-05<br>
Linear solve converged due to CONVERGED_RTOL iterations 47<br>
<br>
Thanks and regards,<br>
<br>
Danyang<br>
<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div>