<div dir="ltr">Hi Matt,<div><br></div><div>Thanks for your suggestions. Here is the output from Stream test on one node which has 20 cores. I run it up to 20. Attached are the dumped output with your suggested options. Really appreciate your help!!! </div><div><div><br></div><div>Number of MPI processes 1<br></div><div>Function Rate (MB/s) <br></div><div>Copy: 13816.9372</div><div>Scale: 8020.1809</div><div>Add: 12762.3830</div><div>Triad: 11852.5016</div><div><br></div><div>Number of MPI processes 2</div><div>Function Rate (MB/s) <br></div><div>Copy: 22748.7681</div><div>Scale: 14081.4906</div><div>Add: 18998.4516</div><div>Triad: 18303.2494</div><div><br></div><div>Number of MPI processes 3</div><div>Function Rate (MB/s) <br></div><div>Copy: 34045.2510</div><div>Scale: 23410.9767</div><div>Add: 30320.2702</div><div>Triad: 30163.7977</div><div><br></div><div>Number of MPI processes 4</div><div>Function Rate (MB/s) <br></div><div>Copy: 36875.5349</div><div>Scale: 29440.1694</div><div>Add: 36971.1860</div><div>Triad: 37377.0103</div><div><br></div><div>Number of MPI processes 5</div><div>Function Rate (MB/s) <br></div><div>Copy: 32272.8763</div><div>Scale: 30316.3435</div><div>Add: 38022.0193</div><div>Triad: 38815.4830</div><div><br></div><div>Number of MPI processes 6</div><div>Function Rate (MB/s) <br></div><div>Copy: 35619.8925</div><div>Scale: 34457.5078</div><div>Add: 41419.3722</div><div>Triad: 35825.3621</div><div><br></div><div>Number of MPI processes 7</div><div>Function Rate (MB/s) <br></div><div>Copy: 55284.2420</div><div>Scale: 47706.8009</div><div>Add: 59076.4735</div><div>Triad: 61680.5559</div><div><br></div><div>Number of MPI processes 8</div><div>Function Rate (MB/s) <br></div><div>Copy: 44525.8901</div><div>Scale: 48949.9599</div><div>Add: 57437.7784</div><div>Triad: 56671.0593</div><div><br></div><div>Number of MPI processes 9</div><div>Function Rate (MB/s) <br></div><div>Copy: 34375.7364</div><div>Scale: 29507.5293</div><div>Add: 45405.3120</div><div>Triad: 39518.7559</div><div><br></div><div>Number of MPI processes 10</div><div>Function Rate (MB/s) <br></div><div>Copy: 34278.0415</div><div>Scale: 41721.7843</div><div>Add: 46642.2465</div><div>Triad: 45454.7000</div><div><br></div><div>Number of MPI processes 11</div><div>Function Rate (MB/s) <br></div><div>Copy: 38093.7244</div><div>Scale: 35147.2412</div><div>Add: 45047.0853</div><div>Triad: 44983.2013</div><div><br></div><div>Number of MPI processes 12</div><div>Function Rate (MB/s) <br></div><div>Copy: 39750.8760</div><div>Scale: 52038.0631</div><div>Add: 55552.9503</div><div>Triad: 54884.3839</div><div><br></div><div>Number of MPI processes 13</div><div>Function Rate (MB/s) <br></div><div>Copy: 60839.0248</div><div>Scale: 74143.7458</div><div>Add: 85545.3135</div><div>Triad: 85667.6551</div><div><br></div><div>Number of MPI processes 14</div><div>Function Rate (MB/s) <br></div><div>Copy: 37766.2343</div><div>Scale: 40279.1928</div><div>Add: 49992.8572</div><div>Triad: 50303.4809</div><div><br></div><div>Number of MPI processes 15</div><div>Function Rate (MB/s) <br></div><div>Copy: 49762.3670</div><div>Scale: 59077.8251</div><div>Add: 60407.9651</div><div>Triad: 61691.9456</div><div><br></div><div>Number of MPI processes 16</div><div>Function Rate (MB/s) <br></div><div>Copy: 31996.7169</div><div>Scale: 36962.4860</div><div>Add: 40183.5060</div><div>Triad: 41096.0512</div><div><br></div><div>Number of MPI processes 17</div><div>Function Rate (MB/s) <br></div><div>Copy: 36348.3839</div><div>Scale: 39108.6761</div><div>Add: 46853.4476</div><div>Triad: 47266.1778</div><div><br></div><div>Number of MPI processes 18</div><div>Function Rate (MB/s) <br></div><div>Copy: 40438.7558</div><div>Scale: 43195.5785</div><div>Add: 53063.4321</div><div>Triad: 53605.0293</div><div><br></div><div>Number of MPI processes 19</div><div>Function Rate (MB/s) <br></div><div>Copy: 30739.4908</div><div>Scale: 34280.8118</div><div>Add: 40710.5155</div><div>Triad: 43330.9503</div><div><br></div><div>Number of MPI processes 20</div><div>Function Rate (MB/s) <br></div><div>Copy: 37488.3777</div><div>Scale: 41791.8999</div><div>Add: 49518.9604</div><div>Triad: 48908.2677</div><div>------------------------------------------------</div><div>np speedup</div><div>1 1.0</div><div>2 1.54</div><div>3 2.54</div><div>4 3.15</div><div>5 3.27</div><div>6 3.02</div><div>7 5.2</div><div>8 4.78</div><div>9 3.33</div><div>10 3.84</div><div>11 3.8</div><div>12 4.63</div><div>13 7.23</div><div>14 4.24</div><div>15 5.2</div><div>16 3.47</div><div>17 3.99</div><div>18 4.52</div><div>19 3.66</div><div>20 4.13</div></div><div><br></div><div><br></div><div><br><img src="https://t.yesware.com/t/d1fcbaa1b12e0f6b1beef0b50d5ebbd873d1b8f9/61ddd503d5285f26a8f55917a35bf6c6/spacer.gif" style="border: 0px; width: 0px; height: 0px; overflow: hidden;" width="0" height="0"><img src="http://t.yesware.com/t/d1fcbaa1b12e0f6b1beef0b50d5ebbd873d1b8f9/61ddd503d5285f26a8f55917a35bf6c6/spacer.gif" style="border: 0px; width: 0px; height: 0px; overflow: hidden;" width="0" height="0"><font face="yw-d1fcbaa1b12e0f6b1beef0b50d5ebbd873d1b8f9-61ddd503d5285f26a8f55917a35bf6c6--to" style></font></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature">Sincerely Yours,<br><br>Lei Shi <br>---------</div></div>
<br><div class="gmail_quote">On Thu, Jun 25, 2015 at 6:44 AM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Thu, Jun 25, 2015 at 5:51 AM, Lei Shi <span dir="ltr"><<a href="mailto:stoneszone@gmail.com" target="_blank">stoneszone@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hello,</div></div></blockquote><div><br></div><div>1) In order to understand this, we have to disentagle the various effect. First, run the STREAMS benchmark</div><div><br></div><div> make NPMAX=4 streams</div><div><br></div><div>This will tell you the maximum speedup you can expect on this machine.</div><div><br></div><div>2) For these test cases, also send the output of</div><div><br></div><div> -ksp_view -ksp_converged_reason -ksp_monitor_true_residual</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I'm trying to improve the parallel efficiency of gmres solve in my. In my CFD solver, Petsc gmres is used to solve the linear system generated by the Newton's method. To test its efficiency, I started with a very simple inviscid subsonic 3D flow as the first testcase. The parallel efficiency of gmres solve with asm as the preconditioner is very bad. The results are from our latest cluster. Right now, I'm only looking at the wclock time of the ksp_solve.</div><div><ol><li>First I tested ASM with gmres and ilu 0 for the sub domain , the cpu time of 2 cores is almost the same as the serial run. Here is the options for this case<br></li></ol></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>-ksp_type gmres -ksp_max_it 100 -ksp_rtol 1e-5 -ksp_atol 1e-50 </div></div><div><div>-ksp_gmres_restart 30 -ksp_pc_side right</div></div><div><div>-pc_type asm -sub_ksp_type gmres -sub_ksp_rtol 0.001 -sub_ksp_atol 1e-30</div></div><div><div>-sub_ksp_max_it 1000 -sub_pc_type ilu -sub_pc_factor_levels 0 </div></div><div><div>-sub_pc_factor_fill 1.9</div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>The iteration numbers increase a lot for parallel run.</div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><table cellspacing="0" cellpadding="0" dir="ltr" border="1" style="table-layout:fixed;font-size:13px;font-family:arial,sans,sans-serif;border-collapse:collapse;border:1px solid rgb(204,204,204)"><colgroup><col width="100"><col width="100"><col width="144"><col width="144"><col width="100"><col width="100"></colgroup><tbody><tr style="height:21px"><td style="padding:2px 3px;vertical-align:bottom">cores</td><td style="padding:2px 3px;vertical-align:bottom">iterations</td><td style="padding:2px 3px;vertical-align:bottom">err</td><td style="padding:2px 3px;vertical-align:bottom">petsc solve wclock time</td><td style="padding:2px 3px;vertical-align:bottom">speedup</td><td style="padding:2px 3px;vertical-align:bottom">efficiency</td></tr><tr style="height:21px"><td style="padding:2px 3px;vertical-align:bottom;text-align:right">1</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">2</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">1.15E-04</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">11.95</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">1</td><td></td></tr><tr style="height:21px"><td style="padding:2px 3px;vertical-align:bottom;text-align:right">2</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">5</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">2.05E-02</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">10.5</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">1.01</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">0.50</td></tr><tr style="height:21px"><td style="padding:2px 3px;vertical-align:bottom;text-align:right">4</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">6</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">2.19E-02</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">7.64</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">1.39</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">0.34</td></tr><tr style="height:21px"><td style="padding:2px 3px;vertical-align:bottom;text-align:right"><br></td><td style="padding:2px 3px;vertical-align:bottom;text-align:right"><br></td><td style="padding:2px 3px;vertical-align:bottom;text-align:right"><br></td><td style="padding:2px 3px;vertical-align:bottom;text-align:right"><br></td><td style="padding:2px 3px;vertical-align:bottom;text-align:right"><br></td><td style="padding:2px 3px;vertical-align:bottom;text-align:right"><br></td></tr></tbody></table></div></blockquote><div><div> </div><div> 2. Then I tested ASM with ilu 0 as the preconditoner only, the cpu time of 2 cores is better than the 1st test, but the speedup is still very bad. Here is the options i'm using<br></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>-ksp_type gmres -ksp_max_it 100 -ksp_rtol 1e-5 -ksp_atol 1e-50 </div></div><div><div>-ksp_gmres_restart 30 -ksp_pc_side right</div></div><div><div>-pc_type asm -sub_pc_type ilu -sub_pc_factor_levels 0 -sub_pc_factor_fill 1.9</div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><table cellspacing="0" cellpadding="0" border="1" style="table-layout:fixed;font-size:13px;font-family:arial,sans,sans-serif;border-collapse:collapse;border:1px solid rgb(204,204,204)"><colgroup><col width="100"><col width="100"><col width="144"><col width="144"><col width="100"><col width="100"></colgroup><tbody><tr style="height:21px"><td style="padding:2px 3px;vertical-align:bottom">cores</td><td style="padding:2px 3px;vertical-align:bottom">iterations</td><td style="padding:2px 3px;vertical-align:bottom">err</td><td style="padding:2px 3px;vertical-align:bottom">petsc solve cpu time</td><td style="padding:2px 3px;vertical-align:bottom">speedup</td><td style="padding:2px 3px;vertical-align:bottom">efficiency</td></tr><tr style="height:21px"><td style="padding:2px 3px;vertical-align:bottom;text-align:left">1</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">10</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">4.54E-04</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">10.68</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">1</td><td></td></tr><tr style="height:21px"><td style="padding:2px 3px;vertical-align:bottom;text-align:left">2</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">11</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">9.55E-04</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">8.2</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">1.30</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">0.65</td></tr><tr style="height:21px"><td style="padding:2px 3px;vertical-align:bottom;text-align:left">4</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">12</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">3.59E-04</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">5.26</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">2.03</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">0.50</td></tr><tr style="height:21px"><td style="padding:2px 3px;vertical-align:bottom;text-align:left"><br></td><td style="padding:2px 3px;vertical-align:bottom;text-align:left"><br></td><td style="padding:2px 3px;vertical-align:bottom;text-align:left"><br></td><td style="padding:2px 3px;vertical-align:bottom;text-align:left"><br></td><td style="padding:2px 3px;vertical-align:bottom;text-align:left"><br></td><td style="padding:2px 3px;vertical-align:bottom;text-align:left"><br></td></tr></tbody></table></blockquote><div><br></div><div> Those results are from a third order "DG" scheme with a very coarse 3D mesh (480 elements). I believe I should get some speedups for this test even on this coarse mesh. </div><div><br></div><div> My question is why does the asm with a local solve take much longer time than the asm as a preconditioner only? Also the accuracy is very bad too I have tested changing the overlap of asm to 2, but make it even worse.</div><div><br></div><div> If I used a larger mesh ~4000 elements, the 2nd case with asm as the preconditioner gives me a better speedup, but still not very good. </div><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><br></div><div><table cellspacing="0" cellpadding="0" border="1" style="table-layout:fixed;font-size:13px;font-family:arial,sans,sans-serif;border-collapse:collapse;border:1px solid rgb(204,204,204)"><colgroup><col width="100"><col width="100"><col width="100"><col width="100"><col width="100"><col width="100"></colgroup><tbody><tr style="height:21px"><td style="padding:2px 3px;vertical-align:bottom">cores</td><td style="padding:2px 3px;vertical-align:bottom">iterations</td><td style="padding:2px 3px;vertical-align:bottom">err</td><td style="padding:2px 3px;vertical-align:bottom">petsc solve cpu time</td><td style="padding:2px 3px;vertical-align:bottom">speedup</td><td style="padding:2px 3px;vertical-align:bottom">efficiency</td></tr><tr style="height:21px"><td style="padding:2px 3px;vertical-align:bottom;text-align:left">1</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">7</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">1.91E-02</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">97.32</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">1</td><td></td></tr><tr style="height:21px"><td style="padding:2px 3px;vertical-align:bottom;text-align:left">2</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">7</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">2.07E-02</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">64.94</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">1.5</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">0.74</td></tr><tr style="height:21px"><td style="padding:2px 3px;vertical-align:bottom;text-align:left">4</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">7</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">2.61E-02</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">36.97</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">2.6</td><td style="padding:2px 3px;vertical-align:bottom;text-align:left">0.65</td></tr></tbody></table></div></blockquote></div><div><br></div><div><br></div><div>Attached are the log_summary dumped from petsc, any suggestions are welcome. I really appreciate it.</div><div><br></div><div><br clear="all"><div><div>Sincerely Yours,<br><br>Lei Shi <br>---------</div></div>
</div><img src="https://t.yesware.com/t/d1fcbaa1b12e0f6b1beef0b50d5ebbd873d1b8f9/e8040b69bc7ca00f2bce0335cfcc6ef8/spacer.gif" style="border:0px;width:0px;min-height:0px;overflow:hidden" width="0" height="0"><img src="http://t.yesware.com/t/d1fcbaa1b12e0f6b1beef0b50d5ebbd873d1b8f9/e8040b69bc7ca00f2bce0335cfcc6ef8/spacer.gif" style="border:0px;width:0px;min-height:0px;overflow:hidden" width="0" height="0"><font face="yw-d1fcbaa1b12e0f6b1beef0b50d5ebbd873d1b8f9-e8040b69bc7ca00f2bce0335cfcc6ef8--to"></font></div>
</blockquote></div></div></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</font></span></div></div>
</blockquote></div><br></div>