<div dir="ltr">Looks something was wrong that you need all cores to reach its bandwidth limit. Run on one node to see what happens.<br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 9, 2020 at 3:45 PM Fande Kong <<a href="mailto:fdkong.jd@gmail.com">fdkong.jd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi All,<br><div><br></div><div><div>I am trying to interpret the results from "make stream" on two compute nodes, where each node has 48 cores. </div><div><br></div><div>If my calculations are memory bandwidth limited, such as AMG, MatVec, GMRES, etc.. </div><div>The best speedup I could get is 16.6938 if I start from one core?? The speedup for function evaluations and Jacobian evaluations can be better than16.6938?</div></div><div><br></div><div>Thanks,</div><div><br></div><div>Fande,</div><div><br></div><div><br></div><div><br></div><div><div>Running streams with 'mpiexec ' using 'NPMAX=96' </div><div>1 19412.4570 Rate (MB/s)</div><div>2 29457.3988 Rate (MB/s) 1.51744 </div><div>3 40483.9318 Rate (MB/s) 2.08546 </div><div>4 51429.3431 Rate (MB/s) 2.64929 </div><div>5 59849.5168 Rate (MB/s) 3.08304 </div><div>6 66124.3461 Rate (MB/s) 3.40628 </div><div>7 70888.1170 Rate (MB/s) 3.65167 </div><div>8 73436.2374 Rate (MB/s) 3.78294 </div><div>9 77441.7622 Rate (MB/s) 3.98927 </div><div>10 78115.3114 Rate (MB/s) 4.02397 </div><div>11 81449.3315 Rate (MB/s) 4.19572 </div><div>12 82812.3471 Rate (MB/s) 4.26593 </div><div>13 81442.2114 Rate (MB/s) 4.19535 </div><div>14 83404.1657 Rate (MB/s) 4.29642 </div><div>15 84165.8536 Rate (MB/s) 4.33565 </div><div>16 83739.2910 Rate (MB/s) 4.31368 </div><div>17 83724.8109 Rate (MB/s) 4.31293 </div><div>18 83225.0743 Rate (MB/s) 4.28719 </div><div>19 81668.2002 Rate (MB/s) 4.20699 </div><div>20 83678.8007 Rate (MB/s) 4.31056 </div><div>21 81400.4590 Rate (MB/s) 4.1932 </div><div>22 81944.8975 Rate (MB/s) 4.22124 </div><div>23 81359.8615 Rate (MB/s) 4.19111 </div><div>24 80674.5064 Rate (MB/s) 4.1558 </div><div>25 83761.3316 Rate (MB/s) 4.31481 </div><div>26 87567.4876 Rate (MB/s) 4.51088 </div><div>27 89605.4435 Rate (MB/s) 4.61586 </div><div>28 94984.9755 Rate (MB/s) 4.89298 </div><div>29 98260.5283 Rate (MB/s) 5.06171 </div><div>30 99852.8790 Rate (MB/s) 5.14374 </div><div>31 102736.3576 Rate (MB/s) 5.29228 </div><div>32 108638.7488 Rate (MB/s) 5.59633 </div><div>33 110431.2938 Rate (MB/s) 5.68867 </div><div>34 112824.2031 Rate (MB/s) 5.81194 </div><div>35 116908.3009 Rate (MB/s) 6.02232 </div><div>36 121312.6574 Rate (MB/s) 6.2492 </div><div>37 122507.3172 Rate (MB/s) 6.31074 </div><div>38 127456.2504 Rate (MB/s) 6.56568 </div><div>39 130098.7056 Rate (MB/s) 6.7018 </div><div>40 134956.4461 Rate (MB/s) 6.95204 </div><div>41 138309.2465 Rate (MB/s) 7.12475 </div><div>42 141779.7997 Rate (MB/s) 7.30353 </div><div>43 145653.3687 Rate (MB/s) 7.50307 </div><div>44 149131.2087 Rate (MB/s) 7.68223 </div><div>45 151611.6104 Rate (MB/s) 7.81 </div><div>46 155554.6394 Rate (MB/s) 8.01312 </div><div>47 159033.1938 Rate (MB/s) 8.19231 </div><div>48 162216.5600 Rate (MB/s) 8.35629 </div><div>49 165034.8116 Rate (MB/s) 8.50147 </div><div>50 168001.4823 Rate (MB/s) 8.65429 </div><div>51 170899.9045 Rate (MB/s) 8.8036 </div><div>52 175687.8033 Rate (MB/s) 9.05024 </div><div>53 178203.9203 Rate (MB/s) 9.17985 </div><div>54 179973.3914 Rate (MB/s) 9.27101 </div><div>55 182207.3495 Rate (MB/s) 9.38608 </div><div>56 185712.9643 Rate (MB/s) 9.56667 </div><div>57 188805.5696 Rate (MB/s) 9.72598 </div><div>58 193360.9158 Rate (MB/s) 9.96064 </div><div>59 198160.8016 Rate (MB/s) 10.2079 </div><div>60 201297.0129 Rate (MB/s) 10.3695 </div><div>61 203618.7672 Rate (MB/s) 10.4891 </div><div>62 209599.2783 Rate (MB/s) 10.7971 </div><div>63 211651.1587 Rate (MB/s) 10.9028 </div><div>64 210254.5035 Rate (MB/s) 10.8309 </div><div>65 218576.4938 Rate (MB/s) 11.2596 </div><div>66 220280.0853 Rate (MB/s) 11.3473 </div><div>67 221281.1867 Rate (MB/s) 11.3989 </div><div>68 228941.1872 Rate (MB/s) 11.7935 </div><div>69 232206.2708 Rate (MB/s) 11.9617 </div><div>70 233569.5866 Rate (MB/s) 12.0319 </div><div>71 238293.6355 Rate (MB/s) 12.2753 </div><div>72 238987.0729 Rate (MB/s) 12.311 </div><div>73 246013.4684 Rate (MB/s) 12.6729 </div><div>74 248850.8942 Rate (MB/s) 12.8191 </div><div>75 249355.6899 Rate (MB/s) 12.8451 </div><div>76 252515.6110 Rate (MB/s) 13.0079 </div><div>77 257489.4268 Rate (MB/s) 13.2641 </div><div>78 260884.2771 Rate (MB/s) 13.439 </div><div>79 264341.8661 Rate (MB/s) 13.6171 </div><div>80 269329.1376 Rate (MB/s) 13.874 </div><div>81 272286.4070 Rate (MB/s) 14.0263 </div><div>82 273325.7822 Rate (MB/s) 14.0799 </div><div>83 277334.6699 Rate (MB/s) 14.2864 </div><div>84 280254.7286 Rate (MB/s) 14.4368 </div><div>85 282219.8194 Rate (MB/s) 14.538 </div><div>86 289039.2677 Rate (MB/s) 14.8893 </div><div>87 291234.4715 Rate (MB/s) 15.0024 </div><div>88 295941.1159 Rate (MB/s) 15.2449 </div><div>89 298136.3163 Rate (MB/s) 15.358 </div><div>90 302820.9080 Rate (MB/s) 15.5993 </div><div>91 306387.5008 Rate (MB/s) 15.783 </div><div>92 310127.0223 Rate (MB/s) 15.9756 </div><div>93 310219.3643 Rate (MB/s) 15.9804 </div><div>94 317089.5971 Rate (MB/s) 16.3343 </div><div>95 315457.0938 Rate (MB/s) 16.2502 </div><div>96 324068.8172 Rate (MB/s) 16.6938 </div></div></div></div></div></div>
</blockquote></div>