<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 5, 2016 at 12:18 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="">On Fri, Feb 5, 2016 at 11:12 AM, Xiangdong <span dir="ltr"><<a href="mailto:epscodes@gmail.com" target="_blank">epscodes@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hello everyone,<div><br></div><div>When I looked at the log_summary, I found that flops ratio of VecDot is 1.1, but the time ratio is 1.8. Also the flops ratio of MatMult is 1.2, but the time ratio is 1.6. I am using the simple SOR preconditioner on 256 procs. Since each of them consumes about 20% of total time, is it worth looking into these imbalance issue? If yes, how do I know which procs are causing the issue? What tools should I start with?</div></div></blockquote><div><br></div></span><div>1) Always send all the output</div><div><br></div><div>2) You have a large number of dot products, and I am guessing you have a slow reduce on this machine (we could see the data at the end of log_summary).</div><div> The waiting time comes from slow collective communication, load imbalance, system noise, etc. You can see this by looking at the difference between</div><div> VecDot() and VecMDot(). For the latter there is enough work to cover up the latency, and its imbalance is only 1.1, not 1.8. Thus I think it is clear that your</div><div> problem is due to a slow communication network rather than load imbalance.</div></div></div></div></blockquote><div><br></div><div>I attached the full summary. At the end, it has </div><div><br></div><div><div>Average time to get PetscTime(): 0</div><div>Average time for MPI_Barrier(): 8.3971e-05</div><div>Average time for zero size MPI_Send(): 7.16746e-06</div></div><div><br></div><div>Is it an indication of slow network?</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>3) Thus you would be better served with a stronger preconditioner (more flops) which resulted in fewer iterations (less synchronization).</div></div></div></div></blockquote><div><br></div><div>Yes, the preconditioner I am using is simple (just sor). I am also trying other advanced preconditioners. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>Thanks.</div><div><br></div><div>Xiangdong</div><div><br></div><div>------------------------------------------------------------------------------------------------------------------------</div><div>Event Count Time (sec) Flops --- Global --- --- Stage --- Total</div><div> Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s</div><div>------------------------------------------------------------------------------------------------------------------------</div><div><br></div><div>--- Event Stage 0: Main Stage</div><div><br></div><div>VecView 44 1.0 3.9026e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.8e+01 1 0 0 0 0 1 0 0 0 0 0</div><div>VecDot 817452 1.0 1.5972e+02 1.8 1.51e+10 1.1 0.0e+00 0.0e+00 8.2e+05 18 3 0 0 40 18 3 0 0 40 22969</div><div>VecMDot 613089 1.0 5.8918e+01 1.1 2.27e+10 1.1 0.0e+00 0.0e+00 6.1e+05 8 4 0 0 30 8 4 0 0 30 93397</div><div>VecNorm 618619 1.0 1.4426e+02 2.2 1.14e+10 1.1 0.0e+00 0.0e+00 6.2e+05 15 2 0 0 30 15 2 0 0 30 19145</div><div>VecScale 938 1.0 8.3907e-03 1.2 8.67e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 250859</div><div>VecCopy 8159 1.0 1.0394e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>VecSet 1884 1.0 3.9701e-02 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>VecAXPY 1021815 1.0 2.2148e+01 2.1 1.89e+10 1.1 0.0e+00 0.0e+00 0.0e+00 2 4 0 0 0 2 4 0 0 0 207057</div><div>VecAYPX 613089 1.0 6.5360e+00 1.9 1.13e+10 1.1 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 420336</div><div>VecAXPBYCZ 191 1.0 3.7365e-03 1.4 7.06e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 458830</div><div>VecWAXPY 938 1.0 3.8502e-02 1.4 8.67e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 54669</div><div>VecMAXPY 613089 1.0 1.3276e+01 2.2 2.27e+10 1.1 0.0e+00 0.0e+00 0.0e+00 1 4 0 0 0 1 4 0 0 0 414499</div><div>VecLoad 5 1.0 2.0913e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>VecScatterBegin 820695 1.0 1.8959e+01 2.8 0.00e+00 0.0 1.1e+09 4.6e+03 0.0e+00 2 0100100 0 2 0100100 0 0</div><div>VecScatterEnd 820695 1.0 5.2002e+01 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 4 0 0 0 0 0</div><div>VecReduceArith 2814 1.0 1.8517e-02 1.2 5.20e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 681995</div><div>VecReduceComm 938 1.0 2.6041e+00 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 9.4e+02 0 0 0 0 0 0 0 0 0 0 0</div><div>MatMult 817452 1.0 1.9379e+02 1.6 2.04e+11 1.2 1.0e+09 4.6e+03 0.0e+00 23 40100100 0 23 40100100 0 253083</div><div>MatSOR 818390 1.0 1.9608e+02 1.5 2.00e+11 1.1 0.0e+00 0.0e+00 0.0e+00 22 40 0 0 0 22 40 0 0 0 247472</div><div>MatAssemblyBegin 939 1.0 7.9671e+00 6.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.9e+03 1 0 0 0 0 1 0 0 0 0 0</div><div>MatAssemblyEnd 939 1.0 2.0570e-01 1.2 0.00e+00 0.0 2.6e+03 1.2e+03 8.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>MatZeroEntries 938 1.0 1.8202e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>SNESSolve 210 1.0 6.8585e+02 1.0 5.06e+11 1.2 1.1e+09 4.6e+03 2.1e+06 98100100100100 98100100100100 178411</div><div>SNESFunctionEval 2296 1.0 2.0512e+01 1.3 0.00e+00 0.0 2.9e+06 3.5e+03 0.0e+00 3 0 0 0 0 3 0 0 0 0 0</div><div>SNESJacobianEval 938 1.0 2.7680e+01 1.0 0.00e+00 0.0 1.2e+06 3.5e+03 1.9e+03 4 0 0 0 0 4 0 0 0 0 0</div><div>SNESLineSearch 938 1.0 9.2491e+00 1.0 6.07e+07 1.1 1.2e+06 3.5e+03 9.4e+02 1 0 0 0 0 1 0 0 0 0 1593</div><div>KSPSetUp 938 1.0 2.7661e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+01 0 0 0 0 0 0 0 0 0 0 0</div><div>KSPSolve 938 1.0 6.3433e+02 1.0 5.06e+11 1.2 1.0e+09 4.6e+03 2.0e+06 91100100100 99 91100100100 99 192852</div><div>PCSetUp 938 1.0 2.1005e-04 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>PCApply 818390 1.0 1.9675e+02 1.5 2.00e+11 1.1 0.0e+00 0.0e+00 0.0e+00 22 40 0 0 0 22 40 0 0 0 246626</div><div>------------------------------------------------------------------------------------------------------------------------</div></div>
</blockquote></div></div></div><span class=""><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</font></span></div></div>
</blockquote></div><br></div></div>