On Tue, Aug 31, 2010 at 7:17 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div style="word-wrap:break-word"><br><div><div class="im"><div>On Aug 31, 2010, at 3:14 PM, Keita Teranishi wrote:</div><br><blockquote type="cite"><span style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:medium"><div lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">Does this PETSc use timers from CUDA?</span></div>
</div></div></span></blockquote><div><br></div></div> No, didn't know there were timers in CUDA. </div></div></blockquote><div><br></div><div>Yes, I use them when I really want to know how well I an utilizing the board, vs. how</div>
<div>much improvement overall I can expect in the code. When compared with PETSc timers,</div><div>they can give us an idea of the transfer overhead, which I do in my GPU FEM code.</div><div><br></div><div> Matt</div><div>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div style="word-wrap:break-word"><div> We actually want to use the real world timers because each method is actually a call on the CPU so real world time is what matters.</div>
<div><br></div><font color="#888888"><div> Barry</div></font><div><div></div><div class="h5"><div><br><blockquote type="cite"><span style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:medium"><div lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)"></span></div>
<div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)"> </span></div>
<div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:10pt;font-family:Arial, sans-serif;color:rgb(31, 73, 125)">================================<br>
Keita Teranishi<br> Scientific Library Group<br> Cray, Inc.<br> <a href="mailto:keita@cray.com" style="color:blue;text-decoration:underline" target="_blank">keita@cray.com</a><br>================================</span><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)"></span></div>
</div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)"> </span></div>
<div><div style="border-right-style:none;border-bottom-style:none;border-left-style:none;border-width:initial;border-color:initial;border-top-style:solid;border-top-color:rgb(181, 196, 223);border-top-width:1pt;padding-top:3pt;padding-right:0in;padding-bottom:0in;padding-left:0in">
<div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><b><span style="font-size:10pt;font-family:Tahoma, sans-serif">From:</span></b><span style="font-size:10pt;font-family:Tahoma, sans-serif"><span> </span><a href="mailto:petsc-dev-bounces@mcs.anl.gov" style="color:blue;text-decoration:underline" target="_blank">petsc-dev-bounces@mcs.anl.gov</a><span> </span>[mailto:<a href="mailto:petsc-dev-bounces@mcs.anl.gov" target="_blank">petsc-dev-bounces@mcs.anl.gov</a>]<span> </span><b>On Behalf Of<span> </span></b>Barry Smith<br>
<b>Sent:</b><span> </span>Tuesday, August 31, 2010 2:03 PM<br><b>To:</b><span> </span>For users of the development version of PETSc<br><b>Subject:</b><span> </span>Re: [petsc-dev] [GPU] Performance of ex19</span></div></div>
</div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"> </div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif">
</div></div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"> Your MatMult is now slower. Are your results reproducible, if you run 5 times how similar are them?</div>
<div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"> </div></div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif">
Barry</div></div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"> </div><div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif">
On Aug 31, 2010, at 2:57 PM, Keita Teranishi wrote:</div></div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><br><br></div>
<div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">VecDot 2 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">VecMDot 2024 1.0 1.1560e+00 1.0 2.54e+09 1.0 0.0e+00 0.0e+00 0.0e+00 18 29 0 0 0 32 29 0 0 0 2201</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">VecNorm 2096 1.0 3.5999e-01 1.0 1.68e+08 1.0 0.0e+00 0.0e+00 0.0e+00 6 2 0 0 0 10 2 0 0 0 466</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">VecScale 2092 1.0 2.1599e-01 1.0 8.37e+07 1.0 0.0e+00 0.0e+00 0.0e+00 3 1 0 0 0 6 1 0 0 0 387</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">VecCopy 2072 1.0 5.5997e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 2 0 0 0 0 0</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">VecSet 70 1.0 8.0004e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">VecAXPY 108 1.0 2.7999e-02 1.0 8.64e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 309</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">VecWAXPY 68 1.0 7.9999e-03 1.0 2.72e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 340</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">VecMAXPY 2092 1.0 5.8399e-01 1.0 2.71e+09 1.0 0.0e+00 0.0e+00 0.0e+00 9 31 0 0 0 16 31 0 0 0 4634</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">VecScatterBegin 5 1.0 4.0002e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">VecReduceArith 2 1.0 3.9999e-03 1.0 1.60e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 40</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">VecReduceComm 1 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">VecCUDACopyTo 10 1.0 3.9999e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">VecCUDACopyFrom 5 1.0 4.0002e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">SNESSolve 1 1.0 3.6119e+00 1.0 8.87e+09 1.0 0.0e+00 0.0e+00 0.0e+00 56100 0 0 0 100100 0 0 0 2456</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">SNESLineSearch 2 1.0 4.0002e-03 1.0 5.49e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1374</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">SNESFunctionEval 3 1.0 4.0002e-03 1.0 2.52e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 630</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">SNESJacobianEval 2 1.0 3.1199e-01 1.0 3.85e+07 1.0 0.0e+00 0.0e+00 0.0e+00 5 0 0 0 0 9 0 0 0 0 123</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">KSPGMRESOrthog 2024 1.0 1.7120e+00 1.0 5.09e+09 1.0 0.0e+00 0.0e+00 0.0e+00 26 57 0 0 0 47 57 0 0 0 2972</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">KSPSetup 2 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">KSPSolve 2 1.0 3.2919e+00 1.0 8.83e+09 1.0 0.0e+00 0.0e+00 0.0e+00 51 99 0 0 0 91 99 0 0 0 2681</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">PCSetUp 2 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">PCApply 2024 1.0 4.7998e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">MatMult 2092 1.0 8.9998e-01 1.0 3.32e+09 1.0 0.0e+00 0.0e+00 0.0e+00 14 37 0 0 0 25 37 0 0 0 3689</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">MatAssemblyBegin 2 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">MatAssemblyEnd 2 1.0 1.2000e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">MatZeroEntries 2 1.0 0.0000e+00 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">MatFDColorApply 2 1.0 3.1199e-01 1.0 3.85e+07 1.0 0.0e+00 0.0e+00 0.0e+00 5 0 0 0 0 9 0 0 0 0 123</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:11pt;font-family:Calibri, sans-serif;color:rgb(31, 73, 125)">MatFDColorFunc 42 1.0 7.9999e-03 1.0 3.53e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4410</span></div>
</div><div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"><span style="font-size:10pt;font-family:Arial, sans-serif;color:rgb(31, 73, 125)"> </span></div>
</div></div><div style="margin-top:0in;margin-right:0in;margin-bottom:0.0001pt;margin-left:0in;font-size:12pt;font-family:'Times New Roman', serif"> </div></div></div></div></span></blockquote></div><br></div></div>
</div></blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<br>