<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Times New Roman \,serif";
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:-webkit-standard;
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Times New Roman",serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">Thank you, Richard and Junchao. This is very helpful info. I’ll try to step through using debugger as you’ve suggested.
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Shri<o:p></o:p></p>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">petsc-dev <petsc-dev-bounces@mcs.anl.gov> on behalf of PETSc Development <petsc-dev@mcs.anl.gov><br>
<b>Reply-To: </b>Richard Tran Mills <rtmills@anl.gov><br>
<b>Date: </b>Tuesday, March 10, 2020 at 11:34 PM<br>
<b>To: </b>PETSc Development <petsc-dev@mcs.anl.gov><br>
<b>Subject: </b>Re: [petsc-dev] Understanding -log_summary with GPUs<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<p style="margin-left:.5in">Hi Shri,<o:p></o:p></p>
<p style="margin-left:.5in">Probably the best way to understand what is going on is to step through things using a debugger, as Junchao suggests. VecAXPY does get used in a lot of places, and maybe it is being called on some vectors that aren't getting their
type from the options database? Also, there are several places where a vector gets "bound" to execute operations on the CPU instead of the GPU (see VecBindToCPU()) either because we know that the vector isn't going to be needed on the CPU for subsequent operations,
or because the size of the vector is too small for it to make sense to do on the GPU because of kernel launch latency. When a vector is bound to the CPU, operations with it will be counted in the CPU MFlops column.<o:p></o:p></p>
<p style="margin-left:.5in">It looks like you are actually getting decent GPU usage for your vector operations. While VecAXPY is showing only 80% of operations on the GPU, it's also accounting for less than one percent of the total flops. I see 100% GPU flops
for the VecMAXPY that accounts for 13% of your flops.<o:p></o:p></p>
<p style="margin-left:.5in">Best regards,<br>
Richard<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">On 3/10/20 3:44 PM, Junchao Zhang via petsc-dev wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="margin-left:.5in">Hi, Shri, <o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"> I don't understand either. But there are many invocations of VecAXPY etc. Is it possible some are done on CPU? Attach a debugger and set a breakpoint on VecAXPY_SeqCUDA to see if it gets a hit. If yes, then see
why. <o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><br clear="all">
<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">--Junchao Zhang<o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">On Tue, Mar 10, 2020 at 2:44 PM Abhyankar, Shrirang G via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov">petsc-dev@mcs.anl.gov</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:"Times New Roman ,serif",serif">Hello all,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:"Times New Roman ,serif",serif"> I need help in understanding the output from -log_summary for the GPU related columns. I am currently simply setting -vec_type seqcuda which I believe performs the vector operations on the GPU. With
-vec_type seqcuda, I presumed all vector operations are being done on the GPU. So, only the GPU MFlops will be logged, and CPU MFlops will be zero. But, -log_summary reports Mflops for both CPU and GPU. I do not understand why Mflops are shown both for CPU
and GPU? </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:"Times New Roman ,serif",serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:"Times New Roman ,serif",serif">What is the meaning of the last column – percent flops on the GPU? For instance, some operations such as VecDot show 100 %F, while others like VecAXPY have less. What is the meaning of this?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:"Times New Roman ,serif",serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:"Times New Roman ,serif",serif">Any other general comments on these numbers?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:"Times New Roman ,serif",serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:"Times New Roman ,serif",serif">Let me know if you need more information.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:"Times New Roman ,serif",serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:"-webkit-standard",serif;color:black">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-family:"-webkit-standard",serif;color:black">Shri</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</body>
</html>