<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<p>Hi Shri,</p>
<p>Probably the best way to understand what is going on is to step
through things using a debugger, as Junchao suggests. VecAXPY does
get used in a lot of places, and maybe it is being called on some
vectors that aren't getting their type from the options database?
Also, there are several places where a vector gets "bound" to
execute operations on the CPU instead of the GPU (see
VecBindToCPU()) either because we know that the vector isn't going
to be needed on the CPU for subsequent operations, or because the
size of the vector is too small for it to make sense to do on the
GPU because of kernel launch latency. When a vector is bound to
the CPU, operations with it will be counted in the CPU MFlops
column.</p>
<p>It looks like you are actually getting decent GPU usage for your
vector operations. While VecAXPY is showing only 80% of operations
on the GPU, it's also accounting for less than one percent of the
total flops. I see 100% GPU flops for the VecMAXPY that accounts
for 13% of your flops.</p>
<p>Best regards,<br>
Richard<br>
</p>
<div class="moz-cite-prefix">On 3/10/20 3:44 PM, Junchao Zhang via
petsc-dev wrote:<br>
</div>
<blockquote type="cite" cite="mid:CA+MQGp9UeVdwFoMSyA_wqi9HJfiSb+A8pmxR_gbpT2yKC4EDcg@mail.gmail.com">
<div dir="ltr">Hi, Shri,
<div> I don't understand either. But there are many invocations
of VecAXPY etc. Is it possible some are done on CPU? Attach a
debugger and set a breakpoint on VecAXPY_SeqCUDA to see if it
gets a hit. If yes, then see why.
<div><br clear="all">
<div>
<div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Mar 10, 2020 at 2:44
PM Abhyankar, Shrirang G via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" moz-do-not-send="true">petsc-dev@mcs.anl.gov</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_6553524128374611606WordSection1">
<p class="MsoNormal"><span style="font-family:"Times
New Roman",serif">Hello all,</span></p>
<p class="MsoNormal"><span style="font-family:"Times
New Roman",serif"> I need help in understanding
the output from -log_summary for the GPU related
columns. I am currently simply setting -vec_type
seqcuda which I believe performs the vector operations
on the GPU. With -vec_type seqcuda, I presumed all
vector operations are being done on the GPU. So, only
the GPU MFlops will be logged, and CPU MFlops will be
zero. But, -log_summary reports Mflops for both CPU
and GPU. I do not understand why Mflops are shown both
for CPU and GPU? </span></p>
<p class="MsoNormal"><span style="font-family:"Times
New Roman",serif"> </span></p>
<p class="MsoNormal"><span style="font-family:"Times
New Roman",serif">What is the meaning of the last
column – percent flops on the GPU? For instance, some
operations such as VecDot show 100 %F, while others
like VecAXPY have less. What is the meaning of this?</span></p>
<p class="MsoNormal"><span style="font-family:"Times
New Roman",serif"> </span></p>
<p class="MsoNormal"><span style="font-family:"Times
New Roman",serif">Any other general comments on
these numbers?</span></p>
<p class="MsoNormal"><span style="font-family:"Times
New Roman",serif"> </span></p>
<p class="MsoNormal"><span style="font-family:"Times
New Roman",serif">Let me know if you need more
information.</span></p>
<p class="MsoNormal"><span style="font-family:"Times
New Roman",serif"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:-webkit-standard;color:black">Thanks,</span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:-webkit-standard;color:black">Shri</span></p>
<p class="MsoNormal"> </p>
</div>
</div>
</blockquote>
</div>
</blockquote>
</body>
</html>