<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="ltr">
<div>*Better to have an abstract for readers to know your intention/conclusion</div>
<div><br>
</div>
*p.5 "We also launch all jobs using the --launch_distribution cyclic option so that MPI ranks are assigned to resource sets in a circular fashion, which we deem appropriate for most high performance computing (HPC) algorithms."
<div>Cyclic distribution is fine for these simple Vec ops since there is almost no communication, but can not be deemed appropriate for most HPC algorithms. I assume packed distribution is better for locality. <br>
</div>
<div><br>
</div>
<div>*Fig. 1 Left. I would use the diagram at p.11 of <a href="https://press3.mcs.anl.gov/atpesc/files/2018/08/ATPESC_2018_Track-1_6_7-30_130pm_Hill-Summit_at_ORNL.pdf">https://press3.mcs.anl.gov/atpesc/files/2018/08/ATPESC_2018_Track-1_6_7-30_130pm_Hill-Summit_at_ORNL.pdf</a>,
which is more informative and contains a lot of numbers we can compare with your results. E.g., peak bandwidth, you mentioned but did not list.</div>
<div><br>
</div>
<div>*2.1 cudaMemcopy ?<br>
</div>
<div> For the two bullets VecAXPY, VecDot, you'd better clearly list how you counted their FLOPS & memory, which you used to calculate bandwidth and performance in the report.</div>
<div><br>
</div>
<div>*p.12 VecACPY ?<br>
</div>
<div>*p.12 I don't the difference of the two GPU launch time.</div>
<div><br>
</div>
<div>*When appropriate, can you draw a line for hardware peak bandwidth or FLOPS/s in the figures.</div>
<div><br>
</div>
<div>*p.13, some bullets are not important and you can mention them earlier in your experimental setup.</div>
<div>bullet 4: I think the reason is: to get peak CPU->GPU bandwidth, the cpu buffer has to be pinned (i.e. non-pageable).</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr" data-smartmail="gmail_signature">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Oct 9, 2019 at 5:34 PM Smith, Barry F. via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div><font size="2"><span style="font-size:11pt">
<div><br>
We've prepared a short report on the performance of vector operations on Summit and would appreciate any feed back including: inconsistencies, lack of clarity, incorrect notation or terminology, etc.<br>
<br>
Thanks<br>
<br>
Barry, Hannah, and Richard<br>
<br>
<br>
<br>
<br>
</div>
</span></font></div>
<div><font size="2"><span style="font-size:11pt">
<div><br>
</div>
</span></font></div>
</div>
</blockquote>
</div>
</body>
</html>