<div dir="ltr">Hello Junchao,<div><br></div><div>Spectacular news!</div><div><br></div><div>I have our production code running on Summit (Power9 + Nvidia V100) and on local x86 workstations, and I can definitely provide comparative benchmark data with this feature once it is ready. Just let me know when it is available for testing and I'll be happy to contribute. </div><div><br></div><div>Thanks,</div><div><br></div><div>-David</div></div><div id="DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2"><br>
<table style="border-top:1px solid #d3d4de">
<tr>
<td style="width:55px;padding-top:13px"><a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon" target="_blank"><img src="https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif" alt="" width="46" height="29" style="width: 46px; height: 29px;"></a></td>
<td style="width:470px;padding-top:12px;color:#41424e;font-size:13px;font-family:Arial,Helvetica,sans-serif;line-height:18px">Virus-free. <a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link" target="_blank" style="color:#4453ea">www.avast.com</a>
</td>
</tr>
</table><a href="#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2" width="1" height="1"></a></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 22, 2019 at 7:22 AM Zhang, Junchao <<a href="mailto:jczhang@mcs.anl.gov">jczhang@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">This feature is under active development. I hope I can make it usable in a couple of weeks. Thanks.
<div>
<div>
<div>
<div dir="ltr" class="gmail-m_2897511617293957267gmail_signature">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Aug 21, 2019 at 3:21 PM David Gutzwiller via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hello,
<div><br>
</div>
<div>I'm currently using PETSc for the GPU acceleration of simple Krylov solver with GMRES, without preconditioning. This is within the framework of our in-house multigrid solver. I am getting a good GPU speedup on the finest grid level but progressively
worse performance on each coarse level. This is not surprising, but I still hope to squeeze out some more performance, hopefully making it worthwhile to run some or all of the coarse grids on the GPU. </div>
<div> </div>
<div>I started investigating with nvprof / nsight and essentially came to the same conclusion that Xiangdong reported in a recent thread (July 16, "MemCpy (HtoD and DtoH) in Krylov solver"). My question is a follow-up to that thread: </div>
<div><br>
</div>
<div>The MPI communication is staged from the host, which results in some H<->D transfers for every mat-vec operation. A CUDA-aware MPI implementation might avoid these transfers for communication between ranks that are assigned to the same accelerator.
Has this been implemented or tested?<br>
</div>
<div><br>
</div>
<div>In our solver we typically run with multiple MPI ranks all assigned to a single device, and running with a single rank is not really feasible as we still have a sizable amount of work for the CPU to chew through. Thus, I think quite a lot of the H<->D
transfers could be avoided if I can skip the MPI staging on the host. I am quite new to PETSc so I wanted to ask around before blindly digging into this. </div>
<div><br>
</div>
<div>Thanks for your help,</div>
<div><br>
</div>
<div>David</div>
<div id="gmail-m_2897511617293957267gmail-m_5808030803546790052m_-3093947404852640465DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2">
<br>
<table style="border-top:1px solid rgb(211,212,222)">
<tbody>
<tr>
<td style="width:55px;padding-top:13px"><a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon" target="_blank"><img src="https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif" alt="" width="46" height="29" style="width: 46px; height: 29px;"></a></td>
<td style="width:470px;padding-top:12px;color:rgb(65,66,78);font-size:13px;font-family:Arial,Helvetica,sans-serif;line-height:18px">
Virus-free. <a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link" style="color:rgb(68,83,234)" target="_blank">
www.avast.com</a> </td>
</tr>
</tbody>
</table>
<a href="#m_2897511617293957267_m_5808030803546790052_m_-3093947404852640465_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2" width="1" height="1"></a></div>
</div>
</blockquote>
</div>
</div>
</blockquote></div>