<div dir="ltr">Hi Junchao,<div><br></div><div>Thanks for letting me know. </div><div><br></div><div>I'm currently in a bit of a crunch for an upcoming product release, but once I have a few days to refocus on this task I'll test the latest master and let you know how it performs.</div><div><br></div><div>-David</div></div><div id="DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2"><br>
<table style="border-top:1px solid #d3d4de">
<tr>
<td style="width:55px;padding-top:13px"><a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon" target="_blank"><img src="https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif" alt="" width="46" height="29" style="width: 46px; height: 29px;"></a></td>
<td style="width:470px;padding-top:12px;color:#41424e;font-size:13px;font-family:Arial,Helvetica,sans-serif;line-height:18px">Virus-free. <a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link" target="_blank" style="color:#4453ea">www.avast.com</a>
</td>
</tr>
</table><a href="#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2" width="1" height="1"></a></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Oct 7, 2019 at 3:09 PM Zhang, Junchao <<a href="mailto:jczhang@mcs.anl.gov">jczhang@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">Hello, David,
<div> It took a longer time than I expected to add the CUDA-aware MPI feature in PETSc. It is now in PETSc-3.12, released last week. I have a little fix after that, so you better use petsc master. Use petsc option -use_gpu_aware_mpi to enable it. On Summit,
you also need jsrun --smpiargs="-gpu" to enable IBM Spectrum MPI's CUDA support. If you run with multiple MPI ranks per GPU, you also need #BSUB -alloc_flags gpumps in your job script.</div>
<div> My experiments (using a simple test doing repeated MatMult) on Summit is mixed. With one MPI rank per GPU, I saw very good performance improvement (up to 25%). But with multiple ranks per GPU, I did not see improvement. That sounds absurd since it should
be easier for MPI ranks communicate data on the same GPU. I'm investigating this issue.</div>
<div> If you can also evaluate this feature with your production code, that would be helpful.</div>
<div> Thanks.</div>
<div>
<div>
<div dir="ltr">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Aug 22, 2019 at 11:34 AM David Gutzwiller <<a href="mailto:david.gutzwiller@gmail.com" target="_blank">david.gutzwiller@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hello Junchao,
<div><br>
</div>
<div>Spectacular news!</div>
<div><br>
</div>
<div>I have our production code running on Summit (Power9 + Nvidia V100) and on local x86 workstations, and I can definitely provide comparative benchmark data with this feature once it is ready. Just let me know when it is available for testing and I'll be
happy to contribute. </div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>-David</div>
</div>
<div id="gmail-m_-4091424993276853060m_5253247854773161582m_1798056796989138033m_-4452134796509464077gmail-m_1768252039860486047DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2">
<br>
<table style="border-top:1px solid rgb(211,212,222)">
<tbody>
<tr>
<td style="width:55px;padding-top:13px"><a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon" target="_blank"><img src="https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif" alt="" width="46" height="29" style="width: 46px; height: 29px;"></a></td>
<td style="width:470px;padding-top:12px;color:rgb(65,66,78);font-size:13px;font-family:Arial,Helvetica,sans-serif;line-height:18px">
Virus-free. <a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link" style="color:rgb(68,83,234)" target="_blank">
www.avast.com</a> </td>
</tr>
</tbody>
</table>
<a href="#m_-4091424993276853060_m_5253247854773161582_m_1798056796989138033_m_-4452134796509464077_m_1768252039860486047_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2" width="1" height="1"></a></div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Aug 22, 2019 at 7:22 AM Zhang, Junchao <<a href="mailto:jczhang@mcs.anl.gov" target="_blank">jczhang@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">This feature is under active development. I hope I can make it usable in a couple of weeks. Thanks.
<div>
<div>
<div>
<div dir="ltr">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Aug 21, 2019 at 3:21 PM David Gutzwiller via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hello,
<div><br>
</div>
<div>I'm currently using PETSc for the GPU acceleration of simple Krylov solver with GMRES, without preconditioning. This is within the framework of our in-house multigrid solver. I am getting a good GPU speedup on the finest grid level but progressively
worse performance on each coarse level. This is not surprising, but I still hope to squeeze out some more performance, hopefully making it worthwhile to run some or all of the coarse grids on the GPU. </div>
<div> </div>
<div>I started investigating with nvprof / nsight and essentially came to the same conclusion that Xiangdong reported in a recent thread (July 16, "MemCpy (HtoD and DtoH) in Krylov solver"). My question is a follow-up to that thread: </div>
<div><br>
</div>
<div>The MPI communication is staged from the host, which results in some H<->D transfers for every mat-vec operation. A CUDA-aware MPI implementation might avoid these transfers for communication between ranks that are assigned to the same accelerator.
Has this been implemented or tested?<br>
</div>
<div><br>
</div>
<div>In our solver we typically run with multiple MPI ranks all assigned to a single device, and running with a single rank is not really feasible as we still have a sizable amount of work for the CPU to chew through. Thus, I think quite a lot of the H<->D
transfers could be avoided if I can skip the MPI staging on the host. I am quite new to PETSc so I wanted to ask around before blindly digging into this. </div>
<div><br>
</div>
<div>Thanks for your help,</div>
<div><br>
</div>
<div>David</div>
<div id="gmail-m_-4091424993276853060m_5253247854773161582m_1798056796989138033m_-4452134796509464077gmail-m_1768252039860486047gmail-m_2897511617293957267gmail-m_5808030803546790052m_-3093947404852640465DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2">
<br>
<table style="border-top:1px solid rgb(211,212,222)">
<tbody>
<tr>
<td style="width:55px;padding-top:13px"><a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon" target="_blank"><img src="https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif" alt="" width="46" height="29" style="width: 46px; height: 29px;"></a></td>
<td style="width:470px;padding-top:12px;color:rgb(65,66,78);font-size:13px;font-family:Arial,Helvetica,sans-serif;line-height:18px">
Virus-free. <a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link" style="color:rgb(68,83,234)" target="_blank">
www.avast.com</a> </td>
</tr>
</tbody>
</table>
<a href="#m_-4091424993276853060_m_5253247854773161582_m_1798056796989138033_m_-4452134796509464077_m_1768252039860486047_m_2897511617293957267_m_5808030803546790052_m_-3093947404852640465_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2" width="1" height="1"></a></div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote></div>