<div dir="ltr"><div>Justin,</div><div> We like to have a wrapper over CUDA/HIP since with that we only need to maintain one code base. Scott (Cc'ed) may introduce his work/thoughts along this line and coordinate with Jacob and your team.</div><div><br></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Nov 10, 2021 at 12:50 PM Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com">jacob.fai@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><blockquote type="cite"><div dir="auto">What’s the plan going forward with this unified cuda/hip branch? </div></blockquote><div><br></div><div>The end goal is to integrate PetscDevice and PetscDeviceContext — new objects which encapsulates physical devices and device-side sets of operations respectively — into PETSc. PetscDeviceContext provides a framework for enqueueing work on device streams, but is far more extensible. <font color="#000000"><span>For example, I extend it to a very basic graph-based “PetscCallGraph" implementation in </span></font><a href="https://gitlab.com/petsc/petsc/-/merge_requests/4217" target="_blank">https://gitlab.com/petsc/petsc/-/merge_requests/4217</a><span style="color:rgb(0,0,0)">. </span><span style="color:rgb(0,0,0)">I believe Junchao is also currently working on integrating SYCL into the PetscDevice framework.</span></div><div><br></div><div><blockquote type="cite"><div dir="auto">Is this related to what some of us have been hearing about PETSc eventually going with a unified wrapper over both the CUDA and HIP API?</div></blockquote><br></div><div>The unified CUDA-HIP wrapper exists mainly <font color="#000000"><span>because both APIs similar enough that it made sense to wrap them, and is </span></font>the man in the middle between direct cuda/hip calls and higher-level PetscDevice/PetscDeviceContext API.</div><br><div>
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div>Best regards,<br><br>Jacob Faibussowitsch<br>(Jacob Fai - booss - oh - vitch)<br></div></div></div>
</div>
<div><br><blockquote type="cite"><div>On Nov 10, 2021, at 12:28, Justin Chang <<a href="mailto:jychang48@gmail.com" target="_blank">jychang48@gmail.com</a>> wrote:</div><br><div><div dir="auto">Jacob, </div><div dir="auto"><br></div><div dir="auto">What’s the plan going forward with this unified cuda/hip branch? Is this related to what some of us have been hearing about PETSc eventually going with a unified wrapper over both the CUDA and HIP API? </div><div dir="auto"><br></div><div dir="auto">We’re interested in making the HIP port more mature, including having support for the HIP part of HYPRE, but we’re unsure what direction you guys want to go with the GPU route.</div><div dir="auto"><br></div><div dir="auto">Thanks,</div><div dir="auto">Justin</div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Nov 10, 2021 at 12:19 PM Justin Chang <<a href="mailto:jychang48@gmail.com" target="_blank">jychang48@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Cc’ing Paul since I misspelled his email address initially</div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Nov 10, 2021 at 12:17 PM Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com" target="_blank">jacob.fai@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">I’m in the process of implementing asynchronous GPU support for petsc. A side effect of this is that I unify the cuda/hip interface such that anywhere we have cuda-like code we will automatically also get the hip variant.<div><br></div><div>The scaffolding is in include/petsc/private/cupminterface.hpp, but for concrete examples see the <span>jacobf/2021-10-21/veccupm-async branch for the WIP port of</span> VecSeq in src/vec/vec/impls/seq/seqcupm/veccupm.hpp.</div><div><div><br><div>
<div dir="auto" style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div dir="auto" style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div>Best regards,<br><br>Jacob Faibussowitsch<br>(Jacob Fai - booss - oh - vitch)<br></div></div></div>
</div></div></div></div><div style="overflow-wrap: break-word;"><div><div>
<div><br><blockquote type="cite"><div>On Nov 10, 2021, at 11:50, Justin Chang <<a href="mailto:jychang48@gmail.com" target="_blank">jychang48@gmail.com</a>> wrote:</div><br><div><div dir="ltr">Paul Bauman was also involved with the HIP port of HYPRE. Several of us at AMD are interested in getting HIP support for PETSc in general, and having HYPRE support would greatly help</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Nov 10, 2021 at 11:47 AM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">I did the work last summer. It's already available in 3.16</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il Mer 10 Nov 2021, 20:44 Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hypre has released HIP support and Ulrike says:<div><br></div><div>I just want to let you know that hypre can now be used through PETSc with GPUs (both Nvidia and AMD).<div><div><br></div></div></div><div>I am guessing we have some work to do to make this happen.</div><div><br></div><div>What should I do?</div></div>
</blockquote></div>
</blockquote></div>
</div></blockquote></div><br></div></div></div></blockquote></div></div>
</blockquote></div></div>
</div></blockquote></div><br></div></blockquote></div>