<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Feb 23, 2022, at 9:48 PM, Qi Yang <<a href="mailto:qiyang@oakland.edu" class="">qiyang@oakland.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Hi Barry,<div class=""><br class=""></div><div class="">Your message is really helpful, really appreciate it.</div><div class=""><br class=""></div><div class=""> Another question is about the hip, it seems the hip is still under development since the only vector is supported according to the PETSc document, so it means we can't run the ksp solver like spmv part under AMD GPU, right ?</div></div></div></blockquote><div><br class=""></div> I think that must be the case. So BoomerAMG runs on the GPU but not the KSP solver.</div><div><br class=""></div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><br class=""></div><div class="">Best Regards,</div><div class="">Qi</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 24, 2022 at 1:19 AM Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><div class=""><br class=""></div> You should use the main git branch of PETSc to utilize the latest hypre GPU support; things change very quickly in terms of what is available and how well it works. <div class=""><br class=""></div><div class=""> hypre's BoomerAMG uses CUDA on Nvidia machines and HIP on AMD machines; it does not use OpenCL. </div><div class=""><br class=""></div><div class=""> When you configure PETSc you should use the option --download-hypre --with-cuda <span style="font-family:Menlo;font-size:14px" class="">--download-hypre-configure-arguments=--enable-unified-memory </span>(or --with-hip) to build the appropriate hypre.</div><div class=""><br class=""></div><div class=""> The PETSc documentation on how to have hypre use the GPU for BoomerAMG pretty much does not exist! Apparently one must be a mind reader to use PETSc. </div><div class=""><br class=""></div><div class=""> By looking at the code it appears that one can get it to use the GPU if the vectors and matrices passed to the solver are for type <span style="font-family:Menlo;font-size:14px" class="">cuda </span>and <span style="font-family:Menlo;font-size:14px" class="">aijcusparse.</span> How you set the vector and matrix types depends on how you create the vectors and matrices in your code. If you are using a DM and create the vectors and matrices from it then it is straightforward, just use -dm_vec_type cuda -dm_mat_type aijcusparse and make sure you call DMSetFromOptions() on your DM. If you use MatCreate() and MatSetFromOptions() you can use -mat_type aijcusparse (and VecCreate() and VecSetFromOptions() and -vec_type cuda). </div><div class=""><br class=""></div><div class=""> I apologize for the sparsity and lack of clarity of the documentation.</div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""></div><div class=""><span style="font-family:Menlo;font-size:14px" class=""><br class=""></span></div><div class=""><br class=""></div><div class=""> <br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Feb 23, 2022, at 8:16 AM, Qi Yang <<a href="mailto:qiyang@oakland.edu" target="_blank" class="">qiyang@oakland.edu</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">Hi,<div class=""><br class=""></div><div class="">I hope this email finds you well.</div><div class=""><br class=""></div><div class=""><div class="">I am a student from Oakland University who is trying to use PETSc in my project.</div><div class=""><br class=""></div><div class="">I found that the PETSc 3.16 now is supporting the AMG preconditioner of hypre on Nvidia and AMD GPU, however, do you use OpenCL to do it? or do you use Cuda on Nvidia and Hip on AMD, if it is not OpenCL, how do you enable it with flag? like -pc_hypre_gamg_mat_type aijcusparse or just -mat_type aijcusparse(which is already used in former edition)? </div><div class=""><br class=""></div><div class="">Sorry for the disturbance, I did not find the answer in your document, looking forward to hearing from you soon.</div><div class=""><br class=""></div><div class="">Appreciate your time.</div><div class=""><br class=""></div><div class="">Best regards,</div><div class="">Qi</div></div></div>
</div></blockquote></div><br class=""></div></div></blockquote></div>
</div></blockquote></div><br class=""></body></html>