[petsc-dev] Petsc support for BoomerAMG from PCHYPRE to run on Nvidia and AMD GPUs

[Barry Smith]

> On Feb 23, 2022, at 9:48 PM, Qi Yang <qiyang at oakland.edu> wrote:
> 
> Hi Barry,
> 
> Your message is really helpful, really appreciate it.
> 
>  Another question is about the hip, it seems the hip is still under development since the only vector is supported according to the PETSc document, so it means we can't run the ksp solver like spmv part under AMD GPU, right ?

   I think that must be the case. So BoomerAMG runs on the GPU but not the KSP solver.


> 
> Best Regards,
> Qi
> 
> On Thu, Feb 24, 2022 at 1:19 AM Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> wrote:
> 
>   You should use the main git branch of PETSc to utilize the latest hypre GPU support; things change very quickly in terms of what is available and how well it works. 
> 
>   hypre's BoomerAMG uses CUDA on Nvidia machines and HIP on AMD machines; it does not use OpenCL. 
> 
>   When you configure PETSc you should use the option --download-hypre --with-cuda --download-hypre-configure-arguments=--enable-unified-memory (or --with-hip) to build the appropriate hypre.
> 
>   The PETSc documentation on how to have hypre use the GPU for BoomerAMG pretty much does not exist! Apparently one must be a mind reader to use PETSc. 
> 
>   By looking at the code it appears that one can get it to use the GPU if the vectors and matrices passed to the solver are for type cuda and aijcusparse. How you set the vector and matrix types depends on how you create the vectors and matrices in your code. If you are using a DM and create the vectors and matrices from it then it is straightforward, just use -dm_vec_type cuda -dm_mat_type aijcusparse and make sure you call DMSetFromOptions() on your DM. If you use MatCreate() and MatSetFromOptions() you can use -mat_type aijcusparse (and VecCreate() and VecSetFromOptions() and -vec_type cuda). 
> 
>   I apologize for the sparsity and lack of clarity of the documentation.
> 
>   Barry
> 
> 
> 
>   
> 
>> On Feb 23, 2022, at 8:16 AM, Qi Yang <qiyang at oakland.edu <mailto:qiyang at oakland.edu>> wrote:
>> 
>> Hi,
>> 
>> I hope this email finds you well.
>> 
>> I am a student from Oakland University who is trying to use PETSc in my project.
>> 
>> I found that the PETSc 3.16 now is supporting the AMG preconditioner of hypre on Nvidia and AMD GPU, however, do you use OpenCL to do it? or do you use Cuda on Nvidia and Hip on AMD, if it is not OpenCL, how do you enable it with flag? like -pc_hypre_gamg_mat_type aijcusparse or just -mat_type aijcusparse(which is already used in former edition)? 
>> 
>> Sorry for the disturbance, I did not find the answer in your document, looking forward to hearing from you soon.
>> 
>> Appreciate your time.
>> 
>> Best regards,
>> Qi
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20220223/31ab8799/attachment.html>