[petsc-users] Error while running with -pc_type gamg on NVIDIA GPU

[Junchao Zhang]

Which version of hypre do you use?  I used petsc's --download-hypre, which
automatically installed hypre-2.29.0. I did not see the error.

$ ./heat_diff_cu -ksp_type gmres -pc_type hypre -pc_hypre_type boomeramg
-use_gpu_aware_mpi 0 -mat_type aijcusparse -vec_type cuda
The start and end indices of mat for each rank: 0       100
Total time taken for KSP solve: rank: 0 0.0112464

--Junchao Zhang


On Tue, Sep 12, 2023 at 10:15 AM Maruthi NH <maruthinh at gmail.com> wrote:

> Hi Junchao Zhang,
>
> Thanks for the help. Updating PETSc fixed the problem. However, if I use
> boomeramg from hypre as follows, I get a similar error.
>
> ./heat_diff_cu --ksp_type gmres -pc_type hypre -pc_hypre_type boomeramg
> -use_gpu_aware_mpi 0 -mat_type aijcusparse -vec_type cuda
>
> ** On entry to cusparseCreateCsr() parameter number 5 (csrRowOffsets) had
> an illegal value: NULL pointer
>
> CUSPARSE ERROR (code = 3, invalid value) at csr_matrix_cuda_utils.c:57
>
> Regards,
> Maruthi
>
>
>
>
>
> On Tue, Sep 12, 2023 at 1:09 AM Junchao Zhang <junchao.zhang at gmail.com>
> wrote:
>
>> Hi, Maruthi,
>>   I could run your example on my machine.  BTW,  I added these at the end
>> of main() to free petsc objects.
>> VecDestroy(&vout);
>> VecDestroy(&x);
>> VecDestroy(&b);
>> VecDestroy(&u);
>> MatDestroy(&A);
>> VecScatterDestroy(&ctx);
>> KSPDestroy(&ksp);
>>
>> If you use cuda-12.2, maybe the problem is already fixed by MR
>> https://gitlab.com/petsc/petsc/-/merge_requests/6828
>> You can use petsc/main branch to try.  Note your petsc version is from
>> Date: 2023-08-13
>>
>> Thanks.
>> --Junchao Zhang
>>
>>
>> On Mon, Sep 11, 2023 at 12:10 PM Maruthi NH <maruthinh at gmail.com> wrote:
>>
>>> Hi Barry Smith,
>>>
>>> Thanks for the quick response.
>>>
>>> Here is the code I used to test PETSc on GPU.
>>> This is the command I used to run
>>> mpiexec.hydra -n 1 ./heat_diff_cu -Nx 10000000 -ksp_type gmres -mat_type
>>> aijcusparse -vec_type cuda -use_gpu_aware_mpi 0 -pc_type gamg
>>> -ksp_converged_reason
>>>
>>> Regards,
>>> Maruthi
>>>
>>>
>>>
>>> On Sun, Sep 10, 2023 at 11:37 PM Barry Smith <bsmith at petsc.dev> wrote:
>>>
>>>>
>>>>
>>>> On Sep 10, 2023, at 5:54 AM, Maruthi NH <maruthinh at gmail.com> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I am trying to accelerate the linear solver with PETSc GPU backend. For
>>>> testing I have a simple 1D heat diffusion solver, here are some
>>>> observations.
>>>> 1. If I use -pc_type gamg it throws the following error
>>>>  ** On entry to cusparseCreateCsr() parameter number 5 (csrRowOffsets)
>>>> had an illegal value: NULL pointer
>>>>
>>>> [0]PETSC ERROR: --------------------- Error Message
>>>> --------------------------------------------------------------
>>>> [0]PETSC ERROR: GPU error
>>>> [0]PETSC ERROR: cuSPARSE errorcode 3 (CUSPARSE_STATUS_INVALID_VALUE) :
>>>> invalid value
>>>> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble
>>>> shooting.
>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.19.4-959-g92f1e92e88
>>>>  GIT Date: 2023-08-13 19:43:04 +0000
>>>>
>>>>    Can you share the code that triggers this?
>>>>
>>>> 2. Default pc ilu takes about 1.2 seconds on a single CPU and it takes
>>>> about 105.9 seconds on a GPU. Similar observations with pc_type asm
>>>> I have NVIDIA RTX A2000 8GB Laptop GPU
>>>>
>>>>
>>>>   This is expected. The triangular solves sequentialize on the GPU so
>>>> naturally are extremely slow since they cannot take advantage of the
>>>> massive parallelism of the GPU.
>>>>
>>>>
>>>> 3. What I could be missing? Also, are there any general guidelines for
>>>> better GPU performance using PETSc?
>>>>
>>>> Regards,
>>>> Maruthi
>>>>
>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230912/7ac4d14a/attachment-0001.html>