[petsc-users] [GPU] Jacobi preconditioner

[LEDAC Pierre]

Thanks, i will give a try with kokkos backend.

I have just seen now that even if we use MatSetPreallocationCOO() with device pointers, it seems that the matrix is preallocated also on host ? Am i wrong or the strategy is to have matrix on host and device even if only the latter is needed ?

Thanks
________________________________
De : Junchao Zhang <junchao.zhang at gmail.com>
Envoyé : lundi 28 juillet 2025 17:43:56
À : LEDAC Pierre
Cc : petsc-users at mcs.anl.gov
Objet : Re: [petsc-users] [GPU] Jacobi preconditioner

Yes, MatGetDiagonal_SeqAIJCUSPARSE hasn't been implemented.  petsc/cuda and petsc/kokkos backends are separate code.
If petsc/kokkos meet your needs, then just use them.  For petsc users, we hope it will be just a difference of extra --download-kokkos --download-kokkos-kernels in configuration.

--Junchao Zhang


On Mon, Jul 28, 2025 at 2:51 AM LEDAC Pierre <Pierre.LEDAC at cea.fr<mailto:Pierre.LEDAC at cea.fr>> wrote:

Hello all,


We are solving with PETSc a linear system updated every time step (constant stencil but coefficients changing).


The matrix is preallocated once with MatSetPreallocationCOO() then filled each time step with MatSetValuesCOO() and we use device pointers for coo_i, coo_j, and coefficients values.


It is working fine with a GMRES Ksp solver and PC Jacobi but we are surprised to see that every time step, during PCSetUp, MatGetDiagonal_SeqAIJ is called whereas the matrix is on the device. Looking at the API, it seems there is no MatGetDiagonal_SeqAIJCUSPARSE() but a MatGetDiagonal_SeqAIJKOKKOS().


Does it mean we should use Kokkos backend in PETSc to have Jacobi preconditioner built directly on device ? Or I am doing something wrong ?

NB: Gmres is running well on device.


I could use -ksp_reuse_preconditioner to avoid Jacobi being recreated each solve on host but it increases significantly the number of iterations.


Thanks,


[cid:ii_19851ad63faf456b1e51]



Pierre LEDAC
Commissariat à l’énergie atomique et aux énergies alternatives
Centre de SACLAY
DES/ISAS/DM2S/SGLS/LCAN
Bâtiment 451 – point courrier n°41
F-91191 Gif-sur-Yvette
+33 1 69 08 04 03
+33 6 83 42 05 79
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20250728/6decbdd5/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pastedImage.png
Type: image/png
Size: 519712 bytes
Desc: pastedImage.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20250728/6decbdd5/attachment-0001.png>