[petsc-dev] Jacobi (smoothing) not staying on GPU

[Mark Adams]

I am looking at TS/SNES/KSP/GAMG solve with Landau, which is all on the
GPU, but it looks like MatGetDiagonal (see attached), and to a lesser
extent VecPointWiseMult (biggest red band on the right side under PCApply),
are resulting in expensive CPU-GPU movement. MatGetDiagonal on the fine
grid is taking about 10x the time of TFQMR/GAMG iteration.

Attached is a view of this with CUDA and an nsys data file with Kokkos that
is pretty much the same.

Any thoughts on how to fix this?

Thanks,
Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20220607/ede91cec/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2022-06-07 at 8.31.20 PM.png
Type: image/png
Size: 210194 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20220607/ede91cec/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: output_ex2_3d_kokkos.nsys-rep
Type: application/octet-stream
Size: 2796831 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20220607/ede91cec/attachment-0001.obj>