[petsc-users] stdpar (nvfortran) managed memory + PETSc CUDA recommended interop pattern?

[Edoardo alinovi]

Hi PETSc friends!

Hope you are all doing great.

With my time I am porting my code to gpu. This is a really instructive
project but it comes with several headaches  as well.

I have some valuable questions you might be able to answer that would help
me a lot understanding how things work in the modern world:

*My goal: *
Assembly loops are *do concurrent* compiled with nvfortran -stdpar=gpu, so
my field/work arrays live in CUDA managed memory (cudaMallocManaged). The
linear solve uses PETSc with MATAIJCUSPARSE + (intended) VECCUDA. PETSc
3.25.1 (git), built with NVHPC 24.11 / CUDA 12.6; targets cc61 and cc86 (no
HMM),. Iwant the whole outer iteration to stay device-resident.

1) Managed pointer as a VECCUDA device array. Can I wrap a
cudaMallocManaged pointer as the device array of a Vec via
VecCreateMPICUDAWithArray (or VecCUDAPlaceArray)? Will the offload-mask
logic treat managed memory coherently, or assume a distinct host array and
issue redundant H2D/D2H copies? Recommended pattern for stdpar-managed +
PETSc-CUDA interop?

2) On-device RHS fill / solution read. Icurrently use
VecSetValues/VecGetValues against host arrays, which flips the mask to CPU
every iteration. With VECCUDA, is the recommended replacement
VecCUDAGetArrayWrite (RHS) and VecCUDAGetArray (solution)? Offload-mask
pitfalls?

3) MatSetValuesCOO from device memory. Ipreallocate with
MatSetPreallocationCOO and assemble with MatSetValuesCOO. For an
AIJCUSPARSE matrix, may coo_v be a device/managed pointer? Memtype-aware
contract, and from which version?

4) Vec type inference. If Iset only mat_type aijcusparse but leave the Vecs
VECMPI, does KSP bounce the vectors each iteration, or promote them? Must I
set vec_type cuda explicitly?

Happy to share the configure line and a minimal reproducer.

Mega Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20260621/ff44e337/attachment.html>