[petsc-users] stdpar (nvfortran) managed memory + PETSc CUDA recommended interop pattern?

[Junchao Zhang]

Hi Edoardo,
  petsc/cuda backend doesn't use cuda managed memory,  so I am just trying
to answer your questions based on my limited experience with it. See the
inlined answers.


On Sun, Jun 21, 2026 at 6:52 AM Edoardo alinovi <edoardo.alinovi at gmail.com>
wrote:

> Hi PETSc friends!
>
> Hope you are all doing great.
>
> With my time I am porting my code to gpu. This is a really instructive
> project but it comes with several headaches  as well.
>
> I have some valuable questions you might be able to answer that would help
> me a lot understanding how things work in the modern world:
>
> *My goal: *
> Assembly loops are *do concurrent* compiled with nvfortran -stdpar=gpu,
> so my field/work arrays live in CUDA managed memory (cudaMallocManaged).
> The linear solve uses PETSc with MATAIJCUSPARSE + (intended) VECCUDA. PETSc
> 3.25.1 (git), built with NVHPC 24.11 / CUDA 12.6; targets cc61 and cc86 (no
> HMM),. Iwant the whole outer iteration to stay device-resident.
>
> 1) Managed pointer as a VECCUDA device array. Can I wrap a
> cudaMallocManaged pointer as the device array of a Vec via
> VecCreateMPICUDAWithArray (or VecCUDAPlaceArray)? Will the offload-mask
> logic treat managed memory coherently, or assume a distinct host array and
> issue redundant H2D/D2H copies? Recommended pattern for stdpar-managed +
> PETSc-CUDA interop?
>

Yes, I think you can use  VecCreateMPICUDAWithArray() and friends.  petsc
will just treat the array you provided as a device accessible array.  If
you don't provide host arrays, petsc will malloc memory on the host to
mirror your cuda managed memory.


>
> 2) On-device RHS fill / solution read. Icurrently use
> VecSetValues/VecGetValues against host arrays, which flips the mask to CPU
> every iteration. With VECCUDA, is the recommended replacement
> VecCUDAGetArrayWrite (RHS) and VecCUDAGetArray (solution)? Offload-mask
> pitfalls?
>
I think you can do it. You just need to pretend your provided memory to
petsc is cude device memory. For example, you can call VecCUDAGetArrayWrite()
to get the array,  then operate on host with the array, and then
VecCUDAReturnArrayWrite().
You need to manage the synchronization yourself.


>
> 3) MatSetValuesCOO from device memory. Ipreallocate with
> MatSetPreallocationCOO and assemble with MatSetValuesCOO. For an
> AIJCUSPARSE matrix, may coo_v be a device/managed pointer? Memtype-aware
> contract, and from which version?
>

petsc checks if the array coo_v is cudaMemoryTypeDevice.  Perhaps we should
also check if it is cudaMemoryTypeManaged.


> 4) Vec type inference. If Iset only mat_type aijcusparse but leave the
> Vecs VECMPI, does KSP bounce the vectors each iteration, or promote them?
> Must I set vec_type cuda explicitly?
>
the vectors must also be device vectors.  If you get the vectors from
MatGetVecs() or VecDuplicate() of the results, the vector types are derived
from matrix types (e.g., aijcusparse --> veccuda)


>
> Happy to share the configure line and a minimal reproducer.
>
Yes, please share one and issues you found, so we can better understand the
problem.


>
> Mega Thanks!
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20260621/722c9080/attachment.html>