[petsc-users] stdpar (nvfortran) managed memory + PETSc CUDA recommended interop pattern?

[Edoardo alinovi]

Thanks Junchao, that clears it up.

One open question: the CUDA device-array getters (VecCUDAGetArrayWrite,
VecCreateMPICUDAWithArray) don't have Fortran stubs in my build. Is there a
supported Fortran path (e.g. VecGetArrayAndMemType from Fortran), or are
thin C shims the recommended approach?

On the managed coo_v point: I rebuilt my PETSc with logging and ran the
attached reproducer. With a cudaMallocManaged coo_v
(cudaPointerGetAttributes → type = Managed) feeding MatSetValuesCOO on an
AIJCUSPARSE matrix, -log_view shows MatSetValuesCOO with CpuToGpu Count =
0; the only H2D is the standard MatCUSPARSCopyTo (8 MB at n=2·10⁵). So I
don't see an extra copy from the managed classification. The assembly is
correct (MatMult checks out) and there's a single matrix to device copy.

One caveat: I'm on WSL2, where cudaMemPrefetchAsync fails ("invalid device
ordinal"), so coo_v stays host-resident. I couldn't exercise a
device-resident managed coo_v, which is the case where a Device-only check
might force a wasteful copy. Does that match your expectation, and is the
device-resident-managed case worth me checking on a native-Linux box?
Probably can double check next week on another machine.

I am attaching a small test just in case it is useful to somebody out
there.

Build Info: PETSc 3.25.1, --with-cuda --with-cuda-arch=61, CUDA 12.6,
gfortran-13.

Thanks again,

Edoardo


Il giorno lun 22 giu 2026 alle ore 06:54 Junchao Zhang <
junchao.zhang at gmail.com> ha scritto:

> Hi Edoardo,
>   petsc/cuda backend doesn't use cuda managed memory,  so I am just trying
> to answer your questions based on my limited experience with it. See the
> inlined answers.
>
>
> On Sun, Jun 21, 2026 at 6:52 AM Edoardo alinovi <edoardo.alinovi at gmail.com>
> wrote:
>
>> Hi PETSc friends!
>>
>> Hope you are all doing great.
>>
>> With my time I am porting my code to gpu. This is a really instructive
>> project but it comes with several headaches  as well.
>>
>> I have some valuable questions you might be able to answer that would
>> help me a lot understanding how things work in the modern world:
>>
>> *My goal: *
>> Assembly loops are *do concurrent* compiled with nvfortran -stdpar=gpu,
>> so my field/work arrays live in CUDA managed memory (cudaMallocManaged).
>> The linear solve uses PETSc with MATAIJCUSPARSE + (intended) VECCUDA. PETSc
>> 3.25.1 (git), built with NVHPC 24.11 / CUDA 12.6; targets cc61 and cc86 (no
>> HMM),. Iwant the whole outer iteration to stay device-resident.
>>
>> 1) Managed pointer as a VECCUDA device array. Can I wrap a
>> cudaMallocManaged pointer as the device array of a Vec via
>> VecCreateMPICUDAWithArray (or VecCUDAPlaceArray)? Will the offload-mask
>> logic treat managed memory coherently, or assume a distinct host array and
>> issue redundant H2D/D2H copies? Recommended pattern for stdpar-managed +
>> PETSc-CUDA interop?
>>
>
> Yes, I think you can use  VecCreateMPICUDAWithArray() and friends.  petsc
> will just treat the array you provided as a device accessible array.  If
> you don't provide host arrays, petsc will malloc memory on the host to
> mirror your cuda managed memory.
>
>
>>
>> 2) On-device RHS fill / solution read. Icurrently use
>> VecSetValues/VecGetValues against host arrays, which flips the mask to CPU
>> every iteration. With VECCUDA, is the recommended replacement
>> VecCUDAGetArrayWrite (RHS) and VecCUDAGetArray (solution)? Offload-mask
>> pitfalls?
>>
> I think you can do it. You just need to pretend your provided memory to
> petsc is cude device memory. For example, you can call VecCUDAGetArrayWrite()
> to get the array,  then operate on host with the array, and then VecCUDAReturnArrayWrite().
> You need to manage the synchronization yourself.
>
>
>>
>> 3) MatSetValuesCOO from device memory. Ipreallocate with
>> MatSetPreallocationCOO and assemble with MatSetValuesCOO. For an
>> AIJCUSPARSE matrix, may coo_v be a device/managed pointer? Memtype-aware
>> contract, and from which version?
>>
>
> petsc checks if the array coo_v is cudaMemoryTypeDevice.  Perhaps we
> should also check if it is cudaMemoryTypeManaged.
>
>
>> 4) Vec type inference. If Iset only mat_type aijcusparse but leave the
>> Vecs VECMPI, does KSP bounce the vectors each iteration, or promote them?
>> Must I set vec_type cuda explicitly?
>>
> the vectors must also be device vectors.  If you get the vectors from
> MatGetVecs() or VecDuplicate() of the results, the vector types are
> derived from matrix types (e.g., aijcusparse --> veccuda)
>
>
>>
>> Happy to share the configure line and a minimal reproducer.
>>
> Yes, please share one and issues you found, so we can better understand
> the problem.
>
>
>>
>> Mega Thanks!
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20260622/99d0e3e5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: coo_managed_test.c
Type: application/octet-stream
Size: 4903 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20260622/99d0e3e5/attachment.obj>