[petsc-dev] using typedef for each different memory space the variable lives in

[Jed Brown]

Barry Smith <bsmith at petsc.dev> writes:

>    Currently we use PetscScalar and PetscScalar * to refer to variables that could be in any memory space. On the CPU, on the GPU, in Kokkos, etc.
>
>    Would it make sense to use typedef to indicate at each location the true type of the memory location when possible? 
>
>    typedef PetscReal PetscKokkosReal   means the variable is in the Kokkos memory space
>    typedef PetscReal PetscCUDAReal
>    typedef PetscReal PetscNVSHEMReal

There's no safety here. I suppose you could do a struct to prevent implicit conversion

typdef struct {PetscReal val;} PetscCUDAReal;

but even this doesn't enforce the invariant.

I think the distinction only makes sense for boxed values/arrays. If you ever have a value, it's in your memory space, but it matters when you dereference a pointer.

A question is how you'd convert. Presumably it requires a cudaMemcpy, so you'd want the array length to be part of the type and some light interfaces to do the conversions.

Some mapping of values to device memory can be piggy-backed on kernel launches so I'm not sure that requiring up-front conversion is the right approach (effectively adding one or more cudaMemcpy to the critical path).

>    etc. 
>
>   Then one could write things like 
>
>   struct {
>      ...
>      PetscNVSHEMReal *values;
>   }
>
>   Similarly inside kernels one would use the type type associated with the kernel, cuda with cuda etc. 
>
>   I think the resulting code will be much clearer and easier to dive into, then having to first figure out where each variable lives.
>
>   I find the current code confusing because one cannot immediately see a variable declaration and know where it lives, even though it does live somewhere in particular..
>
>   Barry