[petsc-dev] Not possible to do a VecPlaceArray for veccusp

[Karl Rupp]

Hi Jose,

 > We are trying to do some GPU developments on the SLEPc side, and we 
would need a way of placing the array of a VECCUSP vector, providing the 
GPU address. Specifically, what we want to do is have a large Vec on GPU 
and slice it in several smaller Vecs.
>
> For the GetArray/RestoreArray we have all possibilities:
> - VecGetArray: gets the pointer to the buffer stored in CPU memory
> - VecCUSPGetArray*: returns a CUSPARRAY object that contains some info, including the buffer allocated in GPU memory
> - VecCUSPGetCUDAArray*: returns a raw pointer of the GPU buffer
>
> The problem comes with PlaceArray equivalents. Using VecPlaceArray we can provide a new pointer to CPU memory. We wanted to implement the equivalent thing for GPU, but we found difficulties due to Thrust. If we wanted to provide a VecCUSPPlaceCUDAArray the problem is that Thrust does not allow wrapping an exisiting GPU buffer with a CUSPARRAY object (when creating a CUSPARRAY it always allocates new memory). On the other hand, VecCUSPPlaceArray is possible to implement, but the problem is that one should provide a CUSPARRAY obtained from a VecCUSPGetArray* without modification (it is not possible to do pointer arithmetic with a CUSPARRAY).

As far as I can see from browsing the documentation and the web there is 
indeed no such option. Ouch.


> Any thoughts?

I'll soon expose the CUDA backend of ViennaCL in PETSc, which will offer 
such functionality. Is this an option? It will take 2-3 weeks, though.

Best regards,
Karli