[petsc-dev] Not possible to do a VecPlaceArray for veccusp

[Jose E. Roman]

We are trying to do some GPU developments on the SLEPc side, and we would need a way of placing the array of a VECCUSP vector, providing the GPU address. Specifically, what we want to do is have a large Vec on GPU and slice it in several smaller Vecs.

For the GetArray/RestoreArray we have all possibilities:
- VecGetArray: gets the pointer to the buffer stored in CPU memory
- VecCUSPGetArray*: returns a CUSPARRAY object that contains some info, including the buffer allocated in GPU memory
- VecCUSPGetCUDAArray*: returns a raw pointer of the GPU buffer

The problem comes with PlaceArray equivalents. Using VecPlaceArray we can provide a new pointer to CPU memory. We wanted to implement the equivalent thing for GPU, but we found difficulties due to Thrust. If we wanted to provide a VecCUSPPlaceCUDAArray the problem is that Thrust does not allow wrapping an exisiting GPU buffer with a CUSPARRAY object (when creating a CUSPARRAY it always allocates new memory). On the other hand, VecCUSPPlaceArray is possible to implement, but the problem is that one should provide a CUSPARRAY obtained from a VecCUSPGetArray* without modification (it is not possible to do pointer arithmetic with a CUSPARRAY).

Any thoughts?