[petsc-dev] Not possible to do a VecPlaceArray for veccusp

[Dominic Meiser]

On Thu, Feb 25, 2016 at 01:13:01PM +0100, Jose E. Roman wrote:
> We are trying to do some GPU developments on the SLEPc side, and we would need a way of placing the array of a VECCUSP vector, providing the GPU address. Specifically, what we want to do is have a large Vec on GPU and slice it in several smaller Vecs.
> 
> For the GetArray/RestoreArray we have all possibilities:
> - VecGetArray: gets the pointer to the buffer stored in CPU memory
> - VecCUSPGetArray*: returns a CUSPARRAY object that contains some info, including the buffer allocated in GPU memory
> - VecCUSPGetCUDAArray*: returns a raw pointer of the GPU buffer
> 
> The problem comes with PlaceArray equivalents. Using VecPlaceArray we can provide a new pointer to CPU memory. We wanted to implement the equivalent thing for GPU, but we found difficulties due to Thrust. If we wanted to provide a VecCUSPPlaceCUDAArray the problem is that Thrust does not allow wrapping an exisiting GPU buffer with a CUSPARRAY object (when creating a CUSPARRAY it always allocates new memory). On the other hand, VecCUSPPlaceArray is possible to implement, but the problem is that one should provide a CUSPARRAY obtained from a VecCUSPGetArray* without modification (it is not possible to do pointer arithmetic with a CUSPARRAY).
> 
> Any thoughts?
> 

I think your and Karli's analysis is correct, this is currently
not supported.  Besides Karli's proposal to use ViennaCL's cuda
backend a different option might be to use cusp's array views.
These have a constructor for sub-ranges of other cusp arrays:

https://github.com/cusplibrary/cusplibrary/blob/master/cusp/array1d.h#L409

However, enabling cusp array views in something like
VecCUSPPlaceArray is not immediately possible.  The CUSPARRAY
type, which is currently hardwired to be
cusp::array1d<PetscScalar,cusp::device_memory>, would have to
become a template parameter.  I'm not sure if we want to go down
that path.

The alternative would be to use raw cuda pointers instead of cusp
arrays for GPU memory in VecCUSP.  That would be a fairly
significant undertaking (certainly more than the 2-3 weeks Karli
is estimating for getting the ViennaCL cuda backend in).

Cheers,
Dominic

-- 
Dominic Meiser
Tech-X Corporation - 5621 Arapahoe Avenue - Boulder, CO 80303