[petsc-users] cudaSetDevice

[Dominic Meiser]

Hi Jed, Harshad,

A different solution to the problem of PETSc and a user code stepping on 
each other's feet with cudaSetDevice might be to use the CUDA driver API 
for device selection rather than the runtime API. If we were to 
explicitly manage a PETSc CUDA context using the driver API we can 
control what devices are being used without interfering with the 
mechanisms used by other parts of a client code for CUDA device 
selection (e.g. cudaSetDevice). PETSc's device management would be 
completely decoupled from the rest of an application.

Of course this approach can be combined with lazy initialization as 
proposed by Karl. Whenever the first device function is called we create 
the PETSc CUDA context. The advantages of lazy initialization mentioned 
by Karl and Jed ensue (e.g. ability to run on machines without GPUs 
provided one is not using GPU functionality).

Another advantage of a solution using the driver API is that device and 
context management would be very similar between CUDA and OpenCL backends.

I realize that this proposal might be impractical as a near term 
solution since it involves a pretty major refactor of the CUDA context 
infrastructure. Furthermore, as far as I can tell, third party libraries 
that we rely on (e.g. cusp and cusparse) assume the runtime api. Perhaps 
these difficulties can be overcome?

A possible near term solution would be to turn this around and to have 
applications with advanced device selection requirements use the driver 
API. Harshad, I'm not familiar with your code but would it be possible 
for you to use the driver API on your end to avoid conflicts with 
cudaSetDevice calls inside PETSc?

Cheers,
Dominic

On 01/14/2014 09:27 AM, Harshad Sahasrabudhe wrote:
> Hi Jed,
>
> Sometime back we talked about an interface which could handle other 
> libraries calling cudaSetDevice simultaneously with PETSc. For 
> example, in our case 2 different instances of PETSc calling cudaSetDevice.
>
> >Sure, but how will we actually share the device between libraries?  What
> >if the other library was not PETSc, but something else, and they also
> >called cudaSetDevice, but with a different default mapping strategy?
>
> >We need an interface that handles this case.
>
> Do we already have any solution for this? If not, can we start looking 
> at this case?
>
> Thanks,
> Harshad


-- 
Dominic Meiser
Tech-X Corporation
5621 Arapahoe Avenue
Boulder, CO 80303
USA
Telephone: 303-996-2036
Fax: 303-448-7756
www.txcorp.com