[petsc-users] cudaSetDevice

[Karl Rupp]

Hi guys,

 > We use external libraries such as MAGMA and cuSPARSE. It looks like they
> use the runtime API as you mentioned above. At the moment, conflict is
> between the two instances of PETSc that we run (one each for real and
> complex). We are planning to write some code in CUDA and will use the
> driver API if need be.
>
> Does moving to the driver API look like something that can be included
> in PETSc 3.5?

I can implement the lazy instantiation mechanism in early February.

As for the CUDA driver API, this depends on the external libraries. 
PETSc's goal should be to stay away from low-level GPU fiddling as much 
as possible and only provide the glue needed to leverage existing GPU 
libraries in an MPI context. For example, I need to check whether e.g. 
thrust and CUSP are able to deal with such a context information. If 
not, then we have no other option but to rely on cudaSetDevice() anyway...

Best regards,
Karli



>
> On Mon, Jan 20, 2014 at 11:27 AM, Dominic Meiser <dmeiser at txcorp.com
> <mailto:dmeiser at txcorp.com>> wrote:
>
>     Hi Jed, Harshad,
>
>     A different solution to the problem of PETSc and a user code
>     stepping on each other's feet with cudaSetDevice might be to use the
>     CUDA driver API for device selection rather than the runtime API. If
>     we were to explicitly manage a PETSc CUDA context using the driver
>     API we can control what devices are being used without interfering
>     with the mechanisms used by other parts of a client code for CUDA
>     device selection (e.g. cudaSetDevice). PETSc's device management
>     would be completely decoupled from the rest of an application.
>
>     Of course this approach can be combined with lazy initialization as
>     proposed by Karl. Whenever the first device function is called we
>     create the PETSc CUDA context. The advantages of lazy initialization
>     mentioned by Karl and Jed ensue (e.g. ability to run on machines
>     without GPUs provided one is not using GPU functionality).
>
>     Another advantage of a solution using the driver API is that device
>     and context management would be very similar between CUDA and OpenCL
>     backends.
>
>     I realize that this proposal might be impractical as a near term
>     solution since it involves a pretty major refactor of the CUDA
>     context infrastructure. Furthermore, as far as I can tell, third
>     party libraries that we rely on (e.g. cusp and cusparse) assume the
>     runtime api. Perhaps these difficulties can be overcome?
>
>     A possible near term solution would be to turn this around and to
>     have applications with advanced device selection requirements use
>     the driver API. Harshad, I'm not familiar with your code but would
>     it be possible for you to use the driver API on your end to avoid
>     conflicts with cudaSetDevice calls inside PETSc?
>
>     Cheers,
>     Dominic
>
>
>     On 01/14/2014 09:27 AM, Harshad Sahasrabudhe wrote:
>
>         Hi Jed,
>
>         Sometime back we talked about an interface which could handle
>         other libraries calling cudaSetDevice simultaneously with PETSc.
>         For example, in our case 2 different instances of PETSc calling
>         cudaSetDevice.
>
>          >Sure, but how will we actually share the device between
>         libraries?  What
>          >if the other library was not PETSc, but something else, and
>         they also
>          >called cudaSetDevice, but with a different default mapping
>         strategy?
>
>          >We need an interface that handles this case.
>
>         Do we already have any solution for this? If not, can we start
>         looking at this case?
>
>         Thanks,
>         Harshad
>
>
>
>     --
>     Dominic Meiser
>     Tech-X Corporation
>     5621 Arapahoe Avenue
>     Boulder, CO 80303
>     USA
>     Telephone: 303-996-2036
>     Fax: 303-448-7756
>     www.txcorp.com <http://www.txcorp.com>
>
>