[petsc-users] cudaSetDevice

[Harshad Sahasrabudhe]

Hi Dominic,

According to Karl, setting up an option for PETSc which specifies whether
to call cudaSetDevice is easy with the lazy instantiation. So this seems to
be the plan.

Harshad


On Mon, Jan 20, 2014 at 12:31 PM, Dominic Meiser <dmeiser at txcorp.com> wrote:

> Hi Harshad,
>
>
> On Mon 20 Jan 2014 10:10:26 AM MST, Harshad Sahasrabudhe wrote:
>
>>
>> Hi Dominic,
>>
>> We use external libraries such as MAGMA and cuSPARSE. It looks like
>> they use the runtime API as you mentioned above. At the moment,
>> conflict is between the two instances of PETSc that we run (one each
>> for real and complex). We are planning to write some code in CUDA and
>> will use the driver API if need be.
>>
> Karl seems to know how to resolve the conflict between the two instances
> of PETSc. I'm not sure how exactly he is going to resolve this issue but
> apparently he thinks it's rather easy to do.
>
>
>> Does moving to the driver API look like something that can be included
>> in PETSc 3.5?
>>
> I doubt it given the challenge of dealing with third party libraries via
> the driver API. Quite possible somebody with a better understanding of the
> PETSc roadmap has a more informed opinion.
>
> Cheers,
> Dominic
>
>>
>> Harshad
>>
>>
>> On Mon, Jan 20, 2014 at 11:27 AM, Dominic Meiser <dmeiser at txcorp.com
>> <mailto:dmeiser at txcorp.com>> wrote:
>>
>> Hi Jed, Harshad,
>>
>> A different solution to the problem of PETSc and a user code
>> stepping on each other's feet with cudaSetDevice might be to use
>> the CUDA driver API for device selection rather than the runtime
>> API. If we were to explicitly manage a PETSc CUDA context using
>> the driver API we can control what devices are being used without
>> interfering with the mechanisms used by other parts of a client
>> code for CUDA device selection (e.g. cudaSetDevice). PETSc's
>> device management would be completely decoupled from the rest of
>> an application.
>>
>> Of course this approach can be combined with lazy initialization
>> as proposed by Karl. Whenever the first device function is called
>> we create the PETSc CUDA context. The advantages of lazy
>> initialization mentioned by Karl and Jed ensue (e.g. ability to
>> run on machines without GPUs provided one is not using GPU
>> functionality).
>>
>> Another advantage of a solution using the driver API is that
>> device and context management would be very similar between CUDA
>> and OpenCL backends.
>>
>> I realize that this proposal might be impractical as a near term
>> solution since it involves a pretty major refactor of the CUDA
>> context infrastructure. Furthermore, as far as I can tell, third
>> party libraries that we rely on (e.g. cusp and cusparse) assume
>> the runtime api. Perhaps these difficulties can be overcome?
>>
>> A possible near term solution would be to turn this around and to
>> have applications with advanced device selection requirements use
>> the driver API. Harshad, I'm not familiar with your code but would
>> it be possible for you to use the driver API on your end to avoid
>> conflicts with cudaSetDevice calls inside PETSc?
>>
>> Cheers,
>> Dominic
>>
>>
>> On 01/14/2014 09:27 AM, Harshad Sahasrabudhe wrote:
>>
>> Hi Jed,
>>
>> Sometime back we talked about an interface which could handle
>> other libraries calling cudaSetDevice simultaneously with
>> PETSc. For example, in our case 2 different instances of PETSc
>> calling cudaSetDevice.
>>
>> >Sure, but how will we actually share the device between
>> libraries? What
>> >if the other library was not PETSc, but something else, and
>> they also
>> >called cudaSetDevice, but with a different default mapping
>> strategy?
>>
>> >We need an interface that handles this case.
>>
>> Do we already have any solution for this? If not, can we start
>> looking at this case?
>>
>> Thanks,
>> Harshad
>>
>>
>>
>> --
>> Dominic Meiser
>> Tech-X Corporation
>> 5621 Arapahoe Avenue
>> Boulder, CO 80303
>> USA
>> Telephone: 303-996-2036
>> Fax: 303-448-7756
>> www.txcorp.com <http://www.txcorp.com>
>>
>>
>>
>
>
> --
> Dominic Meiser
> Tech-X Corporation
> 5621 Arapahoe Avenue
> Boulder, CO 80303
> USA
> Telephone: 303-996-2036
> Fax: 303-448-7756
> www.txcorp.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20140120/4356e19a/attachment.html>