[petsc-users] Cannot eagerly initialize cuda, as doing so results in cuda error 35 (cudaErrorInsufficientDriver) : CUDA driver version is insufficient for CUDA runtime version

[Jed Brown]

Fande Kong <fdkong.jd at gmail.com> writes:

> On Wed, Jan 19, 2022 at 11:39 AM Jacob Faibussowitsch <jacob.fai at gmail.com>
> wrote:
>
>> Are you running on login nodes or compute nodes (I can’t seem to tell from
>> the configure.log)?
>>
>
> I was compiling codes on login nodes, and running codes on compute nodes.
> Login nodes do not have GPUs, but compute nodes do have GPUs.
>
> Just to be clear, the same thing (code, machine) with PETSc-3.16.1 worked
> perfectly. I have this trouble with PETSc-main.

I assume you can

export PETSC_OPTIONS='-device_enable lazy'

and it'll work.

I think this should be the default. The main complaint is that timing the first GPU-using event isn't accurate if it includes initialization, but I think this is mostly hypothetical because you can't trust any timing that doesn't preload in some form and the first GPU-using event will almost always be something uninteresting so I think it will rarely lead to confusion. Meanwhile, eager initialization is viscerally disruptive for lots of people.