[petsc-users] Cannot eagerly initialize cuda, as doing so results in cuda error 35 (cudaErrorInsufficientDriver) : CUDA driver version is insufficient for CUDA runtime version

Jed Brown jed at jedbrown.org
Thu Jan 20 00:03:15 CST 2022

Fande Kong <fdkong.jd at gmail.com> writes:

> On Wed, Jan 19, 2022 at 11:39 AM Jacob Faibussowitsch <jacob.fai at gmail.com>
> wrote:
>> Are you running on login nodes or compute nodes (I can’t seem to tell from
>> the configure.log)?
> I was compiling codes on login nodes, and running codes on compute nodes.
> Login nodes do not have GPUs, but compute nodes do have GPUs.
> Just to be clear, the same thing (code, machine) with PETSc-3.16.1 worked
> perfectly. I have this trouble with PETSc-main.

I assume you can

export PETSC_OPTIONS='-device_enable lazy'

and it'll work.

I think this should be the default. The main complaint is that timing the first GPU-using event isn't accurate if it includes initialization, but I think this is mostly hypothetical because you can't trust any timing that doesn't preload in some form and the first GPU-using event will almost always be something uninteresting so I think it will rarely lead to confusion. Meanwhile, eager initialization is viscerally disruptive for lots of people.

More information about the petsc-users mailing list