[petsc-users] Cannot eagerly initialize cuda, as doing so results in cuda error 35 (cudaErrorInsufficientDriver) : CUDA driver version is insufficient for CUDA runtime version

[Fande Kong]

Hi All,

Upgraded PETSc from 3.16.1 to the current main branch. I suddenly got the
following error message:

2d_diffusion]$ ../../../moose_test-dbg -i 2d_diffusion_test.i
-use_gpu_aware_mpi 0 -gpu_mat_type aijcusparse -gpu_vec_type cuda
 -log_view
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Missing or incorrect user input
[0]PETSC ERROR: Cannot eagerly initialize cuda, as doing so results in cuda
error 35 (cudaErrorInsufficientDriver) : CUDA driver version is
insufficient for CUDA runtime version
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.16.3-618-gad32f7e  GIT
Date: 2022-01-18 16:04:31 +0000
[0]PETSC ERROR: ../../../moose_test-dbg on a arch-linux-c-opt named r8i3n0
by kongf Wed Jan 19 08:30:13 2022
[0]PETSC ERROR: Configure options --with-debugging=no
--with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1
--download-ptscotch=1 --download-parmetis=1 --download-mumps=1
--download-strumpack=1 --download-scalapack=1 --download-slepc=1
--with-mpi=1 --with-cxx-dialect=C++14 --with-fortran-bindings=0
--with-sowing=0 --with-64-bit-indices --with-make-np=24 --with-cuda
--with-cudac=nvcc --with-cuda-arch=70 --download-kokkos=1
[0]PETSC ERROR: #1 initialize() at
/home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:298
[0]PETSC ERROR: #2 PetscDeviceInitializeTypeFromOptions_Private() at
/home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/device/interface/device.cxx:299
[0]PETSC ERROR: #3 PetscDeviceInitializeFromOptions_Internal() at
/home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/device/interface/device.cxx:425
[0]PETSC ERROR: #4 PetscInitialize_Common() at
/home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/pinit.c:963
[0]PETSC ERROR: #5 PetscInitialize() at
/home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/pinit.c:1238
[0]PETSC ERROR: #6 SlepcInitialize() at
/home/kongf/workhome/sawtooth/moosegpu/petsc/arch-linux-c-opt/externalpackages/git.slepc/src/sys/slepcinit.c:275
[0]PETSC ERROR: #7 LibMeshInit() at ../src/base/libmesh.C:522
[r8i3n0:mpi_rank_0][MPIDI_CH3_Abort] application called
MPI_Abort(MPI_COMM_WORLD, 95) - process 0: No such file or directory (2)

Thanks,

Fande
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220119/abb01c67/attachment.html>