[petsc-users] Cannot eagerly initialize cuda, as doing so results in cuda error 35 (cudaErrorInsufficientDriver) : CUDA driver version is insufficient for CUDA runtime version

[Jacob Faibussowitsch]

Hi Fande,

What machine are you running this on? Please attach configure.log so I can troubleshoot this.

Best regards,

Jacob Faibussowitsch
(Jacob Fai - booss - oh - vitch)

> On Jan 19, 2022, at 10:04, Fande Kong <fdkong.jd at gmail.com> wrote:
> 
> Hi All,
> 
> Upgraded PETSc from 3.16.1 to the current main branch. I suddenly got the following error message:
> 
> 2d_diffusion]$ ../../../moose_test-dbg -i 2d_diffusion_test.i -use_gpu_aware_mpi 0 -gpu_mat_type aijcusparse -gpu_vec_type cuda  -log_view 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Missing or incorrect user input 
> [0]PETSC ERROR: Cannot eagerly initialize cuda, as doing so results in cuda error 35 (cudaErrorInsufficientDriver) : CUDA driver version is insufficient for CUDA runtime version
> [0]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.16.3-618-gad32f7e  GIT Date: 2022-01-18 16:04:31 +0000
> [0]PETSC ERROR: ../../../moose_test-dbg on a arch-linux-c-opt named r8i3n0 by kongf Wed Jan 19 08:30:13 2022
> [0]PETSC ERROR: Configure options --with-debugging=no --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 --download-ptscotch=1 --download-parmetis=1 --download-mumps=1 --download-strumpack=1 --download-scalapack=1 --download-slepc=1 --with-mpi=1 --with-cxx-dialect=C++14 --with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices --with-make-np=24 --with-cuda --with-cudac=nvcc --with-cuda-arch=70 --download-kokkos=1
> [0]PETSC ERROR: #1 initialize() at /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:298
> [0]PETSC ERROR: #2 PetscDeviceInitializeTypeFromOptions_Private() at /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/device/interface/device.cxx:299
> [0]PETSC ERROR: #3 PetscDeviceInitializeFromOptions_Internal() at /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/device/interface/device.cxx:425
> [0]PETSC ERROR: #4 PetscInitialize_Common() at /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/pinit.c:963
> [0]PETSC ERROR: #5 PetscInitialize() at /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/pinit.c:1238
> [0]PETSC ERROR: #6 SlepcInitialize() at /home/kongf/workhome/sawtooth/moosegpu/petsc/arch-linux-c-opt/externalpackages/git.slepc/src/sys/slepcinit.c:275
> [0]PETSC ERROR: #7 LibMeshInit() at ../src/base/libmesh.C:522
> [r8i3n0:mpi_rank_0][MPIDI_CH3_Abort] application called MPI_Abort(MPI_COMM_WORLD, 95) - process 0: No such file or directory (2)
> 
> Thanks,
> 
> Fande

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220119/9b05ece5/attachment.html>