<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Are you running on login nodes or compute nodes (I can’t seem to tell from the configure.log)? If running from login nodes, do they support running with GPU’s? Some clusters will install stub versions of cuda runtime on login nodes (such that configuration can find them), but that won’t actually work in practice. <div class=""><br class=""></div><div class="">If this is the case then CUDA will fail to initialize with this exact error. IIRC It wasn’t until CUDA 11.1 that they created a specific error code (cudaErrorStubLibrary) for it.<div class=""><br class=""><div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div>Best regards,<br class=""><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<br class=""></div></div></div>
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">On Jan 19, 2022, at 12:07, Fande Kong <<a href="mailto:fdkong.jd@gmail.com" class="">fdkong.jd@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Thanks, Jacob, and Junchao<br class=""><br class="">The log was attached. I am using Sawtooth at INL <a href="https://hpc.inl.gov/SitePages/Home.aspx" class="">https://hpc.inl.gov/SitePages/Home.aspx</a><br class=""><br class=""><br class="">Thanks,<br class=""><br class="">Fande</div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jan 19, 2022 at 10:32 AM Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com" class="">jacob.fai@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class="">Hi Fande,<div class=""><br class=""></div><div class="">What machine are you running this on? Please attach configure.log so I can troubleshoot this.</div><div class=""><br class=""><div class="">
<div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div class="">Best regards,<br class=""><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<br class=""></div></div></div>
</div>
<div class=""><br class=""><blockquote type="cite" class=""><div class="">On Jan 19, 2022, at 10:04, Fande Kong <<a href="mailto:fdkong.jd@gmail.com" target="_blank" class="">fdkong.jd@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">Hi All,<br class=""><br class="">Upgraded PETSc from 3.16.1 to the current main branch. I suddenly got the following error message:<br class=""><br class="">2d_diffusion]$ ../../../moose_test-dbg -i 2d_diffusion_test.i -use_gpu_aware_mpi 0 -gpu_mat_type aijcusparse -gpu_vec_type cuda -log_view <br class="">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br class="">[0]PETSC ERROR: Missing or incorrect user input <br class="">[0]PETSC ERROR: Cannot eagerly initialize cuda, as doing so results in cuda error 35 (cudaErrorInsufficientDriver) : CUDA driver version is insufficient for CUDA runtime version<br class="">[0]PETSC ERROR: See <a href="https://petsc.org/release/faq/" target="_blank" class="">https://petsc.org/release/faq/</a> for trouble shooting.<br class="">[0]PETSC ERROR: Petsc Development GIT revision: v3.16.3-618-gad32f7e GIT Date: 2022-01-18 16:04:31 +0000<br class="">[0]PETSC ERROR: ../../../moose_test-dbg on a arch-linux-c-opt named r8i3n0 by kongf Wed Jan 19 08:30:13 2022<br class="">[0]PETSC ERROR: Configure options --with-debugging=no --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 --download-ptscotch=1 --download-parmetis=1 --download-mumps=1 --download-strumpack=1 --download-scalapack=1 --download-slepc=1 --with-mpi=1 --with-cxx-dialect=C++14 --with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices --with-make-np=24 --with-cuda --with-cudac=nvcc --with-cuda-arch=70 --download-kokkos=1<br class="">[0]PETSC ERROR: #1 initialize() at /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:298<br class="">[0]PETSC ERROR: #2 PetscDeviceInitializeTypeFromOptions_Private() at /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/device/interface/device.cxx:299<br class="">[0]PETSC ERROR: #3 PetscDeviceInitializeFromOptions_Internal() at /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/device/interface/device.cxx:425<br class="">[0]PETSC ERROR: #4 PetscInitialize_Common() at /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/pinit.c:963<br class="">[0]PETSC ERROR: #5 PetscInitialize() at /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/pinit.c:1238<br class="">[0]PETSC ERROR: #6 SlepcInitialize() at /home/kongf/workhome/sawtooth/moosegpu/petsc/arch-linux-c-opt/externalpackages/git.slepc/src/sys/slepcinit.c:275<br class="">[0]PETSC ERROR: #7 LibMeshInit() at ../src/base/libmesh.C:522<br class="">[r8i3n0:mpi_rank_0][MPIDI_CH3_Abort] application called MPI_Abort(MPI_COMM_WORLD, 95) - process 0: No such file or directory (2)<br class=""><br class="">Thanks,<div class=""><br class="">Fande</div></div>
</div></blockquote></div><br class=""></div></div></blockquote></div>
<span id="cid:f_kyluucf50"><configure.log></span></div></blockquote></div><br class=""></div></div></body></html>