<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr"><div><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: error in cudaSetDevice CUDA driver version is insufficient for CUDA runtime version</span><br></div><div><span style="font-family:"Times New Roman",serif"><br></span></div><div><font face="Times New Roman, serif">That means you need to update your cuda driver for CUDA 10.2. See minimum requirement in Table 1 at </font><a href="https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#major-components">https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#major-components</a></div><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Feb 23, 2020 at 3:33 PM Abhyankar, Shrirang G <<a href="mailto:shrirang.abhyankar@pnnl.gov">shrirang.abhyankar@pnnl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_-2292936989468834930WordSection1">
<p class="MsoNormal"><span style="font-size:12pt;font-family:"Times New Roman",serif">I was using CUDA v10.2. Switching to 9.2 gives a clean make test.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt;font-family:"Times New Roman",serif"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
<p class="MsoNormal">Shri <u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-size:12pt;font-family:"Times New Roman",serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt;font-family:"Times New Roman",serif"><u></u> <u></u></span></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0in 0in">
<p class="MsoNormal" style="margin-left:0.5in"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">petsc-users <<a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.gov</a>> on behalf of "Abhyankar, Shrirang G via petsc-users" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Reply-To: </b>"Abhyankar, Shrirang G" <<a href="mailto:shrirang.abhyankar@pnnl.gov" target="_blank">shrirang.abhyankar@pnnl.gov</a>><br>
<b>Date: </b>Sunday, February 23, 2020 at 3:10 PM<br>
<b>To: </b>petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>, Junchao Zhang <<a href="mailto:jczhang@mcs.anl.gov" target="_blank">jczhang@mcs.anl.gov</a>><br>
<b>Subject: </b>Re: [petsc-users] Using PETSc with GPU supported SuperLU_Dist<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">I am getting an error now for CUDA driver version. Any suggestions?</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">petsc:maint$ make test</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">Running test examples to verify correct installation</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">Using PETSC_DIR=/people/abhy245/software/petsc and PETSC_ARCH=debug-mode-newell</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a></span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: Error in system call</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: error in cudaSetDevice CUDA driver version is insufficient for CUDA runtime version</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: Petsc Release Version 3.12.4, unknown </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: ./ex19 on a debug-mode-newell named <a href="http://newell01.pnl.gov" target="_blank">newell01.pnl.gov</a> by abhy245 Sun Feb 23 12:49:55 2020</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: Configure options --download-fblaslapack --download-make --download-metis --download-parmetis --download-scalapack --download-suitesparse --download-superlu_dist-gpu=1
--download-superlu_dist=1 --with-cc=mpicc --with-clanguage=c++ --with-cuda-dir=/share/apps/cuda/10.2 --with-cuda=1 --with-cxx-dialect=C++11 --with-cxx=mpicxx --with-fc=mpif77 --with-openmp=1 PETSC_ARCH=debug-mode-newell</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: #1 PetscCUDAInitialize() line 261 in /qfs/people/abhy245/software/petsc/src/sys/objects/init.c</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: #2 PetscOptionsCheckInitial_Private() line 652 in /qfs/people/abhy245/software/petsc/src/sys/objects/init.c</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: #3 PetscInitialize() line 1010 in /qfs/people/abhy245/software/petsc/src/sys/objects/pinit.c</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">--------------------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">Primary job terminated normally, but 1 process returned</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">a non-zero exit code. Per user-direction, the job has been aborted.</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">--------------------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">--------------------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">mpiexec detected that one or more processes exited with non-zero status, thus causing</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">the job to be terminated. The first process to do so was:</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif"> Process name: [[46518,1],0]</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif"> Exit code: 88</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">--------------------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a></span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[1]PETSC ERROR: Error in system call</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[1]PETSC ERROR: [0]PETSC ERROR: Error in system call</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: error in cudaGetDeviceCount CUDA driver version is insufficient for CUDA runtime version</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">error in cudaGetDeviceCount CUDA driver version is insufficient for CUDA runtime version</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[1]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[1]PETSC ERROR: [0]PETSC ERROR: Petsc Release Version 3.12.4, unknown </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: ./ex19 on a debug-mode-newell named <a href="http://newell01.pnl.gov" target="_blank">newell01.pnl.gov</a> by abhy245 Sun Feb 23 12:49:57 2020</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: Configure options --download-fblaslapack --download-make --download-metis --download-parmetis --download-scalapack --download-suitesparse --download-superlu_dist-gpu=1
--download-superlu_dist=1 --with-cc=mpicc --with-clanguage=c++ --with-cuda-dir=/share/apps/cuda/10.2 --with-cuda=1 --with-cxx-dialect=C++11 --with-cxx=mpicxx --with-fc=mpif77 --with-openmp=1 PETSC_ARCH=debug-mode-newell</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: #1 PetscCUDAInitialize() line 254 in /qfs/people/abhy245/software/petsc/src/sys/objects/init.c</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: #2 PetscOptionsCheckInitial_Private() line 652 in /qfs/people/abhy245/software/petsc/src/sys/objects/init.c</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: #3 PetscInitialize() line 1010 in /qfs/people/abhy245/software/petsc/src/sys/objects/pinit.c</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">Petsc Release Version 3.12.4, unknown </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[1]PETSC ERROR: ./ex19 on a debug-mode-newell named <a href="http://newell01.pnl.gov" target="_blank">newell01.pnl.gov</a> by abhy245 Sun Feb 23 12:49:57 2020</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[1]PETSC ERROR: Configure options --download-fblaslapack --download-make --download-metis --download-parmetis --download-scalapack --download-suitesparse --download-superlu_dist-gpu=1
--download-superlu_dist=1 --with-cc=mpicc --with-clanguage=c++ --with-cuda-dir=/share/apps/cuda/10.2 --with-cuda=1 --with-cxx-dialect=C++11 --with-cxx=mpicxx --with-fc=mpif77 --with-openmp=1 PETSC_ARCH=debug-mode-newell</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[1]PETSC ERROR: #1 PetscCUDAInitialize() line 254 in /qfs/people/abhy245/software/petsc/src/sys/objects/init.c</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[1]PETSC ERROR: #2 PetscOptionsCheckInitial_Private() line 652 in /qfs/people/abhy245/software/petsc/src/sys/objects/init.c</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[1]PETSC ERROR: #3 PetscInitialize() line 1010 in /qfs/people/abhy245/software/petsc/src/sys/objects/pinit.c</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">--------------------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">Primary job terminated normally, but 1 process returned</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">a non-zero exit code. Per user-direction, the job has been aborted.</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">--------------------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">--------------------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">mpiexec detected that one or more processes exited with non-zero status, thus causing</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">the job to be terminated. The first process to do so was:</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif"> Process name: [[46522,1],0]</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif"> Exit code: 88</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">--------------------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">1,2c1,21</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">< lid velocity = 0.0025, prandtl # = 1., grashof # = 1.</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">< Number of SNES iterations = 2</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">---</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> [0]PETSC ERROR: Error in system call</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> [0]PETSC ERROR: error in cudaSetDevice CUDA driver version is insufficient for CUDA runtime version</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> [0]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> [0]PETSC ERROR: Petsc Release Version 3.12.4, unknown </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> [0]PETSC ERROR: ./ex19 on a debug-mode-newell named <a href="http://newell01.pnl.gov" target="_blank">newell01.pnl.gov</a> by abhy245 Sun Feb 23 12:50:00 2020</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> [0]PETSC ERROR: Configure options --download-fblaslapack --download-make --download-metis --download-parmetis --download-scalapack --download-suitesparse --download-superlu_dist-gpu=1
--download-superlu_dist=1 --with-cc=mpicc --with-clanguage=c++ --with-cuda-dir=/share/apps/cuda/10.2 --with-cuda=1 --with-cxx-dialect=C++11 --with-cxx=mpicxx --with-fc=mpif77 --with-openmp=1 PETSC_ARCH=debug-mode-newell</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> [0]PETSC ERROR: #1 PetscCUDAInitialize() line 261 in /qfs/people/abhy245/software/petsc/src/sys/objects/init.c</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> [0]PETSC ERROR: #2 PetscOptionsCheckInitial_Private() line 652 in /qfs/people/abhy245/software/petsc/src/sys/objects/init.c</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> [0]PETSC ERROR: #3 PetscInitialize() line 1010 in /qfs/people/abhy245/software/petsc/src/sys/objects/pinit.c</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> --------------------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> Primary job terminated normally, but 1 process returned</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> a non-zero exit code. Per user-direction, the job has been aborted.</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> --------------------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> --------------------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> mpiexec detected that one or more processes exited with non-zero status, thus causing</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> the job to be terminated. The first process to do so was:</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> Process name: [[46545,1],0]</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> Exit code: 88</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">> --------------------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">/people/abhy245/software/petsc/src/snes/examples/tutorials</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">Possible problem with ex19 running with superlu_dist, diffs above</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">=========================================</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a></span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: Error in system call</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: error in cudaSetDevice CUDA driver version is insufficient for CUDA runtime version</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: Petsc Release Version 3.12.4, unknown </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: ./ex5f on a debug-mode-newell named <a href="http://newell01.pnl.gov" target="_blank">newell01.pnl.gov</a> by abhy245 Sun Feb 23 12:50:04 2020</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: Configure options --download-fblaslapack --download-make --download-metis --download-parmetis --download-scalapack --download-suitesparse --download-superlu_dist-gpu=1
--download-superlu_dist=1 --with-cc=mpicc --with-clanguage=c++ --with-cuda-dir=/share/apps/cuda/10.2 --with-cuda=1 --with-cxx-dialect=C++11 --with-cxx=mpicxx --with-fc=mpif77 --with-openmp=1 PETSC_ARCH=debug-mode-newell</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: #1 PetscCUDAInitialize() line 261 in /qfs/people/abhy245/software/petsc/src/sys/objects/init.c</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: #2 PetscOptionsCheckInitial_Private() line 652 in /qfs/people/abhy245/software/petsc/src/sys/objects/init.c</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: PetscInitialize:Checking initial options</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif"> Unable to initialize PETSc</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">--------------------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">mpiexec has exited due to process rank 0 with PID 0 on</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">node newell01 exiting improperly. There are three reasons this could occur:</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">1. this process did not call "init" before exiting, but others in</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">the job did. This can cause a job to hang indefinitely while it waits</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">for all processes to call "init". By rule, if one process calls "init",</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">then ALL processes must call "init" prior to termination.</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">2. this process called "init", but exited without calling "finalize".</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">By rule, all processes that call "init" MUST call "finalize" prior to</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">exiting or it will be considered an "abnormal termination"</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">3. this process called "MPI_Abort" or "orte_abort" and the mca parameter</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">orte_create_session_dirs is set to false. In this case, the run-time cannot</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">detect that the abort call was an abnormal termination. Hence, the only</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">error message you will receive is this one.</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">This may have caused other processes in the application to be</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">terminated by signals sent by mpiexec (as reported here).</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">You can avoid this message by specifying -quiet on the mpiexec command line.</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">--------------------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-family:"Times New Roman",serif">Completed test examples</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1in"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">Satish Balay <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>><br>
<b>Reply-To: </b>petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Date: </b>Saturday, February 22, 2020 at 9:00 PM<br>
<b>To: </b>Junchao Zhang <<a href="mailto:jczhang@mcs.anl.gov" target="_blank">jczhang@mcs.anl.gov</a>><br>
<b>Cc: </b>"Abhyankar, Shrirang G" <<a href="mailto:shrirang.abhyankar@pnnl.gov" target="_blank">shrirang.abhyankar@pnnl.gov</a>>, petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject: </b>Re: [petsc-users] Using PETSc with GPU supported SuperLU_Dist</span><u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:1in"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">The fix is now in both maint and master<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in"><a href="https://gitlab.com/petsc/petsc/-/merge_requests/2555" target="_blank">https://gitlab.com/petsc/petsc/-/merge_requests/2555</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">Satish<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">On Sat, 22 Feb 2020, Junchao Zhang via petsc-users wrote:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in"> <u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:4.5pt solid rgb(181,196,223);padding:0in 0in 0in 4pt;margin:5pt 0in 5pt 3.75pt" id="gmail-m_-2292936989468834930MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<p class="MsoNormal" style="margin-left:1in">We met the error before and knew why. Will fix it soon.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">--Junchao Zhang<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">On Sat, Feb 22, 2020 at 11:43 AM Abhyankar, Shrirang G via petsc-users <<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in"><a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> Thanks, Satish. Configure and make go through fine. Getting an undefined<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> reference error for VecGetArrayWrite_SeqCUDA.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> Shri<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> *From: *Satish Balay <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> *Reply-To: *petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> *Date: *Saturday, February 22, 2020 at 8:25 AM<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> *To: *"Abhyankar, Shrirang G" <<a href="mailto:shrirang.abhyankar@pnnl.gov" target="_blank">shrirang.abhyankar@pnnl.gov</a>><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> *Cc: *"<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> *Subject: *Re: [petsc-users] Using PETSc with GPU supported SuperLU_Dist<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> On Sat, 22 Feb 2020, Abhyankar, Shrirang G via petsc-users wrote:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> Hi,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> I want to install PETSc with GPU supported SuperLU_Dist. What are the<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> configure options I should be using?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> Shri,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> if self.framework.argDB['download-superlu_dist-gpu']:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> self.cuda = framework.require('config.packages.cuda',self)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> self.openmp =<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> framework.require('config.packages.openmp',self)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> self.deps =<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> [self.mpi,self.blasLapack,self.cuda,self.openmp]<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <<<<<<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> So try:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> --with-cuda=1 --download-superlu_dist=1 --download-superlu_dist-gpu=1<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> --with-openmp=1 [and usual MPI, blaslapack]<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> Satish<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in">> <u></u><u></u></p>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:1in"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1in"> <u></u><u></u></p>
</div>
</div>
</div>
</blockquote></div>