<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Times New Roman",serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">I was using CUDA v10.2. Switching to 9.2 gives a clean make test.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal">Shri <o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">petsc-users <petsc-users-bounces@mcs.anl.gov> on behalf of "Abhyankar, Shrirang G via petsc-users" <petsc-users@mcs.anl.gov><br>
<b>Reply-To: </b>"Abhyankar, Shrirang G" <shrirang.abhyankar@pnnl.gov><br>
<b>Date: </b>Sunday, February 23, 2020 at 3:10 PM<br>
<b>To: </b>petsc-users <petsc-users@mcs.anl.gov>, Junchao Zhang <jczhang@mcs.anl.gov><br>
<b>Subject: </b>Re: [petsc-users] Using PETSc with GPU supported SuperLU_Dist<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">I am getting an error now for CUDA driver version. Any suggestions?</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">petsc:maint$ make test</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">Running test examples to verify correct installation</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">Using PETSC_DIR=/people/abhy245/software/petsc and PETSC_ARCH=debug-mode-newell</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">See http://www.mcs.anl.gov/petsc/documentation/faq.html</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: Error in system call</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: error in cudaSetDevice CUDA driver version is insufficient for CUDA runtime version</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: Petsc Release Version 3.12.4, unknown </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: ./ex19 on a debug-mode-newell named newell01.pnl.gov by abhy245 Sun Feb 23 12:49:55 2020</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: Configure options --download-fblaslapack --download-make --download-metis --download-parmetis --download-scalapack --download-suitesparse --download-superlu_dist-gpu=1
--download-superlu_dist=1 --with-cc=mpicc --with-clanguage=c++ --with-cuda-dir=/share/apps/cuda/10.2 --with-cuda=1 --with-cxx-dialect=C++11 --with-cxx=mpicxx --with-fc=mpif77 --with-openmp=1 PETSC_ARCH=debug-mode-newell</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: #1 PetscCUDAInitialize() line 261 in /qfs/people/abhy245/software/petsc/src/sys/objects/init.c</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: #2 PetscOptionsCheckInitial_Private() line 652 in /qfs/people/abhy245/software/petsc/src/sys/objects/init.c</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: #3 PetscInitialize() line 1010 in /qfs/people/abhy245/software/petsc/src/sys/objects/pinit.c</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">--------------------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">Primary job terminated normally, but 1 process returned</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">a non-zero exit code. Per user-direction, the job has been aborted.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">--------------------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">--------------------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">mpiexec detected that one or more processes exited with non-zero status, thus causing</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">the job to be terminated. The first process to do so was:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif"> Process name: [[46518,1],0]</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif"> Exit code: 88</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">--------------------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">See http://www.mcs.anl.gov/petsc/documentation/faq.html</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[1]PETSC ERROR: Error in system call</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[1]PETSC ERROR: [0]PETSC ERROR: Error in system call</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: error in cudaGetDeviceCount CUDA driver version is insufficient for CUDA runtime version</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">error in cudaGetDeviceCount CUDA driver version is insufficient for CUDA runtime version</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[1]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[1]PETSC ERROR: [0]PETSC ERROR: Petsc Release Version 3.12.4, unknown </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: ./ex19 on a debug-mode-newell named newell01.pnl.gov by abhy245 Sun Feb 23 12:49:57 2020</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: Configure options --download-fblaslapack --download-make --download-metis --download-parmetis --download-scalapack --download-suitesparse --download-superlu_dist-gpu=1
--download-superlu_dist=1 --with-cc=mpicc --with-clanguage=c++ --with-cuda-dir=/share/apps/cuda/10.2 --with-cuda=1 --with-cxx-dialect=C++11 --with-cxx=mpicxx --with-fc=mpif77 --with-openmp=1 PETSC_ARCH=debug-mode-newell</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: #1 PetscCUDAInitialize() line 254 in /qfs/people/abhy245/software/petsc/src/sys/objects/init.c</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: #2 PetscOptionsCheckInitial_Private() line 652 in /qfs/people/abhy245/software/petsc/src/sys/objects/init.c</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: #3 PetscInitialize() line 1010 in /qfs/people/abhy245/software/petsc/src/sys/objects/pinit.c</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">Petsc Release Version 3.12.4, unknown </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[1]PETSC ERROR: ./ex19 on a debug-mode-newell named newell01.pnl.gov by abhy245 Sun Feb 23 12:49:57 2020</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[1]PETSC ERROR: Configure options --download-fblaslapack --download-make --download-metis --download-parmetis --download-scalapack --download-suitesparse --download-superlu_dist-gpu=1
--download-superlu_dist=1 --with-cc=mpicc --with-clanguage=c++ --with-cuda-dir=/share/apps/cuda/10.2 --with-cuda=1 --with-cxx-dialect=C++11 --with-cxx=mpicxx --with-fc=mpif77 --with-openmp=1 PETSC_ARCH=debug-mode-newell</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[1]PETSC ERROR: #1 PetscCUDAInitialize() line 254 in /qfs/people/abhy245/software/petsc/src/sys/objects/init.c</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[1]PETSC ERROR: #2 PetscOptionsCheckInitial_Private() line 652 in /qfs/people/abhy245/software/petsc/src/sys/objects/init.c</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[1]PETSC ERROR: #3 PetscInitialize() line 1010 in /qfs/people/abhy245/software/petsc/src/sys/objects/pinit.c</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">--------------------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">Primary job terminated normally, but 1 process returned</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">a non-zero exit code. Per user-direction, the job has been aborted.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">--------------------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">--------------------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">mpiexec detected that one or more processes exited with non-zero status, thus causing</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">the job to be terminated. The first process to do so was:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif"> Process name: [[46522,1],0]</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif"> Exit code: 88</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">--------------------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">1,2c1,21</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">< lid velocity = 0.0025, prandtl # = 1., grashof # = 1.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">< Number of SNES iterations = 2</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">---</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> [0]PETSC ERROR: Error in system call</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> [0]PETSC ERROR: error in cudaSetDevice CUDA driver version is insufficient for CUDA runtime version</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> [0]PETSC ERROR: Petsc Release Version 3.12.4, unknown </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> [0]PETSC ERROR: ./ex19 on a debug-mode-newell named newell01.pnl.gov by abhy245 Sun Feb 23 12:50:00 2020</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> [0]PETSC ERROR: Configure options --download-fblaslapack --download-make --download-metis --download-parmetis --download-scalapack --download-suitesparse --download-superlu_dist-gpu=1
--download-superlu_dist=1 --with-cc=mpicc --with-clanguage=c++ --with-cuda-dir=/share/apps/cuda/10.2 --with-cuda=1 --with-cxx-dialect=C++11 --with-cxx=mpicxx --with-fc=mpif77 --with-openmp=1 PETSC_ARCH=debug-mode-newell</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> [0]PETSC ERROR: #1 PetscCUDAInitialize() line 261 in /qfs/people/abhy245/software/petsc/src/sys/objects/init.c</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> [0]PETSC ERROR: #2 PetscOptionsCheckInitial_Private() line 652 in /qfs/people/abhy245/software/petsc/src/sys/objects/init.c</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> [0]PETSC ERROR: #3 PetscInitialize() line 1010 in /qfs/people/abhy245/software/petsc/src/sys/objects/pinit.c</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> --------------------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> Primary job terminated normally, but 1 process returned</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> a non-zero exit code. Per user-direction, the job has been aborted.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> --------------------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> --------------------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> mpiexec detected that one or more processes exited with non-zero status, thus causing</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> the job to be terminated. The first process to do so was:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> Process name: [[46545,1],0]</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> Exit code: 88</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">> --------------------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">/people/abhy245/software/petsc/src/snes/examples/tutorials</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">Possible problem with ex19 running with superlu_dist, diffs above</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">=========================================</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">See http://www.mcs.anl.gov/petsc/documentation/faq.html</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: Error in system call</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: error in cudaSetDevice CUDA driver version is insufficient for CUDA runtime version</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: Petsc Release Version 3.12.4, unknown </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: ./ex5f on a debug-mode-newell named newell01.pnl.gov by abhy245 Sun Feb 23 12:50:04 2020</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: Configure options --download-fblaslapack --download-make --download-metis --download-parmetis --download-scalapack --download-suitesparse --download-superlu_dist-gpu=1
--download-superlu_dist=1 --with-cc=mpicc --with-clanguage=c++ --with-cuda-dir=/share/apps/cuda/10.2 --with-cuda=1 --with-cxx-dialect=C++11 --with-cxx=mpicxx --with-fc=mpif77 --with-openmp=1 PETSC_ARCH=debug-mode-newell</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: #1 PetscCUDAInitialize() line 261 in /qfs/people/abhy245/software/petsc/src/sys/objects/init.c</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: #2 PetscOptionsCheckInitial_Private() line 652 in /qfs/people/abhy245/software/petsc/src/sys/objects/init.c</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">[0]PETSC ERROR: PetscInitialize:Checking initial options</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif"> Unable to initialize PETSc</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">--------------------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">mpiexec has exited due to process rank 0 with PID 0 on</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">node newell01 exiting improperly. There are three reasons this could occur:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">1. this process did not call "init" before exiting, but others in</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">the job did. This can cause a job to hang indefinitely while it waits</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">for all processes to call "init". By rule, if one process calls "init",</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">then ALL processes must call "init" prior to termination.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">2. this process called "init", but exited without calling "finalize".</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">By rule, all processes that call "init" MUST call "finalize" prior to</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">exiting or it will be considered an "abnormal termination"</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">3. this process called "MPI_Abort" or "orte_abort" and the mca parameter</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">orte_create_session_dirs is set to false. In this case, the run-time cannot</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">detect that the abort call was an abnormal termination. Hence, the only</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">error message you will receive is this one.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">This may have caused other processes in the application to be</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">terminated by signals sent by mpiexec (as reported here).</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">You can avoid this message by specifying -quiet on the mpiexec command line.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">--------------------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Times New Roman",serif">Completed test examples</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Satish Balay <balay@mcs.anl.gov><br>
<b>Reply-To: </b>petsc-users <petsc-users@mcs.anl.gov><br>
<b>Date: </b>Saturday, February 22, 2020 at 9:00 PM<br>
<b>To: </b>Junchao Zhang <jczhang@mcs.anl.gov><br>
<b>Cc: </b>"Abhyankar, Shrirang G" <shrirang.abhyankar@pnnl.gov>, petsc-users <petsc-users@mcs.anl.gov><br>
<b>Subject: </b>Re: [petsc-users] Using PETSc with GPU supported SuperLU_Dist</span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">The fix is now in both maint and master<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in"><a href="https://gitlab.com/petsc/petsc/-/merge_requests/2555">https://gitlab.com/petsc/petsc/-/merge_requests/2555</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">Satish<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">On Sat, 22 Feb 2020, Junchao Zhang via petsc-users wrote:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<p class="MsoNormal" style="margin-left:1.0in">We met the error before and knew why. Will fix it soon.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">--Junchao Zhang<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">On Sat, Feb 22, 2020 at 11:43 AM Abhyankar, Shrirang G via petsc-users <<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in"><a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> Thanks, Satish. Configure and make go through fine. Getting an undefined<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> reference error for VecGetArrayWrite_SeqCUDA.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> Shri<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> *From: *Satish Balay <<a href="mailto:balay@mcs.anl.gov">balay@mcs.anl.gov</a>><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> *Reply-To: *petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> *Date: *Saturday, February 22, 2020 at 8:25 AM<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> *To: *"Abhyankar, Shrirang G" <<a href="mailto:shrirang.abhyankar@pnnl.gov">shrirang.abhyankar@pnnl.gov</a>><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> *Cc: *"<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> *Subject: *Re: [petsc-users] Using PETSc with GPU supported SuperLU_Dist<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> On Sat, 22 Feb 2020, Abhyankar, Shrirang G via petsc-users wrote:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> Hi,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> I want to install PETSc with GPU supported SuperLU_Dist. What are the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> configure options I should be using?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> Shri,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> if self.framework.argDB['download-superlu_dist-gpu']:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> self.cuda = framework.require('config.packages.cuda',self)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> self.openmp =<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> framework.require('config.packages.openmp',self)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> self.deps =<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> [self.mpi,self.blasLapack,self.cuda,self.openmp]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <<<<<<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> So try:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> --with-cuda=1 --download-superlu_dist=1 --download-superlu_dist-gpu=1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> --with-openmp=1 [and usual MPI, blaslapack]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> Satish<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">> <o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
</div>
</div>
</body>
</html>