<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hello,<div class=""><br class=""/></div><div class="">I was hoping to get help regarding a runtime error I am encountering <span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">on a cluster node with 4 Tesla K40m GPUs</span> after configuring PETSc with the following command:</div><div class=""><br class=""/></div><div class="">$./configure --force \</div><div class=""> --with-precision=double \</div><div class=""> --with-debugging=0 \</div><div class=""> --COPTFLAGS=-O3 \</div><div class=""> --CXXOPTFLAGS=-O3 \</div><div class=""> --FOPTFLAGS=-O3 \</div><div class=""> PETSC_ARCH=linux64GccDPInt32-spack \</div><div class=""> --download-fblaslapack \</div><div class=""> --download-openblas \</div><div class=""> --download-hypre \</div><div class=""> --download-hypre-configure-arguments=--enable-unified-memory \</div><div class=""> --with-mpi-dir=/opt/ohpc/pub/mpi/openmpi4-gnu9/4.0.4 \</div><div class=""> --with-cuda=1 \</div><div class=""> --download-suitesparse \</div><div class=""> --download-dir=downloads \</div><div class=""> --with-cudac=/opt/ohpc/admin/spack/0.15.0/opt/spack/linux-centos8-ivybridge/gcc-9.3.0/cuda-11.7.0-hel25vgwc7fixnvfl5ipvnh34fnskw3m/bin/nvcc \</div><div class=""> --with-packages-download-dir=downloads \</div><div class=""> --download-sowing=downloads/v1.1.26-p4.tar.gz \</div><div class=""> --with-cuda-arch=35</div><br class=""/><div class="">When I run</div><div class=""><br class=""/></div><div class="">$ make PETSC_DIR=/home/juan/OpenFOAM/juan-v2206/petsc-cuda PETSC_ARCH=linux64GccDPInt32-spack check</div><div class="">Running check examples to verify correct installation<br class=""/>Using PETSC_DIR=/home/juan/OpenFOAM/juan-v2206/petsc-cuda and PETSC_ARCH=linux64GccDPInt32-spack<br class=""/>C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process<br class=""/>C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes<br class=""/>3,5c3,15<br class=""/>< 1 SNES Function norm 4.12227e-06 <br class=""/>< 2 SNES Function norm 6.098e-11 <br class=""/>< Number of SNES iterations = 2<br class=""/>---<br class=""/>> CUDA ERROR (code = 101, invalid device ordinal) at memory.c:139<br class=""/>> CUDA ERROR (code = 101, invalid device ordinal) at memory.c:139<br class=""/>> --------------------------------------------------------------------------<br class=""/>> Primary job terminated normally, but 1 process returned<br class=""/>> a non-zero exit code. Per user-direction, the job has been aborted.<br class=""/>> --------------------------------------------------------------------------<br class=""/>> --------------------------------------------------------------------------<br class=""/>> mpiexec detected that one or more processes exited with non-zero status, thus causing<br class=""/>> the job to be terminated. The first process to do so was:<br class=""/>> <br class=""/>> Process name: [[52712,1],0]<br class=""/>> Exit code: 1<br class=""/>> --------------------------------------------------------------------------<br class=""/>/home/juan/OpenFOAM/juan-v2206/petsc-cuda/src/snes/tutorials<br class=""/>Possible problem with ex19 running with hypre, diffs above<br class=""/>=========================================<br class=""/>C/C++ example src/snes/tutorials/ex19 run successfully with cuda<br class=""/>C/C++ example src/snes/tutorials/ex19 run successfully with suitesparse<br class=""/>Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process<br class=""/>Completed test examples<br class=""/><br class=""/></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">I have compiled the code on the head node (without GPUs) and on the compute node where there are 4 GPUs. </div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""/></div><div class=""><font color="#000000" class=""><span style="caret-color: rgb(0, 0, 0);" class="">$nvidia-debugdump -l<br class=""/>Found 4 NVIDIA devices<br class=""/><span class="Apple-tab-span" style="white-space: pre;"> </span>Device ID: 0<br class=""/><span class="Apple-tab-span" style="white-space: pre;"> </span>Device name: Tesla K40m<br class=""/><span class="Apple-tab-span" style="white-space: pre;"> </span>GPU internal ID: 0320717032250<br class=""/><br class=""/><span class="Apple-tab-span" style="white-space: pre;"> </span>Device ID: 1<br class=""/><span class="Apple-tab-span" style="white-space: pre;"> </span>Device name: Tesla K40m<br class=""/><span class="Apple-tab-span" style="white-space: pre;"> </span>GPU internal ID: 0320717031968<br class=""/><br class=""/><span class="Apple-tab-span" style="white-space: pre;"> </span>Device ID: 2<br class=""/><span class="Apple-tab-span" style="white-space: pre;"> </span>Device name: Tesla K40m<br class=""/><span class="Apple-tab-span" style="white-space: pre;"> </span>GPU internal ID: 0320717032246<br class=""/><br class=""/><span class="Apple-tab-span" style="white-space: pre;"> </span>Device ID: 3<br class=""/><span class="Apple-tab-span" style="white-space: pre;"> </span>Device name: Tesla K40m<br class=""/><span class="Apple-tab-span" style="white-space: pre;"> </span>GPU internal ID: 0320717032235</span></font></div><div class=""><font color="#000000" class=""><span style="caret-color: rgb(0, 0, 0);" class=""><br class=""/></span></font></div><div class=""><font color="#000000" class=""><span style="caret-color: rgb(0, 0, 0);" class="">Attached are the log files form configure and make.</span></font></div><div class=""><font color="#000000" class=""><span style="caret-color: rgb(0, 0, 0);" class=""><br class=""/></span></font></div><div class=""><font color="#000000" class="">Any pointers are highly appreciated. My intention is to use PETSc as a linear solver for OpenFOAM, leveraging the availability of GPUs at the same time. Currently I can run PETSc without GPU support. </font></div><div class=""><font color="#000000" class=""><br class=""/></font></div><div class=""><font color="#000000" class="">Cheers,</font></div><div class=""><font color="#000000" class="">Juan S.</font></div><div class=""><font color="#000000" class=""><br class=""/></font></div><div class=""></div></body></html><html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/><div></div></body></html><html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""></div><div class=""><font color="#000000" class=""><br class=""/></font></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""/></div><div class=""><br class=""/></div><div class=""><br class=""/></div></body></html>