[petsc-users] Problem with NVIDIA compiler and OpenACC

Fri Apr 5 15:14:59 CDT 2024

> On Apr 5, 2024, at 4:04 PM, Frank Bramkamp <bramkamp at nsc.liu.se> wrote:
> 
> Hi Barry,
> 
> The problem is that libnvJitlink.so.12 also appears in the libraries in petsclib.so (when I check with ldd).
> But there, libnvJitlink.no <https://urldefense.us/v3/__http://libnvjitlink.no/__;!!G_uCfscf7eWS!bS26tKsQ1ISdECsLypFMo2kft9mzG57JQkvudkUd5J9ja4oPFPbS7nunTNVXArezMHasPwVhH7BSoPfWAEckhs8$ >.12 is stated as not found. And then I cannot execute my program, since this
> Library is not found. We typically do not want to add LD_LIBRARY_PATH. 

  Ok, it is a bit confusing. 

Contents /software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64:

 ['libnppif.so.12', 'libcudart.so.12', 'libnppitc.so.12.1.1.14', 'libcuinj64.so.12.2', 'libnppc.so.12.1.1.14', 'libcuinj64.so.12.2.60', 'libnppim.so.12.1.1.14', 'libnppig.so', 'libnvjpeg.so.12.1.1.14', 'libnppidei.so.12.1.1.14', 'libnppidei.so.12', 'libnppial.so.12.1.1.14', 'libnppisu_static.a', 'libcudadevrt.a', 'libnppim.so', 'libnvToolsExt.so.1.0.0', 'libnpps.so.12.1.1.14', 'libaccinj64.so.12.2', 'libnppidei_static.a', 'libnvJitLink_static.a', 'libnppif.so', 'libnppial.so', 'libnvrtc-builtins.so', 'libcufilt.a', 'libnppig.so.12.1.1.14', 'libnvjpeg.so', 'libnvjpeg.so.12', 'libnppial_static.a', 'libnppicc_static.a', 'libnppidei.so', 'libnppist.so.12', 'cmake', 'libOpenCL.so.1.0', 'libnppisu.so', 'libnvrtc-builtins.so.12.2.91', 'libnpps.so', 'libnvrtc.so.12', 'libnppim_static.a', 'libcudart_static.a', 'libcuinj64.so', 'libnppicc.so.12.1.1.14', 'libOpenCL.so', 'libnvToolsExt.so', 'libnppig_static.a', 'libOpenCL.so.1.0.0', 'libnppc.so', 'libnppif_static.a', 'libaccinj64.so', 'libnvrtc_static.a', 'stubs', 'libnvJitLink.so.12.2.91', 'libnvrtc.so', 'libnvrtc-builtins_static.a', 'libnppig.so.12', 'libnppist.so', 'libnppif.so.12.1.1.14', 'libnppicc.so.12', 'libnppitc_static.a', 'libnppial.so.12', 'libnppim.so.12', 'libnppitc.so', 'libnpps.so.12', 'libnppist_static.a', 'libnpps_static.a', 'libnppist.so.12.1.1.14', 'libnppc.so.12', 'libcudart.so.12.2.53', 'libnvrtc-builtins.so.12.2', 'libnppisu.so.12.1.1.14', 'libculibos.a', 'libnvJitLink.so', 'libnvptxcompiler_static.a', 'libaccinj64.so.12.2.60', 'libnppitc.so.12', 'libnppisu.so.12', 'libOpenCL.so.1', 'libnvToolsExt.so.1', 'libnvrtc.so.12.2.91', 'libnppicc.so', 'libnppc_static.a', 'libcudart.so', 'libnvJitLink.so.12', 'libnvjpeg_static.a']

Executing: mpicc  -o /tmp/petsc-h7tpd5_s/config.packages.BlasLapack/conftest    -KPIC -O2 -g /tmp/petsc-h7tpd5_s/config.packages.BlasLapack/conftest.o -Wl,-rpath,/proj/nsc/users/bramkamp/petsc_install/petsc_barry_fix_nvclib_with_cuda_fixed_module/lib -L/proj/nsc/users/bramkamp/petsc_install/petsc_barry_fix_nvclib_with_cuda_fixed_module/lib -lflapack -Wl,-rpath,/proj/nsc/users/bramkamp/petsc_install/petsc_barry_fix_nvclib_with_cuda_fixed_module/lib -L/proj/nsc/users/bramkamp/petsc_install/petsc_barry_fix_nvclib_with_cuda_fixed_module/lib -lfblas

 -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64

 -L/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64 -lcudart -lnvToolsExt -lcufft -lcublas -lcusparse -lcusolver -lcurand -L/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64/stubs -lcuda -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/comm_libs/mpi/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/comm_libs/mpi/lib -L/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/comm_libs/mpi/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/FFTW/3.3.10/nv23.7/hpc1/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/FFTW/3.3.10/nv23.7/hpc1/lib -L/software/sse2/tetralith_el9/manual/FFTW/3.3.10/nv23.7/hpc1/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/comm_libs/nvshmem/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/comm_libs/nvshmem/lib -L/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/comm_libs/nvshmem/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/comm_libs/nccl/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/comm_libs/nccl/lib -L/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/comm_libs/nccl/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/math_libs/lib64 -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/math_libs/lib64 -L/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/math_libs/lib64 -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/compilers/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/compilers/lib -L/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/compilers/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/compilers/extras/qd/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/compilers/extras/qd/lib -L/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/compilers/extras/qd/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/extras/CUPTI/lib64 -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/extras/CUPTI/lib64 -L/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/extras/CUPTI/lib64 -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64 -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64 -L/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/11 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/11 -L/usr/lib/gcc/x86_64-redhat-linux/11 -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/comm_libs/mpi/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/compilers/lib -lnvf -lnvomp -ldl -lnvhpcatm -latomic -lpthread -lnvcpumath -lnsnvc -lrt -lgcc_s -lm -lquadmath 

We see that libnvJitLink.so.12 is in /software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64 

Then whenit links the executable (above) it passes -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64 to the linker so that at
run time, it should be able to find all the libraries in /software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64. 

Is  /software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64/libnvJitLink.so.12 a link to something that actually exists.

> 
> At the moment I am not sure where it tries to find that library. Therefore I thought that maybe the problem
> is that BlasLapack could put some path in the library, which does not exist. At the beginning of configure.log
> it mentions  libnvJitlink.so.12, but then it seems to get lost somewhere.
> 
> I have to see again if there is already a problem when I make petsc check, or if it is just in my program later.
> Not quite sure anymore.
> 
> 
> I will write back next week, Frank
> 
> 
> 
> 
> 
>> On 5 Apr 2024, at 19:47, Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> wrote:
>> 
>> 
>>   Thanks for the configure.log Send the configure.log for the failed nvJitlink problem.
>> 
>> 
>>> On Apr 5, 2024, at 12:58 PM, Frank Bramkamp <bramkamp at nsc.liu.se <mailto:bramkamp at nsc.liu.se>> wrote:
>>> 
>>> Hi Barry,
>>> 
>>> Here comes the latest configure.log file
>>> 
>>> My cuda nvJitlink problem unfortunately still exists. 
>>> I will try it on a different cluster to see if this a specific problem of the actual nvhpc installation.
>>> 
>>> 
>>> Have a nice weekend, Frank
>>> 
>>> 
>>> 
>>> <configure.log>
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240405/20dfdae5/attachment.html>