<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
On Apr 5, 2024, at 4: 04 PM, Frank Bramkamp <bramkamp@ nsc. liu. se> wrote: Hi Barry, The problem is that libnvJitlink. so. 12 also appears in the libraries in petsclib. so (when I check with ldd). But there, libnvJitlink. no. 12 is stated as
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannernduy1qe" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannernduy1qe" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannernduy1qe" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannernduy1qe" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html><head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannernduy1qe { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonnduy1qe:hover, .pfptPrimaryButtonnduy1qe:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonnduy1qe:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><br id="lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Apr 5, 2024, at 4:04 PM, Frank Bramkamp <bramkamp@nsc.liu.se> wrote:</div><br class="Apple-interchange-newline"><div><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">Hi Barry,<div><br></div><div>The problem is that libnvJitlink.so.12 also appears in the libraries in petsclib.so (when I check with ldd).</div><div>But there, <a href="https://urldefense.us/v3/__http://libnvjitlink.no/__;!!G_uCfscf7eWS!bS26tKsQ1ISdECsLypFMo2kft9mzG57JQkvudkUd5J9ja4oPFPbS7nunTNVXArezMHasPwVhH7BSoPfWAEckhs8$">libnvJitlink.no</a>.12 is stated as not found. And then I cannot execute my program, since this</div><div>Library is not found. We typically do not want to add LD_LIBRARY_PATH. </div></div></div></blockquote><div><br></div> Ok, it is a bit confusing. </div><div><br></div><div>Contents /software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64:</div><div><br></div><div> ['libnppif.so.12', 'libcudart.so.12', 'libnppitc.so.12.1.1.14', 'libcuinj64.so.12.2', 'libnppc.so.12.1.1.14', 'libcuinj64.so.12.2.60', 'libnppim.so.12.1.1.14', 'libnppig.so', 'libnvjpeg.so.12.1.1.14', 'libnppidei.so.12.1.1.14', 'libnppidei.so.12', 'libnppial.so.12.1.1.14', 'libnppisu_static.a', 'libcudadevrt.a', 'libnppim.so', 'libnvToolsExt.so.1.0.0', 'libnpps.so.12.1.1.14', 'libaccinj64.so.12.2', 'libnppidei_static.a', 'libnvJitLink_static.a', 'libnppif.so', 'libnppial.so', 'libnvrtc-builtins.so', 'libcufilt.a', 'libnppig.so.12.1.1.14', 'libnvjpeg.so', 'libnvjpeg.so.12', 'libnppial_static.a', 'libnppicc_static.a', 'libnppidei.so', 'libnppist.so.12', 'cmake', 'libOpenCL.so.1.0', 'libnppisu.so', 'libnvrtc-builtins.so.12.2.91', 'libnpps.so', 'libnvrtc.so.12', 'libnppim_static.a', 'libcudart_static.a', 'libcuinj64.so', 'libnppicc.so.12.1.1.14', 'libOpenCL.so', 'libnvToolsExt.so', 'libnppig_static.a', 'libOpenCL.so.1.0.0', 'libnppc.so', 'libnppif_static.a', 'libaccinj64.so', 'libnvrtc_static.a', 'stubs', 'libnvJitLink.so.12.2.91', 'libnvrtc.so', 'libnvrtc-builtins_static.a', 'libnppig.so.12', 'libnppist.so', 'libnppif.so.12.1.1.14', 'libnppicc.so.12', 'libnppitc_static.a', 'libnppial.so.12', 'libnppim.so.12', 'libnppitc.so', 'libnpps.so.12', 'libnppist_static.a', 'libnpps_static.a', 'libnppist.so.12.1.1.14', 'libnppc.so.12', 'libcudart.so.12.2.53', 'libnvrtc-builtins.so.12.2', 'libnppisu.so.12.1.1.14', 'libculibos.a', 'libnvJitLink.so', 'libnvptxcompiler_static.a', 'libaccinj64.so.12.2.60', 'libnppitc.so.12', 'libnppisu.so.12', 'libOpenCL.so.1', 'libnvToolsExt.so.1', 'libnvrtc.so.12.2.91', 'libnppicc.so', 'libnppc_static.a', 'libcudart.so', 'libnvJitLink.so.12', 'libnvjpeg_static.a']</div><div><br></div><div>Executing: mpicc -o /tmp/petsc-h7tpd5_s/config.packages.BlasLapack/conftest -KPIC -O2 -g /tmp/petsc-h7tpd5_s/config.packages.BlasLapack/conftest.o -Wl,-rpath,/proj/nsc/users/bramkamp/petsc_install/petsc_barry_fix_nvclib_with_cuda_fixed_module/lib -L/proj/nsc/users/bramkamp/petsc_install/petsc_barry_fix_nvclib_with_cuda_fixed_module/lib -lflapack -Wl,-rpath,/proj/nsc/users/bramkamp/petsc_install/petsc_barry_fix_nvclib_with_cuda_fixed_module/lib -L/proj/nsc/users/bramkamp/petsc_install/petsc_barry_fix_nvclib_with_cuda_fixed_module/lib -lfblas</div><div><br></div><div> -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64</div><div><br></div><div> -L/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64 -lcudart -lnvToolsExt -lcufft -lcublas -lcusparse -lcusolver -lcurand -L/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64/stubs -lcuda -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/comm_libs/mpi/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/comm_libs/mpi/lib -L/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/comm_libs/mpi/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/FFTW/3.3.10/nv23.7/hpc1/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/FFTW/3.3.10/nv23.7/hpc1/lib -L/software/sse2/tetralith_el9/manual/FFTW/3.3.10/nv23.7/hpc1/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/comm_libs/nvshmem/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/comm_libs/nvshmem/lib -L/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/comm_libs/nvshmem/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/comm_libs/nccl/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/comm_libs/nccl/lib -L/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/comm_libs/nccl/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/math_libs/lib64 -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/math_libs/lib64 -L/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/math_libs/lib64 -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/compilers/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/compilers/lib -L/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/compilers/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/compilers/extras/qd/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/compilers/extras/qd/lib -L/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/compilers/extras/qd/lib -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/extras/CUPTI/lib64 -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/extras/CUPTI/lib64 -L/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/extras/CUPTI/lib64 -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64 -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64 -L/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/11 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/11 -L/usr/lib/gcc/x86_64-redhat-linux/11 -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/comm_libs/mpi/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/compilers/lib -lnvf -lnvomp -ldl -lnvhpcatm -latomic -lpthread -lnvcpumath -lnsnvc -lrt -lgcc_s -lm -lquadmath </div><div><br></div><div><br></div><div>We see that libnvJitLink.so.12 is in /software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64 </div><div><br></div><div>Then whenit links the executable (above) it passes -Wl,-rpath,/software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64 to the linker so that at</div><div>run time, it should be able to find all the libraries in /software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64. </div><div><br></div><div>Is /software/sse2/tetralith_el9/manual/nvhpc/23.7/Linux_x86_64/23.7/cuda/12.2/lib64/libnvJitLink.so.12 a link to something that actually exists.</div><div><br></div><div><blockquote type="cite"><div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div><div>At the moment I am not sure where it tries to find that library. Therefore I thought that maybe the problem</div><div>is that BlasLapack could put some path in the library, which does not exist. At the beginning of configure.log</div><div>it mentions libnvJitlink.so.12, but then it seems to get lost somewhere.</div><div><br></div><div>I have to see again if there is already a problem when I make petsc check, or if it is just in my program later.</div><div>Not quite sure anymore.</div><div><br></div><div><br></div><div>I will write back next week, Frank</div><div><br></div><div><br></div><div><br></div><div><br><div><br><blockquote type="cite"><div>On 5 Apr 2024, at 19:47, Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>> wrote:</div><br class="Apple-interchange-newline"><div><div><br> Thanks for the configure.log Send the configure.log for the failed nvJitlink problem.<br><br><br><blockquote type="cite">On Apr 5, 2024, at 12:58 PM, Frank Bramkamp <<a href="mailto:bramkamp@nsc.liu.se">bramkamp@nsc.liu.se</a>> wrote:<br><br>Hi Barry,<br><br>Here comes the latest configure.log file<br><br>My cuda nvJitlink problem unfortunately still exists. <br>I will try it on a different cluster to see if this a specific problem of the actual nvhpc installation.<br><br><br>Have a nice weekend, Frank<br><br><br><br><configure.log><br></blockquote><br></div></div></blockquote></div><br></div></div></div></blockquote></div><br></body></html>