[petsc-users] Problem with NVIDIA compiler and OpenACC

[Frank Bramkamp]

Dear PETSC Team,

I found the following problem:
I compile petsc 3.20.5 with Nvidia compiler 23.7.


I use a pretty standard configuration, including

--with-cc=mpicc --with-cxx=mpicxx --with-fc=mpifort COPTFLAGS="-O2 -g" CXXOPTFLAGS="-O2 -g" FOPTFLAGS="-O2 -g"  --with-debugging=0 --with-log=1 --download-fblaslapack --with-cuda=0

I exclude cuda, since I was not sure if the problem was cuda related. 


The problem is now, if I have s simple fortran program where I link the petsc library, but I actually do not use petsc in that program
(Just for testing). I want to use OpenACC directives in my program, e.g. !$acc parallel loop .
The problem is now, as soon I link with the petsc library, the openacc commands do not work anymore.
It seems that openacc is not initialised and hence it cannot find a GPU.

The problem seems that you link with -lnvc.
In “petscvariables” => PETSC_WITH_EXTERNAL_LIB you include “-lnvc”.
If I take this out, then openacc works. With “-lnvc” something gets messed up.

The problem is also discussed here:
https://urldefense.us/v3/__https://forums.developer.nvidia.com/t/failed-cuda-device-detection-when-explicitly-linking-libnvc/203225/1__;!!G_uCfscf7eWS!dlXNyKBzSbximQ13OXxwO506OF71yRM_H5KEnarqXE75D6Vg-ePZr2u6SJ5V3YpRETatvb9pMOUVmpyN0-19SFlbug$  <https://urldefense.us/v3/__https://forums.developer.nvidia.com/t/failed-cuda-device-detection-when-explicitly-linking-libnvc/203225/1__;!!G_uCfscf7eWS!dlXNyKBzSbximQ13OXxwO506OF71yRM_H5KEnarqXE75D6Vg-ePZr2u6SJ5V3YpRETatvb9pMOUVmpyN0-19SFlbug$ >

My understanding is that libnvc is more a runtime library that does not need to be included by the linker.
Not sure if there is a specific reason to include libnvc (I am not so familiar what this library does).

If I take out -lnvc from “petscvariables”, then my program with openacc works as expected. I did not try any more realistic program that includes petsc.



2)
When compiling petsc with cuda support, I also found that in the petsc library the library libnvJitLink.so.12
Is not found. On my system this library is in $CUDA_ROOT/lib64
I am not sure where this library is on your system ?! 


Thanks a lot, Frank Bramkamp











-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240404/9b0d9775/attachment-0001.html>