<div dir="ltr"><div dir="ltr">Yes, I had the hsa lib commented out but that did not help (appended).</div><div dir="ltr"><br></div><div>I now see that I had this problem in July and Junchao was helping. I was able to fix it with PrgEnv-gnu.</div><div><br></div><div>THe fortran test actually worked.</div><div><br></div><div>Oh well, the application does their own linking so maybe that will fix it up. (They do use OMP).</div><div><br></div><div>Thanks,</div><div>Mark</div><div dir="ltr"><br></div><div dir="ltr">gmake[3]: [/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib/petsc/conf/rules:301: ex19.PETSc] Error 2 (ignored)<br>*******************Error detected during compile or link!*******************<br>See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><br>/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19<br>*********************************************************************************<br>cc <b>-L/opt/rocm-4.2.0/lib -lhsa-runtime64</b> -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O2 -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/include -I/opt/rocm-4.2.0/include ex19.c -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib -Wl,-rpath,/opt/rocm-4.2.0/lib -L/opt/rocm-4.2.0/lib -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib64 -L/opt/cray/pe/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/<a href="http://21.06.1.1/CRAY/9.0/x86_64/lib">21.06.1.1/CRAY/9.0/x86_64/lib</a> -L/opt/cray/pe/libsci/<a href="http://21.06.1.1/CRAY/9.0/x86_64/lib">21.06.1.1/CRAY/9.0/x86_64/lib</a> -Wl,-rpath,/opt/cray/pe/mpich/8.1.7/ofi/cray/10.0/lib -L/opt/cray/pe/mpich/8.1.7/ofi/cray/10.0/lib -Wl,-rpath,/opt/cray/pe/mpich/default/gtl/lib -L/opt/cray/pe/mpich/default/gtl/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.5/dsmml/lib -L/opt/cray/pe/dsmml/0.1.5/dsmml/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.12/lib -L/opt/cray/pe/pmi/6.0.12/lib -Wl,-rpath,/opt/cray/pe/cce/12.0.1/cce/x86_64/lib -L/opt/cray/pe/cce/12.0.1/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/cray/pe/cce/12.0.1/cce-clang/x86_64/lib/clang/12.0.0/lib/linux -L/opt/cray/pe/cce/12.0.1/cce-clang/x86_64/lib/clang/12.0.0/lib/linux -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -Wl,-rpath,/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-unknown-linux-gnu/lib -Wl,-rpath,/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -lpetsc -lparmetis -lmetis -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lrocrand -lamdhip64<b> -lhsa-runtime64 </b>-lstdc++ -ldl -lmpifort_cray -lmpi_cray -lmpi_gtl_hsa -ldsmml -lpmi -lxpmem -lpgas-shmem -lquadmath -lcrayacc_amdgpu -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -o ex19<br>ld.lld: error: /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib/libpetsc.so: undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined]<br>ld.lld: error: /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib/libpetsc.so: undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined]<br>ld.lld: error: /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib/libpetsc.so: undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined]<br>clang-12: error: linker command failed with exit code 1 (use -v to see invocation)<br>gmake[4]: *** [<builtin>: ex19] Error 1<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Sep 19, 2021 at 9:44 AM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Are you following the user advices here <a href="https://docs.olcf.ornl.gov/systems/spock_quick_start_guide.html#compiling-with-the-cray-compiler-wrappers-cc-or-cc" target="_blank">https://docs.olcf.ornl.gov/systems/spock_quick_start_guide.html#compiling-with-the-cray-compiler-wrappers-cc-or-cc</a>?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno dom 19 set 2021 alle ore 16:30 Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I am getting to see this error. It seems to be suggesting that I turn --no-allow-shlib-undefined off.<div>Any ideas?</div><div>Thanks,</div><div>Mark<br><div><br></div><div>09:09 main= /gpfs/alpine/csc314/scratch/adams/petsc$ make PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new PETSC_ARCH="" check<br>Running check examples to verify correct installation<br>Using PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new and PETSC_ARCH=<br>gmake[3]: [/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib/petsc/conf/rules:301: ex19.PETSc] Error 2 (ignored)<br>*******************Error detected during compile or link!*******************<br>See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><br>/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19<br>*********************************************************************************<br>cc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O2 -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/include -I/opt/rocm-4.2.0/include ex19.c -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib -Wl,-rpath,/opt/rocm-4.2.0/lib -L/opt/rocm-4.2.0/lib -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib64 -L/opt/cray/pe/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/<a href="http://21.06.1.1/CRAY/9.0/x86_64/lib" target="_blank">21.06.1.1/CRAY/9.0/x86_64/lib</a> -L/opt/cray/pe/libsci/<a href="http://21.06.1.1/CRAY/9.0/x86_64/lib" target="_blank">21.06.1.1/CRAY/9.0/x86_64/lib</a> -Wl,-rpath,/opt/cray/pe/mpich/8.1.7/ofi/cray/10.0/lib -L/opt/cray/pe/mpich/8.1.7/ofi/cray/10.0/lib -Wl,-rpath,/opt/cray/pe/mpich/default/gtl/lib -L/opt/cray/pe/mpich/default/gtl/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.5/dsmml/lib -L/opt/cray/pe/dsmml/0.1.5/dsmml/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.12/lib -L/opt/cray/pe/pmi/6.0.12/lib -Wl,-rpath,/opt/cray/pe/cce/12.0.1/cce/x86_64/lib -L/opt/cray/pe/cce/12.0.1/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/cray/pe/cce/12.0.1/cce-clang/x86_64/lib/clang/12.0.0/lib/linux -L/opt/cray/pe/cce/12.0.1/cce-clang/x86_64/lib/clang/12.0.0/lib/linux -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -Wl,-rpath,/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-unknown-linux-gnu/lib -Wl,-rpath,/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -lpetsc -lparmetis -lmetis -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lrocrand -lamdhip64 -lstdc++ -ldl -lmpifort_cray -lmpi_cray -lmpi_gtl_hsa -ldsmml -lpmi -lxpmem -lpgas-shmem -lquadmath -lcrayacc_amdgpu -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -o ex19<br><b>ld.lld: error: /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib/libpetsc.so: undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined]<br>ld.lld: error: /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib/libpetsc.so: undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined]<br>ld.lld: error: /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib/libpetsc.so: undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined]</b><br>clang-12: error: linker command failed with exit code 1 (use -v to see invocation)<br>gmake[4]: *** [<builtin>: ex19] Error 1<br>*******************Error detected during compile or link!*******************<br>See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><br>/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f<br>*********************************************************<br>ftn -fPIC -g -O2 -fPIC -g -O2 -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/include -I/opt/rocm-4.2.0/include ex5f.F90 -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib -Wl,-rpath,/opt/rocm-4.2.0/lib -L/opt/rocm-4.2.0/lib -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib64 -L/opt/cray/pe/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/<a href="http://21.06.1.1/CRAY/9.0/x86_64/lib" target="_blank">21.06.1.1/CRAY/9.0/x86_64/lib</a> -L/opt/cray/pe/libsci/<a href="http://21.06.1.1/CRAY/9.0/x86_64/lib" target="_blank">21.06.1.1/CRAY/9.0/x86_64/lib</a> -Wl,-rpath,/opt/cray/pe/mpich/8.1.7/ofi/cray/10.0/lib -L/opt/cray/pe/mpich/8.1.7/ofi/cray/10.0/lib -Wl,-rpath,/opt/cray/pe/mpich/default/gtl/lib -L/opt/cray/pe/mpich/default/gtl/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.5/dsmml/lib -L/opt/cray/pe/dsmml/0.1.5/dsmml/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.12/lib -L/opt/cray/pe/pmi/6.0.12/lib -Wl,-rpath,/opt/cray/pe/cce/12.0.1/cce/x86_64/lib -L/opt/cray/pe/cce/12.0.1/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/cray/pe/cce/12.0.1/cce-clang/x86_64/lib/clang/12.0.0/lib/linux -L/opt/cray/pe/cce/12.0.1/cce-clang/x86_64/lib/clang/12.0.0/lib/linux -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -Wl,-rpath,/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-unknown-linux-gnu/lib -Wl,-rpath,/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -lpetsc -lparmetis -lmetis -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lrocrand -lamdhip64 -lstdc++ -ldl -lmpifort_cray -lmpi_cray -lmpi_gtl_hsa -ldsmml -lpmi -lxpmem -lpgas-shmem -lquadmath -lcrayacc_amdgpu -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -o ex5f<br>/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: warning: alignment 128 of symbol `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in /opt/cray/pe/cce/12.0.1/cce/x86_64/lib/libmodules.so is smaller than 256 in /tmp/pe_39617/ex5f_1.o<br>/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: warning: alignment 64 of symbol `$data_init$iso_c_binding_' in /opt/cray/pe/cce/12.0.1/cce/x86_64/lib/libmodules.so is smaller than 256 in /tmp/pe_39617/ex5f_1.o<br>Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process<br>Completed test examples<br>09:12 main= /gpfs/alpine/csc314/scratch/adams/petsc$ module list<br><br>Currently Loaded Modules:<br> 1) craype-x86-rome 4) perftools-base/21.05.0 7) cray-pmi-lib/6.0.12 10) cray-dsmml/0.1.5 13) PrgEnv-cray/8.1.0 16) rocm/4.2.0 19) autoconf/2.69<br> 2) libfabric/1.11.0.4.75 5) xpmem/2.2.40-2.1_2.44__g3cf3325.shasta 8) cce/12.0.1 11) cray-mpich/8.1.7 14) DefApps/default 17) emacs/27.2 20) automake/1.16.3<br> 3) craype-network-ofi 6) cray-pmi/6.0.12 9) craype/2.7.8 12) cray-libsci/<a href="http://21.06.1.1" target="_blank">21.06.1.1</a> 15) craype-accel-amd-gfx908 18) zlib/1.2.11 21) libtool/2.4.6<br><br><br></div></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr">Stefano</div>
</blockquote></div></div>