<div dir="ltr">Mark, I can reproduce this error with PrgEnv-cray, i.e., using the Cray compiler (clang-11). Previously I used PrgEnv-gnu, which did not have this error. <div>Probably it is a problem of Spock. But I am not sure.<div><div><br><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">And I can run a fortran test, with warnings, but C tests fail:<div><br></div><div>11:15 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc$ make PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test search='ts_utils_dmplexlandau_tutorials-ex1f90_0'<br>Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0 PETSC_ARCH=arch-spock-opt-cray-kokkos PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc<br> FC arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o<br> FLINKER arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90<br>/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: warning: alignment 128 of symbol `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o<br>/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: warning: alignment 64 of symbol `$data_init$iso_c_binding_' in /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o<br> TEST arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts<br> ok ts_utils_dmplexlandau_tutorials-ex1f90_0<br> ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0<br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">HUmm, I can not reproduce this.<div><br></div><div>10:49 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc$ make PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc PETSC_ARCH=arch-spock-dbg-kokkos check<br>Running check examples to verify correct installation<br>Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and PETSC_ARCH=arch-spock-dbg-kokkos<br>gmake[3]: [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: ex19.PETSc] Error 2 (ignored)<br>*******************Error detected during compile or link!*******************<br>See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><br>/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19<br>*********************************************************************************<br>cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/gpfs/alpine/csc314/scratch/adams/petsc/include -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/lib" target="_blank">21.04.1.1/CRAY/9.0/x86_64/lib</a> -L/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/lib" target="_blank">21.04.1.1/CRAY/9.0/x86_64/lib</a> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -o ex19<br>ld.lld: error: /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined]<br>ld.lld: error: /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined]<br>ld.lld: error: /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined]<br>clang-11: error: linker command failed with exit code 1 (use -v to see invocation)<br>gmake[4]: *** [<builtin>: ex19] Error 1<br>gmake[3]: [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: ex3k.PETSc] Error 2 (ignored)<br>*******************Error detected during compile or link!*******************<br>See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><br>/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k<br>*********************************************************************************<br>hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/include" target="_blank">21.04.1.1/CRAY/9.0/x86_64/include</a> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include -I/opt/cray/pe/pmi/6.0.10/include -I/opt/cray/pe/dsmml/0.1.4/dsmml//include -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include -L/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/lib" target="_blank">21.04.1.1/CRAY/9.0/x86_64/lib</a> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../ -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed -Wl,--as-needed,-lsci_cray,--no-as-needed -Wl,--as-needed,-lmpi_cray,--no-as-needed -Wl,--as-needed,-ldsmml,--no-as-needed -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl -lmpi_gtl_hsa -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup ex3k.kokkos.cxx -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/lib" target="_blank">21.04.1.1/CRAY/9.0/x86_64/lib</a> -L/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/lib" target="_blank">21.04.1.1/CRAY/9.0/x86_64/lib</a> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -o ex3k<br>/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: undefined reference to `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa'<br>/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: undefined reference to `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa'<br>/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: undefined reference to `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa'<br>clang-12: error: linker command failed with exit code 1 (use -v to see invocation)<br>gmake[4]: *** [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: ex3k] Error 1<br>gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored)<br>1,25c1,4<br>< atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000<br>< Vec Object: Exact Solution 2 MPI processes<br>< type: mpikokkos<br>< Process [0]<br>< 0.<br>< 0.015625<br>< 0.125<br>< Process [1]<br>< 0.421875<br>< 1.<br>< Vec Object: Forcing function 2 MPI processes<br>< type: mpikokkos<br>< Process [0]<br>< 1e-72<br>< 1.50024<br>< 3.01563<br>< Process [1]<br>< 4.67798<br>< 7.<br>< 0 SNES Function norm 5.414682427127e+00<br>< 1 SNES Function norm 2.952582418265e-01<br>< 2 SNES Function norm 4.502293658739e-04<br>< 3 SNES Function norm 1.389665806646e-09<br>< Number of SNES iterations = 3<br>< Norm of error 1.49752e-10 Iterations 3<br>---<br>> slurmstepd: error: execve(): /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such file or directory<br>> slurmstepd: error: execve(): /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such file or directory<br>> srun: error: spock01: tasks 0-1: Exited with exit code 2<br>> srun: launch/slurm: _step_signal: Terminating StepId=272979.0<br>/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials<br>Possible problem with ex3k running with kokkos-kernels, diffs above<br>=========================================<br>*******************Error detected during compile or link!*******************<br>See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><br>/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f<br>*********************************************************<br>ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 -I/gpfs/alpine/csc314/scratch/adams/petsc/include -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90 -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/lib" target="_blank">21.04.1.1/CRAY/9.0/x86_64/lib</a> -L/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/lib" target="_blank">21.04.1.1/CRAY/9.0/x86_64/lib</a> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -o ex5f<br>/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: warning: alignment 128 of symbol `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in /tmp/pe_30429/ex5f_1.o<br>/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: warning: alignment 64 of symbol `$data_init$iso_c_binding_' in /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in /tmp/pe_30429/ex5f_1.o<br>Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process<br>Completed test examples<br>10:49 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc$ ll<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Mark, I configured with Fortran binding enabled with main + MR <a href="https://gitlab.com/petsc/petsc/-/merge_requests/4175" style="font-size:12px;font-family:-apple-system,system-ui,"Segoe UI",Roboto,"Noto Sans",Ubuntu,Cantarell,"Helvetica Neue",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";background-color:transparent;box-sizing:border-box;color:rgb(48,48,48);text-decoration-line:none" target="_blank">!4175</a></div><div><br></div><div><div style="color:rgb(0,0,0);font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;line-height:21px;white-space:pre-wrap"><div> <span style="color:rgb(163,21,21)">'--with-mpiexec=srun'</span>,</div><div> <span style="color:rgb(163,21,21)">'--with-shared-libraries=1'</span>,</div><div> <span style="color:rgb(163,21,21)">'--with-cc=cc'</span>,</div><div> <span style="color:rgb(163,21,21)">'--with-cxx=CC'</span>,</div><div> <span style="color:rgb(163,21,21)">'--with-fc=ftn'</span>,</div><div> <span style="color:rgb(163,21,21)">'--with-fortran-bindings'</span>,</div><div> <span style="color:rgb(163,21,21)">'--with-hip'</span>,</div><div> <span style="color:rgb(163,21,21)">'--with-hipc=hipcc'</span>,</div><div> <span style="color:rgb(163,21,21)">'--with-debugging'</span>,</div><div> <span style="color:rgb(163,21,21)">'--CPPFLAGS=-I${ROCM_PATH}/include'</span>,</div><div> <span style="color:rgb(163,21,21)">'--CXXPPFLAGS=-I${ROCM_PATH}/include'</span>,<br></div><div> <span style="color:rgb(163,21,21)">'--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64'</span>,</div><div> <span style="color:rgb(163,21,21)">'--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64'</span>,<br></div><div> <span style="color:rgb(163,21,21)">'--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64'</span>,</div><div> <span style="color:rgb(163,21,21)">'--COPTFLAGS=-g -O0'</span>,</div><div> <span style="color:rgb(163,21,21)">'--CXXOPTFLAGS=-g -O0'</span>,</div><div> <span style="color:rgb(163,21,21)">'--FOPTFLAGS=-g -O0'</span>,</div><div> <span style="color:rgb(163,21,21)">'--download-kokkos'</span>,</div><div> <span style="color:rgb(163,21,21)">'--download-kokkos-kernels'</span>,</div><div> <span style="color:rgb(163,21,21)">'--download-kokkos-commit=3.4.01'</span>,</div><div> <span style="color:rgb(163,21,21)">'--download-kokkos-kernels-commit=3.4.01'</span>,</div><div> <span style="color:rgb(163,21,21)">'--with-kokkos-hip-arch=VEGA908'</span>,</div></div></div><div><br></div><div>and 'make check' ran smoothly on a compute node</div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>$ make check</div><div>Running check examples to verify correct installation</div><div>Using PETSC_DIR=/ccs/home/jczhang/petsc and PETSC_ARCH=arch-spock-cray-kokkos-dbg</div><div>C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process</div><div>C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes</div><div>C/C++ example src/snes/tutorials/ex3k run successfully with kokkos-kernels</div><div>Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process</div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>Completed test examples</div></blockquote><br clear="all"><div><div dir="ltr"><div dir="ltr">--Junchao Zhang</div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">And I find that this error, on non-Kokkos C tests, is fixed by turning the fortran bindings off:<div><br></div><div>ld.lld: error: /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined]<br>ld.lld: error: /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined]<br>ld.lld: error: /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined]<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Not complex. THis has some overlap with my problem w/o Kokkos.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Do you use complex? post your configure.log. <div><br clear="all"><div><div dir="ltr"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>The simple Kokkos example is failing for me on Spock.</div><div>Any ideas?</div><div>Thanks,</div><div><br></div>10:44 main *= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k<br>MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/lib" target="_blank">21.04.1.1/CRAY/9.0/x86_64/lib</a> -L/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/lib" target="_blank">21.04.1.1/CRAY/9.0/x86_64/lib</a> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -o ex3k<br>In file included from ex3k.kokkos.cxx:3:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:<br>In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:<br>In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:<br>/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: error: no member named 'lock_address_host_space' in namespace 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'?<br> while (!Impl::lock_address_host_space((void*)dest))<br> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br> lock_address_host_space<br>/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: note: 'lock_address_host_space' declared here<br>bool lock_address_host_space(void* ptr);<br> ^<br>In file included from ex3k.kokkos.cxx:3:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:<br>In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:<br>In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:<br>/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; did you mean simply 'memory_fence'?<br> Kokkos::memory_fence();<br> ^~~~~~~~~~~~~~~~~~~~<br> memory_fence<br>/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: note: 'memory_fence' declared here<br>void memory_fence() {<br> ^<br>In file included from ex3k.kokkos.cxx:3:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:<br>In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:<br>In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:<br>/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; did you mean simply 'memory_fence'?<br> Kokkos::memory_fence();<br> ^~~~~~~~~~~~~~~~~~~~<br> memory_fence<br>/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: note: 'memory_fence' declared here<br>void memory_fence() {<br> ^<br></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>