[petsc-dev] Kokkos make error on Spock
Mark Adams
mfadams at lbl.gov
Sat Jul 17 10:17:12 CDT 2021
And I can run a fortran test, with warnings, but C tests fail:
11:15 jczhang/fix-cray-mpicxx-includes/main=
/gpfs/alpine/csc314/scratch/adams/petsc$ make
PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test
search='ts_utils_dmplexlandau_tutorials-ex1f90_0'
Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0
PETSC_ARCH=arch-spock-opt-cray-kokkos
PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
FC
arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
FLINKER
arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90
/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
warning: alignment 128 of symbol
`$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in
/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
warning: alignment 64 of symbol `$data_init$iso_c_binding_' in
/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
TEST
arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts
ok ts_utils_dmplexlandau_tutorials-ex1f90_0
ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0
On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfadams at lbl.gov> wrote:
> HUmm, I can not reproduce this.
>
> 10:49 jczhang/fix-cray-mpicxx-includes/main=
> /gpfs/alpine/csc314/scratch/adams/petsc$ make
> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
> PETSC_ARCH=arch-spock-dbg-kokkos check
> Running check examples to verify correct installation
> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and
> PETSC_ARCH=arch-spock-dbg-kokkos
> gmake[3]:
> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301:
> ex19.PETSc] Error 2 (ignored)
> *******************Error detected during compile or
> link!*******************
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19
>
> *********************************************************************************
> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64
> -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 -fPIC
> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0
> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c
> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
> -lquadmath -lstdc++ -ldl -o ex19
> ld.lld: error:
> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
> [--no-allow-shlib-undefined]
> ld.lld: error:
> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
> [--no-allow-shlib-undefined]
> ld.lld: error:
> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
> [--no-allow-shlib-undefined]
> clang-11: error: linker command failed with exit code 1 (use -v to see
> invocation)
> gmake[4]: *** [<builtin>: ex19] Error 1
> gmake[3]:
> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301:
> ex3k.PETSc] Error 2 (ignored)
> *******************Error detected during compile or
> link!*******************
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k
>
> *********************************************************************************
> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing
> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14
> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
> -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/opt/cray/pe/libsci/
> 21.04.1.1/CRAY/9.0/x86_64/include
> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include
> -I/opt/cray/pe/pmi/6.0.10/include -I/opt/cray/pe/dsmml/0.1.4/dsmml//include
> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include
> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib
> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib
> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../
> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed
> -Wl,--as-needed,-lsci_cray,--no-as-needed
> -Wl,--as-needed,-lmpi_cray,--no-as-needed
> -Wl,--as-needed,-ldsmml,--no-as-needed
> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl -lmpi_gtl_hsa
> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup
> ex3k.kokkos.cxx
> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
> -lquadmath -lstdc++ -ldl -o ex3k
> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
> undefined reference to `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa'
> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
> undefined reference to `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa'
> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
> undefined reference to `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa'
> clang-12: error: linker command failed with exit code 1 (use -v to see
> invocation)
> gmake[4]: ***
> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: ex3k]
> Error 1
> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored)
> 1,25c1,4
> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000
> < Vec Object: Exact Solution 2 MPI processes
> < type: mpikokkos
> < Process [0]
> < 0.
> < 0.015625
> < 0.125
> < Process [1]
> < 0.421875
> < 1.
> < Vec Object: Forcing function 2 MPI processes
> < type: mpikokkos
> < Process [0]
> < 1e-72
> < 1.50024
> < 3.01563
> < Process [1]
> < 4.67798
> < 7.
> < 0 SNES Function norm 5.414682427127e+00
> < 1 SNES Function norm 2.952582418265e-01
> < 2 SNES Function norm 4.502293658739e-04
> < 3 SNES Function norm 1.389665806646e-09
> < Number of SNES iterations = 3
> < Norm of error 1.49752e-10 Iterations 3
> ---
> > slurmstepd: error: execve():
> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such
> file or directory
> > slurmstepd: error: execve():
> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such
> file or directory
> > srun: error: spock01: tasks 0-1: Exited with exit code 2
> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0
> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials
> Possible problem with ex3k running with kokkos-kernels, diffs above
> =========================================
> *******************Error detected during compile or
> link!*******************
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f
> *********************************************************
> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64
> -fPIC -g -O0 -fPIC -g -O0
> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90
> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
> -lquadmath -lstdc++ -ldl -o ex5f
> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
> warning: alignment 128 of symbol
> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in
> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
> /tmp/pe_30429/ex5f_1.o
> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in
> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
> /tmp/pe_30429/ex5f_1.o
> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process
> Completed test examples
> 10:49 jczhang/fix-cray-mpicxx-includes/main=
> /gpfs/alpine/csc314/scratch/adams/petsc$ ll
>
> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang <junchao.zhang at gmail.com>
> wrote:
>
>> Mark, I configured with Fortran binding enabled with main + MR !4175
>> <https://gitlab.com/petsc/petsc/-/merge_requests/4175>
>>
>> '--with-mpiexec=srun',
>> '--with-shared-libraries=1',
>> '--with-cc=cc',
>> '--with-cxx=CC',
>> '--with-fc=ftn',
>> '--with-fortran-bindings',
>> '--with-hip',
>> '--with-hipc=hipcc',
>> '--with-debugging',
>> '--CPPFLAGS=-I${ROCM_PATH}/include',
>> '--CXXPPFLAGS=-I${ROCM_PATH}/include',
>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
>> '--COPTFLAGS=-g -O0',
>> '--CXXOPTFLAGS=-g -O0',
>> '--FOPTFLAGS=-g -O0',
>> '--download-kokkos',
>> '--download-kokkos-kernels',
>> '--download-kokkos-commit=3.4.01',
>> '--download-kokkos-kernels-commit=3.4.01',
>> '--with-kokkos-hip-arch=VEGA908',
>>
>> and 'make check' ran smoothly on a compute node
>>
>> $ make check
>> Running check examples to verify correct installation
>> Using PETSC_DIR=/ccs/home/jczhang/petsc and
>> PETSC_ARCH=arch-spock-cray-kokkos-dbg
>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI
>> processes
>> C/C++ example src/snes/tutorials/ex3k run successfully with kokkos-kernels
>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI
>> process
>>
>> Completed test examples
>>
>>
>> --Junchao Zhang
>>
>>
>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfadams at lbl.gov> wrote:
>>
>>> And I find that this error, on non-Kokkos C tests, is fixed by turning
>>> the fortran bindings off:
>>>
>>> ld.lld: error:
>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
>>> [--no-allow-shlib-undefined]
>>> ld.lld: error:
>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
>>> [--no-allow-shlib-undefined]
>>> ld.lld: error:
>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
>>> [--no-allow-shlib-undefined]
>>>
>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>> Not complex. THis has some overlap with my problem w/o Kokkos.
>>>>
>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang <junchao.zhang at gmail.com>
>>>> wrote:
>>>>
>>>>> Do you use complex? post your configure.log.
>>>>>
>>>>> --Junchao Zhang
>>>>>
>>>>>
>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams at lbl.gov> wrote:
>>>>>
>>>>>> The simple Kokkos example is failing for me on Spock.
>>>>>> Any ideas?
>>>>>> Thanks,
>>>>>>
>>>>>> 10:44 main *=
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k
>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC
>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64
>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64
>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings
>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3
>>>>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include
>>>>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include
>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx
>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
>>>>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
>>>>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
>>>>>> -lquadmath -lstdc++ -ldl -o ex3k
>>>>>> In file included from ex3k.kokkos.cxx:3:
>>>>>> In file included from
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>>>>>> In file included from
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>>>>>> In file included from
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>>>>>> In file included from
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>>>>>> In file included from
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>>>>>> In file included from
>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>>>>>> In file included from
>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11:
>>>>>> error: no member named 'lock_address_host_space' in namespace
>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'?
>>>>>> while (!Impl::lock_address_host_space((void*)dest))
>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>> lock_address_host_space
>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6:
>>>>>> note: 'lock_address_host_space' declared here
>>>>>> bool lock_address_host_space(void* ptr);
>>>>>> ^
>>>>>> In file included from ex3k.kokkos.cxx:3:
>>>>>> In file included from
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>>>>>> In file included from
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>>>>>> In file included from
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>>>>>> In file included from
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>>>>>> In file included from
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>>>>>> In file included from
>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>>>>>> In file included from
>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3:
>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos';
>>>>>> did you mean simply 'memory_fence'?
>>>>>> Kokkos::memory_fence();
>>>>>> ^~~~~~~~~~~~~~~~~~~~
>>>>>> memory_fence
>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6:
>>>>>> note: 'memory_fence' declared here
>>>>>> void memory_fence() {
>>>>>> ^
>>>>>> In file included from ex3k.kokkos.cxx:3:
>>>>>> In file included from
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>>>>>> In file included from
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>>>>>> In file included from
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>>>>>> In file included from
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>>>>>> In file included from
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>>>>>> In file included from
>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>>>>>> In file included from
>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3:
>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos';
>>>>>> did you mean simply 'memory_fence'?
>>>>>> Kokkos::memory_fence();
>>>>>> ^~~~~~~~~~~~~~~~~~~~
>>>>>> memory_fence
>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6:
>>>>>> note: 'memory_fence' declared here
>>>>>> void memory_fence() {
>>>>>> ^
>>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20210717/32d98254/attachment-0001.html>
More information about the petsc-dev
mailing list