[petsc-dev] Kokkos make error on Spock
Junchao Zhang
junchao.zhang at gmail.com
Sat Jul 17 22:26:45 CDT 2021
Mark, I can reproduce this error with PrgEnv-cray, i.e., using the Cray
compiler (clang-11). Previously I used PrgEnv-gnu, which did not have this
error.
Probably it is a problem of Spock. But I am not sure.
--Junchao Zhang
On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <mfadams at lbl.gov> wrote:
> And I can run a fortran test, with warnings, but C tests fail:
>
> 11:15 jczhang/fix-cray-mpicxx-includes/main=
> /gpfs/alpine/csc314/scratch/adams/petsc$ make
> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
> PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test
> search='ts_utils_dmplexlandau_tutorials-ex1f90_0'
> Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0
> PETSC_ARCH=arch-spock-opt-cray-kokkos
> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
> FC
> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
> FLINKER
> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90
> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
> warning: alignment 128 of symbol
> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in
> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in
> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
> TEST
> arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts
> ok ts_utils_dmplexlandau_tutorials-ex1f90_0
> ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0
>
>
> On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfadams at lbl.gov> wrote:
>
>> HUmm, I can not reproduce this.
>>
>> 10:49 jczhang/fix-cray-mpicxx-includes/main=
>> /gpfs/alpine/csc314/scratch/adams/petsc$ make
>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>> PETSC_ARCH=arch-spock-dbg-kokkos check
>> Running check examples to verify correct installation
>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and
>> PETSC_ARCH=arch-spock-dbg-kokkos
>> gmake[3]:
>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301:
>> ex19.PETSc] Error 2 (ignored)
>> *******************Error detected during compile or
>> link!*******************
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19
>>
>> *********************************************************************************
>> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64
>> -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
>> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 -fPIC
>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
>> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0
>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c
>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
>> -lquadmath -lstdc++ -ldl -o ex19
>> ld.lld: error:
>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
>> [--no-allow-shlib-undefined]
>> ld.lld: error:
>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
>> [--no-allow-shlib-undefined]
>> ld.lld: error:
>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
>> [--no-allow-shlib-undefined]
>> clang-11: error: linker command failed with exit code 1 (use -v to see
>> invocation)
>> gmake[4]: *** [<builtin>: ex19] Error 1
>> gmake[3]:
>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301:
>> ex3k.PETSc] Error 2 (ignored)
>> *******************Error detected during compile or
>> link!*******************
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k
>>
>> *********************************************************************************
>> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing
>> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14
>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/opt/cray/pe/libsci/
>> 21.04.1.1/CRAY/9.0/x86_64/include
>> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include
>> -I/opt/cray/pe/pmi/6.0.10/include -I/opt/cray/pe/dsmml/0.1.4/dsmml//include
>> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include
>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib
>> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib
>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../
>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed
>> -Wl,--as-needed,-lsci_cray,--no-as-needed
>> -Wl,--as-needed,-lmpi_cray,--no-as-needed
>> -Wl,--as-needed,-ldsmml,--no-as-needed
>> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl -lmpi_gtl_hsa
>> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup
>> ex3k.kokkos.cxx
>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
>> -lquadmath -lstdc++ -ldl -o ex3k
>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>> undefined reference to `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa'
>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>> undefined reference to `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa'
>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>> undefined reference to `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa'
>> clang-12: error: linker command failed with exit code 1 (use -v to see
>> invocation)
>> gmake[4]: ***
>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: ex3k]
>> Error 1
>> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored)
>> 1,25c1,4
>> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000
>> < Vec Object: Exact Solution 2 MPI processes
>> < type: mpikokkos
>> < Process [0]
>> < 0.
>> < 0.015625
>> < 0.125
>> < Process [1]
>> < 0.421875
>> < 1.
>> < Vec Object: Forcing function 2 MPI processes
>> < type: mpikokkos
>> < Process [0]
>> < 1e-72
>> < 1.50024
>> < 3.01563
>> < Process [1]
>> < 4.67798
>> < 7.
>> < 0 SNES Function norm 5.414682427127e+00
>> < 1 SNES Function norm 2.952582418265e-01
>> < 2 SNES Function norm 4.502293658739e-04
>> < 3 SNES Function norm 1.389665806646e-09
>> < Number of SNES iterations = 3
>> < Norm of error 1.49752e-10 Iterations 3
>> ---
>> > slurmstepd: error: execve():
>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such
>> file or directory
>> > slurmstepd: error: execve():
>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such
>> file or directory
>> > srun: error: spock01: tasks 0-1: Exited with exit code 2
>> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0
>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials
>> Possible problem with ex3k running with kokkos-kernels, diffs above
>> =========================================
>> *******************Error detected during compile or
>> link!*******************
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f
>> *********************************************************
>> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64
>> -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0
>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90
>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
>> -lquadmath -lstdc++ -ldl -o ex5f
>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
>> warning: alignment 128 of symbol
>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in
>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
>> /tmp/pe_30429/ex5f_1.o
>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in
>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
>> /tmp/pe_30429/ex5f_1.o
>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI
>> process
>> Completed test examples
>> 10:49 jczhang/fix-cray-mpicxx-includes/main=
>> /gpfs/alpine/csc314/scratch/adams/petsc$ ll
>>
>> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang <junchao.zhang at gmail.com>
>> wrote:
>>
>>> Mark, I configured with Fortran binding enabled with main + MR !4175
>>> <https://gitlab.com/petsc/petsc/-/merge_requests/4175>
>>>
>>> '--with-mpiexec=srun',
>>> '--with-shared-libraries=1',
>>> '--with-cc=cc',
>>> '--with-cxx=CC',
>>> '--with-fc=ftn',
>>> '--with-fortran-bindings',
>>> '--with-hip',
>>> '--with-hipc=hipcc',
>>> '--with-debugging',
>>> '--CPPFLAGS=-I${ROCM_PATH}/include',
>>> '--CXXPPFLAGS=-I${ROCM_PATH}/include',
>>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
>>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
>>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
>>> '--COPTFLAGS=-g -O0',
>>> '--CXXOPTFLAGS=-g -O0',
>>> '--FOPTFLAGS=-g -O0',
>>> '--download-kokkos',
>>> '--download-kokkos-kernels',
>>> '--download-kokkos-commit=3.4.01',
>>> '--download-kokkos-kernels-commit=3.4.01',
>>> '--with-kokkos-hip-arch=VEGA908',
>>>
>>> and 'make check' ran smoothly on a compute node
>>>
>>> $ make check
>>> Running check examples to verify correct installation
>>> Using PETSC_DIR=/ccs/home/jczhang/petsc and
>>> PETSC_ARCH=arch-spock-cray-kokkos-dbg
>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI
>>> processes
>>> C/C++ example src/snes/tutorials/ex3k run successfully with
>>> kokkos-kernels
>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI
>>> process
>>>
>>> Completed test examples
>>>
>>>
>>> --Junchao Zhang
>>>
>>>
>>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>> And I find that this error, on non-Kokkos C tests, is fixed by turning
>>>> the fortran bindings off:
>>>>
>>>> ld.lld: error:
>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
>>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
>>>> [--no-allow-shlib-undefined]
>>>> ld.lld: error:
>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
>>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
>>>> [--no-allow-shlib-undefined]
>>>> ld.lld: error:
>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
>>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
>>>> [--no-allow-shlib-undefined]
>>>>
>>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>>
>>>>> Not complex. THis has some overlap with my problem w/o Kokkos.
>>>>>
>>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang <
>>>>> junchao.zhang at gmail.com> wrote:
>>>>>
>>>>>> Do you use complex? post your configure.log.
>>>>>>
>>>>>> --Junchao Zhang
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams at lbl.gov> wrote:
>>>>>>
>>>>>>> The simple Kokkos example is failing for me on Spock.
>>>>>>> Any ideas?
>>>>>>> Thanks,
>>>>>>>
>>>>>>> 10:44 main *=
>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
>>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k
>>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC
>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64
>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64
>>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
>>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings
>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3
>>>>>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include
>>>>>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include
>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx
>>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
>>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
>>>>>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
>>>>>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
>>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
>>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
>>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
>>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
>>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
>>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
>>>>>>> -lquadmath -lstdc++ -ldl -o ex3k
>>>>>>> In file included from ex3k.kokkos.cxx:3:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11:
>>>>>>> error: no member named 'lock_address_host_space' in namespace
>>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'?
>>>>>>> while (!Impl::lock_address_host_space((void*)dest))
>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>> lock_address_host_space
>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6:
>>>>>>> note: 'lock_address_host_space' declared here
>>>>>>> bool lock_address_host_space(void* ptr);
>>>>>>> ^
>>>>>>> In file included from ex3k.kokkos.cxx:3:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3:
>>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos';
>>>>>>> did you mean simply 'memory_fence'?
>>>>>>> Kokkos::memory_fence();
>>>>>>> ^~~~~~~~~~~~~~~~~~~~
>>>>>>> memory_fence
>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6:
>>>>>>> note: 'memory_fence' declared here
>>>>>>> void memory_fence() {
>>>>>>> ^
>>>>>>> In file included from ex3k.kokkos.cxx:3:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>>>>>>> In file included from
>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3:
>>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos';
>>>>>>> did you mean simply 'memory_fence'?
>>>>>>> Kokkos::memory_fence();
>>>>>>> ^~~~~~~~~~~~~~~~~~~~
>>>>>>> memory_fence
>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6:
>>>>>>> note: 'memory_fence' declared here
>>>>>>> void memory_fence() {
>>>>>>> ^
>>>>>>>
>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20210717/28f5f8d2/attachment-0001.html>
More information about the petsc-dev
mailing list