[petsc-dev] Kokkos make error on Spock

Fri Jul 16 22:28:12 CDT 2021

Mark, I configured with Fortran binding enabled with main + MR !4175
<https://gitlab.com/petsc/petsc/-/merge_requests/4175>

'--with-mpiexec=srun',
'--with-shared-libraries=1',
'--with-cc=cc',
'--with-cxx=CC',
'--with-fc=ftn',
'--with-fortran-bindings',
'--with-hip',
'--with-hipc=hipcc',
'--with-debugging',
'--CPPFLAGS=-I${ROCM_PATH}/include',
'--CXXPPFLAGS=-I${ROCM_PATH}/include',
'--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
'--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
'--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
'--COPTFLAGS=-g -O0',
'--CXXOPTFLAGS=-g -O0',
'--FOPTFLAGS=-g -O0',
'--download-kokkos',
'--download-kokkos-kernels',
'--download-kokkos-commit=3.4.01',
'--download-kokkos-kernels-commit=3.4.01',
'--with-kokkos-hip-arch=VEGA908',

and 'make check' ran smoothly on a compute node

$ make check
Running check examples to verify correct installation
Using PETSC_DIR=/ccs/home/jczhang/petsc and
PETSC_ARCH=arch-spock-cray-kokkos-dbg
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes
C/C++ example src/snes/tutorials/ex3k run successfully with kokkos-kernels
Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process

Completed test examples

--Junchao Zhang

On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfadams at lbl.gov> wrote:

> And I find that this error, on non-Kokkos C tests, is fixed by turning the
> fortran bindings off:
>
> ld.lld: error:
> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
> [--no-allow-shlib-undefined]
> ld.lld: error:
> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
> [--no-allow-shlib-undefined]
> ld.lld: error:
> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
> [--no-allow-shlib-undefined]
>
> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams at lbl.gov> wrote:
>
>> Not complex. THis has some overlap with my problem w/o Kokkos.
>>
>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang <junchao.zhang at gmail.com>
>> wrote:
>>
>>> Do you use complex? post your configure.log.
>>>
>>> --Junchao Zhang
>>>
>>>
>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>> The simple Kokkos example is failing for me on Spock.
>>>> Any ideas?
>>>> Thanks,
>>>>
>>>> 10:44 main *=
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k
>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC
>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64
>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64
>>>>  -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
>>>> -fstack-protector -fvisibility=hidden -g -O2   -fPIC -Wall -Wwrite-strings
>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3
>>>> -std=c++14  -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include
>>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include
>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>>>>  -I/sw/spock/spack-envs/views/rocm-4.1.0/include    ex3k.kokkos.cxx
>>>>  -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
>>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
>>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
>>>> -lquadmath -lstdc++ -ldl -o ex3k
>>>> In file included from ex3k.kokkos.cxx:3:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>>>> In file included from
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>>>> In file included from
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11:
>>>> error: no member named 'lock_address_host_space' in namespace
>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'?
>>>>   while (!Impl::lock_address_host_space((void*)dest))
>>>>           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>           lock_address_host_space
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6:
>>>> note: 'lock_address_host_space' declared here
>>>> bool lock_address_host_space(void* ptr);
>>>>      ^
>>>> In file included from ex3k.kokkos.cxx:3:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>>>> In file included from
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>>>> In file included from
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3:
>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos';
>>>> did you mean simply 'memory_fence'?
>>>>   Kokkos::memory_fence();
>>>>   ^~~~~~~~~~~~~~~~~~~~
>>>>   memory_fence
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6:
>>>> note: 'memory_fence' declared here
>>>> void memory_fence() {
>>>>      ^
>>>> In file included from ex3k.kokkos.cxx:3:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>>>> In file included from
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>>>> In file included from
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3:
>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos';
>>>> did you mean simply 'memory_fence'?
>>>>   Kokkos::memory_fence();
>>>>   ^~~~~~~~~~~~~~~~~~~~
>>>>   memory_fence
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6:
>>>> note: 'memory_fence' declared here
>>>> void memory_fence() {
>>>>      ^
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20210716/5383630b/attachment-0001.html>