[petsc-dev] Kokkos make error on Spock

Fri Jul 16 17:42:13 CDT 2021

Ha, Landau is working with the Kokkos backend and it is running the the GPU!

My Kokkos build was clean, but let me try it again.

On Fri, Jul 16, 2021 at 5:19 PM Junchao Zhang <junchao.zhang at gmail.com>
wrote:

> I don't understand this when linking ex19
>    ld.lld: error:
> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib/libpetsc.so:
> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
> [--no-allow-shlib-undefined]
> Do you have a stale libpetsc.so?
>
> For ex3k, it is a makefile problem (I am fixing). If you do not directly
> build an executable from *.kokkos.cxx, then you can avoid this problem.
>  For example,  snes/tests/ex13 works with kokkos options on Spock.
>
> --Junchao Zhang
>
>
> On Fri, Jul 16, 2021 at 2:53 PM Mark Adams <mfadams at lbl.gov> wrote:
>
>> Not complex. THis has some overlap with my problem w/o Kokkos.
>>
>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang <junchao.zhang at gmail.com>
>> wrote:
>>
>>> Do you use complex? post your configure.log.
>>>
>>> --Junchao Zhang
>>>
>>>
>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>> The simple Kokkos example is failing for me on Spock.
>>>> Any ideas?
>>>> Thanks,
>>>>
>>>> 10:44 main *=
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k
>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC
>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64
>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64
>>>>  -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
>>>> -fstack-protector -fvisibility=hidden -g -O2   -fPIC -Wall -Wwrite-strings
>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3
>>>> -std=c++14  -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include
>>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include
>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>>>>  -I/sw/spock/spack-envs/views/rocm-4.1.0/include    ex3k.kokkos.cxx
>>>>  -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
>>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
>>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
>>>> -lquadmath -lstdc++ -ldl -o ex3k
>>>> In file included from ex3k.kokkos.cxx:3:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>>>> In file included from
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>>>> In file included from
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11:
>>>> error: no member named 'lock_address_host_space' in namespace
>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'?
>>>>   while (!Impl::lock_address_host_space((void*)dest))
>>>>           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>           lock_address_host_space
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6:
>>>> note: 'lock_address_host_space' declared here
>>>> bool lock_address_host_space(void* ptr);
>>>>      ^
>>>> In file included from ex3k.kokkos.cxx:3:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>>>> In file included from
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>>>> In file included from
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3:
>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos';
>>>> did you mean simply 'memory_fence'?
>>>>   Kokkos::memory_fence();
>>>>   ^~~~~~~~~~~~~~~~~~~~
>>>>   memory_fence
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6:
>>>> note: 'memory_fence' declared here
>>>> void memory_fence() {
>>>>      ^
>>>> In file included from ex3k.kokkos.cxx:3:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>>>> In file included from
>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>>>> In file included from
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>>>> In file included from
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3:
>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos';
>>>> did you mean simply 'memory_fence'?
>>>>   Kokkos::memory_fence();
>>>>   ^~~~~~~~~~~~~~~~~~~~
>>>>   memory_fence
>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6:
>>>> note: 'memory_fence' declared here
>>>> void memory_fence() {
>>>>      ^
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20210716/27e50584/attachment-0001.html>