<div dir="ltr">Ha, Landau is working with the Kokkos backend and it is running the the GPU!<div><br></div><div>My Kokkos build was clean, but let me try it again.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 16, 2021 at 5:19 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I don't understand this when linking ex19<div> ld.lld: error: /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib/libpetsc.so: undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined]</div><div>Do you have a stale libpetsc.so?</div><div><br></div><div>For ex3k, it is a makefile problem (I am fixing). If you do not directly build an executable from *.kokkos.cxx, then you can avoid this problem. For example, snes/tests/ex13 works with kokkos options on Spock.</div><div> </div><div><div><div dir="ltr"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 16, 2021 at 2:53 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Not complex. THis has some overlap with my problem w/o Kokkos.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Do you use complex? post your configure.log. <div><br clear="all"><div><div dir="ltr"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>The simple Kokkos example is failing for me on Spock.</div><div>Any ideas?</div><div>Thanks,</div><div><br></div>10:44 main *= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k<br>MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/lib" target="_blank">21.04.1.1/CRAY/9.0/x86_64/lib</a> -L/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/lib" target="_blank">21.04.1.1/CRAY/9.0/x86_64/lib</a> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -o ex3k<br>In file included from ex3k.kokkos.cxx:3:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:<br>In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:<br>In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:<br>/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: error: no member named 'lock_address_host_space' in namespace 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'?<br> while (!Impl::lock_address_host_space((void*)dest))<br> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br> lock_address_host_space<br>/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: note: 'lock_address_host_space' declared here<br>bool lock_address_host_space(void* ptr);<br> ^<br>In file included from ex3k.kokkos.cxx:3:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:<br>In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:<br>In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:<br>/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; did you mean simply 'memory_fence'?<br> Kokkos::memory_fence();<br> ^~~~~~~~~~~~~~~~~~~~<br> memory_fence<br>/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: note: 'memory_fence' declared here<br>void memory_fence() {<br> ^<br>In file included from ex3k.kokkos.cxx:3:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:<br>In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:<br>In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:<br>In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:<br>/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; did you mean simply 'memory_fence'?<br> Kokkos::memory_fence();<br> ^~~~~~~~~~~~~~~~~~~~<br> memory_fence<br>/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: note: 'memory_fence' declared here<br>void memory_fence() {<br> ^<br></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>