[petsc-dev] Kokkos make error on Spock
Stefano Zampini
stefano.zampini at gmail.com
Sun Jul 18 11:00:21 CDT 2021
This is probably kokkos pulling in the dependencies where compiling kokkos
source within PETSc.
Il Dom 18 Lug 2021, 16:29 Mark Adams <mfadams at lbl.gov> ha scritto:
> Whoops, this error was just from not telling gfortran to allow long lines.
>
> Anway, I did find that the when fortran bindings are enabled this OMP
> thing gets into the library. No idea how. I'll askm ORNL.
>
> 10:26 jczhang/fix-cray-mpicxx-includes/main=
> /gpfs/alpine/csc314/scratch/adams/petsc2$ nm
> /gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so
> |g offload
> U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
> U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
> U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
> 0000000001d457b0 T vecgetoffloadmask_
>
>
> On Sun, Jul 18, 2021 at 8:43 AM Mark Adams <mfadams at lbl.gov> wrote:
>
>> Ah, your test was not on Spock ... I have something working but this is
>> strange.
>>
>> I switched to the GNU ProgEnv. and it passed the Fortran test in 'check',
>> but this failed:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *08:26 2 jczhang/fix-cray-mpicxx-includes/main=
>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
>> PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu
>> PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> -lamdhip64 -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0
>> -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include
>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90
>> -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
>> -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
>> -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib
>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib
>> <http://21.04.1.1/GNU/9.1/x86_64/lib>
>> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib
>> <http://21.04.1.1/GNU/9.1/x86_64/lib>
>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib
>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0
>> -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0
>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64
>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib
>> -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib
>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib -lpetsc
>> -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64
>> -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem -lgfortran -lm
>> -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml
>> -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++ -ldl -o
>> ex40f90ex40f90.F90:35:132: 35 | call
>> DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one,
>> & |
>> 1Error:
>> Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:*
>> * ....*
>>
>> But ex5f (the Fortran check test) does work manually also (sanity check).
>>
>> Oddly I seem to have the cray compilers:
>>
>> 08:34 jczhang/fix-cray-mpicxx-includes/main=
>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which ftn
>> /opt/cray/pe/craype/2.7.6/bin/ftn
>>
>> yet it is GNU:
>>
>> 08:34 jczhang/fix-cray-mpicxx-includes/main=
>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ ftn --version
>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.)
>> Copyright (C) 2020 Free Software Foundation, Inc.
>> This is free software; see the source for copying conditions. There is NO
>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
>> PURPOSE.
>>
>> The documentation says ftn should be here:
>>
>> 08:34 jczhang/fix-cray-mpicxx-includes/main=
>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$
>> $GCC_PATH/bin/gfortran --version
>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.)
>> Copyright (C) 2020 Free Software Foundation, Inc.
>> This is free software; see the source for copying conditions. There is NO
>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
>> PURPOSE.
>>
>> Same, but the path is not the same:
>>
>> 08:39 jczhang/fix-cray-mpicxx-includes/main=
>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ echo $GCC_PATH
>> /opt/gcc/10.2.0
>>
>> Anyway, thanks for your help,
>> Mark
>>
>> On Sat, Jul 17, 2021 at 11:26 PM Junchao Zhang <junchao.zhang at gmail.com>
>> wrote:
>>
>>> Mark, I can reproduce this error with PrgEnv-cray, i.e., using the Cray
>>> compiler (clang-11). Previously I used PrgEnv-gnu, which did not have this
>>> error.
>>> Probably it is a problem of Spock. But I am not sure.
>>>
>>> --Junchao Zhang
>>>
>>>
>>> On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>> And I can run a fortran test, with warnings, but C tests fail:
>>>>
>>>> 11:15 jczhang/fix-cray-mpicxx-includes/main=
>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make
>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test
>>>> search='ts_utils_dmplexlandau_tutorials-ex1f90_0'
>>>> Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0
>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos
>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>>>> FC
>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
>>>> FLINKER
>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90
>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
>>>> warning: alignment 128 of symbol
>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in
>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in
>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
>>>> TEST
>>>> arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts
>>>> ok ts_utils_dmplexlandau_tutorials-ex1f90_0
>>>> ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0
>>>>
>>>>
>>>> On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfadams at lbl.gov> wrote:
>>>>
>>>>> HUmm, I can not reproduce this.
>>>>>
>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main=
>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make
>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>>>>> PETSC_ARCH=arch-spock-dbg-kokkos check
>>>>> Running check examples to verify correct installation
>>>>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and
>>>>> PETSC_ARCH=arch-spock-dbg-kokkos
>>>>> gmake[3]:
>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301:
>>>>> ex19.PETSc] Error 2 (ignored)
>>>>> *******************Error detected during compile or
>>>>> link!*******************
>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19
>>>>>
>>>>> *********************************************************************************
>>>>> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64
>>>>> -lhsa-runtime64 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing
>>>>> -Wno-unknown-pragmas -fstack-protector -Qunused-arguments
>>>>> -fvisibility=hidden -g -O0 -fPIC -Wall -Wwrite-strings
>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector
>>>>> -Qunused-arguments -fvisibility=hidden -g -O0
>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c
>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
>>>>> -lquadmath -lstdc++ -ldl -o ex19
>>>>> ld.lld: error:
>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>>>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
>>>>> [--no-allow-shlib-undefined]
>>>>> ld.lld: error:
>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>>>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
>>>>> [--no-allow-shlib-undefined]
>>>>> ld.lld: error:
>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>>>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
>>>>> [--no-allow-shlib-undefined]
>>>>> clang-11: error: linker command failed with exit code 1 (use -v to see
>>>>> invocation)
>>>>> gmake[4]: *** [<builtin>: ex19] Error 1
>>>>> gmake[3]:
>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301:
>>>>> ex3k.PETSc] Error 2 (ignored)
>>>>> *******************Error detected during compile or
>>>>> link!*******************
>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k
>>>>>
>>>>> *********************************************************************************
>>>>> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing
>>>>> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14
>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/opt/cray/pe/libsci/
>>>>> 21.04.1.1/CRAY/9.0/x86_64/include
>>>>> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include
>>>>> -I/opt/cray/pe/pmi/6.0.10/include -I/opt/cray/pe/dsmml/0.1.4/dsmml//include
>>>>> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include
>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib
>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib
>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../
>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed
>>>>> -Wl,--as-needed,-lsci_cray,--no-as-needed
>>>>> -Wl,--as-needed,-lmpi_cray,--no-as-needed
>>>>> -Wl,--as-needed,-ldsmml,--no-as-needed
>>>>> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl -lmpi_gtl_hsa
>>>>> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup
>>>>> ex3k.kokkos.cxx
>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
>>>>> -lquadmath -lstdc++ -ldl -o ex3k
>>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>>>>> undefined reference to `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa'
>>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>>>>> undefined reference to `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa'
>>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>>>>> undefined reference to `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa'
>>>>> clang-12: error: linker command failed with exit code 1 (use -v to see
>>>>> invocation)
>>>>> gmake[4]: ***
>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: ex3k]
>>>>> Error 1
>>>>> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored)
>>>>> 1,25c1,4
>>>>> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000
>>>>> < Vec Object: Exact Solution 2 MPI processes
>>>>> < type: mpikokkos
>>>>> < Process [0]
>>>>> < 0.
>>>>> < 0.015625
>>>>> < 0.125
>>>>> < Process [1]
>>>>> < 0.421875
>>>>> < 1.
>>>>> < Vec Object: Forcing function 2 MPI processes
>>>>> < type: mpikokkos
>>>>> < Process [0]
>>>>> < 1e-72
>>>>> < 1.50024
>>>>> < 3.01563
>>>>> < Process [1]
>>>>> < 4.67798
>>>>> < 7.
>>>>> < 0 SNES Function norm 5.414682427127e+00
>>>>> < 1 SNES Function norm 2.952582418265e-01
>>>>> < 2 SNES Function norm 4.502293658739e-04
>>>>> < 3 SNES Function norm 1.389665806646e-09
>>>>> < Number of SNES iterations = 3
>>>>> < Norm of error 1.49752e-10 Iterations 3
>>>>> ---
>>>>> > slurmstepd: error: execve():
>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such
>>>>> file or directory
>>>>> > slurmstepd: error: execve():
>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such
>>>>> file or directory
>>>>> > srun: error: spock01: tasks 0-1: Exited with exit code 2
>>>>> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0
>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials
>>>>> Possible problem with ex3k running with kokkos-kernels, diffs above
>>>>> =========================================
>>>>> *******************Error detected during compile or
>>>>> link!*******************
>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f
>>>>> *********************************************************
>>>>> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64
>>>>> -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0
>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90
>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
>>>>> -lquadmath -lstdc++ -ldl -o ex5f
>>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
>>>>> warning: alignment 128 of symbol
>>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in
>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
>>>>> /tmp/pe_30429/ex5f_1.o
>>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
>>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in
>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
>>>>> /tmp/pe_30429/ex5f_1.o
>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI
>>>>> process
>>>>> Completed test examples
>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main=
>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ ll
>>>>>
>>>>> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang <
>>>>> junchao.zhang at gmail.com> wrote:
>>>>>
>>>>>> Mark, I configured with Fortran binding enabled with main + MR !4175
>>>>>> <https://gitlab.com/petsc/petsc/-/merge_requests/4175>
>>>>>>
>>>>>> '--with-mpiexec=srun',
>>>>>> '--with-shared-libraries=1',
>>>>>> '--with-cc=cc',
>>>>>> '--with-cxx=CC',
>>>>>> '--with-fc=ftn',
>>>>>> '--with-fortran-bindings',
>>>>>> '--with-hip',
>>>>>> '--with-hipc=hipcc',
>>>>>> '--with-debugging',
>>>>>> '--CPPFLAGS=-I${ROCM_PATH}/include',
>>>>>> '--CXXPPFLAGS=-I${ROCM_PATH}/include',
>>>>>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
>>>>>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
>>>>>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
>>>>>> '--COPTFLAGS=-g -O0',
>>>>>> '--CXXOPTFLAGS=-g -O0',
>>>>>> '--FOPTFLAGS=-g -O0',
>>>>>> '--download-kokkos',
>>>>>> '--download-kokkos-kernels',
>>>>>> '--download-kokkos-commit=3.4.01',
>>>>>> '--download-kokkos-kernels-commit=3.4.01',
>>>>>> '--with-kokkos-hip-arch=VEGA908',
>>>>>>
>>>>>> and 'make check' ran smoothly on a compute node
>>>>>>
>>>>>> $ make check
>>>>>> Running check examples to verify correct installation
>>>>>> Using PETSC_DIR=/ccs/home/jczhang/petsc and
>>>>>> PETSC_ARCH=arch-spock-cray-kokkos-dbg
>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI
>>>>>> process
>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI
>>>>>> processes
>>>>>> C/C++ example src/snes/tutorials/ex3k run successfully with
>>>>>> kokkos-kernels
>>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI
>>>>>> process
>>>>>>
>>>>>> Completed test examples
>>>>>>
>>>>>>
>>>>>> --Junchao Zhang
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>>>>
>>>>>>> And I find that this error, on non-Kokkos C tests, is fixed by
>>>>>>> turning the fortran bindings off:
>>>>>>>
>>>>>>> ld.lld: error:
>>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
>>>>>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
>>>>>>> [--no-allow-shlib-undefined]
>>>>>>> ld.lld: error:
>>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
>>>>>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
>>>>>>> [--no-allow-shlib-undefined]
>>>>>>> ld.lld: error:
>>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
>>>>>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
>>>>>>> [--no-allow-shlib-undefined]
>>>>>>>
>>>>>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>>>>>
>>>>>>>> Not complex. THis has some overlap with my problem w/o Kokkos.
>>>>>>>>
>>>>>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang <
>>>>>>>> junchao.zhang at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Do you use complex? post your configure.log.
>>>>>>>>>
>>>>>>>>> --Junchao Zhang
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams at lbl.gov>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> The simple Kokkos example is failing for me on Spock.
>>>>>>>>>> Any ideas?
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> 10:44 main *=
>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
>>>>>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>>>>>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k
>>>>>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC
>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64
>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64
>>>>>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
>>>>>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings
>>>>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3
>>>>>>>>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>>>>>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include
>>>>>>>>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include
>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx
>>>>>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
>>>>>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
>>>>>>>>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
>>>>>>>>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
>>>>>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>>>>>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>>>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>>>>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>>>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>>>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
>>>>>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
>>>>>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
>>>>>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
>>>>>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
>>>>>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
>>>>>>>>>> -lquadmath -lstdc++ -ldl -o ex3k
>>>>>>>>>> In file included from ex3k.kokkos.cxx:3:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11:
>>>>>>>>>> error: no member named 'lock_address_host_space' in namespace
>>>>>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'?
>>>>>>>>>> while (!Impl::lock_address_host_space((void*)dest))
>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>>> lock_address_host_space
>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6:
>>>>>>>>>> note: 'lock_address_host_space' declared here
>>>>>>>>>> bool lock_address_host_space(void* ptr);
>>>>>>>>>> ^
>>>>>>>>>> In file included from ex3k.kokkos.cxx:3:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3:
>>>>>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos';
>>>>>>>>>> did you mean simply 'memory_fence'?
>>>>>>>>>> Kokkos::memory_fence();
>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~
>>>>>>>>>> memory_fence
>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6:
>>>>>>>>>> note: 'memory_fence' declared here
>>>>>>>>>> void memory_fence() {
>>>>>>>>>> ^
>>>>>>>>>> In file included from ex3k.kokkos.cxx:3:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>>>>>>>>>> In file included from
>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3:
>>>>>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos';
>>>>>>>>>> did you mean simply 'memory_fence'?
>>>>>>>>>> Kokkos::memory_fence();
>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~
>>>>>>>>>> memory_fence
>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6:
>>>>>>>>>> note: 'memory_fence' declared here
>>>>>>>>>> void memory_fence() {
>>>>>>>>>> ^
>>>>>>>>>>
>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20210718/fa55893b/attachment-0001.html>
More information about the petsc-dev
mailing list