[petsc-dev] Kokkos make error on Spock
Mark Adams
mfadams at lbl.gov
Mon Jul 19 06:19:52 CDT 2021
Thanks, but this happens w/o Kokkos.
I've stripped this down and attached good/bad logs without/with Fortran
bindings.
Hope this helps,
Thanks again,
Mark
On Sun, Jul 18, 2021 at 12:00 PM Stefano Zampini <stefano.zampini at gmail.com>
wrote:
> This is probably kokkos pulling in the dependencies where compiling kokkos
> source within PETSc.
>
> Il Dom 18 Lug 2021, 16:29 Mark Adams <mfadams at lbl.gov> ha scritto:
>
>> Whoops, this error was just from not telling gfortran to allow long lines.
>>
>> Anway, I did find that the when fortran bindings are enabled this OMP
>> thing gets into the library. No idea how. I'll askm ORNL.
>>
>> 10:26 jczhang/fix-cray-mpicxx-includes/main=
>> /gpfs/alpine/csc314/scratch/adams/petsc2$ nm
>> /gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so
>> |g offload
>> U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
>> U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
>> U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
>> 0000000001d457b0 T vecgetoffloadmask_
>>
>>
>> On Sun, Jul 18, 2021 at 8:43 AM Mark Adams <mfadams at lbl.gov> wrote:
>>
>>> Ah, your test was not on Spock ... I have something working but this is
>>> strange.
>>>
>>> I switched to the GNU ProgEnv. and it passed the Fortran test in
>>> 'check', but this failed:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *08:26 2 jczhang/fix-cray-mpicxx-includes/main=
>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
>>> PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu
>>> PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>> -lamdhip64 -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0
>>> -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include
>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90
>>> -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
>>> -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
>>> -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib
>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib
>>> <http://21.04.1.1/GNU/9.1/x86_64/lib>
>>> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib
>>> <http://21.04.1.1/GNU/9.1/x86_64/lib>
>>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib
>>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0
>>> -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0
>>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64
>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib
>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib
>>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib -lpetsc
>>> -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64
>>> -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem -lgfortran -lm
>>> -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml
>>> -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++ -ldl -o
>>> ex40f90ex40f90.F90:35:132: 35 | call
>>> DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one,
>>> & |
>>> 1Error:
>>> Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:*
>>> * ....*
>>>
>>> But ex5f (the Fortran check test) does work manually also (sanity check).
>>>
>>> Oddly I seem to have the cray compilers:
>>>
>>> 08:34 jczhang/fix-cray-mpicxx-includes/main=
>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which ftn
>>> /opt/cray/pe/craype/2.7.6/bin/ftn
>>>
>>> yet it is GNU:
>>>
>>> 08:34 jczhang/fix-cray-mpicxx-includes/main=
>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ ftn --version
>>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.)
>>> Copyright (C) 2020 Free Software Foundation, Inc.
>>> This is free software; see the source for copying conditions. There is
>>> NO
>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
>>> PURPOSE.
>>>
>>> The documentation says ftn should be here:
>>>
>>> 08:34 jczhang/fix-cray-mpicxx-includes/main=
>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$
>>> $GCC_PATH/bin/gfortran --version
>>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.)
>>> Copyright (C) 2020 Free Software Foundation, Inc.
>>> This is free software; see the source for copying conditions. There is
>>> NO
>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
>>> PURPOSE.
>>>
>>> Same, but the path is not the same:
>>>
>>> 08:39 jczhang/fix-cray-mpicxx-includes/main=
>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ echo $GCC_PATH
>>> /opt/gcc/10.2.0
>>>
>>> Anyway, thanks for your help,
>>> Mark
>>>
>>> On Sat, Jul 17, 2021 at 11:26 PM Junchao Zhang <junchao.zhang at gmail.com>
>>> wrote:
>>>
>>>> Mark, I can reproduce this error with PrgEnv-cray, i.e., using the
>>>> Cray compiler (clang-11). Previously I used PrgEnv-gnu, which did not have
>>>> this error.
>>>> Probably it is a problem of Spock. But I am not sure.
>>>>
>>>> --Junchao Zhang
>>>>
>>>>
>>>> On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <mfadams at lbl.gov> wrote:
>>>>
>>>>> And I can run a fortran test, with warnings, but C tests fail:
>>>>>
>>>>> 11:15 jczhang/fix-cray-mpicxx-includes/main=
>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make
>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test
>>>>> search='ts_utils_dmplexlandau_tutorials-ex1f90_0'
>>>>> Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0
>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos
>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>>>>> FC
>>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
>>>>> FLINKER
>>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90
>>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
>>>>> warning: alignment 128 of symbol
>>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in
>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
>>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
>>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
>>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in
>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
>>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
>>>>> TEST
>>>>> arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts
>>>>> ok ts_utils_dmplexlandau_tutorials-ex1f90_0
>>>>> ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0
>>>>>
>>>>>
>>>>> On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfadams at lbl.gov> wrote:
>>>>>
>>>>>> HUmm, I can not reproduce this.
>>>>>>
>>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main=
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make
>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>>>>>> PETSC_ARCH=arch-spock-dbg-kokkos check
>>>>>> Running check examples to verify correct installation
>>>>>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and
>>>>>> PETSC_ARCH=arch-spock-dbg-kokkos
>>>>>> gmake[3]:
>>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301:
>>>>>> ex19.PETSc] Error 2 (ignored)
>>>>>> *******************Error detected during compile or
>>>>>> link!*******************
>>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19
>>>>>>
>>>>>> *********************************************************************************
>>>>>> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64
>>>>>> -lhsa-runtime64 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing
>>>>>> -Wno-unknown-pragmas -fstack-protector -Qunused-arguments
>>>>>> -fvisibility=hidden -g -O0 -fPIC -Wall -Wwrite-strings
>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector
>>>>>> -Qunused-arguments -fvisibility=hidden -g -O0
>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c
>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
>>>>>> -lquadmath -lstdc++ -ldl -o ex19
>>>>>> ld.lld: error:
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>>>>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
>>>>>> [--no-allow-shlib-undefined]
>>>>>> ld.lld: error:
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>>>>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
>>>>>> [--no-allow-shlib-undefined]
>>>>>> ld.lld: error:
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>>>>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
>>>>>> [--no-allow-shlib-undefined]
>>>>>> clang-11: error: linker command failed with exit code 1 (use -v to
>>>>>> see invocation)
>>>>>> gmake[4]: *** [<builtin>: ex19] Error 1
>>>>>> gmake[3]:
>>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301:
>>>>>> ex3k.PETSc] Error 2 (ignored)
>>>>>> *******************Error detected during compile or
>>>>>> link!*******************
>>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k
>>>>>>
>>>>>> *********************************************************************************
>>>>>> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing
>>>>>> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14
>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/opt/cray/pe/libsci/
>>>>>> 21.04.1.1/CRAY/9.0/x86_64/include
>>>>>> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include
>>>>>> -I/opt/cray/pe/pmi/6.0.10/include -I/opt/cray/pe/dsmml/0.1.4/dsmml//include
>>>>>> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include
>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib
>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib
>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../
>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>>> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed
>>>>>> -Wl,--as-needed,-lsci_cray,--no-as-needed
>>>>>> -Wl,--as-needed,-lmpi_cray,--no-as-needed
>>>>>> -Wl,--as-needed,-ldsmml,--no-as-needed
>>>>>> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl -lmpi_gtl_hsa
>>>>>> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup
>>>>>> ex3k.kokkos.cxx
>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
>>>>>> -lquadmath -lstdc++ -ldl -o ex3k
>>>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>>>>>> undefined reference to `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa'
>>>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>>>>>> undefined reference to `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa'
>>>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>>>>>> undefined reference to `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa'
>>>>>> clang-12: error: linker command failed with exit code 1 (use -v to
>>>>>> see invocation)
>>>>>> gmake[4]: ***
>>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: ex3k]
>>>>>> Error 1
>>>>>> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored)
>>>>>> 1,25c1,4
>>>>>> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000
>>>>>> < Vec Object: Exact Solution 2 MPI processes
>>>>>> < type: mpikokkos
>>>>>> < Process [0]
>>>>>> < 0.
>>>>>> < 0.015625
>>>>>> < 0.125
>>>>>> < Process [1]
>>>>>> < 0.421875
>>>>>> < 1.
>>>>>> < Vec Object: Forcing function 2 MPI processes
>>>>>> < type: mpikokkos
>>>>>> < Process [0]
>>>>>> < 1e-72
>>>>>> < 1.50024
>>>>>> < 3.01563
>>>>>> < Process [1]
>>>>>> < 4.67798
>>>>>> < 7.
>>>>>> < 0 SNES Function norm 5.414682427127e+00
>>>>>> < 1 SNES Function norm 2.952582418265e-01
>>>>>> < 2 SNES Function norm 4.502293658739e-04
>>>>>> < 3 SNES Function norm 1.389665806646e-09
>>>>>> < Number of SNES iterations = 3
>>>>>> < Norm of error 1.49752e-10 Iterations 3
>>>>>> ---
>>>>>> > slurmstepd: error: execve():
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such
>>>>>> file or directory
>>>>>> > slurmstepd: error: execve():
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such
>>>>>> file or directory
>>>>>> > srun: error: spock01: tasks 0-1: Exited with exit code 2
>>>>>> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials
>>>>>> Possible problem with ex3k running with kokkos-kernels, diffs above
>>>>>> =========================================
>>>>>> *******************Error detected during compile or
>>>>>> link!*******************
>>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f
>>>>>> *********************************************************
>>>>>> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64
>>>>>> -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0
>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90
>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
>>>>>> -lquadmath -lstdc++ -ldl -o ex5f
>>>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
>>>>>> warning: alignment 128 of symbol
>>>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in
>>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
>>>>>> /tmp/pe_30429/ex5f_1.o
>>>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
>>>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in
>>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
>>>>>> /tmp/pe_30429/ex5f_1.o
>>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI
>>>>>> process
>>>>>> Completed test examples
>>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main=
>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ ll
>>>>>>
>>>>>> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang <
>>>>>> junchao.zhang at gmail.com> wrote:
>>>>>>
>>>>>>> Mark, I configured with Fortran binding enabled with main + MR !4175
>>>>>>> <https://gitlab.com/petsc/petsc/-/merge_requests/4175>
>>>>>>>
>>>>>>> '--with-mpiexec=srun',
>>>>>>> '--with-shared-libraries=1',
>>>>>>> '--with-cc=cc',
>>>>>>> '--with-cxx=CC',
>>>>>>> '--with-fc=ftn',
>>>>>>> '--with-fortran-bindings',
>>>>>>> '--with-hip',
>>>>>>> '--with-hipc=hipcc',
>>>>>>> '--with-debugging',
>>>>>>> '--CPPFLAGS=-I${ROCM_PATH}/include',
>>>>>>> '--CXXPPFLAGS=-I${ROCM_PATH}/include',
>>>>>>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
>>>>>>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
>>>>>>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
>>>>>>> '--COPTFLAGS=-g -O0',
>>>>>>> '--CXXOPTFLAGS=-g -O0',
>>>>>>> '--FOPTFLAGS=-g -O0',
>>>>>>> '--download-kokkos',
>>>>>>> '--download-kokkos-kernels',
>>>>>>> '--download-kokkos-commit=3.4.01',
>>>>>>> '--download-kokkos-kernels-commit=3.4.01',
>>>>>>> '--with-kokkos-hip-arch=VEGA908',
>>>>>>>
>>>>>>> and 'make check' ran smoothly on a compute node
>>>>>>>
>>>>>>> $ make check
>>>>>>> Running check examples to verify correct installation
>>>>>>> Using PETSC_DIR=/ccs/home/jczhang/petsc and
>>>>>>> PETSC_ARCH=arch-spock-cray-kokkos-dbg
>>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI
>>>>>>> process
>>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI
>>>>>>> processes
>>>>>>> C/C++ example src/snes/tutorials/ex3k run successfully with
>>>>>>> kokkos-kernels
>>>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI
>>>>>>> process
>>>>>>>
>>>>>>> Completed test examples
>>>>>>>
>>>>>>>
>>>>>>> --Junchao Zhang
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>>>>>
>>>>>>>> And I find that this error, on non-Kokkos C tests, is fixed by
>>>>>>>> turning the fortran bindings off:
>>>>>>>>
>>>>>>>> ld.lld: error:
>>>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
>>>>>>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
>>>>>>>> [--no-allow-shlib-undefined]
>>>>>>>> ld.lld: error:
>>>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
>>>>>>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
>>>>>>>> [--no-allow-shlib-undefined]
>>>>>>>> ld.lld: error:
>>>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
>>>>>>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
>>>>>>>> [--no-allow-shlib-undefined]
>>>>>>>>
>>>>>>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>>>>>>
>>>>>>>>> Not complex. THis has some overlap with my problem w/o Kokkos.
>>>>>>>>>
>>>>>>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang <
>>>>>>>>> junchao.zhang at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Do you use complex? post your configure.log.
>>>>>>>>>>
>>>>>>>>>> --Junchao Zhang
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams at lbl.gov>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> The simple Kokkos example is failing for me on Spock.
>>>>>>>>>>> Any ideas?
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>> 10:44 main *=
>>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
>>>>>>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>>>>>>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k
>>>>>>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC
>>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64
>>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64
>>>>>>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
>>>>>>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings
>>>>>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3
>>>>>>>>>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>>>>>>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include
>>>>>>>>>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include
>>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx
>>>>>>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
>>>>>>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
>>>>>>>>>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
>>>>>>>>>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
>>>>>>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>>>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>>>>>>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>>>>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>>>>>>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>>>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>>>>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>>>>>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
>>>>>>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
>>>>>>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
>>>>>>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
>>>>>>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
>>>>>>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
>>>>>>>>>>> -lquadmath -lstdc++ -ldl -o ex3k
>>>>>>>>>>> In file included from ex3k.kokkos.cxx:3:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11:
>>>>>>>>>>> error: no member named 'lock_address_host_space' in namespace
>>>>>>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'?
>>>>>>>>>>> while (!Impl::lock_address_host_space((void*)dest))
>>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>>>> lock_address_host_space
>>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6:
>>>>>>>>>>> note: 'lock_address_host_space' declared here
>>>>>>>>>>> bool lock_address_host_space(void* ptr);
>>>>>>>>>>> ^
>>>>>>>>>>> In file included from ex3k.kokkos.cxx:3:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3:
>>>>>>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos';
>>>>>>>>>>> did you mean simply 'memory_fence'?
>>>>>>>>>>> Kokkos::memory_fence();
>>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~
>>>>>>>>>>> memory_fence
>>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6:
>>>>>>>>>>> note: 'memory_fence' declared here
>>>>>>>>>>> void memory_fence() {
>>>>>>>>>>> ^
>>>>>>>>>>> In file included from ex3k.kokkos.cxx:3:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>>>>>>>>>>> In file included from
>>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3:
>>>>>>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos';
>>>>>>>>>>> did you mean simply 'memory_fence'?
>>>>>>>>>>> Kokkos::memory_fence();
>>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~
>>>>>>>>>>> memory_fence
>>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6:
>>>>>>>>>>> note: 'memory_fence' declared here
>>>>>>>>>>> void memory_fence() {
>>>>>>>>>>> ^
>>>>>>>>>>>
>>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20210719/2574e106/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.good.log
Type: application/octet-stream
Size: 1702460 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20210719/2574e106/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.bad.log
Type: application/octet-stream
Size: 1720607 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20210719/2574e106/attachment-0003.obj>
More information about the petsc-dev
mailing list