[petsc-dev] Kokkos make error on Spock
Mark Adams
mfadams at lbl.gov
Tue Jul 20 10:26:23 CDT 2021
OK, it looks like this flag caused the problem and does not seem to be
necessary:
'--LDFLAGS=-L'+os.environ['ROCM_PATH']+'/lib -lhsa-runtime64',
On Tue, Jul 20, 2021 at 9:00 AM Mark Adams <mfadams at lbl.gov> wrote:
>
>
> On Mon, Jul 19, 2021 at 6:41 PM Scott Kruger <kruger at txcorp.com> wrote:
>
>>
>> Mark,
>>
>> On tulip, things with fortran went wonky when a `-fast` flag snuck into
>> the flags (snuck in being me copying a previous configure file and not
>> questioning the flags). The reason is that for clang, `-fast` implies
>> link time optimization (lto) for C/C++ code, but flang doesn't support
>> lto so things got weird. I suspect that gfortran does not either (but
>> perhaps the real question is why not use flang?)
>>
>
> It does work with GNU. I am doing this for an application and I want to
> avoid dictating the program env.
>
> I'm (re)trying with -O0 just to check.
>
>
>>
>> I look at configure and I don't see anything in your flags that would
>> trigger lto, but as Junchao says, it might be picking something up from
>> Kokkos, so perhaps this is the issue.
>>
>>
> The configure files that I sent did not have Kokkos. They were stripped
> down.
>
> I have asked ORNL, but don't expect help.
> The person (Matt) that is helping me for other things is very helpful so
> perhaps he can come up with some ideas.
> Otherwise I will tell the app that they need to use GNU for now.
>
> Thanks,
> Mark
>
>
>
>> Scott
>>
>>
>> On 2021-07-19 07:19, Mark Adams did write:
>> > Thanks, but this happens w/o Kokkos.
>> > I've stripped this down and attached good/bad logs without/with Fortran
>> > bindings.
>> > Hope this helps,
>> > Thanks again,
>> > Mark
>> >
>> > On Sun, Jul 18, 2021 at 12:00 PM Stefano Zampini <
>> stefano.zampini at gmail.com>
>> > wrote:
>> >
>> > > This is probably kokkos pulling in the dependencies where compiling
>> kokkos
>> > > source within PETSc.
>> > >
>> > > Il Dom 18 Lug 2021, 16:29 Mark Adams <mfadams at lbl.gov> ha scritto:
>> > >
>> > >> Whoops, this error was just from not telling gfortran to allow long
>> lines.
>> > >>
>> > >> Anway, I did find that the when fortran bindings are enabled this OMP
>> > >> thing gets into the library. No idea how. I'll askm ORNL.
>> > >>
>> > >> 10:26 jczhang/fix-cray-mpicxx-includes/main=
>> > >> /gpfs/alpine/csc314/scratch/adams/petsc2$ nm
>> > >>
>> /gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so
>> > >> |g offload
>> > >> U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
>> > >> U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
>> > >> U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
>> > >> 0000000001d457b0 T vecgetoffloadmask_
>> > >>
>> > >>
>> > >> On Sun, Jul 18, 2021 at 8:43 AM Mark Adams <mfadams at lbl.gov> wrote:
>> > >>
>> > >>> Ah, your test was not on Spock ... I have something working but
>> this is
>> > >>> strange.
>> > >>>
>> > >>> I switched to the GNU ProgEnv. and it passed the Fortran test in
>> > >>> 'check', but this failed:
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>> *08:26 2 jczhang/fix-cray-mpicxx-includes/main=
>> > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
>> > >>>
>> PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu
>> > >>> PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> > >>> -lamdhip64 -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0
>> > >>>
>> -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include
>> > >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90
>> > >>>
>> -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
>> > >>>
>> -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
>> > >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> > >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> > >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
>> > >>> -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
>> > >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>> > >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib
>> > >>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib
>> > >>> <http://21.04.1.1/GNU/9.1/x86_64/lib>
>> > >>> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib
>> > >>> <http://21.04.1.1/GNU/9.1/x86_64/lib>
>> > >>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib
>> > >>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> > >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> > >>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> > >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0
>> > >>> -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0
>> > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64
>> > >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib
>> > >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib
>> > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib
>> -lpetsc
>> > >>> -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64
>> > >>> -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem
>> -lgfortran -lm
>> > >>> -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml
>> > >>> -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++
>> -ldl -o
>> > >>> ex40f90ex40f90.F90:35:132: 35 | call
>> > >>>
>> DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one,
>> > >>> & |
>> > >>>
>> 1Error:
>> > >>> Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:*
>> > >>> * ....*
>> > >>>
>> > >>> But ex5f (the Fortran check test) does work manually also (sanity
>> check).
>> > >>>
>> > >>> Oddly I seem to have the cray compilers:
>> > >>>
>> > >>> 08:34 jczhang/fix-cray-mpicxx-includes/main=
>> > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which
>> ftn
>> > >>> /opt/cray/pe/craype/2.7.6/bin/ftn
>> > >>>
>> > >>> yet it is GNU:
>> > >>>
>> > >>> 08:34 jczhang/fix-cray-mpicxx-includes/main=
>> > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ ftn
>> --version
>> > >>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.)
>> > >>> Copyright (C) 2020 Free Software Foundation, Inc.
>> > >>> This is free software; see the source for copying conditions.
>> There is
>> > >>> NO
>> > >>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
>> > >>> PURPOSE.
>> > >>>
>> > >>> The documentation says ftn should be here:
>> > >>>
>> > >>> 08:34 jczhang/fix-cray-mpicxx-includes/main=
>> > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$
>> > >>> $GCC_PATH/bin/gfortran --version
>> > >>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.)
>> > >>> Copyright (C) 2020 Free Software Foundation, Inc.
>> > >>> This is free software; see the source for copying conditions.
>> There is
>> > >>> NO
>> > >>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
>> > >>> PURPOSE.
>> > >>>
>> > >>> Same, but the path is not the same:
>> > >>>
>> > >>> 08:39 jczhang/fix-cray-mpicxx-includes/main=
>> > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ echo
>> $GCC_PATH
>> > >>> /opt/gcc/10.2.0
>> > >>>
>> > >>> Anyway, thanks for your help,
>> > >>> Mark
>> > >>>
>> > >>> On Sat, Jul 17, 2021 at 11:26 PM Junchao Zhang <
>> junchao.zhang at gmail.com>
>> > >>> wrote:
>> > >>>
>> > >>>> Mark, I can reproduce this error with PrgEnv-cray, i.e., using the
>> > >>>> Cray compiler (clang-11). Previously I used PrgEnv-gnu, which did
>> not have
>> > >>>> this error.
>> > >>>> Probably it is a problem of Spock. But I am not sure.
>> > >>>>
>> > >>>> --Junchao Zhang
>> > >>>>
>> > >>>>
>> > >>>> On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <mfadams at lbl.gov>
>> wrote:
>> > >>>>
>> > >>>>> And I can run a fortran test, with warnings, but C tests fail:
>> > >>>>>
>> > >>>>> 11:15 jczhang/fix-cray-mpicxx-includes/main=
>> > >>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make
>> > >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>> > >>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test
>> > >>>>> search='ts_utils_dmplexlandau_tutorials-ex1f90_0'
>> > >>>>> Using MAKEFLAGS: --
>> search=ts_utils_dmplexlandau_tutorials-ex1f90_0
>> > >>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos
>> > >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>> > >>>>> FC
>> > >>>>>
>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
>> > >>>>> FLINKER
>> > >>>>>
>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90
>> > >>>>>
>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
>> > >>>>> warning: alignment 128 of symbol
>> > >>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in
>> > >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller
>> than 256 in
>> > >>>>>
>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
>> > >>>>>
>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
>> > >>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in
>> > >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller
>> than 256 in
>> > >>>>>
>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
>> > >>>>> TEST
>> > >>>>>
>> arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts
>> > >>>>> ok ts_utils_dmplexlandau_tutorials-ex1f90_0
>> > >>>>> ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0
>> > >>>>>
>> > >>>>>
>> > >>>>> On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfadams at lbl.gov>
>> wrote:
>> > >>>>>
>> > >>>>>> HUmm, I can not reproduce this.
>> > >>>>>>
>> > >>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main=
>> > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make
>> > >>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>> > >>>>>> PETSC_ARCH=arch-spock-dbg-kokkos check
>> > >>>>>> Running check examples to verify correct installation
>> > >>>>>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and
>> > >>>>>> PETSC_ARCH=arch-spock-dbg-kokkos
>> > >>>>>> gmake[3]:
>> > >>>>>>
>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301:
>> > >>>>>> ex19.PETSc] Error 2 (ignored)
>> > >>>>>> *******************Error detected during compile or
>> > >>>>>> link!*******************
>> > >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19
>> > >>>>>>
>> > >>>>>>
>> *********************************************************************************
>> > >>>>>> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64
>> > >>>>>> -lhsa-runtime64 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing
>> > >>>>>> -Wno-unknown-pragmas -fstack-protector -Qunused-arguments
>> > >>>>>> -fvisibility=hidden -g -O0 -fPIC -Wall -Wwrite-strings
>> > >>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector
>> > >>>>>> -Qunused-arguments -fvisibility=hidden -g -O0
>> > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>> > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>> > >>>>>>
>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
>> > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>> > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c
>> > >>>>>>
>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> > >>>>>>
>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> > >>>>>>
>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> > >>>>>>
>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> > >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> > >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>> > >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>> > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>> > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>> > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib
>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>> > >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib
>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>> > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>> > >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> > >>>>>>
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>> > >>>>>>
>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>> > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>> > >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>> > >>>>>>
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>> > >>>>>>
>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>> > >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore
>> -lhipsparse
>> > >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64
>> -lhsa-runtime64
>> > >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray
>> -lmpi_cray
>> > >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath
>> -lcrayacc_amdgpu
>> > >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran
>> -lpthread
>> > >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
>> > >>>>>> -lquadmath -lstdc++ -ldl -o ex19
>> > >>>>>> ld.lld: error:
>> > >>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>> > >>>>>> undefined reference to
>> .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
>> > >>>>>> [--no-allow-shlib-undefined]
>> > >>>>>> ld.lld: error:
>> > >>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>> > >>>>>> undefined reference to
>> .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
>> > >>>>>> [--no-allow-shlib-undefined]
>> > >>>>>> ld.lld: error:
>> > >>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>> > >>>>>> undefined reference to
>> .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
>> > >>>>>> [--no-allow-shlib-undefined]
>> > >>>>>> clang-11: error: linker command failed with exit code 1 (use -v
>> to
>> > >>>>>> see invocation)
>> > >>>>>> gmake[4]: *** [<builtin>: ex19] Error 1
>> > >>>>>> gmake[3]:
>> > >>>>>>
>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301:
>> > >>>>>> ex3k.PETSc] Error 2 (ignored)
>> > >>>>>> *******************Error detected during compile or
>> > >>>>>> link!*******************
>> > >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k
>> > >>>>>>
>> > >>>>>>
>> *********************************************************************************
>> > >>>>>> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing
>> > >>>>>> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14
>> > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>> > >>>>>>
>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
>> > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>> > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>> -I/opt/cray/pe/libsci/
>> > >>>>>> 21.04.1.1/CRAY/9.0/x86_64/include
>> > >>>>>> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include
>> > >>>>>> -I/opt/cray/pe/pmi/6.0.10/include
>> -I/opt/cray/pe/dsmml/0.1.4/dsmml//include
>> > >>>>>> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include
>> > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib
>> > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib
>> > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../
>> > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> > >>>>>> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed
>> > >>>>>> -Wl,--as-needed,-lsci_cray,--no-as-needed
>> > >>>>>> -Wl,--as-needed,-lmpi_cray,--no-as-needed
>> > >>>>>> -Wl,--as-needed,-ldsmml,--no-as-needed
>> > >>>>>> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl
>> -lmpi_gtl_hsa
>> > >>>>>> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup
>> > >>>>>> ex3k.kokkos.cxx
>> > >>>>>>
>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> > >>>>>>
>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> > >>>>>>
>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> > >>>>>>
>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> > >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> > >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>> > >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>> > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>> > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>> > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib
>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>> > >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib
>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>> > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>> > >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> > >>>>>>
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>> > >>>>>>
>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>> > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>> > >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>> > >>>>>>
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>> > >>>>>>
>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>> > >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore
>> -lhipsparse
>> > >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64
>> -lhsa-runtime64
>> > >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray
>> -lmpi_cray
>> > >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath
>> -lcrayacc_amdgpu
>> > >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran
>> -lpthread
>> > >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
>> > >>>>>> -lquadmath -lstdc++ -ldl -o ex3k
>> > >>>>>>
>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
>> > >>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>> > >>>>>> undefined reference to
>> `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa'
>> > >>>>>>
>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
>> > >>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>> > >>>>>> undefined reference to
>> `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa'
>> > >>>>>>
>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
>> > >>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
>> > >>>>>> undefined reference to
>> `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa'
>> > >>>>>> clang-12: error: linker command failed with exit code 1 (use -v
>> to
>> > >>>>>> see invocation)
>> > >>>>>> gmake[4]: ***
>> > >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26:
>> ex3k]
>> > >>>>>> Error 1
>> > >>>>>> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored)
>> > >>>>>> 1,25c1,4
>> > >>>>>> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000
>> > >>>>>> < Vec Object: Exact Solution 2 MPI processes
>> > >>>>>> < type: mpikokkos
>> > >>>>>> < Process [0]
>> > >>>>>> < 0.
>> > >>>>>> < 0.015625
>> > >>>>>> < 0.125
>> > >>>>>> < Process [1]
>> > >>>>>> < 0.421875
>> > >>>>>> < 1.
>> > >>>>>> < Vec Object: Forcing function 2 MPI processes
>> > >>>>>> < type: mpikokkos
>> > >>>>>> < Process [0]
>> > >>>>>> < 1e-72
>> > >>>>>> < 1.50024
>> > >>>>>> < 3.01563
>> > >>>>>> < Process [1]
>> > >>>>>> < 4.67798
>> > >>>>>> < 7.
>> > >>>>>> < 0 SNES Function norm 5.414682427127e+00
>> > >>>>>> < 1 SNES Function norm 2.952582418265e-01
>> > >>>>>> < 2 SNES Function norm 4.502293658739e-04
>> > >>>>>> < 3 SNES Function norm 1.389665806646e-09
>> > >>>>>> < Number of SNES iterations = 3
>> > >>>>>> < Norm of error 1.49752e-10 Iterations 3
>> > >>>>>> ---
>> > >>>>>> > slurmstepd: error: execve():
>> > >>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such
>> > >>>>>> file or directory
>> > >>>>>> > slurmstepd: error: execve():
>> > >>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such
>> > >>>>>> file or directory
>> > >>>>>> > srun: error: spock01: tasks 0-1: Exited with exit code 2
>> > >>>>>> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0
>> > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials
>> > >>>>>> Possible problem with ex3k running with kokkos-kernels, diffs
>> above
>> > >>>>>> =========================================
>> > >>>>>> *******************Error detected during compile or
>> > >>>>>> link!*******************
>> > >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f
>> > >>>>>> *********************************************************
>> > >>>>>> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64
>> > >>>>>> -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0
>> > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>> > >>>>>>
>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
>> > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90
>> > >>>>>>
>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> > >>>>>>
>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> > >>>>>>
>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> > >>>>>>
>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> > >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> > >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>> > >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>> > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>> > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>> > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib
>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>> > >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib
>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>> > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>> > >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> > >>>>>>
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>> > >>>>>>
>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>> > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>> > >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>> > >>>>>>
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>> > >>>>>>
>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>> > >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore
>> -lhipsparse
>> > >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64
>> -lhsa-runtime64
>> > >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray
>> -lmpi_cray
>> > >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath
>> -lcrayacc_amdgpu
>> > >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran
>> -lpthread
>> > >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
>> > >>>>>> -lquadmath -lstdc++ -ldl -o ex5f
>> > >>>>>>
>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
>> > >>>>>> warning: alignment 128 of symbol
>> > >>>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in
>> > >>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller
>> than 256 in
>> > >>>>>> /tmp/pe_30429/ex5f_1.o
>> > >>>>>>
>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
>> > >>>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in
>> > >>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller
>> than 256 in
>> > >>>>>> /tmp/pe_30429/ex5f_1.o
>> > >>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1
>> MPI
>> > >>>>>> process
>> > >>>>>> Completed test examples
>> > >>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main=
>> > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ ll
>> > >>>>>>
>> > >>>>>> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang <
>> > >>>>>> junchao.zhang at gmail.com> wrote:
>> > >>>>>>
>> > >>>>>>> Mark, I configured with Fortran binding enabled with main + MR
>> !4175
>> > >>>>>>> <https://gitlab.com/petsc/petsc/-/merge_requests/4175>
>> > >>>>>>>
>> > >>>>>>> '--with-mpiexec=srun',
>> > >>>>>>> '--with-shared-libraries=1',
>> > >>>>>>> '--with-cc=cc',
>> > >>>>>>> '--with-cxx=CC',
>> > >>>>>>> '--with-fc=ftn',
>> > >>>>>>> '--with-fortran-bindings',
>> > >>>>>>> '--with-hip',
>> > >>>>>>> '--with-hipc=hipcc',
>> > >>>>>>> '--with-debugging',
>> > >>>>>>> '--CPPFLAGS=-I${ROCM_PATH}/include',
>> > >>>>>>> '--CXXPPFLAGS=-I${ROCM_PATH}/include',
>> > >>>>>>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64
>> -lhsa-runtime64',
>> > >>>>>>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64
>> -lhsa-runtime64',
>> > >>>>>>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64
>> -lhsa-runtime64',
>> > >>>>>>> '--COPTFLAGS=-g -O0',
>> > >>>>>>> '--CXXOPTFLAGS=-g -O0',
>> > >>>>>>> '--FOPTFLAGS=-g -O0',
>> > >>>>>>> '--download-kokkos',
>> > >>>>>>> '--download-kokkos-kernels',
>> > >>>>>>> '--download-kokkos-commit=3.4.01',
>> > >>>>>>> '--download-kokkos-kernels-commit=3.4.01',
>> > >>>>>>> '--with-kokkos-hip-arch=VEGA908',
>> > >>>>>>>
>> > >>>>>>> and 'make check' ran smoothly on a compute node
>> > >>>>>>>
>> > >>>>>>> $ make check
>> > >>>>>>> Running check examples to verify correct installation
>> > >>>>>>> Using PETSC_DIR=/ccs/home/jczhang/petsc and
>> > >>>>>>> PETSC_ARCH=arch-spock-cray-kokkos-dbg
>> > >>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1
>> MPI
>> > >>>>>>> process
>> > >>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2
>> MPI
>> > >>>>>>> processes
>> > >>>>>>> C/C++ example src/snes/tutorials/ex3k run successfully with
>> > >>>>>>> kokkos-kernels
>> > >>>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1
>> MPI
>> > >>>>>>> process
>> > >>>>>>>
>> > >>>>>>> Completed test examples
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>> --Junchao Zhang
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfadams at lbl.gov>
>> wrote:
>> > >>>>>>>
>> > >>>>>>>> And I find that this error, on non-Kokkos C tests, is fixed by
>> > >>>>>>>> turning the fortran bindings off:
>> > >>>>>>>>
>> > >>>>>>>> ld.lld: error:
>> > >>>>>>>>
>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
>> > >>>>>>>> undefined reference to
>> .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
>> > >>>>>>>> [--no-allow-shlib-undefined]
>> > >>>>>>>> ld.lld: error:
>> > >>>>>>>>
>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
>> > >>>>>>>> undefined reference to
>> .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
>> > >>>>>>>> [--no-allow-shlib-undefined]
>> > >>>>>>>> ld.lld: error:
>> > >>>>>>>>
>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
>> > >>>>>>>> undefined reference to
>> .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
>> > >>>>>>>> [--no-allow-shlib-undefined]
>> > >>>>>>>>
>> > >>>>>>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams at lbl.gov>
>> wrote:
>> > >>>>>>>>
>> > >>>>>>>>> Not complex. THis has some overlap with my problem w/o Kokkos.
>> > >>>>>>>>>
>> > >>>>>>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang <
>> > >>>>>>>>> junchao.zhang at gmail.com> wrote:
>> > >>>>>>>>>
>> > >>>>>>>>>> Do you use complex? post your configure.log.
>> > >>>>>>>>>>
>> > >>>>>>>>>> --Junchao Zhang
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams at lbl.gov>
>> > >>>>>>>>>> wrote:
>> > >>>>>>>>>>
>> > >>>>>>>>>>> The simple Kokkos example is failing for me on Spock.
>> > >>>>>>>>>>> Any ideas?
>> > >>>>>>>>>>> Thanks,
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> 10:44 main *=
>> > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$
>> make
>> > >>>>>>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>> > >>>>>>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k
>> > >>>>>>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC
>> > >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64
>> > >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64
>> -lhsa-runtime64
>> > >>>>>>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing
>> -Wno-unknown-pragmas
>> > >>>>>>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall
>> -Wwrite-strings
>> > >>>>>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas
>> -fvisibility=hidden -g -O3
>> > >>>>>>>>>>> -std=c++14
>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>> > >>>>>>>>>>>
>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include
>> > >>>>>>>>>>>
>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include
>> > >>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>> > >>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>> ex3k.kokkos.cxx
>> > >>>>>>>>>>>
>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
>> > >>>>>>>>>>>
>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
>> > >>>>>>>>>>>
>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
>> > >>>>>>>>>>>
>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
>> > >>>>>>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> > >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> > >>>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64
>> -L/opt/gcc/8.1.0/snos/lib64
>> > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/
>> 21.04.1.1/CRAY/9.0/x86_64/lib
>> > >>>>>>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>> > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>> > >>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>> > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>> > >>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib
>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>> > >>>>>>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib
>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> > >>>>>>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>> > >>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>> > >>>>>>>>>>>
>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> > >>>>>>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> > >>>>>>>>>>>
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>> > >>>>>>>>>>>
>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>> > >>>>>>>>>>>
>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>> > >>>>>>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>> > >>>>>>>>>>>
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>> > >>>>>>>>>>>
>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>> > >>>>>>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore
>> -lhipsparse
>> > >>>>>>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64
>> -lhsa-runtime64
>> > >>>>>>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray
>> -lmpifort_cray -lmpi_cray
>> > >>>>>>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath
>> -lcrayacc_amdgpu
>> > >>>>>>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup
>> -lgfortran -lpthread
>> > >>>>>>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64
>> -lclang_rt.builtins-x86_64
>> > >>>>>>>>>>> -lquadmath -lstdc++ -ldl -o ex3k
>> > >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>> > >>>>>>>>>>>
>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11:
>> > >>>>>>>>>>> error: no member named 'lock_address_host_space' in
>> namespace
>> > >>>>>>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply
>> 'lock_address_host_space'?
>> > >>>>>>>>>>> while (!Impl::lock_address_host_space((void*)dest))
>> > >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> > >>>>>>>>>>> lock_address_host_space
>> > >>>>>>>>>>>
>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6:
>> > >>>>>>>>>>> note: 'lock_address_host_space' declared here
>> > >>>>>>>>>>> bool lock_address_host_space(void* ptr);
>> > >>>>>>>>>>> ^
>> > >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>> > >>>>>>>>>>>
>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3:
>> > >>>>>>>>>>> error: no member named 'memory_fence' in namespace
>> 'Kokkos::Impl::Kokkos';
>> > >>>>>>>>>>> did you mean simply 'memory_fence'?
>> > >>>>>>>>>>> Kokkos::memory_fence();
>> > >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~
>> > >>>>>>>>>>> memory_fence
>> > >>>>>>>>>>>
>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6:
>> > >>>>>>>>>>> note: 'memory_fence' declared here
>> > >>>>>>>>>>> void memory_fence() {
>> > >>>>>>>>>>> ^
>> > >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>> > >>>>>>>>>>> In file included from
>> > >>>>>>>>>>>
>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>> > >>>>>>>>>>>
>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3:
>> > >>>>>>>>>>> error: no member named 'memory_fence' in namespace
>> 'Kokkos::Impl::Kokkos';
>> > >>>>>>>>>>> did you mean simply 'memory_fence'?
>> > >>>>>>>>>>> Kokkos::memory_fence();
>> > >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~
>> > >>>>>>>>>>> memory_fence
>> > >>>>>>>>>>>
>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6:
>> > >>>>>>>>>>> note: 'memory_fence' declared here
>> > >>>>>>>>>>> void memory_fence() {
>> > >>>>>>>>>>> ^
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>>
>>
>>
>>
>> --
>> Scott Kruger
>> Tech-X Corporation kruger at txcorp.com
>> 5621 Arapahoe Ave, Suite A Phone: (720) 466-3196
>> Boulder, CO 80303 Fax: (303) 448-7756
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20210720/7a7d9d79/attachment-0001.html>
More information about the petsc-dev
mailing list