<div dir="ltr">OK, it looks like this flag caused the problem and does not seem to be necessary:<div><br></div><div>'--LDFLAGS=-L'+os.environ['ROCM_PATH']+'/lib -lhsa-runtime64',<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jul 20, 2021 at 9:00 AM Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 19, 2021 at 6:41 PM Scott Kruger <<a href="mailto:kruger@txcorp.com" target="_blank">kruger@txcorp.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Mark,<br>
<br>
On tulip, things with fortran went wonky when a `-fast` flag snuck into<br>
the flags (snuck in being me copying a previous configure file and not<br>
questioning the flags). The reason is that for clang, `-fast` implies<br>
link time optimization (lto) for C/C++ code, but flang doesn't support<br>
lto so things got weird. I suspect that gfortran does not either (but<br>
perhaps the real question is why not use flang?)<br></blockquote><div><br></div><div>It does work with GNU. I am doing this for an application and I want to avoid dictating the program env.</div><div><br></div><div>I'm (re)trying with -O0 just to check.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
I look at configure and I don't see anything in your flags that would<br>
trigger lto, but as Junchao says, it might be picking something up from<br>
Kokkos, so perhaps this is the issue.<br>
<br></blockquote><div><br></div><div>The configure files that I sent did not have Kokkos. They were stripped down.</div><div><br></div><div>I have asked ORNL, but don't expect help. </div><div>The person (Matt) that is helping me for other things is very helpful so perhaps he can come up with some ideas. </div><div>Otherwise I will tell the app that they need to use GNU for now.</div><div><br></div><div>Thanks,</div><div>Mark</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Scott<br>
<br>
<br>
On 2021-07-19 07:19, Mark Adams did write:<br>
> Thanks, but this happens w/o Kokkos.<br>
> I've stripped this down and attached good/bad logs without/with Fortran<br>
> bindings.<br>
> Hope this helps,<br>
> Thanks again,<br>
> Mark<br>
> <br>
> On Sun, Jul 18, 2021 at 12:00 PM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>><br>
> wrote:<br>
> <br>
> > This is probably kokkos pulling in the dependencies where compiling kokkos<br>
> > source within PETSc.<br>
> ><br>
> > Il Dom 18 Lug 2021, 16:29 Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> ha scritto:<br>
> ><br>
> >> Whoops, this error was just from not telling gfortran to allow long lines.<br>
> >><br>
> >> Anway, I did find that the when fortran bindings are enabled this OMP<br>
> >> thing gets into the library. No idea how. I'll askm ORNL.<br>
> >><br>
> >> 10:26 jczhang/fix-cray-mpicxx-includes/main=<br>
> >> /gpfs/alpine/csc314/scratch/adams/petsc2$ nm<br>
> >> /gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so<br>
> >> |g offload<br>
> >> U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa<br>
> >> U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa<br>
> >> U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa<br>
> >> 0000000001d457b0 T vecgetoffloadmask_<br>
> >><br>
> >><br>
> >> On Sun, Jul 18, 2021 at 8:43 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
> >><br>
> >>> Ah, your test was not on Spock ... I have something working but this is<br>
> >>> strange.<br>
> >>><br>
> >>> I switched to the GNU ProgEnv. and it passed the Fortran test in<br>
> >>> 'check', but this failed:<br>
> >>><br>
> >>><br>
> >>><br>
> >>><br>
> >>><br>
> >>><br>
> >>><br>
> >>><br>
> >>><br>
> >>> *08:26 2 jczhang/fix-cray-mpicxx-includes/main=<br>
> >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make<br>
> >>> PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu<br>
> >>> PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib<br>
> >>> -lamdhip64 -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0<br>
> >>> -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include<br>
> >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90<br>
> >>> -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib<br>
> >>> -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib<br>
> >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib<br>
> >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib<br>
> >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib<br>
> >>> -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib<br>
> >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib<br>
> >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib<br>
> >>> -Wl,-rpath,/opt/cray/pe/libsci/<a href="http://21.04.1.1/GNU/9.1/x86_64/lib" rel="noreferrer" target="_blank">21.04.1.1/GNU/9.1/x86_64/lib</a><br>
> >>> <<a href="http://21.04.1.1/GNU/9.1/x86_64/lib" rel="noreferrer" target="_blank">http://21.04.1.1/GNU/9.1/x86_64/lib</a>><br>
> >>> -L/opt/cray/pe/libsci/<a href="http://21.04.1.1/GNU/9.1/x86_64/lib" rel="noreferrer" target="_blank">21.04.1.1/GNU/9.1/x86_64/lib</a><br>
> >>> <<a href="http://21.04.1.1/GNU/9.1/x86_64/lib" rel="noreferrer" target="_blank">http://21.04.1.1/GNU/9.1/x86_64/lib</a>><br>
> >>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib<br>
> >>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib<br>
> >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib<br>
> >>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64<br>
> >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64<br>
> >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0<br>
> >>> -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0<br>
> >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64<br>
> >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib<br>
> >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib<br>
> >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib -lpetsc<br>
> >>> -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64<br>
> >>> -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem -lgfortran -lm<br>
> >>> -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml<br>
> >>> -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++ -ldl -o<br>
> >>> ex40f90ex40f90.F90:35:132: 35 | call<br>
> >>> DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one,<br>
> >>> & |<br>
> >>> 1Error:<br>
> >>> Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:*<br>
> >>> * ....*<br>
> >>><br>
> >>> But ex5f (the Fortran check test) does work manually also (sanity check).<br>
> >>><br>
> >>> Oddly I seem to have the cray compilers:<br>
> >>><br>
> >>> 08:34 jczhang/fix-cray-mpicxx-includes/main=<br>
> >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which ftn<br>
> >>> /opt/cray/pe/craype/2.7.6/bin/ftn<br>
> >>><br>
> >>> yet it is GNU:<br>
> >>><br>
> >>> 08:34 jczhang/fix-cray-mpicxx-includes/main=<br>
> >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ ftn --version<br>
> >>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.)<br>
> >>> Copyright (C) 2020 Free Software Foundation, Inc.<br>
> >>> This is free software; see the source for copying conditions. There is<br>
> >>> NO<br>
> >>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR<br>
> >>> PURPOSE.<br>
> >>><br>
> >>> The documentation says ftn should be here:<br>
> >>><br>
> >>> 08:34 jczhang/fix-cray-mpicxx-includes/main=<br>
> >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$<br>
> >>> $GCC_PATH/bin/gfortran --version<br>
> >>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.)<br>
> >>> Copyright (C) 2020 Free Software Foundation, Inc.<br>
> >>> This is free software; see the source for copying conditions. There is<br>
> >>> NO<br>
> >>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR<br>
> >>> PURPOSE.<br>
> >>><br>
> >>> Same, but the path is not the same:<br>
> >>><br>
> >>> 08:39 jczhang/fix-cray-mpicxx-includes/main=<br>
> >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ echo $GCC_PATH<br>
> >>> /opt/gcc/10.2.0<br>
> >>><br>
> >>> Anyway, thanks for your help,<br>
> >>> Mark<br>
> >>><br>
> >>> On Sat, Jul 17, 2021 at 11:26 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>><br>
> >>> wrote:<br>
> >>><br>
> >>>> Mark, I can reproduce this error with PrgEnv-cray, i.e., using the<br>
> >>>> Cray compiler (clang-11). Previously I used PrgEnv-gnu, which did not have<br>
> >>>> this error.<br>
> >>>> Probably it is a problem of Spock. But I am not sure.<br>
> >>>><br>
> >>>> --Junchao Zhang<br>
> >>>><br>
> >>>><br>
> >>>> On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
> >>>><br>
> >>>>> And I can run a fortran test, with warnings, but C tests fail:<br>
> >>>>><br>
> >>>>> 11:15 jczhang/fix-cray-mpicxx-includes/main=<br>
> >>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make<br>
> >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc<br>
> >>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test<br>
> >>>>> search='ts_utils_dmplexlandau_tutorials-ex1f90_0'<br>
> >>>>> Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0<br>
> >>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos<br>
> >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc<br>
> >>>>> FC<br>
> >>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o<br>
> >>>>> FLINKER<br>
> >>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90<br>
> >>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:<br>
> >>>>> warning: alignment 128 of symbol<br>
> >>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in<br>
> >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in<br>
> >>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o<br>
> >>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:<br>
> >>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in<br>
> >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in<br>
> >>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o<br>
> >>>>> TEST<br>
> >>>>> arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts<br>
> >>>>> ok ts_utils_dmplexlandau_tutorials-ex1f90_0<br>
> >>>>> ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0<br>
> >>>>><br>
> >>>>><br>
> >>>>> On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
> >>>>><br>
> >>>>>> HUmm, I can not reproduce this.<br>
> >>>>>><br>
> >>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main=<br>
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make<br>
> >>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc<br>
> >>>>>> PETSC_ARCH=arch-spock-dbg-kokkos check<br>
> >>>>>> Running check examples to verify correct installation<br>
> >>>>>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and<br>
> >>>>>> PETSC_ARCH=arch-spock-dbg-kokkos<br>
> >>>>>> gmake[3]:<br>
> >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301:<br>
> >>>>>> ex19.PETSc] Error 2 (ignored)<br>
> >>>>>> *******************Error detected during compile or<br>
> >>>>>> link!*******************<br>
> >>>>>> See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><br>
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19<br>
> >>>>>><br>
> >>>>>> *********************************************************************************<br>
> >>>>>> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64<br>
> >>>>>> -lhsa-runtime64 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing<br>
> >>>>>> -Wno-unknown-pragmas -fstack-protector -Qunused-arguments<br>
> >>>>>> -fvisibility=hidden -g -O0 -fPIC -Wall -Wwrite-strings<br>
> >>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector<br>
> >>>>>> -Qunused-arguments -fvisibility=hidden -g -O0<br>
> >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include<br>
> >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include<br>
> >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include<br>
> >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include<br>
> >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c<br>
> >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib<br>
> >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib<br>
> >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib<br>
> >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib<br>
> >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib<br>
> >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib<br>
> >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64<br>
> >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/lib" rel="noreferrer" target="_blank">21.04.1.1/CRAY/9.0/x86_64/lib</a><br>
> >>>>>> -L/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/lib" rel="noreferrer" target="_blank">21.04.1.1/CRAY/9.0/x86_64/lib</a><br>
> >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib<br>
> >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib<br>
> >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib<br>
> >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib<br>
> >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib<br>
> >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib<br>
> >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib<br>
> >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib<br>
> >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64<br>
> >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64<br>
> >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux<br>
> >>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux<br>
> >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0<br>
> >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0<br>
> >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib<br>
> >>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib<br>
> >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse<br>
> >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64<br>
> >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray<br>
> >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu<br>
> >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread<br>
> >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64<br>
> >>>>>> -lquadmath -lstdc++ -ldl -o ex19<br>
> >>>>>> ld.lld: error:<br>
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:<br>
> >>>>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa<br>
> >>>>>> [--no-allow-shlib-undefined]<br>
> >>>>>> ld.lld: error:<br>
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:<br>
> >>>>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa<br>
> >>>>>> [--no-allow-shlib-undefined]<br>
> >>>>>> ld.lld: error:<br>
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:<br>
> >>>>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa<br>
> >>>>>> [--no-allow-shlib-undefined]<br>
> >>>>>> clang-11: error: linker command failed with exit code 1 (use -v to<br>
> >>>>>> see invocation)<br>
> >>>>>> gmake[4]: *** [<builtin>: ex19] Error 1<br>
> >>>>>> gmake[3]:<br>
> >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301:<br>
> >>>>>> ex3k.PETSc] Error 2 (ignored)<br>
> >>>>>> *******************Error detected during compile or<br>
> >>>>>> link!*******************<br>
> >>>>>> See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><br>
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k<br>
> >>>>>><br>
> >>>>>> *********************************************************************************<br>
> >>>>>> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing<br>
> >>>>>> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14<br>
> >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include<br>
> >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include<br>
> >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include<br>
> >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/opt/cray/pe/libsci/<br>
> >>>>>> <a href="http://21.04.1.1/CRAY/9.0/x86_64/include" rel="noreferrer" target="_blank">21.04.1.1/CRAY/9.0/x86_64/include</a><br>
> >>>>>> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include<br>
> >>>>>> -I/opt/cray/pe/pmi/6.0.10/include -I/opt/cray/pe/dsmml/0.1.4/dsmml//include<br>
> >>>>>> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include<br>
> >>>>>> -L/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/lib" rel="noreferrer" target="_blank">21.04.1.1/CRAY/9.0/x86_64/lib</a><br>
> >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib<br>
> >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib<br>
> >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib<br>
> >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../<br>
> >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64<br>
> >>>>>> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed<br>
> >>>>>> -Wl,--as-needed,-lsci_cray,--no-as-needed<br>
> >>>>>> -Wl,--as-needed,-lmpi_cray,--no-as-needed<br>
> >>>>>> -Wl,--as-needed,-ldsmml,--no-as-needed<br>
> >>>>>> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl -lmpi_gtl_hsa<br>
> >>>>>> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup<br>
> >>>>>> ex3k.kokkos.cxx<br>
> >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib<br>
> >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib<br>
> >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib<br>
> >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib<br>
> >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib<br>
> >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib<br>
> >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64<br>
> >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/lib" rel="noreferrer" target="_blank">21.04.1.1/CRAY/9.0/x86_64/lib</a><br>
> >>>>>> -L/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/lib" rel="noreferrer" target="_blank">21.04.1.1/CRAY/9.0/x86_64/lib</a><br>
> >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib<br>
> >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib<br>
> >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib<br>
> >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib<br>
> >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib<br>
> >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib<br>
> >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib<br>
> >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib<br>
> >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64<br>
> >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64<br>
> >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux<br>
> >>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux<br>
> >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0<br>
> >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0<br>
> >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib<br>
> >>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib<br>
> >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse<br>
> >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64<br>
> >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray<br>
> >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu<br>
> >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread<br>
> >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64<br>
> >>>>>> -lquadmath -lstdc++ -ldl -o ex3k<br>
> >>>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:<br>
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:<br>
> >>>>>> undefined reference to `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa'<br>
> >>>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:<br>
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:<br>
> >>>>>> undefined reference to `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa'<br>
> >>>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:<br>
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:<br>
> >>>>>> undefined reference to `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa'<br>
> >>>>>> clang-12: error: linker command failed with exit code 1 (use -v to<br>
> >>>>>> see invocation)<br>
> >>>>>> gmake[4]: ***<br>
> >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: ex3k]<br>
> >>>>>> Error 1<br>
> >>>>>> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored)<br>
> >>>>>> 1,25c1,4<br>
> >>>>>> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000<br>
> >>>>>> < Vec Object: Exact Solution 2 MPI processes<br>
> >>>>>> < type: mpikokkos<br>
> >>>>>> < Process [0]<br>
> >>>>>> < 0.<br>
> >>>>>> < 0.015625<br>
> >>>>>> < 0.125<br>
> >>>>>> < Process [1]<br>
> >>>>>> < 0.421875<br>
> >>>>>> < 1.<br>
> >>>>>> < Vec Object: Forcing function 2 MPI processes<br>
> >>>>>> < type: mpikokkos<br>
> >>>>>> < Process [0]<br>
> >>>>>> < 1e-72<br>
> >>>>>> < 1.50024<br>
> >>>>>> < 3.01563<br>
> >>>>>> < Process [1]<br>
> >>>>>> < 4.67798<br>
> >>>>>> < 7.<br>
> >>>>>> < 0 SNES Function norm 5.414682427127e+00<br>
> >>>>>> < 1 SNES Function norm 2.952582418265e-01<br>
> >>>>>> < 2 SNES Function norm 4.502293658739e-04<br>
> >>>>>> < 3 SNES Function norm 1.389665806646e-09<br>
> >>>>>> < Number of SNES iterations = 3<br>
> >>>>>> < Norm of error 1.49752e-10 Iterations 3<br>
> >>>>>> ---<br>
> >>>>>> > slurmstepd: error: execve():<br>
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such<br>
> >>>>>> file or directory<br>
> >>>>>> > slurmstepd: error: execve():<br>
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such<br>
> >>>>>> file or directory<br>
> >>>>>> > srun: error: spock01: tasks 0-1: Exited with exit code 2<br>
> >>>>>> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0<br>
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials<br>
> >>>>>> Possible problem with ex3k running with kokkos-kernels, diffs above<br>
> >>>>>> =========================================<br>
> >>>>>> *******************Error detected during compile or<br>
> >>>>>> link!*******************<br>
> >>>>>> See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><br>
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f<br>
> >>>>>> *********************************************************<br>
> >>>>>> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64<br>
> >>>>>> -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0<br>
> >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include<br>
> >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include<br>
> >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90<br>
> >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib<br>
> >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib<br>
> >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib<br>
> >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib<br>
> >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib<br>
> >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib<br>
> >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64<br>
> >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/lib" rel="noreferrer" target="_blank">21.04.1.1/CRAY/9.0/x86_64/lib</a><br>
> >>>>>> -L/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/lib" rel="noreferrer" target="_blank">21.04.1.1/CRAY/9.0/x86_64/lib</a><br>
> >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib<br>
> >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib<br>
> >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib<br>
> >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib<br>
> >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib<br>
> >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib<br>
> >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib<br>
> >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib<br>
> >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64<br>
> >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64<br>
> >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux<br>
> >>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux<br>
> >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0<br>
> >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0<br>
> >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib<br>
> >>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib<br>
> >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse<br>
> >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64<br>
> >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray<br>
> >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu<br>
> >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread<br>
> >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64<br>
> >>>>>> -lquadmath -lstdc++ -ldl -o ex5f<br>
> >>>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:<br>
> >>>>>> warning: alignment 128 of symbol<br>
> >>>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in<br>
> >>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in<br>
> >>>>>> /tmp/pe_30429/ex5f_1.o<br>
> >>>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:<br>
> >>>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in<br>
> >>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in<br>
> >>>>>> /tmp/pe_30429/ex5f_1.o<br>
> >>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI<br>
> >>>>>> process<br>
> >>>>>> Completed test examples<br>
> >>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main=<br>
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ ll<br>
> >>>>>><br>
> >>>>>> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang <<br>
> >>>>>> <a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br>
> >>>>>><br>
> >>>>>>> Mark, I configured with Fortran binding enabled with main + MR !4175<br>
> >>>>>>> <<a href="https://gitlab.com/petsc/petsc/-/merge_requests/4175" rel="noreferrer" target="_blank">https://gitlab.com/petsc/petsc/-/merge_requests/4175</a>><br>
> >>>>>>><br>
> >>>>>>> '--with-mpiexec=srun',<br>
> >>>>>>> '--with-shared-libraries=1',<br>
> >>>>>>> '--with-cc=cc',<br>
> >>>>>>> '--with-cxx=CC',<br>
> >>>>>>> '--with-fc=ftn',<br>
> >>>>>>> '--with-fortran-bindings',<br>
> >>>>>>> '--with-hip',<br>
> >>>>>>> '--with-hipc=hipcc',<br>
> >>>>>>> '--with-debugging',<br>
> >>>>>>> '--CPPFLAGS=-I${ROCM_PATH}/include',<br>
> >>>>>>> '--CXXPPFLAGS=-I${ROCM_PATH}/include',<br>
> >>>>>>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',<br>
> >>>>>>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',<br>
> >>>>>>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',<br>
> >>>>>>> '--COPTFLAGS=-g -O0',<br>
> >>>>>>> '--CXXOPTFLAGS=-g -O0',<br>
> >>>>>>> '--FOPTFLAGS=-g -O0',<br>
> >>>>>>> '--download-kokkos',<br>
> >>>>>>> '--download-kokkos-kernels',<br>
> >>>>>>> '--download-kokkos-commit=3.4.01',<br>
> >>>>>>> '--download-kokkos-kernels-commit=3.4.01',<br>
> >>>>>>> '--with-kokkos-hip-arch=VEGA908',<br>
> >>>>>>><br>
> >>>>>>> and 'make check' ran smoothly on a compute node<br>
> >>>>>>><br>
> >>>>>>> $ make check<br>
> >>>>>>> Running check examples to verify correct installation<br>
> >>>>>>> Using PETSC_DIR=/ccs/home/jczhang/petsc and<br>
> >>>>>>> PETSC_ARCH=arch-spock-cray-kokkos-dbg<br>
> >>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI<br>
> >>>>>>> process<br>
> >>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI<br>
> >>>>>>> processes<br>
> >>>>>>> C/C++ example src/snes/tutorials/ex3k run successfully with<br>
> >>>>>>> kokkos-kernels<br>
> >>>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI<br>
> >>>>>>> process<br>
> >>>>>>><br>
> >>>>>>> Completed test examples<br>
> >>>>>>><br>
> >>>>>>><br>
> >>>>>>> --Junchao Zhang<br>
> >>>>>>><br>
> >>>>>>><br>
> >>>>>>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
> >>>>>>><br>
> >>>>>>>> And I find that this error, on non-Kokkos C tests, is fixed by<br>
> >>>>>>>> turning the fortran bindings off:<br>
> >>>>>>>><br>
> >>>>>>>> ld.lld: error:<br>
> >>>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:<br>
> >>>>>>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa<br>
> >>>>>>>> [--no-allow-shlib-undefined]<br>
> >>>>>>>> ld.lld: error:<br>
> >>>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:<br>
> >>>>>>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa<br>
> >>>>>>>> [--no-allow-shlib-undefined]<br>
> >>>>>>>> ld.lld: error:<br>
> >>>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:<br>
> >>>>>>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa<br>
> >>>>>>>> [--no-allow-shlib-undefined]<br>
> >>>>>>>><br>
> >>>>>>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
> >>>>>>>><br>
> >>>>>>>>> Not complex. THis has some overlap with my problem w/o Kokkos.<br>
> >>>>>>>>><br>
> >>>>>>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang <<br>
> >>>>>>>>> <a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br>
> >>>>>>>>><br>
> >>>>>>>>>> Do you use complex? post your configure.log.<br>
> >>>>>>>>>><br>
> >>>>>>>>>> --Junchao Zhang<br>
> >>>>>>>>>><br>
> >>>>>>>>>><br>
> >>>>>>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br>
> >>>>>>>>>> wrote:<br>
> >>>>>>>>>><br>
> >>>>>>>>>>> The simple Kokkos example is failing for me on Spock.<br>
> >>>>>>>>>>> Any ideas?<br>
> >>>>>>>>>>> Thanks,<br>
> >>>>>>>>>>><br>
> >>>>>>>>>>> 10:44 main *=<br>
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make<br>
> >>>>>>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc<br>
> >>>>>>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k<br>
> >>>>>>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC<br>
> >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64<br>
> >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64<br>
> >>>>>>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas<br>
> >>>>>>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings<br>
> >>>>>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3<br>
> >>>>>>>>>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include<br>
> >>>>>>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include<br>
> >>>>>>>>>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include<br>
> >>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include<br>
> >>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx<br>
> >>>>>>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib<br>
> >>>>>>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib<br>
> >>>>>>>>>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib<br>
> >>>>>>>>>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib<br>
> >>>>>>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib<br>
> >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib<br>
> >>>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64<br>
> >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/lib" rel="noreferrer" target="_blank">21.04.1.1/CRAY/9.0/x86_64/lib</a><br>
> >>>>>>>>>>> -L/opt/cray/pe/libsci/<a href="http://21.04.1.1/CRAY/9.0/x86_64/lib" rel="noreferrer" target="_blank">21.04.1.1/CRAY/9.0/x86_64/lib</a><br>
> >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib<br>
> >>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib<br>
> >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib<br>
> >>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib<br>
> >>>>>>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib<br>
> >>>>>>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib<br>
> >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib<br>
> >>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib<br>
> >>>>>>>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64<br>
> >>>>>>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64<br>
> >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux<br>
> >>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux<br>
> >>>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0<br>
> >>>>>>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0<br>
> >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib<br>
> >>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib<br>
> >>>>>>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse<br>
> >>>>>>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64<br>
> >>>>>>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray<br>
> >>>>>>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu<br>
> >>>>>>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread<br>
> >>>>>>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64<br>
> >>>>>>>>>>> -lquadmath -lstdc++ -ldl -o ex3k<br>
> >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:<br>
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11:<br>
> >>>>>>>>>>> error: no member named 'lock_address_host_space' in namespace<br>
> >>>>>>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'?<br>
> >>>>>>>>>>> while (!Impl::lock_address_host_space((void*)dest))<br>
> >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
> >>>>>>>>>>> lock_address_host_space<br>
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6:<br>
> >>>>>>>>>>> note: 'lock_address_host_space' declared here<br>
> >>>>>>>>>>> bool lock_address_host_space(void* ptr);<br>
> >>>>>>>>>>> ^<br>
> >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:<br>
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3:<br>
> >>>>>>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos';<br>
> >>>>>>>>>>> did you mean simply 'memory_fence'?<br>
> >>>>>>>>>>> Kokkos::memory_fence();<br>
> >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~<br>
> >>>>>>>>>>> memory_fence<br>
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6:<br>
> >>>>>>>>>>> note: 'memory_fence' declared here<br>
> >>>>>>>>>>> void memory_fence() {<br>
> >>>>>>>>>>> ^<br>
> >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:<br>
> >>>>>>>>>>> In file included from<br>
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:<br>
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3:<br>
> >>>>>>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos';<br>
> >>>>>>>>>>> did you mean simply 'memory_fence'?<br>
> >>>>>>>>>>> Kokkos::memory_fence();<br>
> >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~<br>
> >>>>>>>>>>> memory_fence<br>
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6:<br>
> >>>>>>>>>>> note: 'memory_fence' declared here<br>
> >>>>>>>>>>> void memory_fence() {<br>
> >>>>>>>>>>> ^<br>
> >>>>>>>>>>><br>
> >>>>>>>>>><br>
<br>
<br>
<br>
<br>
-- <br>
Scott Kruger<br>
Tech-X Corporation <a href="mailto:kruger@txcorp.com" target="_blank">kruger@txcorp.com</a><br>
5621 Arapahoe Ave, Suite A Phone: (720) 466-3196<br>
Boulder, CO 80303 Fax: (303) 448-7756<br>
</blockquote></div></div>
</blockquote></div>