<div dir="ltr"><div>Satish is right.  ex55k is not standalone.  </div><div>There are many examples in petsc tests that can run with kokkos.   Just search "requires: kokkos_kernels"  and you will find them.  ex55k.kokkos.cxx is an example showing how to use petsc/kokkos functions like DMDAVecRestoreKokkosOffsetView, MatSetValuesCOO etc.</div><div><br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 14, 2024 at 11:23 AM Satish Balay via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Looks like ex55 is the one to use - that links in with ex55k<br>
<br>
But it needs a fix for a build from 'make'<br>
<br>
>>>>>>>>>><br>
balay@petsc-gpu-01:/scratch/balay/petsc/src/snes/tutorials$ git diff<br>
diff --git a/src/snes/tutorials/makefile b/src/snes/tutorials/makefile<br>
index 672a62aa5a0..eed127f7eae 100644<br>
--- a/src/snes/tutorials/makefile<br>
+++ b/src/snes/tutorials/makefile<br>
@@ -8,6 +8,8 @@ CLEANFILES       = ex5f90t<br>
 include ${PETSC_DIR}/lib/petsc/conf/variables<br>
 include ${PETSC_DIR}/lib/petsc/conf/rules<br>
<br>
+ex55: ex55.o ex55k.o<br>
+<br>
 #-------------------------------------------------------------------------<br>
<br>
 #  these tests are used by the makefile in PETSC_DIR for basic tests of the install and should not be removed<br>
balay@petsc-gpu-01:/scratch/balay/petsc/src/snes/tutorials$ make ex55<br>
mpicc -o ex55.o -c -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -g3 -O0  -I/scratch/balay/petsc/include -I/scratch/balay/petsc/arch-linux-c-debug/include -I/usr/local/cuda/include    `pwd`/ex55.c<br>
PATH=`dirname /usr/local/cuda/bin/nvcc`:/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/bin:/usr/lib/ccache:/home/balay/.local/bin:/home/balay/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin::/nfs/gce/projects/gce/bin NVCC_WRAPPER_DEFAULT_COMPILER="mpicxx" /scratch/balay/petsc/arch-linux-c-debug/bin/nvcc_wrapper --expt-extended-lambda -c -ccbin mpicxx -std=c++17 -Xcompiler -fPIC -g -lineinfo -gencode arch=compute_86,code=sm_86 -I/scratch/balay/petsc/include -I/scratch/balay/petsc/arch-linux-c-debug/include -I/usr/local/cuda/include -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-psabi -fstack-protector -g -I/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/include -o ex55k.o `pwd`/ex55k.kokkos.cxx<br>
mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -g3 -O0   -Wl,-export-dynamic ex55.o ex55k.o  -Wl,-rpath,/scratch/balay/petsc/arch-linux-c-debug/lib -L/scratch/balay/petsc/arch-linux-c-debug/lib -Wl,-rpath,/scratch/balay/petsc/arch-linux-c-debug/lib -L/scratch/balay/petsc/arch-linux-c-debug/lib -Wl,-rpath,/usr/local/cuda/lib64 -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib64/stubs -Wl,-rpath,/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/lib -L/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/11 -L/usr/lib/gcc/x86_64-linux-gnu/11 -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lkokkossimd -llapack -lblas -lm -lcudart -lnvToolsExt -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lX11 -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -lquadmath -o ex55<br>
balay@petsc-gpu-01:/scratch/balay/petsc/src/snes/tutorials$ <br>
<<<<<<<<br>
<br>
Satish<br>
<br>
<br>
On Wed, 14 Feb 2024, Uralovich, Ibragimov Iskander wrote:<br>
<br>
> Hello!<br>
> <br>
> I want to start porting part of our PETSc-based code to GPU through kokkos and I want to start with excercises given in tutorials folder.<br>
> <br>
> For that I found two examples of using PETSC with kokkos in folder petsc/src/snes/tutorials<br>
> <br>
> I managed to succesfully compile and run ex3k.kokkos.cxx, but I cannot compile ex55k.kokkos.cxx.<br>
> when I do make i got a following error:<br>
> <br>
> <br>
> iskander@apollo:~/software/petsc3_20/src/snes/tutorials$ make ex55k<br>
> PATH=`dirname nvcc`:/local/home/iskander/software/petsc3_20/petsc-3.20-deb/bin:/usr/local/cuda-12.2/bin:/local/home/iskander/.vscode-server/bin/05047486b6df5eb8d44b2ecd70ea3bdf775fd937/bin/remote-cli:/local/home/iskander/.local/bin:/local/home/iskander/software/petsc3_20/petsc-3.20-deb/bin:/usr/local/cuda-12.2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/paraview/v5.10.0/bin NVCC_WRAPPER_DEFAULT_COMPILER="/local/home/iskander/software/petsc3_20/petsc-3.20-deb/bin"/mpicxx /local/home/iskander/software/petsc3_20/petsc-3.20-deb/bin/nvcc_wrapper --expt-extended-lambda -ccbin /local/home/iskander/software/petsc3_20/petsc-3.20-deb/bin/mpicxx -std=c++17 -Xcompiler -fPIC -Xcompiler -fvisibility=hidden -g -lineinfo -gencode arch=compute_80,code=sm_80  -I/local/home/iskander/software/petsc3_20/petsc-3.20-deb/include  -I/local/home/iskander/software/petsc3_20/include -I/local/home/iskander/software/petsc3_20/arch-linux-c-d<br>
 ebug/inc<br>
 lude -I/local/home/iskander/software/petsc3_20/petsc-3.20-deb/include -I/usr/local/cuda-12.2/include    -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-psabi -fstack-protector -fvisibility=hidden -march=znver2    -L/local/home/iskander/software/petsc3_20/petsc-3.20-deb/lib -Wl,-rpath -Wl,/local/home/iskander/software/petsc3_20/petsc-3.20-deb/lib -Wl,--enable-new-dtags  -lmpi   ex55k.kokkos.cxx  -Wl,-rpath,/local/home/iskander/software/petsc3_20/arch-linux-c-debug/lib -L/local/home/iskander/software/petsc3_20/arch-linux-c-debug/lib -Wl,-rpath,/local/home/iskander/software/petsc3_20/petsc-3.20-deb/lib -L/local/home/iskander/software/petsc3_20/petsc-3.20-deb/lib -Wl,-rpath,/opt/amd -L/opt/amd -Wl,-rpath,/usr/local/cuda-12.2/lib64 -L/usr/local/cuda-12.2/lib64 -L/usr/local/cuda-12.2/lib64/stubs -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/11 -L/usr/lib/gcc/x86_64-linux-gnu/11 -lpetsc -ldmumps -lmumps_common -lpord -lpthread -lscalapack -lsu<br>
 perlu_di<br>
 st -lkokkoskernels -lkokkoscontainers -lkokkoscore -lkokkossimd -lflame -lblis -lptesmumps -lptscotchparmetisv3 -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lparmetis -lmetis -lm -lcudart -lnvToolsExt -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lX11 -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -lrt -o ex55k<br>
> /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/Scrt1.o: in function `_start':<br>
> (.text+0x1b): undefined reference to `main'<br>
> collect2: error: ld returned 1 exit status<br>
> make: *** [/local/home/iskander/software/petsc3_20/lib/petsc/conf/rules:218: ex55k] Error 1<br>
> <br>
> <br>
> please let me know on how I can resolve a given error.<br>
> <br>
> I also would be happy if you know more resources and examples on implementation of PETSc with kokkos.<br>
> <br>
> <br>
> <br>
> Have a nice day!<br>
> <br>
> <br>
> <br>
> Regards,<br>
> <br>
> Ibragimov Iskander, PhD student of<br>
> <br>
> Johannes Gutenberg-Universität Mainz<br>
> Institute of Geosciences<br>
> J.-J.-Becher-Weg 21<br>
> D-55128 Mainz<br>
> Room: 03-264<br>
> <br>
> <br>
</blockquote></div>