<div dir="ltr">Sarat, is there anything special that you do for Kokkos - OpenMP?<div><br></div><div>Just set OMP_NUM_THREADS=48 ?</div><div><br></div><div>Also, I am confused about the number of cores here. Is 48 or 64 per node/socket?</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 16, 2021 at 2:03 PM Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Cool, I have it running too. Need to add Sarat's flags and test ex2.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 16, 2021 at 1:57 PM Satish Balay via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Mark,<br>
<br>
The following build works for me:<br>
<br>
Satish<br>
<br>
----<br>
<br>
pjsub --interact -L "node=1" -L "rscunit=rscunit_ft01" -L "elapse=1:00:00" --sparam "wait-time=1200"<br>
<br>
. /vol0004/apps/oss/spack/share/spack/setup-env.sh<br>
spack load fujitsu-mpi%gcc<br>
spack load gcc@10.2.0 arch=linux-rhel8-a64fx<br>
./configure COPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' CXXOPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' FOPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' --with-openmp=1 --download-p4est --download-zlib --download-kokkos --download-kokkos-kernels --download-kokkos-commit=origin/develop --download-kokkos-kernels-commit=origin/develop '--download-kokkos-cmake-arguments=-DBUILD_TESTING=OFF -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON' --download-cmake=<a href="https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar.gz" rel="noreferrer" target="_blank">https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar.gz</a>  --download-fblaslapack=1<br>
make PETSC_DIR=/vol0004/ra010009/a04201/petsc.z PETSC_ARCH=arch-linux-c-debug all<br>
<br>
<br>
To test - redo job allocation using max-proc-per-node:<br>
<br>
login6$ pjsub --interact -L "node=1" -L "rscunit=rscunit_ft01" -L "elapse=1:00:00" --sparam "wait-time=1200" --mpi "max-proc-per-node=16"<br>
<br>
[a04201@c31-3201c petsc.z]$ . /vol0004/apps/oss/spack/share/spack/setup-env.sh<br>
[a04201@c31-3201c petsc.z]$ spack load fujitsu-mpi%gcc<br>
[a04201@c31-3201c petsc.z]$ spack load gcc@10.2.0 arch=linux-rhel8-a64fx<br>
[a04201@c31-3201c petsc.z]$ make check<br>
Running check examples to verify correct installation<br>
Using PETSC_DIR=/vol0004/ra010009/a04201/petsc.z and PETSC_ARCH=arch-linux-c-debug<br>
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process<br>
C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes<br>
C/C++ example src/snes/tutorials/ex3k run successfully with kokkos-kernels<br>
Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process<br>
Completed test examples<br>
[a04201@c31-3201c petsc.z]$ <br>
</blockquote></div>
</blockquote></div>