<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">My turn: did you folks figure out tips for performant hybrid MPI+OMP core binding? I tried some from the documentation but that didn’t seem to help.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-Sarat.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Sreepathi, Sarat <br>
<b>Sent:</b> Friday, April 16, 2021 3:02 PM<br>
<b>To:</b> Mark Adams <mfadams@lbl.gov>; petsc-dev <petsc-dev@mcs.anl.gov><br>
<b>Cc:</b> Satish Balay <balay@mcs.anl.gov><br>
<b>Subject:</b> RE: [petsc-dev] [EXTERNAL] Re: building on Fugaku<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">It’s 48 cores but there are 4 NUMA domains (CMGs). So, you may want to experiment in hybrid mode (4x12 etc.) if possible.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-Sarat.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>>
<br>
<b>Sent:</b> Friday, April 16, 2021 2:10 PM<br>
<b>To:</b> petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov">petsc-dev@mcs.anl.gov</a>><br>
<b>Cc:</b> Satish Balay <<a href="mailto:balay@mcs.anl.gov">balay@mcs.anl.gov</a>>; Sreepathi, Sarat <<a href="mailto:sarat@ornl.gov">sarat@ornl.gov</a>><br>
<b>Subject:</b> Re: [petsc-dev] [EXTERNAL] Re: building on Fugaku<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Sarat, is there anything special that you do for Kokkos - OpenMP?<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Just set OMP_NUM_THREADS=48 ?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Also, I am confused about the number of cores here. Is 48 or 64 per node/socket?<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, Apr 16, 2021 at 2:03 PM Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Cool, I have it running too. Need to add Sarat's flags and test ex2.<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, Apr 16, 2021 at 1:57 PM Satish Balay via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal">Mark,<br>
<br>
The following build works for me:<br>
<br>
Satish<br>
<br>
----<br>
<br>
pjsub --interact -L "node=1" -L "rscunit=rscunit_ft01" -L "elapse=1:00:00" --sparam "wait-time=1200"<br>
<br>
. /vol0004/apps/oss/spack/share/spack/setup-env.sh<br>
spack load fujitsu-mpi%gcc<br>
spack load <a href="mailto:gcc@10.2.0">gcc@10.2.0</a> arch=linux-rhel8-a64fx<br>
./configure COPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' CXXOPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' FOPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' --with-openmp=1 --download-p4est --download-zlib --download-kokkos
--download-kokkos-kernels --download-kokkos-commit=origin/develop --download-kokkos-kernels-commit=origin/develop '--download-kokkos-cmake-arguments=-DBUILD_TESTING=OFF -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON' --download-cmake=<a href="https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar.gz" target="_blank">https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar.gz</a>
--download-fblaslapack=1<br>
make PETSC_DIR=/vol0004/ra010009/a04201/petsc.z PETSC_ARCH=arch-linux-c-debug all<br>
<br>
<br>
To test - redo job allocation using max-proc-per-node:<br>
<br>
login6$ pjsub --interact -L "node=1" -L "rscunit=rscunit_ft01" -L "elapse=1:00:00" --sparam "wait-time=1200" --mpi "max-proc-per-node=16"<br>
<br>
[a04201@c31-3201c petsc.z]$ . /vol0004/apps/oss/spack/share/spack/setup-env.sh<br>
[a04201@c31-3201c petsc.z]$ spack load fujitsu-mpi%gcc<br>
[a04201@c31-3201c petsc.z]$ spack load <a href="mailto:gcc@10.2.0">gcc@10.2.0</a> arch=linux-rhel8-a64fx<br>
[a04201@c31-3201c petsc.z]$ make check<br>
Running check examples to verify correct installation<br>
Using PETSC_DIR=/vol0004/ra010009/a04201/petsc.z and PETSC_ARCH=arch-linux-c-debug<br>
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process<br>
C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes<br>
C/C++ example src/snes/tutorials/ex3k run successfully with kokkos-kernels<br>
Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process<br>
Completed test examples<br>
[a04201@c31-3201c petsc.z]$ <o:p></o:p></p>
</blockquote>
</div>
</blockquote>
</div>
</div>
</body>
</html>