<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 16, 2021 at 12:31 PM Sreepathi, Sarat <<a href="mailto:sarat@ornl.gov">sarat@ornl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_5956879895548332163WordSection1">
<p class="MsoNormal">Hi Mark,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I couldn’t follow all the issues in the other thread. Anyway, you are in luck.
<br>
<br>
<u></u><u></u></p>
<p class="MsoNormal">I have pushed for GNU compiler support with Fujitsu MPI since November after hitting multiple issues. Finally, got an update from Fugaku helpdesk earlier this week.<u></u><u></u></p>
<p class="MsoNormal">As you are aware, you will probably get better performance with GNU than Fujitsu on A64FX. My latest experiments confirmed that *<b>again</b>* for climate workloads.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">See "Users Guide - Language and development environment Ver. 1.02" - Chapter 4 for details.<u></u><u></u></p>
<p class="MsoNormal"><a href="https://www.fugaku.r-ccs.riken.jp/doc_root/en/user_guides/lang_1.02/GCC_MPI/index.html" target="_blank">https://www.fugaku.r-ccs.riken.jp/doc_root/en/user_guides/lang_1.02/GCC_MPI/index.html</a><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Additionally, please note that you will need to `spack load <a href="mailto:gcc@10.2.0%60" target="_blank">
gcc@10.2.0`</a> to load the latest GNU compiler.<u></u><u></u></p>
<p class="MsoNormal">To summarize, you need to do the following on a compute node to get the requisite environment.<u></u><u></u></p>
<p class="MsoNormal">```<u></u><u></u></p>
<p class="MsoNormal">. /vol0004/apps/oss/spack/share/spack/setup-env.sh<u></u><u></u></p>
<p class="MsoNormal">spack load fujitsu-mpi%gcc<u></u><u></u></p>
<p class="MsoNormal">spack load <a href="mailto:gcc@10.2.0" target="_blank">gcc@10.2.0</a><u></u><u></u></p>
<p class="MsoNormal">```</p></div></div></blockquote><div><br></div><div>Yes, I go this this AM from Kokkos:</div><div><br></div><div> . /vol0004/apps/oss/spack/share/spack/setup-env.sh<br> spack load gcc@10.2.0%gcc@8.3.1 arch=linux-rhel8-a64fx<br> spack load fujitsu-mpi@4.5.0%gcc@8.3.1 arch=linux-rhel8-a64fx<br></div><div><br></div><div>Looks like the same thing + @8.3.1 arch=linux-rhel8-a64fx</div><div> </div><div>It has passed PETSc 'make check'</div><div><br></div><div>Thanks,</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_5956879895548332163WordSection1"><p class="MsoNormal"><u></u><u></u></p>
<p class="MsoNormal"><br>
Hope this helps (cc’ing petsc-dev),<br>
Sarat.<br>
<br>
<u></u><u></u></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> <br>
<b>Sent:</b> Friday, April 16, 2021 10:27 AM<br>
<b>To:</b> Sreepathi, Sarat <<a href="mailto:sarat@ornl.gov" target="_blank">sarat@ornl.gov</a>><br>
<b>Subject:</b> Re: [EXTERNAL] Re: building on Fugaku<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Sarat, <u></u><u></u></p>
<div>
<p class="MsoNormal">I am making progress. Kokkos is working w/o OMP, but Kokkos Kernels + OMP is barfing and Kokkos people suggested GNU.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">But one of my third party libraries needs MPI. Do you know of some sort of MPI library on Fugaku? <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">It just has to work with one process.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Mark<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Wed, Apr 14, 2021 at 3:21 PM Sreepathi, Sarat <<a href="mailto:sarat@ornl.gov" target="_blank">sarat@ornl.gov</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">Hi Mark,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">From earlier notes, I used the Kokkos “develop” branch.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal" style="margin-bottom:12pt">cmake -DCMAKE_BUILD_TYPE=Release \<br>
-DCMAKE_INSTALL_PREFIX=${KOKKOS_SRC_DIR}/install \<br>
-DBUILD_TESTING=OFF \<br>
-DKokkos_ENABLE_LIBDL=OFF \<br>
-DKokkos_ENABLE_OPENMP=ON \<br>
-DKokkos_ENABLE_SERIAL=ON \<br>
-DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON \<br>
..<u></u><u></u></p>
<p class="MsoNormal">-Sarat.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>>
<br>
<b>Sent:</b> Wednesday, April 14, 2021 3:09 PM<br>
<b>To:</b> Sreepathi, Sarat <<a href="mailto:sarat@ornl.gov" target="_blank">sarat@ornl.gov</a>><br>
<b>Cc:</b> Richard Tran Mills <<a href="mailto:rtmills@anl.gov" target="_blank">rtmills@anl.gov</a>><br>
<b>Subject:</b> [EXTERNAL] Re: building on Fugaku<u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">Sarat,<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">I have made some progress but am getting a C++ error:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">CMake Warning at cmake/kokkos_test_cxx_std.cmake:47 (MESSAGE):<br>
C++14 is not supported as a compiler feature. We will choose custom flags<br>
for now, but this behavior has been deprecated. Please open an issue at<br>
<a href="https://github.com/kokkos/kokkos/issues" target="_blank">https://github.com/kokkos/kokkos/issues</a> reporting that Fujitsu failed for<br>
14, preferrably including your CMake command.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I am using:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p class="MsoNormal">04:08 adams/landau-gpu-opt= ~/petsc$ mpiFCC --version<br>
FCC (FCC) 4.5.0 20210304<br>
simulating gcc version 6.1<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Do you have any ideas?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Mark <u></u><u></u></p>
</div>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">On Wed, Apr 14, 2021 at 7:55 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt">
<div>
<p class="MsoNormal">Hi Sarat,<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">My colleague, Rich Mills, and I are coming back to Fugaku after a while, and I wanted to ask you how you build, with Kokkos, on Fugaku. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I have tried Arm and Fugitzu compilers and was able to get Fugitzu to work a few months ago, but I wanted to see what you use. Especially for Kokkos.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Can you share with us your build details?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Mark<u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote></div></div>