[petsc-dev] [EXTERNAL] Re: building on Fugaku

[Sreepathi, Sarat]

Hi Mark,

I couldn’t follow all the issues in the other thread.  Anyway, you are in luck.

I have pushed for GNU compiler support with Fujitsu MPI since November after hitting multiple issues. Finally, got an update from Fugaku helpdesk earlier this week.
As you are aware, you will probably get better performance with GNU than Fujitsu on A64FX. My latest experiments confirmed that *again* for climate workloads.

See "Users Guide - Language and development environment Ver. 1.02" - Chapter 4 for details.
https://www.fugaku.r-ccs.riken.jp/doc_root/en/user_guides/lang_1.02/GCC_MPI/index.html

Additionally, please note that you will need to `spack load gcc at 10.2.0`<mailto:gcc at 10.2.0%60> to load the latest GNU compiler.
To summarize, you need to do the following on a compute node to get the requisite environment.
```
. /vol0004/apps/oss/spack/share/spack/setup-env.sh
spack load fujitsu-mpi%gcc
spack load gcc at 10.2.0<mailto:gcc at 10.2.0>
```

Hope this helps (cc’ing petsc-dev),
Sarat.

From: Mark Adams <mfadams at lbl.gov>
Sent: Friday, April 16, 2021 10:27 AM
To: Sreepathi, Sarat <sarat at ornl.gov>
Subject: Re: [EXTERNAL] Re: building on Fugaku

Sarat,
I am making progress. Kokkos is working w/o OMP,  but Kokkos Kernels + OMP is barfing and Kokkos people suggested GNU.
But one of my third party libraries needs MPI. Do you know of some sort of MPI library on Fugaku?
It just has to work with one process.
Thanks,
Mark

On Wed, Apr 14, 2021 at 3:21 PM Sreepathi, Sarat <sarat at ornl.gov<mailto:sarat at ornl.gov>> wrote:
Hi Mark,

From earlier notes, I used the Kokkos “develop” branch.

cmake -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=${KOKKOS_SRC_DIR}/install \
    -DBUILD_TESTING=OFF \
    -DKokkos_ENABLE_LIBDL=OFF \
    -DKokkos_ENABLE_OPENMP=ON \
    -DKokkos_ENABLE_SERIAL=ON \
    -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON \
    ..
-Sarat.

From: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Sent: Wednesday, April 14, 2021 3:09 PM
To: Sreepathi, Sarat <sarat at ornl.gov<mailto:sarat at ornl.gov>>
Cc: Richard Tran Mills <rtmills at anl.gov<mailto:rtmills at anl.gov>>
Subject: [EXTERNAL] Re: building on Fugaku

Sarat,

I have made some progress but am getting a C++ error:

CMake Warning at cmake/kokkos_test_cxx_std.cmake:47 (MESSAGE):
  C++14 is not supported as a compiler feature.  We will choose custom flags
  for now, but this behavior has been deprecated.  Please open an issue at
  https://github.com/kokkos/kokkos/issues reporting that Fujitsu failed for
  14, preferrably including your CMake command.

I am using:

04:08 adams/landau-gpu-opt= ~/petsc$ mpiFCC --version
FCC (FCC) 4.5.0 20210304
simulating gcc version 6.1

Do you have any ideas?

Thanks,
Mark

On Wed, Apr 14, 2021 at 7:55 AM Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>> wrote:
Hi Sarat,

My colleague, Rich Mills, and I are coming back to Fugaku after a while, and I wanted to ask you how you build, with Kokkos, on Fugaku.
I have tried Arm and Fugitzu compilers and was able to get Fugitzu to work a few months ago, but I wanted to see what you use. Especially for Kokkos.
Can you share with us your build details?

Thanks,
Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20210416/05f20f0a/attachment.html>