[petsc-dev] [EXTERNAL] Re: building on Fugaku

[Satish Balay]

Thanks!

Are you able to use cmake from this spack install? I get this error:

[a04201 at a33-5211c ~]$ spack load cmake
==> Error: cmake matches multiple packages.
  Matching packages:
    ioepv7h cmake at 3.18.4%gcc at 8.3.1 arch=linux-rhel8-skylake_avx512
    bae6gkh cmake at 3.18.4%fj at 4.3.1 arch=linux-rhel8-a64fx
  Use a more specific spec.
[a04201 at a33-5211c ~]$ spack load cmake at 3.18.4%fj at 4.3.1
[WARN] xos LPG 2002 - Failed to map HugeTLBfs for data/bss: /bin/bash
The e_type of elf header must be ET_EXEC when using libmpg. You can check it on your load module by readelf -h command.

Satish

On Fri, 16 Apr 2021, Sreepathi, Sarat via petsc-dev wrote:

> For GNU in general,
> -Ofast -march=armv8.2-a+sve -msve-vector-bits=512
> 
> You may want to use -O3 if you run into FP exceptions etc. or look into turning on/off specific optimizations.
> -Sarat.
> 
> -----Original Message-----
> From: Satish Balay <balay at mcs.anl.gov> 
> Sent: Friday, April 16, 2021 12:41 PM
> To: Sreepathi, Sarat <sarat at ornl.gov>
> Cc: Mark Adams <mfadams at lbl.gov>; petsc-dev <petsc-dev at mcs.anl.gov>
> Subject: Re: [petsc-dev] [EXTERNAL] Re: building on Fugaku
> 
> Sarat,
> 
> I can give this a try. I see the instructions say "Execute the following commands on the command line in an interactive job"
> 
> What optimization flags do you use here?
> 
> Satish
> 
> On Fri, 16 Apr 2021, Sreepathi, Sarat via petsc-dev wrote:
> 
> > Hi Mark,
> > 
> > I couldn’t follow all the issues in the other thread.  Anyway, you are in luck.
> > 
> > I have pushed for GNU compiler support with Fujitsu MPI since November after hitting multiple issues. Finally, got an update from Fugaku helpdesk earlier this week.
> > As you are aware, you will probably get better performance with GNU than Fujitsu on A64FX. My latest experiments confirmed that *again* for climate workloads.
> > 
> > See "Users Guide - Language and development environment Ver. 1.02" - Chapter 4 for details.
> > https://www.fugaku.r-ccs.riken.jp/doc_root/en/user_guides/lang_1.02/GC
> > C_MPI/index.html
> > 
> > Additionally, please note that you will need to `spack load gcc at 10.2.0`<mailto:gcc at 10.2.0%60> to load the latest GNU compiler.
> > To summarize, you need to do the following on a compute node to get the requisite environment.
> > ```
> > . /vol0004/apps/oss/spack/share/spack/setup-env.sh
> > spack load fujitsu-mpi%gcc
> > spack load gcc at 10.2.0<mailto:gcc at 10.2.0> ```
> > 
> > Hope this helps (cc’ing petsc-dev),
> > Sarat.
> > 
> > From: Mark Adams <mfadams at lbl.gov>
> > Sent: Friday, April 16, 2021 10:27 AM
> > To: Sreepathi, Sarat <sarat at ornl.gov>
> > Subject: Re: [EXTERNAL] Re: building on Fugaku
> > 
> > Sarat,
> > I am making progress. Kokkos is working w/o OMP,  but Kokkos Kernels + OMP is barfing and Kokkos people suggested GNU.
> > But one of my third party libraries needs MPI. Do you know of some sort of MPI library on Fugaku?
> > It just has to work with one process.
> > Thanks,
> > Mark
> > 
> > On Wed, Apr 14, 2021 at 3:21 PM Sreepathi, Sarat <sarat at ornl.gov<mailto:sarat at ornl.gov>> wrote:
> > Hi Mark,
> > 
> > From earlier notes, I used the Kokkos “develop” branch.
> > 
> > cmake -DCMAKE_BUILD_TYPE=Release \
> >     -DCMAKE_INSTALL_PREFIX=${KOKKOS_SRC_DIR}/install \
> >     -DBUILD_TESTING=OFF \
> >     -DKokkos_ENABLE_LIBDL=OFF \
> >     -DKokkos_ENABLE_OPENMP=ON \
> >     -DKokkos_ENABLE_SERIAL=ON \
> >     -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON \
> >     ..
> > -Sarat.
> > 
> > From: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
> > Sent: Wednesday, April 14, 2021 3:09 PM
> > To: Sreepathi, Sarat <sarat at ornl.gov<mailto:sarat at ornl.gov>>
> > Cc: Richard Tran Mills <rtmills at anl.gov<mailto:rtmills at anl.gov>>
> > Subject: [EXTERNAL] Re: building on Fugaku
> > 
> > Sarat,
> > 
> > I have made some progress but am getting a C++ error:
> > 
> > CMake Warning at cmake/kokkos_test_cxx_std.cmake:47 (MESSAGE):
> >   C++14 is not supported as a compiler feature.  We will choose custom flags
> >   for now, but this behavior has been deprecated.  Please open an issue at
> >   https://github.com/kokkos/kokkos/issues reporting that Fujitsu failed for
> >   14, preferrably including your CMake command.
> > 
> > I am using:
> > 
> > 04:08 adams/landau-gpu-opt= ~/petsc$ mpiFCC --version FCC (FCC) 4.5.0 
> > 20210304 simulating gcc version 6.1
> > 
> > Do you have any ideas?
> > 
> > Thanks,
> > Mark
> > 
> > On Wed, Apr 14, 2021 at 7:55 AM Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>> wrote:
> > Hi Sarat,
> > 
> > My colleague, Rich Mills, and I are coming back to Fugaku after a while, and I wanted to ask you how you build, with Kokkos, on Fugaku.
> > I have tried Arm and Fugitzu compilers and was able to get Fugitzu to work a few months ago, but I wanted to see what you use. Especially for Kokkos.
> > Can you share with us your build details?
> > 
> > Thanks,
> > Mark
> > 
>