[petsc-dev] Error on Fugaku
Mark Adams
mfadams at lbl.gov
Wed Apr 14 17:43:00 CDT 2021
PETSc seems to be skipping the compiler that it should use:
-- Check for working CXX compiler:
/opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC - skipped
I see -fopenmp was added. This might be wrong. I use -Kopenmp.
--help says:
-fopenmp
The -fopenmp option specifies to enable Specification of OpenMP
Appli-
cation Program Interface.
When the -fopenmp option is specified, -mt is set.
The -fopenmp option is needed if an object program compiled with the
-fopenmp option exists in the command line as input files.
Any ideas?
On Wed, Apr 14, 2021 at 5:17 PM Satish Balay <balay at mcs.anl.gov> wrote:
> The following is working for me on the compute node [its completed kokkos
> (core) build - and is now building kokkos-kernels
>
> ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py
> --download-p4est --download-zlib --download-kokkos
> --download-kokkos-kernels --download-kokkos-commit=origin/develop
> --download-cmake=
> https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar.gz
>
> -DKokkos_ENABLE_OPENMP=ON give me some issue - so I didn't try any of the
> additional options..
>
> Satish
>
> On Wed, 14 Apr 2021, Mark Adams wrote:
>
> > Satish, I get this error.
> >
> > I wonder if this syntax is wrong:
> >
> > '--download-kokkos-cmake-arguments=-DCMAKE_BUILD_TYPE=Release
> > -DBUILD_TESTING=OFF -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON
> > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON',
> >
> >
> >
> =============================================================================================
> > Configuring KOKKOS with cmake; this may
> take
> > several minutes
> >
> >
> =============================================================================================
> >
> > Executing: /usr/bin/cmake ..
> > -DCMAKE_INSTALL_PREFIX=/home/ra010009/a04199/petsc/arch-arm
> >
> -DCMAKE_INSTALL_NAME_DIR:STRING="/home/ra010009/a04199/petsc/arch-arm/lib"
> > -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1
> > -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpifcc"
> > -DMPI_C_COMPILER="mpifcc" -DCMAKE_AR=/usr/bin/ar
> > -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC
> -Kfast,openmp
> > -fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Kfast,openmp -fopenmp"
> > -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Kfast,openmp -fopenmp"
> > -DCMAKE_CXX_COMPILER="mpiFCC" -DMPI_CXX_COMPILER="mpiFCC"
> > -DCMAKE_CXX_FLAGS:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp"
> > -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp"
> > -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp"
> > -DCMAKE_Fortran_COMPILER="mpifrt" -DMPI_Fortran_COMPILER="mpifrt"
> > -DCMAKE_Fortran_FLAGS:STRING="-fPIC -O -fopenmp"
> > -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -O -fopenmp"
> > -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -O -fopenmp"
> > -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON
> > -DUSE_XSDK_DEFAULTS=YES -DXSDK_ENABLE_DEBUG=NO
> > -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON
> > -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON -DKokkos_ENABLE_MPI=ON
> > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_OPENMP=ON
> > -DCMAKE_CXX_STANDARD="14" -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF
> > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON
> > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON
> > stdout:
> > -- Setting default Kokkos CXX standard to 14
> > -- The CXX compiler identification is Fujitsu
> > -- Check for working CXX compiler:
> /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC
> > -- Check for working CXX compiler:
> > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC -- works
> > -- Detecting CXX compiler ABI info
> > -- Detecting CXX compiler ABI info - done
> > -- The project name is: Kokkos
> > -- Configuring incomplete, errors occurred!
> > See also
> >
> "/vol0004/ra010009/a04199/petsc/arch-arm/externalpackages/git.kokkos/petsc-build/CMakeFiles/CMakeOutput.log".
> > Error configuring KOKKOS with cmake Could not execute
> > "['/usr/bin/cmake ..
> > -DCMAKE_INSTALL_PREFIX=/home/ra010009/a04199/petsc/arch-arm
> >
> -DCMAKE_INSTALL_NAME_DIR:STRING="/home/ra010009/a04199/petsc/arch-arm/lib"
> > -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1
> > -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpifcc"
> > -DMPI_C_COMPILER="mpifcc" -DCMAKE_AR=/usr/bin/ar
> > -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC
> -Kfast,openmp
> > -fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Kfast,openmp -fopenmp"
> > -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Kfast,openmp -fopenmp"
> > -DCMAKE_CXX_COMPILER="mpiFCC" -DMPI_CXX_COMPILER="mpiFCC"
> > -DCMAKE_CXX_FLAGS:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp"
> > -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp"
> > -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp"
> > -DCMAKE_Fortran_COMPILER="mpifrt" -DMPI_Fortran_COMPILER="mpifrt"
> > -DCMAKE_Fortran_FLAGS:STRING="-fPIC -O -fopenmp"
> > -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -O -fopenmp"
> > -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -O -fopenmp"
> > -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON
> > -DUSE_XSDK_DEFAULTS=YES -DXSDK_ENABLE_DEBUG=NO
> > -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON
> > -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON -DKokkos_ENABLE_MPI=ON
> > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_OPENMP=ON
> > -DCMAKE_CXX_STANDARD="14" -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF
> > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON
> > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON']":
> > -- Setting default Kokkos CXX standard to 14
> > -- The CXX compiler identification is Fujitsu
> > -- Check for working CXX compiler:
> /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC
> > -- Check for working CXX compiler:
> > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC -- works
> > -- Detecting CXX compiler ABI info
> > -- Detecting CXX compiler ABI info - done
> > -- The project name is: Kokkos
> > -- Configuring incomplete, errors occurred!
> > See also
> >
> "/vol0004/ra010009/a04199/petsc/arch-arm/externalpackages/git.kokkos/petsc-build/CMakeFiles/CMakeOutput.log".CMake
> > Error at cmake/kokkos_compiler_id.cmake:129 (STRING):
> >
> > * STRING sub-command REPLACE requires at least four arguments.*Call
> Stack
> > (most recent call first):
> > cmake/kokkos_tribits.cmake:174 (INCLUDE)
> > CMakeLists.txt:166 (KOKKOS_SETUP_BUILD_ENVIRONMENT)
> >
> > On Wed, Apr 14, 2021 at 3:34 PM Satish Balay <balay at mcs.anl.gov> wrote:
> >
> > > Additional kokkos cmake arguments can be passed in via
> > > --download-kokkos-cmake-arguments=string option.
> > >
> > > Satish
> > >
> > > On Wed, 14 Apr 2021, Mark Adams wrote:
> > >
> > > > Satish,
> > > >
> > > > For the fujitsu compiler OMP is -Kopenmp.
> > > >
> > > > Sarat (cc'ed) tells me that he built Kokkos with:
> > > >
> > > > cmake -DCMAKE_BUILD_TYPE=Release \
> > > > -DCMAKE_INSTALL_PREFIX=${KOKKOS_SRC_DIR}/install \
> > > > -DBUILD_TESTING=OFF \
> > > > -DKokkos_ENABLE_LIBDL=OFF \
> > > > -DKokkos_ENABLE_OPENMP=ON \
> > > > -DKokkos_ENABLE_SERIAL=ON \
> > > > -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON \
> > > > ..
> > > >
> > > > How might I make this happen in PETSc?
> > > >
> > > > Thanks,
> > > > Mark
> > > >
> > > >
> > > > On Wed, Apr 14, 2021 at 2:44 PM Satish Balay <balay at mcs.anl.gov>
> wrote:
> > > >
> > > > > On Wed, 14 Apr 2021, Mark Adams wrote:
> > > > >
> > > > > > I have this building now.
> > > > > > Do you know anything about OpenMP?
> > > > > > I can add --with-openmp
> > > > > > That should get Kokkos to be made with OpenMP.
> > > > > > Should PETSc deal with the compilers correctly?
> > > > >
> > > > > Well it tries the following compiler options for openmp.
> > > > >
> > > > > oflags = ["-fopenmp", # Gnu
> > > > > "-qsmp=omp",# IBM XL C/C++
> > > > > "-h omp", # Cray. Must come after XL because XL
> > > interprets
> > > > > this option as meaning "-soname omp"
> > > > > "-mp", # Portland Group
> > > > > "-Qopenmp", # Intel windows
> > > > > "-openmp", # Intel
> > > > > "-xopenmp", # Sun
> > > > > "+Oopenmp", # HP
> > > > > "/openmp" # Microsoft Visual Studio
> > > > > ]
> > > > >
> > > > > I don't know what the flag for fugaku compiler is.
> > > > >
> > > > > Satish
> > > > >
> > > > >
> > > > > > Thanks,
> > > > > > Mark
> > > > > >
> > > > > > On Wed, Apr 14, 2021 at 1:45 PM Mark Adams <mfadams at lbl.gov>
> wrote:
> > > > > >
> > > > > > > Thanks,
> > > > > > > If you feel inspired you could try Kokkos :||
> > > > > > > I am in a parking lot waiting for my daughter but can try this
> > > when I
> > > > > get
> > > > > > > home,
> > > > > > > Thanks again,
> > > > > > > Mark
> > > > > > >
> > > > > > > On Wed, Apr 14, 2021 at 1:33 PM Satish Balay <
> balay at mcs.anl.gov>
> > > > > wrote:
> > > > > > >
> > > > > > >> I think I allocated a single node - and did the build on it.
> > > > > > >>
> > > > > > >> Now I'm getting an error - don't know what changed..
> > > > > > >>
> > > > > > >> login6$ pjsub --interact -L "node=1" -L
> "rscunit=rscunit_ft01" -L
> > > > > > >> "rscgrp=eap-int" -L "elapse=1:00:00" --sparam "wait-time=600"
> > > > > > >> [ERR.] PJM 0059 pjsub rscgrp=eap-int is disabled.
> > > > > > >>
> > > > > > >> Ok - the following worked..
> > > > > > >>
> > > > > > >> login6$ pjsub --interact -L "node=1" -L
> "rscunit=rscunit_ft01" -L
> > > > > > >> "elapse=1:00:00" --sparam "wait-time=600"
> > > > > > >> [INFO] PJM 0000 pjsub Job 6301572 submitted.
> > > > > > >> [INFO] PJM 0081 .connected.
> > > > > > >> [INFO] PJM 0082 pjsub Interactive job 6301572 started.
> > > > > > >> [a04201 at j31-3110s petsc]$
> > > > > > >>
> > > > > > >> Ok - trying this build now.
> > > > > > >>
> > > > > > >> [a04201 at j31-3110s petsc]$ cat
> > > > > > >> ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py
> > > > > > >> #!/usr/bin/python3
> > > > > > >> if __name__ == '__main__':
> > > > > > >> import sys
> > > > > > >> import os
> > > > > > >> sys.path.insert(0, os.path.abspath('config'))
> > > > > > >> import configure
> > > > > > >> configure_options = [
> > > > > > >> '--with-blaslapack-lib=-lfjlapack',
> > > > > > >> '--with-debugging=0',
> > > > > > >> 'CC=mpifcc',
> > > > > > >> 'CXX=mpiFCC',
> > > > > > >> 'FC=mpifrt',
> > > > > > >> 'PETSC_ARCH=arch-arm',
> > > > > > >> ]
> > > > > > >> configure.petsc_configure(configure_options)
> > > > > > >> [a04201 at j31-3110s petsc]$
> > > > > > >> ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py
> > > > > > >> --download-p4est --download-zlib
> > > > > > >> <snip>
> > > > > > >> p4est:
> > > > > > >> Includes: -I/vol0004/ra010009/a04201/petsc/arch-arm/include
> > > > > > >> Library:
> -Wl,-rpath,/vol0004/ra010009/a04201/petsc/arch-arm/lib
> > > > > > >> -L/vol0004/ra010009/a04201/petsc/arch-arm/lib -lp4est -lsc
> > > > > > >>
> > > > > > >> Ok - this worked for me.
> > > > > > >>
> > > > > > >> Satish
> > > > > > >>
> > > > > > >> On Wed, 14 Apr 2021, Mark Adams wrote:
> > > > > > >>
> > > > > > >> > Do you recall what nodes you use to build on a "compute"
> node,
> > > to
> > > > > avoid
> > > > > > >> > cross compilation?
> > > > > > >> >
> > > > > > >> > On Wed, Apr 14, 2021 at 12:08 PM Satish Balay <
> > > balay at mcs.anl.gov>
> > > > > > >> wrote:
> > > > > > >> >
> > > > > > >> > > looks like p4est cannot be cross-compiled.
> > > > > > >> > >
> > > > > > >> > > Satish
> > > > > > >> > >
> > > > > > >> > > On Wed, 14 Apr 2021, Mark Adams wrote:
> > > > > > >> > >
> > > > > > >> > > > I get this error with p4est on Fugaku.
> > > > > > >> > > > It is a Fortran error. Odd.
> > > > > > >> > > > Mark
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20210414/a40c99f6/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 771949 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20210414/a40c99f6/attachment-0001.obj>
More information about the petsc-dev
mailing list