[petsc-users] installing petsc via spack on ac922 (a summit like system)

[Satish Balay]

Created an MR for this change

https://gitlab.com/petsc/petsc/-/merge_requests/2593

Satish

On Tue, 10 Mar 2020, Satish Balay via petsc-users wrote:

> Another option is to strip out -lxlomp_ser when parsing 'xlc -V' [as its a common library for all 3 language compilers]. i.e the following change [this change is on current maint - but similar change can be done with petsc-3.7]..
> 
> [This is easier than changing the order in which --with-openmp option is processed]
> 
> Satish
> -----------
> 
> diff --git a/config/BuildSystem/config/compilers.py b/config/BuildSystem/config/compilers.py
> index b4cbc183c0..cfffb1a80e 100644
> --- a/config/BuildSystem/config/compilers.py
> +++ b/config/BuildSystem/config/compilers.py
> @@ -308,7 +308,7 @@ class Configure(config.base.Configure):
>              self.logPrint('already in lflags: '+arg, 4, 'compilers')
>            continue
>          # Check for system libraries
> -        m = re.match(r'^-l(ang.*|crt[0-9].o|crtbegin.o|c|gcc|gcc_ext(.[0-9]+)*|System|cygwin|crt[0-9].[0-9][0-9].[0-9].o)$', arg)
> +        m = re.match(r'^-l(ang.*|crt[0-9].o|crtbegin.o|c|gcc|gcc_ext(.[0-9]+)*|System|cygwin|xlomp_ser|crt[0-9].[0-9][0-9].[0-9].o)$', arg)
>          if m:
>            self.logPrint('Skipping system library: '+arg, 4, 'compilers')
>            continue
> @@ -687,7 +687,7 @@ class Configure(config.base.Configure):
>              self.logPrint('already in lflags: '+arg, 4, 'compilers')
>            continue
>          # Check for system libraries
> -        m = re.match(r'^-l(ang.*|crt[0-9].o|crtbegin.o|c|gcc|gcc_ext(.[0-9]+)*|System|cygwin|crt[0-9].[0-9][0-9].[0-9].o)$', arg)
> +        m = re.match(r'^-l(ang.*|crt[0-9].o|crtbegin.o|c|gcc|gcc_ext(.[0-9]+)*|System|cygwin|xlomp_ser|crt[0-9].[0-9][0-9].[0-9].o)$', arg)
>          if m:
>            self.logPrint('Skipping system library: '+arg, 4, 'compilers')
>            continue
> @@ -1085,7 +1085,7 @@ Otherwise you need a different combination of C, C++, and Fortran compilers")
>              self.logPrint('Already in lflags so skipping: '+arg, 4, 'compilers')
>            continue
>          # Check for system libraries
> -        m = re.match(r'^-l(ang.*|crt[0-9].o|crtbegin.o|c|gcc|gcc_ext(.[0-9]+)*|System|cygwin|crt[0-9].[0-9][0-9].[0-9].o)$', arg)
> +        m = re.match(r'^-l(ang.*|crt[0-9].o|crtbegin.o|c|gcc|gcc_ext(.[0-9]+)*|System|cygwin|xlomp_ser|crt[0-9].[0-9][0-9].[0-9].o)$', arg)
>          if m:
>            self.logPrint('Found system library therefore skipping: '+arg, 4, 'compilers')
>            continue
> 
> On Tue, 10 Mar 2020, Cameron Smith wrote:
> 
> > That makes sense.  Thank you.
> > 
> > -Cameron
> > 
> > On 3/10/20 9:25 AM, Satish Balay wrote:
> > > PETSc configure attempts to check for compiler libraries [so that one
> > > can mix and use c,c++,fortran codes with a c linker] by running
> > > compilers in verbose mode - and parsing the output. [i.e checkCLibraries(),
> > > checkFortranLibraries() ..]
> > > 
> > > Here IBM compiler is using different library internally based on 'xlc -V' vs
> > > 'xlc -fopenmp -V'. For most other compilers - its just an additional library
> > > [with corresponding include file]
> > > 
> > > So -fopenmp needs to be set before this step [checkCLibraries()] in
> > > configure.
> > > 
> > > PETSc configure treats -with-openmp as an additional package - and process
> > > this option after the above checkCLibraries() check.
> > > 
> > > However CFLAGS etc get processed and set before the call to
> > > checkCLibraries().
> > > 
> > > My suggestion is a workaround to get -fopenmp option set before
> > > checkCLibraries() are called.
> > > 
> > > Satish
> > > 
> > > On Tue, 10 Mar 2020, Cameron Smith wrote:
> > > 
> > >> Thank you.  I'll give that a shot.
> > >>
> > >> Out of curiosity, how does passing the openmp flags relate to the
> > >> '--with-openmp' option described here:
> > >>
> > >> https://www.mcs.anl.gov/petsc/documentation/installation.html
> > >>
> > >> under 'Installing packages that utilize OpenMP'?  Is this just passing the
> > >> openmp flags into compile/link commands of the packages that petsc builds
> > >> (via, --download-<package> options) and not to the petsc compile/link?
> > >>
> > >> -Cameron
> > >>
> > >> On 3/10/20 9:01 AM, Satish Balay wrote:
> > >>> BTW: You might be able to do the same via spack.
> > >>>
> > >>> spack install petsc at 3.7.7 ~hdf5 ~hypre ~superlu-dist cflags=-fopenmp
> > >>> fflags=-fopenmp cxxflags=-fopenmp
> > >>>
> > >>> Satish
> > >>>
> > >>> On Tue, 10 Mar 2020, Satish Balay via petsc-users wrote:
> > >>>
> > >>>> Cameron,
> > >>>>
> > >>>> You can try changing following petsc configure options and see if that
> > >>>> works.. [i.e build petsc manually]
> > >>>>
> > >>>> CFLAGS=-fopenmp FFLAGS=-fopenmp CXXFLAGS=-fopenmp
> > >>>>
> > >>>> Satish
> > >>>>
> > >>>> On Tue, 10 Mar 2020, Cameron Smith wrote:
> > >>>>
> > >>>>> Thank you Mark.
> > >>>>>
> > >>>>> The configure.log is attached.
> > >>>>>
> > >>>>> Please let me know if any other info is needed.
> > >>>>>
> > >>>>> -Cameron
> > >>>>>
> > >>>>> On 3/10/20 7:31 AM, Mark Adams wrote:
> > >>>>>> Hi Cameron,
> > >>>>>>
> > >>>>>> This can go on the list and we always want the configure.log file.
> > >>>>>>
> > >>>>>> I build on Summit, but have not used the XL compilers. I've built 3.7.7
> > >>>>>> with
> > >>>>>> GNU and PGI. (XGC usually wants PGI)
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Mon, Mar 9, 2020 at 11:27 PM Cameron Smith <smithc11 at rpi.edu
> > >>>>>> <mailto:smithc11 at rpi.edu>> wrote:
> > >>>>>>
> > >>>>>>       Hello,
> > >>>>>>
> > >>>>>>       I'm installing petsc 3.7.7 on a summit like system with the
> > >>>>>>       following
> > >>>>>>       spack spec:
> > >>>>>>
> > >>>>>>       petsc at 3.7.7 ~hdf5 ~hypre ~superlu-dist
> > >>>>>>
> > >>>>>>       with the XL 16.1.1 compiler and Spectrum MPI 10.3 .  This install
> > >>>>>>       produces a `/path/to/petsc/install/lib/petsc/conf/petscvariables`
> > >>>>>>       file
> > >>>>>>       that contains '-lxlomp_ser' in the 'PETSC_EXTERNAL_LIB_BASIC' and
> > >>>>>>       'PETSC_WITH_EXTERNAL_LIB' variables.
> > >>>>>>
> > >>>>>>       The application I'm building, XGC, has a makefile based build
> > >>>>>>       system
> > >>>>>>       that includes '/path/to/petsc/install/lib/petsc/conf/variables'
> > >>>>>>       which in
> > >>>>>>       turn includes '/lib/petsc/conf/petscvariables'.
> > >>>>>>
> > >>>>>>        From what I can tell, xlomp_ser is a serial implementation of
> > >>>>>>       the
> > >>>>>>       openmp library.  When XGC links this library it satisfies the
> > >>>>>>       openmp
> > >>>>>>       symbols XGC wants and at run time results in openmp API calls
> > >>>>>>       like
> > >>>>>>       'omp_get_max_threads()' returning 1 regardless of the
> > >>>>>>       OMP_NUM_THREADS
> > >>>>>>       setting.
> > >>>>>>
> > >>>>>>       Do you know how I can build petsc, with or without spack, and
> > >>>>>>       avoid
> > >>>>>>       this
> > >>>>>>       library being listed in 'lib/petsc/conf/petscvariables'?
> > >>>>>>
> > >>>>>>       If this should go to a petsc mailing list or git repo issues page
> > >>>>>>       I
> > >>>>>>       can
> > >>>>>>       send it there.
> > >>>>>>
> > >>>>>>       Thank-you,
> > >>>>>>       Cameron
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>
> > >>
> > 
>