[petsc-users] installing petsc via spack on ac922 (a summit like system)

[Satish Balay]

Another option is to strip out -lxlomp_ser when parsing 'xlc -V' [as its a common library for all 3 language compilers]. i.e the following change [this change is on current maint - but similar change can be done with petsc-3.7]..

[This is easier than changing the order in which --with-openmp option is processed]

Satish
-----------

diff --git a/config/BuildSystem/config/compilers.py b/config/BuildSystem/config/compilers.py
index b4cbc183c0..cfffb1a80e 100644
--- a/config/BuildSystem/config/compilers.py
+++ b/config/BuildSystem/config/compilers.py
@@ -308,7 +308,7 @@ class Configure(config.base.Configure):
             self.logPrint('already in lflags: '+arg, 4, 'compilers')
           continue
         # Check for system libraries
-        m = re.match(r'^-l(ang.*|crt[0-9].o|crtbegin.o|c|gcc|gcc_ext(.[0-9]+)*|System|cygwin|crt[0-9].[0-9][0-9].[0-9].o)$', arg)
+        m = re.match(r'^-l(ang.*|crt[0-9].o|crtbegin.o|c|gcc|gcc_ext(.[0-9]+)*|System|cygwin|xlomp_ser|crt[0-9].[0-9][0-9].[0-9].o)$', arg)
         if m:
           self.logPrint('Skipping system library: '+arg, 4, 'compilers')
           continue
@@ -687,7 +687,7 @@ class Configure(config.base.Configure):
             self.logPrint('already in lflags: '+arg, 4, 'compilers')
           continue
         # Check for system libraries
-        m = re.match(r'^-l(ang.*|crt[0-9].o|crtbegin.o|c|gcc|gcc_ext(.[0-9]+)*|System|cygwin|crt[0-9].[0-9][0-9].[0-9].o)$', arg)
+        m = re.match(r'^-l(ang.*|crt[0-9].o|crtbegin.o|c|gcc|gcc_ext(.[0-9]+)*|System|cygwin|xlomp_ser|crt[0-9].[0-9][0-9].[0-9].o)$', arg)
         if m:
           self.logPrint('Skipping system library: '+arg, 4, 'compilers')
           continue
@@ -1085,7 +1085,7 @@ Otherwise you need a different combination of C, C++, and Fortran compilers")
             self.logPrint('Already in lflags so skipping: '+arg, 4, 'compilers')
           continue
         # Check for system libraries
-        m = re.match(r'^-l(ang.*|crt[0-9].o|crtbegin.o|c|gcc|gcc_ext(.[0-9]+)*|System|cygwin|crt[0-9].[0-9][0-9].[0-9].o)$', arg)
+        m = re.match(r'^-l(ang.*|crt[0-9].o|crtbegin.o|c|gcc|gcc_ext(.[0-9]+)*|System|cygwin|xlomp_ser|crt[0-9].[0-9][0-9].[0-9].o)$', arg)
         if m:
           self.logPrint('Found system library therefore skipping: '+arg, 4, 'compilers')
           continue

On Tue, 10 Mar 2020, Cameron Smith wrote:

> That makes sense.  Thank you.
> 
> -Cameron
> 
> On 3/10/20 9:25 AM, Satish Balay wrote:
> > PETSc configure attempts to check for compiler libraries [so that one
> > can mix and use c,c++,fortran codes with a c linker] by running
> > compilers in verbose mode - and parsing the output. [i.e checkCLibraries(),
> > checkFortranLibraries() ..]
> > 
> > Here IBM compiler is using different library internally based on 'xlc -V' vs
> > 'xlc -fopenmp -V'. For most other compilers - its just an additional library
> > [with corresponding include file]
> > 
> > So -fopenmp needs to be set before this step [checkCLibraries()] in
> > configure.
> > 
> > PETSc configure treats -with-openmp as an additional package - and process
> > this option after the above checkCLibraries() check.
> > 
> > However CFLAGS etc get processed and set before the call to
> > checkCLibraries().
> > 
> > My suggestion is a workaround to get -fopenmp option set before
> > checkCLibraries() are called.
> > 
> > Satish
> > 
> > On Tue, 10 Mar 2020, Cameron Smith wrote:
> > 
> >> Thank you.  I'll give that a shot.
> >>
> >> Out of curiosity, how does passing the openmp flags relate to the
> >> '--with-openmp' option described here:
> >>
> >> https://www.mcs.anl.gov/petsc/documentation/installation.html
> >>
> >> under 'Installing packages that utilize OpenMP'?  Is this just passing the
> >> openmp flags into compile/link commands of the packages that petsc builds
> >> (via, --download-<package> options) and not to the petsc compile/link?
> >>
> >> -Cameron
> >>
> >> On 3/10/20 9:01 AM, Satish Balay wrote:
> >>> BTW: You might be able to do the same via spack.
> >>>
> >>> spack install petsc at 3.7.7 ~hdf5 ~hypre ~superlu-dist cflags=-fopenmp
> >>> fflags=-fopenmp cxxflags=-fopenmp
> >>>
> >>> Satish
> >>>
> >>> On Tue, 10 Mar 2020, Satish Balay via petsc-users wrote:
> >>>
> >>>> Cameron,
> >>>>
> >>>> You can try changing following petsc configure options and see if that
> >>>> works.. [i.e build petsc manually]
> >>>>
> >>>> CFLAGS=-fopenmp FFLAGS=-fopenmp CXXFLAGS=-fopenmp
> >>>>
> >>>> Satish
> >>>>
> >>>> On Tue, 10 Mar 2020, Cameron Smith wrote:
> >>>>
> >>>>> Thank you Mark.
> >>>>>
> >>>>> The configure.log is attached.
> >>>>>
> >>>>> Please let me know if any other info is needed.
> >>>>>
> >>>>> -Cameron
> >>>>>
> >>>>> On 3/10/20 7:31 AM, Mark Adams wrote:
> >>>>>> Hi Cameron,
> >>>>>>
> >>>>>> This can go on the list and we always want the configure.log file.
> >>>>>>
> >>>>>> I build on Summit, but have not used the XL compilers. I've built 3.7.7
> >>>>>> with
> >>>>>> GNU and PGI. (XGC usually wants PGI)
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Mar 9, 2020 at 11:27 PM Cameron Smith <smithc11 at rpi.edu
> >>>>>> <mailto:smithc11 at rpi.edu>> wrote:
> >>>>>>
> >>>>>>       Hello,
> >>>>>>
> >>>>>>       I'm installing petsc 3.7.7 on a summit like system with the
> >>>>>>       following
> >>>>>>       spack spec:
> >>>>>>
> >>>>>>       petsc at 3.7.7 ~hdf5 ~hypre ~superlu-dist
> >>>>>>
> >>>>>>       with the XL 16.1.1 compiler and Spectrum MPI 10.3 .  This install
> >>>>>>       produces a `/path/to/petsc/install/lib/petsc/conf/petscvariables`
> >>>>>>       file
> >>>>>>       that contains '-lxlomp_ser' in the 'PETSC_EXTERNAL_LIB_BASIC' and
> >>>>>>       'PETSC_WITH_EXTERNAL_LIB' variables.
> >>>>>>
> >>>>>>       The application I'm building, XGC, has a makefile based build
> >>>>>>       system
> >>>>>>       that includes '/path/to/petsc/install/lib/petsc/conf/variables'
> >>>>>>       which in
> >>>>>>       turn includes '/lib/petsc/conf/petscvariables'.
> >>>>>>
> >>>>>>        From what I can tell, xlomp_ser is a serial implementation of
> >>>>>>       the
> >>>>>>       openmp library.  When XGC links this library it satisfies the
> >>>>>>       openmp
> >>>>>>       symbols XGC wants and at run time results in openmp API calls
> >>>>>>       like
> >>>>>>       'omp_get_max_threads()' returning 1 regardless of the
> >>>>>>       OMP_NUM_THREADS
> >>>>>>       setting.
> >>>>>>
> >>>>>>       Do you know how I can build petsc, with or without spack, and
> >>>>>>       avoid
> >>>>>>       this
> >>>>>>       library being listed in 'lib/petsc/conf/petscvariables'?
> >>>>>>
> >>>>>>       If this should go to a petsc mailing list or git repo issues page
> >>>>>>       I
> >>>>>>       can
> >>>>>>       send it there.
> >>>>>>
> >>>>>>       Thank-you,
> >>>>>>       Cameron
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> >>
>