[petsc-users] installing petsc via spack on ac922 (a summit like system)

[Cameron Smith]

Thank you.

Using the spack spec

petsc at 3.7.7 ~hdf5 ~hypre ~superlu-dist cflags=-qsmp=omp fflags=-qsmp=omp 
cxxflags=-qsmp=omp

and passing the same flags to a manual (non-spack) petsc 3.7.7 build 
produced installs that do not list xlomp_ser in the configure.log.

Thank-you,
Cameron

On 3/10/20 9:48 AM, Satish Balay wrote:
> Created an MR for this change
> 
> https://gitlab.com/petsc/petsc/-/merge_requests/2593
> 
> Satish
> 
> On Tue, 10 Mar 2020, Satish Balay via petsc-users wrote:
> 
>> Another option is to strip out -lxlomp_ser when parsing 'xlc -V' [as its a common library for all 3 language compilers]. i.e the following change [this change is on current maint - but similar change can be done with petsc-3.7]..
>>
>> [This is easier than changing the order in which --with-openmp option is processed]
>>
>> Satish
>> -----------
>>
>> diff --git a/config/BuildSystem/config/compilers.py b/config/BuildSystem/config/compilers.py
>> index b4cbc183c0..cfffb1a80e 100644
>> --- a/config/BuildSystem/config/compilers.py
>> +++ b/config/BuildSystem/config/compilers.py
>> @@ -308,7 +308,7 @@ class Configure(config.base.Configure):
>>               self.logPrint('already in lflags: '+arg, 4, 'compilers')
>>             continue
>>           # Check for system libraries
>> -        m = re.match(r'^-l(ang.*|crt[0-9].o|crtbegin.o|c|gcc|gcc_ext(.[0-9]+)*|System|cygwin|crt[0-9].[0-9][0-9].[0-9].o)$', arg)
>> +        m = re.match(r'^-l(ang.*|crt[0-9].o|crtbegin.o|c|gcc|gcc_ext(.[0-9]+)*|System|cygwin|xlomp_ser|crt[0-9].[0-9][0-9].[0-9].o)$', arg)
>>           if m:
>>             self.logPrint('Skipping system library: '+arg, 4, 'compilers')
>>             continue
>> @@ -687,7 +687,7 @@ class Configure(config.base.Configure):
>>               self.logPrint('already in lflags: '+arg, 4, 'compilers')
>>             continue
>>           # Check for system libraries
>> -        m = re.match(r'^-l(ang.*|crt[0-9].o|crtbegin.o|c|gcc|gcc_ext(.[0-9]+)*|System|cygwin|crt[0-9].[0-9][0-9].[0-9].o)$', arg)
>> +        m = re.match(r'^-l(ang.*|crt[0-9].o|crtbegin.o|c|gcc|gcc_ext(.[0-9]+)*|System|cygwin|xlomp_ser|crt[0-9].[0-9][0-9].[0-9].o)$', arg)
>>           if m:
>>             self.logPrint('Skipping system library: '+arg, 4, 'compilers')
>>             continue
>> @@ -1085,7 +1085,7 @@ Otherwise you need a different combination of C, C++, and Fortran compilers")
>>               self.logPrint('Already in lflags so skipping: '+arg, 4, 'compilers')
>>             continue
>>           # Check for system libraries
>> -        m = re.match(r'^-l(ang.*|crt[0-9].o|crtbegin.o|c|gcc|gcc_ext(.[0-9]+)*|System|cygwin|crt[0-9].[0-9][0-9].[0-9].o)$', arg)
>> +        m = re.match(r'^-l(ang.*|crt[0-9].o|crtbegin.o|c|gcc|gcc_ext(.[0-9]+)*|System|cygwin|xlomp_ser|crt[0-9].[0-9][0-9].[0-9].o)$', arg)
>>           if m:
>>             self.logPrint('Found system library therefore skipping: '+arg, 4, 'compilers')
>>             continue
>>
>> On Tue, 10 Mar 2020, Cameron Smith wrote:
>>
>>> That makes sense.  Thank you.
>>>
>>> -Cameron
>>>
>>> On 3/10/20 9:25 AM, Satish Balay wrote:
>>>> PETSc configure attempts to check for compiler libraries [so that one
>>>> can mix and use c,c++,fortran codes with a c linker] by running
>>>> compilers in verbose mode - and parsing the output. [i.e checkCLibraries(),
>>>> checkFortranLibraries() ..]
>>>>
>>>> Here IBM compiler is using different library internally based on 'xlc -V' vs
>>>> 'xlc -fopenmp -V'. For most other compilers - its just an additional library
>>>> [with corresponding include file]
>>>>
>>>> So -fopenmp needs to be set before this step [checkCLibraries()] in
>>>> configure.
>>>>
>>>> PETSc configure treats -with-openmp as an additional package - and process
>>>> this option after the above checkCLibraries() check.
>>>>
>>>> However CFLAGS etc get processed and set before the call to
>>>> checkCLibraries().
>>>>
>>>> My suggestion is a workaround to get -fopenmp option set before
>>>> checkCLibraries() are called.
>>>>
>>>> Satish
>>>>
>>>> On Tue, 10 Mar 2020, Cameron Smith wrote:
>>>>
>>>>> Thank you.  I'll give that a shot.
>>>>>
>>>>> Out of curiosity, how does passing the openmp flags relate to the
>>>>> '--with-openmp' option described here:
>>>>>
>>>>> https://www.mcs.anl.gov/petsc/documentation/installation.html
>>>>>
>>>>> under 'Installing packages that utilize OpenMP'?  Is this just passing the
>>>>> openmp flags into compile/link commands of the packages that petsc builds
>>>>> (via, --download-<package> options) and not to the petsc compile/link?
>>>>>
>>>>> -Cameron
>>>>>
>>>>> On 3/10/20 9:01 AM, Satish Balay wrote:
>>>>>> BTW: You might be able to do the same via spack.
>>>>>>
>>>>>> spack install petsc at 3.7.7 ~hdf5 ~hypre ~superlu-dist cflags=-fopenmp
>>>>>> fflags=-fopenmp cxxflags=-fopenmp
>>>>>>
>>>>>> Satish
>>>>>>
>>>>>> On Tue, 10 Mar 2020, Satish Balay via petsc-users wrote:
>>>>>>
>>>>>>> Cameron,
>>>>>>>
>>>>>>> You can try changing following petsc configure options and see if that
>>>>>>> works.. [i.e build petsc manually]
>>>>>>>
>>>>>>> CFLAGS=-fopenmp FFLAGS=-fopenmp CXXFLAGS=-fopenmp
>>>>>>>
>>>>>>> Satish
>>>>>>>
>>>>>>> On Tue, 10 Mar 2020, Cameron Smith wrote:
>>>>>>>
>>>>>>>> Thank you Mark.
>>>>>>>>
>>>>>>>> The configure.log is attached.
>>>>>>>>
>>>>>>>> Please let me know if any other info is needed.
>>>>>>>>
>>>>>>>> -Cameron
>>>>>>>>
>>>>>>>> On 3/10/20 7:31 AM, Mark Adams wrote:
>>>>>>>>> Hi Cameron,
>>>>>>>>>
>>>>>>>>> This can go on the list and we always want the configure.log file.
>>>>>>>>>
>>>>>>>>> I build on Summit, but have not used the XL compilers. I've built 3.7.7
>>>>>>>>> with
>>>>>>>>> GNU and PGI. (XGC usually wants PGI)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Mar 9, 2020 at 11:27 PM Cameron Smith <smithc11 at rpi.edu
>>>>>>>>> <mailto:smithc11 at rpi.edu>> wrote:
>>>>>>>>>
>>>>>>>>>        Hello,
>>>>>>>>>
>>>>>>>>>        I'm installing petsc 3.7.7 on a summit like system with the
>>>>>>>>>        following
>>>>>>>>>        spack spec:
>>>>>>>>>
>>>>>>>>>        petsc at 3.7.7 ~hdf5 ~hypre ~superlu-dist
>>>>>>>>>
>>>>>>>>>        with the XL 16.1.1 compiler and Spectrum MPI 10.3 .  This install
>>>>>>>>>        produces a `/path/to/petsc/install/lib/petsc/conf/petscvariables`
>>>>>>>>>        file
>>>>>>>>>        that contains '-lxlomp_ser' in the 'PETSC_EXTERNAL_LIB_BASIC' and
>>>>>>>>>        'PETSC_WITH_EXTERNAL_LIB' variables.
>>>>>>>>>
>>>>>>>>>        The application I'm building, XGC, has a makefile based build
>>>>>>>>>        system
>>>>>>>>>        that includes '/path/to/petsc/install/lib/petsc/conf/variables'
>>>>>>>>>        which in
>>>>>>>>>        turn includes '/lib/petsc/conf/petscvariables'.
>>>>>>>>>
>>>>>>>>>         From what I can tell, xlomp_ser is a serial implementation of
>>>>>>>>>        the
>>>>>>>>>        openmp library.  When XGC links this library it satisfies the
>>>>>>>>>        openmp
>>>>>>>>>        symbols XGC wants and at run time results in openmp API calls
>>>>>>>>>        like
>>>>>>>>>        'omp_get_max_threads()' returning 1 regardless of the
>>>>>>>>>        OMP_NUM_THREADS
>>>>>>>>>        setting.
>>>>>>>>>
>>>>>>>>>        Do you know how I can build petsc, with or without spack, and
>>>>>>>>>        avoid
>>>>>>>>>        this
>>>>>>>>>        library being listed in 'lib/petsc/conf/petscvariables'?
>>>>>>>>>
>>>>>>>>>        If this should go to a petsc mailing list or git repo issues page
>>>>>>>>>        I
>>>>>>>>>        can
>>>>>>>>>        send it there.
>>>>>>>>>
>>>>>>>>>        Thank-you,
>>>>>>>>>        Cameron
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>