[petsc-dev] EXTERNAL: Re: PETSc optimization

[Barry Smith]

  This is a good idea. We can put a couple of specific examples of spack commands for PETSc builds in our instructions.

  Thanks

   Barry


> On Aug 8, 2023, at 4:55 PM, Liu Wei AWE via petsc-dev <petsc-dev at mcs.anl.gov> wrote:
> 
>> I see 'spack install cflags=-O3' is working - but not spack install cflags='-O3 -g' [something to debug]
> 
> I got the same issue with 0.19/0.20, likely to be a bug, I will raise the issue with Todd after evaluating the dev branch.
> 
>> The other issue is mapping cflags from spack to CFLAGS/COPTFLAGS - in some cases petsc build requires some default CLFAGS to work - and overriding them from spack might cause issues. [again this part is not properly handed]..
> 
> mfem team implemented the option to pass spack cflags as COPTFLAGS
> https://github.com/spack/spack/blame/develop/var/spack/repos/builtin/packages/petsc/package.py#L380
> 
> 
>>> The question is should optimisation be amended at spack recipe stage or should PETSc configure enforce some compiler optimisation (if -with-debugging=0 is set) ?
> 
>> This sounds reasonable, however there are pitfalls to automatically enabling optimization.
> 
>> The naive approach would be to check if `--with-debugging=0` is set, and if so, append `-O3 -march=native -mtune=native` (assuming those work with the compiler). This might work well enough for GCC/Clang (and probably MSVC, but who knows), but not all compilers play nice. For example:
> 
>> - Intel compilers (before the clang transition) automatically (and silently!) enable the equivalent of `-ffast-math` on any optimization level greater than base.
>> - PGI compilers have had some long running bugs when optimization has been enabled. For example, there has been a bug with __attribute__((visibility("hidden"))) (conversely, "default" if using `-fvisibility=hidden`) variables, where their order in the source file (i.e. is variable A literally defined before or after variable B) causes their definition to be lost at link-time leading to "undefined reference to Foo" errors.
> 
>> All this is to say, configure now has to keep track of, and add special rules for all these compilers (and versions). Since there is no clear answer we have found it's easier to punt responsibility to the user.
> 
> With spack, build and deployment is decentralized, whilst traditionally we can apply optimization + appropriate dependency packages (e.g. MKL/libsci) for the specific HPC platform and compilers; changing to spack means individual software teams builds their own package chain, with their own internal PETSc which by default likely to be reliant on the package recipe / configure, and they are certainly not going to examine the dependency stack in detail until spack build performs poorly to the non-spack version.
> 
> 
>> The space of "best" choices for optimization flags for PETSc is huge; each OS, each compiler, and each particular piece of hardware may have different "best" choices, in addition, the previous choice may not even allow compiling with any small change in any of these things. Finally particular choices may not be compatible with choices may be other packages being linked with PETSc. Hence PETSc's configure punts on trying to select a good choice (and just uses -O).  Surely this is an issue with most portable open source packages not being able to select "good" optimization flags in a portable way?
> 
> We did conducted some limited evaluation a while back, performance difference from "-g -O" -> "-g -O3 -march=native" was noticeable (although that was Tealeaf + older PETSc 3.10 / gcc 9.3).
> 
>> Does the Spack system contain any infrastructure to help solve this common problem? That ensures all the packages it builds have good compatible optimization flags for the particular system?
> I don't know which takes precedence, optimization enforced by cmake within the package, or package recipes, or via spack compiler spec.
> 
>> Perhaps the configure message you mention printed by PETSc is inappropriate for Spack builds of PETSc and we should print something different to help guide the user to the correct Spack solution to the issue?
> 
> If the consensus is setting optimization via spack (instead of configure), possibly also give a more detailed breakdown of "spack petsc install" on PETSc install page (maybe even a separate page)?
> 
> e.g.
> # gnu optimized version
> spack install petsc %gcc at 10 cflags='-g -O3 -march=native -mtune=native' fflags='-g -O3 -march=native -mtune=native'  cxxflags='-g -O3 -march=native -mtune=native'
> 
> # petsc without default spack extras
> spack install petsc ~superlu-dist ~metis ~hypre ~hdf5
> 
> 
> Regards
> Wei Liu
> 
> 
> 
> 
> The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR