[petsc-users] Does an optimized compilation with -march=native on multi-CPU HPC cluster make sense?

[Barry Smith]

> On Aug 11, 2020, at 6:58 AM, Bastian Löhrer <bastian.loehrer at tu-dresden.de> wrote:
> 
> Dear PETSc users,
> 
> we use PETSc in our code.
> Therefore, we have multiple PETSc compilations lying around, each compiled differently,
> e.g debug build using an Intel-compiler, optimized build for Intel-CPUs using an Intel-compiler, optimized build for AMD-Rome-CPUs using an Intel-compiler, several builds using GNU-compilers ...
> 
> Prior to compiling our code, we essentially set $PETSC_DIR and $PETSC_ARCH to point to a suitable PETSc build.
> 
> When I compile such an optimized PETSc build aimed at Intel-CPUs and using an Intel-compiler I do so with
> 
>     COPTFLAGS="-axCOMMON-AVX512,CORE-AVX2,AVX ..."
> 
> to address all Intel processors available on our cluster.
> 
> However, I recently noticed that the PETSc compilation provided by our HPC-administrators was compiled with -march=native.
> Does that make sense? If so this implies that my optimization flags are unnecessary, does it not?
> 
> I imagine that when using a PETSc previously compiled with march=native and I compile that into my code on a CPU different from the one that has been used when compiling PETSc, I end up without optimizations of PETSc. Is that correct?

   This is a good question that suggests maybe we should add a check in PETSc.

 At  runtime could we compare the options used for the compile (and what was built in the .o) to the hardware it is being run on and produce a warning if they are out of wack or one could do better? Presumably one could also do this with non-Intel hardware and GPUs also? Then at least users would know when they copied excutables to other hardware that they are not the best possible optimization for what they are running on. 

  Barry

> 
> Best,
> Bastian
>