[petsc-users] Does an optimized compilation with -march=native on multi-CPU HPC cluster make sense?

[Jed Brown]

-march=native is whatever architecture it was built on.  That might be a login node.

You might note that without -mprefer-vector-width=512 (gcc/clang) or -qopt-zmm-usage=high (icc), the compiler will rarely if ever actually use AVX-512 (because it causes huge stalls while the frequency is dropped).  It likely doesn't pay off for most sparse matrix work.  In that case, you can just build for AVX2/FMA (Haswell/Broadwell, Rome) and it'll presumably work across all machines, with comparable performance on your skylake-avx512 systems.  I'd encourage you to double-check this with benchmarking of your entire app.  You can check if any AVX-512 instructions have been generated using

objdump -d --prefix-addresses -M intel libpetsc.so | grep zmm0

Bastian Löhrer <bastian.loehrer at tu-dresden.de> writes:

> Dear PETSc users,
>
> we use PETSc in our code.
> Therefore, we have multiple PETSc compilations lying around, each 
> compiled differently,
> e.g debug build using an Intel-compiler, optimized build for Intel-CPUs 
> using an Intel-compiler, optimized build for AMD-Rome-CPUs using an 
> Intel-compiler, several builds using GNU-compilers ...
>
> Prior to compiling our code, we essentially set $PETSC_DIR and 
> $PETSC_ARCH to point to a suitable PETSc build.
>
> When I compile such an optimized PETSc build aimed at Intel-CPUs and 
> using an Intel-compiler I do so with
>
>      COPTFLAGS="-axCOMMON-AVX512,CORE-AVX2,AVX ..."
>
> to address all Intel processors available on our cluster.
>
> However, I recently noticed that the PETSc compilation provided by our 
> HPC-administrators was compiled with -march=native.
> Does that make sense? If so this implies that my optimization flags are 
> unnecessary, does it not?
>
> I imagine that when using a PETSc previously compiled with march=native 
> and I compile that into my code on a CPU different from the one that has 
> been used when compiling PETSc, I end up without optimizations of PETSc. 
> Is that correct?
>
> Best,
> Bastian