[petsc-users] Compile flags for a Blue Gene\Q

[Jed Brown]

Christian Klettner <ucemckl at ucl.ac.uk> writes:

> Dear PETSc group,
> We have been given access to a Blue Gene\Q system to run code we normally
> run on a traditional cluster architecture. We have ported the code
> successfully and run some jobs using 16 cores per node however the
> performance is roughly four times slower to that of a Xeon processor. We
> expect less performance (due to the slower chips) however this seems a bit
> excessive. One problem we think is that we are not using all four hardware
> threads per core. 

The cache architecture is different, so the factor of 4x is not
unreasonable.  Sparse linear algebra is bandwidth limited and you don't
need to use threads for bandwidth.

http://www.alcf.anl.gov/sites/www.alcf.anl.gov/files/miracon_AppPerform_BobWalkup_0.pdf

You can configure petsc-dev to use pthreads and/or OpenMP (it was
disabled in the release), but that code is experimental and still in
flux.  You can also experiment with this fork to compare the performance
of an all-OpenMP approach to threading.

  https://bitbucket.org/ggorman/petsc-3.3-omp

> To achieve this do we need to use a threaded version of PETSc? Could
> someone suggest the additional arguments required to make use of the
> pthreads when launching an MPI job? We are using the Blue Gene MPI
> libraries but are currently unable to use the ESSL blas libraries. We
> found the example compile flags for a Blue Gene \P in the PETSc
> package but were wondering if anyone had compile flags which they
> would recommend for a Blue Gene\Q?  Best regards, Christian

You could start with something like this (fixing paths as appropriate;
choose your optimization flags).

./configure --with-blas-lapack-lib=/soft/libraries/essl/5.1.1-0.beta/lib64/libesslbg.a --with-mpi-dir=/bgsys/drivers/ppcfloor/comm/xl/ --with-mpi-dir=/bgsys/drivers/ppcfloor/comm/xl.legacy.ndebug --with-debugging=0 COPTFLAGS="-O3 -qnohot -qsimd=noauto -qsmp=omp:noauto" FOPTFLAGS="-O3 -qnohot -qsimd=noauto -qsmp=omp:noauto" PETSC_ARCH=xl-opt