[petsc-users] Compile flags for a Blue Gene\Q

[Christian Klettner]

Thanks Jed, for this information.
We'll try it out.
Best regards,
Christian

> Christian Klettner <ucemckl at ucl.ac.uk> writes:
>
>> Dear PETSc group,
>> We have been given access to a Blue Gene\Q system to run code we
>> normally
>> run on a traditional cluster architecture. We have ported the code
>> successfully and run some jobs using 16 cores per node however the
>> performance is roughly four times slower to that of a Xeon processor. We
>> expect less performance (due to the slower chips) however this seems a
>> bit
>> excessive. One problem we think is that we are not using all four
>> hardware
>> threads per core.
>
> The cache architecture is different, so the factor of 4x is not
> unreasonable.  Sparse linear algebra is bandwidth limited and you don't
> need to use threads for bandwidth.
>
> http://www.alcf.anl.gov/sites/www.alcf.anl.gov/files/miracon_AppPerform_BobWalkup_0.pdf
>
> You can configure petsc-dev to use pthreads and/or OpenMP (it was
> disabled in the release), but that code is experimental and still in
> flux.  You can also experiment with this fork to compare the performance
> of an all-OpenMP approach to threading.
>
>   https://bitbucket.org/ggorman/petsc-3.3-omp
>
>> To achieve this do we need to use a threaded version of PETSc? Could
>> someone suggest the additional arguments required to make use of the
>> pthreads when launching an MPI job? We are using the Blue Gene MPI
>> libraries but are currently unable to use the ESSL blas libraries. We
>> found the example compile flags for a Blue Gene \P in the PETSc
>> package but were wondering if anyone had compile flags which they
>> would recommend for a Blue Gene\Q?  Best regards, Christian
>
> You could start with something like this (fixing paths as appropriate;
> choose your optimization flags).
>
> ./configure
> --with-blas-lapack-lib=/soft/libraries/essl/5.1.1-0.beta/lib64/libesslbg.a
> --with-mpi-dir=/bgsys/drivers/ppcfloor/comm/xl/
> --with-mpi-dir=/bgsys/drivers/ppcfloor/comm/xl.legacy.ndebug
> --with-debugging=0 COPTFLAGS="-O3 -qnohot -qsimd=noauto -qsmp=omp:noauto"
> FOPTFLAGS="-O3 -qnohot -qsimd=noauto -qsmp=omp:noauto" PETSC_ARCH=xl-opt
>