[petsc-dev] Moving from BG/L to BG/P

[Satish Balay]

On Thu, 13 Oct 2011, Jed Brown wrote:

> On Thu, Oct 13, 2011 at 23:01, <Kevin.Buckley at ecs.vuw.ac.nz> wrote:
> 
> > Hi there,
> >
> > Some time ago I built a PETSc (3.0.0-p9) on the BG/P down at NZ's
> > Univ of Canterbury, for a researcher here at VUW who wanted to
> > run the PISM ice-sheet modelling code on top of it, and in the
> > process discovered that the version installed by the sys admins
> > at the facility was buggy, so the researcher carried on compiling
> > later PISMs against my 3.0.0-p9.
> >
> > UoC are in the process of upgrading to a BG/P and the researcher
> > has asked me to see if I can coddle things together, is keen to
> > run with what he had before and is under "time pressure" to get
> > the remaining results needed for a paper, so is hoping I can
> > install something ahead of the facility's sys admins install
> > a system wide version.
> >
> > Notwithstanding the obvious upgrade in release of PETSc from our
> > 3.0 series to the current 3.2, I notice that the petsc-bgl-tools
> > wrapper package only (not surprisingly I guess, given the bgl bit)
> > provided for IBM 7.0 and 8.0 compiler suites, so, have you guys
> > tried out PETSc on a BG/P yet?
> >
> 
> BG/P has been out for a while, so of course people have been running PETSc
> on it for years. You can look at
> 
> config/examples/arch-bgp-ibm-opt.py
> 
> in the source tree. Here is a configuration that we used for some benchmarks
> on Shaheen last year:
> 
> Configure options: --with-x=0 --with-is-color-value-type=short
> --with-debugging=1 --with-fortran-kernels=1
> --with-mpi-dir=/bgsys/drivers/ppcfloor/comm --with-batch=1
> --known-mpi-shared-libraries=1 --known-memcmp-ok --known-sizeof-char=1
> --known-sizeof-void-p=4 --known-sizeof-short=2 --known-sizeof-int=4
> --known-sizeof-long=4 --known-sizeof-size_t=4 --known-sizeof-long-long=8
> --known-sizeof-float=4 --known-sizeof-double=8 --known-bits-per-byte=8
> --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4
> --known-mpi-long-double=1 --known-level1-dcache-assoc=0
> --known-level1-dcache-linesize=32 --known-level1-dcache-size=32768
> --download-hypre=1 --with-shared=0
> --prefix=/opt/share/ksl/petsc/dev-dec9-hypre/ppc450d-bgp_xlc_hypre_fast
> --with-clanguage=c --COPTFLAGS="     -O3 -qhot" --CXXOPTFLAGS="     -O3
> -qhot" --FOPTFLAGS="     -O3 -qhot" --LIBS="
> -L/bgsys/ibm_essl/sles10/prod/opt/ibmmath/lib -L/opt/ibmcmp/xlsmp/bg/1.7/lib
> -L/opt/ibmcmp/xlmass/bg/4.4/bglib -L/opt/ibmcmp/xlf/bg/11.1/bglib
> -L/bgsys/ibm_essl/sles10/prod/opt/ibmmath/lib -L/opt/ibmcmp/xlsmp/bg/1.7/lib
> -L/opt/ibmcmp/xlmass/bg/4.4/bglib -L/opt/ibmcmp/xlf/bg/11.1/bglib   -lesslbg
>  -lxlf90_r -lxlopt -lxlsmp -lxl -lxlfmath  -lesslbg  -lxlf90_r -lxlopt
> -lxlsmp -lxl -lxlfmath -O3"
> --CC=/bgsys/drivers/ppcfloor/comm/xl/bin/mpixlc_r
> --CXX=/bgsys/drivers/ppcfloor/comm/xl/bin/mpixlcxx_r
> --FC=/bgsys/drivers/ppcfloor/comm/xl/bin/mpixlf90_r --with-debugging=0
> PETSC_ARCH=bgp-xlc-hypre-fast

Just to clarify - petsc-bgl-tools were useful for the first generation
of BlueGene [bg/l] with the first generation of software stack - where
mpixlc/mpixlf77/mpixlf90 wrappers to use IBM compilers with MPI on
these machines were absent.

Subsequent machines [bg/p] and perhaps software stack update to bg/l
do provide proper mpixlc/mpixlf77/mpixlf90 warppers. [so
petsc-bgl-tools isn't needed anymore]

As Jed suggested - use config/examples/arch-bgp-ibm-opt.py - modify it
appropriately for the externalpackages requred - and run it.

Satish