[petsc-dev] Seeking OLCF users complaining about poor build times

[Satish Balay]

On Thu, 26 Feb 2015, Nathan Collier wrote:

> Ok, so I am not sure I got all that, but here goes some more information
> (attached log).
> 
> No packages / master
> 
> [nate at titan] time python no_packages.py
> ...
> real 14m32.169s
> user 1m51.527s
> sys 3m40.734s
> 
> [nate at titan] time make all
> ...
> real 15m56.004s
> user 8m8.971s
> sys 52m42.734s

Ok - here are the numbers I get.

These builds avoid --download-package stuff completely. i.e - I'm
avoding auto download/install of sowing by installing it separately
and adding bfort to PATH.

The default TMPDIR setting on hopper [i.e luster fs] is causing
configure to run 4 times slower..

Satish

------------
Laptop MAKE_NP = 4
------------
$ time ./configure --with-c2html=0 --with-x=0 --with-debugging=0 COPTFLAGS="-O3 -fast" CXXOPTFLAGS="-O3 -fast" FOPTFLAGS="-O3 -fast"
real 0m53.781s
user 0m22.082s
sys  0m19.343s
$ time    make PETSC_DIR=/home/balay/petsc PETSC_ARCH=arch-linux2-c-opt all
real 1m13.324s
user 3m43.320s
sys  0m47.678s
$ uptime
 19:59:43 up 3 days,  9:29, 11 users,  load average: 0.05, 0.28, 0.26

------------
mira  MAKE_NP = 32
------------
[balay at miralac1 petsc]$ time ./reconfigure-arch-time-test-nopkg.py
real    6m59.320s
user    3m0.473s
sys     3m29.692s
[balay at miralac1 petsc]$ time   make PETSC_DIR=/gpfs/mira-home/balay/petsc PETSC_ARCH=arch-time-test-nopkg all
real    1m49.837s
user    26m1.417s
sys     9m11.808s
[balay at miralac1 petsc]$ uptime
 01:59:24 up 10 days,  6:23, 75 users,  load average: 5.27, 8.19, 10.83

------------
hopper  MAKE_NP = 13 TMPDIR=/tmp
------------
balay at hopper03:~/petsc.clone> time ./reconfigure-arch-cray-xt6-pkgs-opt.py 
real 6m17.133s
user 3m8.348s
sys  1m33.378s

balay at hopper03:~/petsc.clone> time   make PETSC_DIR=/global/u2/b/balay/petsc.clone PETSC_ARCH=arch-cray-xt6-pkgs-opt all
real 1m39.624s
user 13m20.154s
sys  1m39.854s

------------
hopper  MAKE_NP = 13 TMPDIR=/scratch/scratchdirs/balay [default]
------------
balay at hopper03:~/petsc.clone> time ./reconfigure-arch-cray-xt6-pkgs-opt.py 
real                          23m17.471s
user                          3m27.725s
sys                           17m53.559s
balay at hopper03:~/petsc.clone> time    make PETSC_DIR=/global/u2/b/balay/petsc.clone PETSC_ARCH=arch-cray-xt6-pkgs-opt all
real                          1m45.691s
user                          13m29.499s
sys                           1m47.615s
balay at hopper03:~/petsc.clone> uptime
 18:21pm  up 8 days  3:31,  21 users,  load average: 4.05, 3.96, 3.21


------------------------------------------------------------------------------

[balay at miralac1 petsc]$ cat reconfigure-arch-time-test-nopkg.py 
#!/usr/bin/python
if __name__ == '__main__':
  import sys
  import os
  sys.path.insert(0, os.path.abspath('config'))
  import configure
  configure_options = [
    '--with-blas-lapack-lib=-L/soft/libraries/alcf/current/xl/LAPACK/lib -llapack -L/soft/libraries/alcf/current/xl/BLAS/lib -lblas',
    '--with-c2html=0',
    '--with-cc=mpixlc_r',
    '--with-cxx=mpixlcxx_r',
    '--with-debugging=0',
    '--with-fc=mpixlf77_r -qnosave',
    '--with-shared-libraries=0',
    '--with-x=0',
    '-COPTFLAGS= -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction',
    '-CXXOPTFLAGS= -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction',
    '-FOPTFLAGS= -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction',
    'PETSC_ARCH=arch-time-test-nopkg',
  ]
  configure.petsc_configure(configure_options)
[balay at miralac1 petsc]$ 

------------------------------------------------------------------------------

balay at hopper01:~/petsc.clone> cat reconfigure-arch-cray-xt6-pkgs-opt.py 
#!/usr/bin/python
if __name__ == '__main__':
  import sys
  import os
  sys.path.insert(0, os.path.abspath('config'))
  import configure
  configure_options = [
    '--COPTFLAGS=-fast -mp',
    '--CXXOPTFLAGS=-fast -mp',
    '--FOPTFLAGS=-fast -mp',
    '--with-blas-lapack-lib=-L/opt/acml/5.3.1/pgi64/lib -lacml',
    '--with-cc=cc',
    '--with-clib-autodetect=0',
    '--with-cxx=CC',
    '--with-cxxlib-autodetect=0',
    '--with-debugging=0',
    '--with-fc=ftn',
    '--with-fortranlib-autodetect=0',
    '--with-shared-libraries=0',
    '--with-x=0',
    '--with-c2html=0',
    'PETSC_ARCH=arch-cray-xt6-pkgs-opt',
  ]
  configure.petsc_configure(configure_options)
balay at hopper01:~/petsc.clone> 

------------------------------------------------------------------------------