[petsc-dev] Seeking OLCF users complaining about poor build times

[Barry Smith]

  Ok, based on what we have so far we need to completely rethink our process, our previous test results are pretty much useless Kill any tests you have outstanding at the moment.

  1) Forget about maint branch!  
  2) Forget about --download-superlu_dist since it does not have a parallel make

   So we want runs on master with --download-hypre --download-metis --download-parmetis --with-debugging=0 and good optimization flags and proper TMPDIRs, if appropriate, on all systems.

  Barry

  
> On Feb 26, 2015, at 5:47 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> And the following is for hopper at nersc. Note: the TMPDIR is on
> luster - so that might be contributing to long times - I'll have to
> redo some of this with a different TMPDIR
> 
> Satish
> 
> ----------------
> hopper.nersc MAKE_NP = 13
> ---------------
> 
> balay at hopper03:~/petsc.clone> time ./reconfigure-arch-cray-xt6-pkgs-opt.py
> real                          58m13.350s
> user                          18m8.168s
> sys                           28m3.945s
> 
> balay at hopper03:~/petsc.clone> time   make PETSC_DIR=/global/u2/b/balay/petsc.clone PETSC_ARCH=arch-cray-xt6-pkgs-opt all
> 
> real    1m40.348s
> user    12m5.281s
> sys     1m38.422s
> 
> balay at hopper03:~/petsc.clone> uptime
> 15:45pm  up 8 days  0:55,  22 users,  load average: 2.37, 3.14, 2.42
> 
> balay at hopper03:~/petsc.clone> cat reconfigure-arch-cray-xt6-pkgs-opt.py 
> #!/usr/bin/python
> if __name__ == '__main__':
>  import sys
>  import os
>  sys.path.insert(0, os.path.abspath('config'))
>  import configure
>  configure_options = [
>    '--COPTFLAGS=-fast -mp',
>    '--CXXOPTFLAGS=-fast -mp',
>    '--FOPTFLAGS=-fast -mp',
>    '--download-hypre',
>    '--download-metis',
>    '--download-parmetis',
>    '--download-superlu_dist',
>    #'--with-blas-lapack-lib=-L/opt/acml/5.3.1/pgi64/lib -lacml',
>    '--download-fblaslapack',
>    '--with-cc=cc',
>    '--with-clib-autodetect=0',
>    '--with-cxx=CC',
>    '--with-cxxlib-autodetect=0',
>    '--with-debugging=0',
>    '--with-fc=ftn',
>    '--with-fortranlib-autodetect=0',
>    '--with-shared-libraries=0',
>    '--with-x=0',
>    '--with-c2html=0',
>    'PETSC_ARCH=arch-cray-xt6-pkgs-opt',
>  ]
>  configure.petsc_configure(configure_options)
> 
> On Thu, 26 Feb 2015, Satish Balay wrote:
> 
>> Attaching logs.
>> 
>> Note: 'uptime' is run a bit after the builds completed..
>> 
>> Satish
>> 
>> --------------
>> mira.alcf build: MAKE_NP = 32
>> --------------
>> [balay at miralac1 petsc.maint.35]$ time ./reconfigure-arch-time-test.py
>> <snip>
>> real    25m57.240s
>> user    17m33.944s
>> sys     6m41.065s
>> 
>> [balay at miralac1 petsc.maint.35]$ time    make PETSC_DIR=/gpfs/mira-home/balay/petsc.maint.35 PETSC_ARCH=arch-time-test all
>> <snip>
>> real    1m39.067s
>> user    22m56.248s
>> sys     7m42.658s
>> 
>> [balay at miralac1 petsc.maint.35]$ uptime
>> 21:50:08 up 10 days,  2:14, 87 users,  load average: 3.86, 5.02, 5.03
>> 
>> --------------
>> Laptop build: MAKE_NP = 4
>> --------------
>> $ time ./configure --download-hypre --download-metis --download-parmetis --download-superlu_dist --download-fblaslapack --with-debugging=0 --with-c2html=0 PETSC_ARCH=arch-test COPTFLAGS="-O3 -fast" FOPTFLAGS="-O3 -fast" CXXOPTFLAGS="-O3 -fast"
>> <snip>
>> real    3m56.755s
>> user    2m47.458s
>> sys     0m57.206s
>> 
>> $ time    make PETSC_DIR=/home/balay/petsc PETSC_ARCH=arch-test all
>> <snip>
>> real    1m13.519s
>> user    3m40.708s
>> sys     0m47.002s
>> 
>> $ uptime
>> 15:50:32 up 3 days,  5:20, 11 users,  load average: 0.05, 0.30, 0.49
>> 
>> --------------
>> 
>> [balay at miralac1 petsc.maint.35]$ cat reconfigure-arch-time-test.py 
>> #!/usr/bin/python
>> if __name__ == '__main__':
>>  import sys
>>  import os
>>  sys.path.insert(0, os.path.abspath('config'))
>>  import configure
>>  configure_options = [
>>    '--download-hypre=1',
>>    '--download-metis=1',
>>    '--download-parmetis=1',
>>    '--download-superlu_dist=1',
>>    #'--with-blas-lapack-lib=-L/soft/libraries/alcf/current/xl/LAPACK/lib -llapack -L/soft/libraries/alcf/current/xl/BLAS/lib -lblas',
>>    '--download-fblaslapack=1',
>>    '--with-c2html=0',
>>    '--with-cc=mpixlc_r',
>>    '--with-cxx=mpixlcxx_r',
>>    '--with-debugging=0',
>>    '--with-fc=mpixlf77_r -qnosave',
>>    '--with-shared-libraries=0',
>>    '--with-x=0',
>>    '-COPTFLAGS= -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction',
>>    '-CXXOPTFLAGS= -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction',
>>    '-FOPTFLAGS= -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction',
>>    'PETSC_ARCH=arch-time-test',
>>  ]
>>  configure.petsc_configure(configure_options)
>> 
>> 
>>>>>> On Feb 26, 2015, at 8:13 AM, Nathan Collier <nathaniel.collier at gmail.com>
>>>>> wrote:
>>>>>> 
>>>>>> Ok, so I built PETSc with metis, parmetis, superlu_dist, and hyper on
>>>>> Titan. The configure time is the second configure--when you run the
>>>>> reconfigure script that the batch submission generates for you.
>>>>>> 
>>>>>> configure:  38m15.488s
>>>>>> make: 15m37.610s
>>>>>> 
>> 
> <configure_hopper.log.gz><make_hopper.log.gz>