[petsc-dev] Seeking OLCF users complaining about poor build times

[Satish Balay]

On Thu, 26 Feb 2015, Victor Eijkhout wrote:

> 	
> > On Feb 26, 2015, at 3:52 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> > 
> >  time ./configure --download-hypre --download-metis --download-parmetis --download-superlu_dist --download-fblaslapack --with-debugging=0 --with-c2html=0 PETSC_ARCH=arch-test COPTFLAGS="-O3 -fast" FOPTFLAGS="-O3 -fast" CXXOPTFLAGS="-O3 -fast"
> 
> [login3:petsc-3.5.3:7]  time ./configure --download-hypre --download-metis --download-parmetis --download-superlu_dist --download-fblaslapack --with-debugging=0 --with-c2html=0 PETSC_ARCH=arch-test COPTFLAGS="-O3 -fast" FOPTFLAGS="-O3 -fast" CXXOPTFLAGS="-O3 -fast"
> ===============================================================================
>              Configuring PETSc to compile on your system                       
> ===============================================================================
> ===============================================================================                 WARNING! Compiling PETSc with no debugging, this should                               only be done for timing and production runs. All developme                      be done when configured using --with-debugging=1          =============================================================================== TESTING: checkCLibraries from config.compilers(config/BuildSystem/config/compile*******************************************************************************
>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> -------------------------------------------------------------------------------
> C libraries cannot directly be used from Fortran
> 
> 
> ???
> 
> Successful compile:
> Source:
>       program main
> 
>       end
> Executing: mpif90 -c -o /tmp/petsc-KqceB4/config.setCompilers/conftest.o -I/tmp/petsc-KqceB4/config.setCompilers  -fPIC -O3 -fast  /tmp/petsc-KqceB4/config.setCompilers/conftest.F 
> Successful compile:
> Source:
>       program main
> 
>       end
>                     Pushing language FC
>                     Popping language FC
> Executing: mpif90  -o /tmp/petsc-KqceB4/config.setCompilers/conftest    -fPIC -O3 -fast /tmp/petsc-KqceB4/config.setCompilers/conftest.o -Wl,-rpath,/opt/ofed/lib64 -L/opt/ofed/lib64 -Wl,-rpath,/opt/apps/limic2/0.5.5/lib -L/opt/apps/limic2/0.5.5/lib -Wl,-rpath,/opt/ofed/lib64 -L/opt/ofed/lib64 -Wl,-rpath,/opt/apps/limic2/0.5.5/lib -L/opt/apps/limic2/0.5.5/lib -ldl -Wl,-rpath,/opt/apps/intel13/mvapich2/1.9/lib -L/opt/apps/intel13/mvapich2/1.9/lib -lmpich -lopa -lmpl -libmad -lrdmacm -libumad -libverbs -lrt -llimic2 -lpthread -Wl,-rpath,/opt/ofed/lib64 -L/opt/ofed/lib64 -Wl,-rpath,/opt/apps/limic2/0.5.5/lib -L/opt/apps/limic2/0.5.5/lib -Wl,-rpath,/opt/ofed/lib64 -L/opt/ofed/lib64 -Wl,-rpath,/opt/apps/limic2/0.5.5/lib -L/opt/apps/limic2/0.5.5/lib -Wl,-rpath,/opt/apps/intel13/mvapich2/1.9/lib -L/opt/apps/intel13/mvapich2/1.9/lib -Wl,-rpath,/opt/apps/intel/13/composer_xe_2013.2.146/compiler/lib/intel64 -L/opt/apps/intel/13/composer_xe_2013.2.146/compiler/lib/intel64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4..7 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -Wl,-rpath,/opt/apps/intel/13/composer_xe_2013.2.146/compiler/lib/intel64 -Wl,-rpath,/opt/apps/limic2/0.5.5/lib -Wl,-rpath,/opt/ofed/lib64 -Wl,-rpath,/opt/apps/intel13/mvapich2/1.9/lib -limf -lsvml -lirng -lipgo -ldecimal -lgcc_eh -lirc -lirc_s -Wl,-rpath,/opt/ofed/lib64 -L/opt/ofed/lib64 -Wl,-rpath,/opt/apps/limic2/0.5.5/lib -L/opt/apps/limic2/0.5.5/lib -Wl,-rpath,/opt/ofed/lib64 -L/opt/ofed/lib64 -Wl,-rpath,/opt/apps/limic2/0.5.5/lib -L/opt/apps/limic2/0.5.5/lib -Wl,-rpath,/opt/apps/intel13/mvapich2/1.9/lib -L/opt/apps/intel13/mvapich2/1.9/lib -Wl,-rpath,/opt/apps/intel/13/composer_xe_2013.2.146/compiler/lib/intel64 -L/opt/apps/intel/13/composer_xe_2013.2.146/compiler/lib/intel64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -ldl 
> Possible ERROR while running linker: exit code 256
> stderr:
> ipo: remark #11001: performing single-file optimizations
> ipo: remark #11006: generating object file /tmp/ipo_ifortNhWLZj.o
> /usr/bin/ld: cannot find -ldl
>             Popping language FC
>         compilers: Error message from compiling {Cannot compile/link FC with mpif90.}
> **** Configure header /tmp/petsc-KqceB4/confdefs.h ****
> 
> 
> Seem the "-fast" confuses the intel compiler. We have bad experiences with it anyway, so we don't advise people to use it.

Yeah - intel compilers messup with -fast.

Nathan [on titan] used: CXXOPTFLAGS="-O3 -msse3 -fpic "

[my usage was with gcc] - in my attempt to stress gcc's optimizer - I used -fast.

On mira I used: '-COPTFLAGS= -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict -qstrict_induction',

Satish