[petsc-dev] Compilation with ESSL SMP

[Satish Balay]

On Mon, 20 Feb 2017, Jeff Hammond wrote:

> The following ALCF documentation may be relevant to folks who want to link
> ESSL (note the need for path modifications for installations with different
> filesystem organization).
> 
> Linking ESSL with GCC is a bit more complicated because the compiler
> runtime libraries will not be added automatically.
> 
> Please also note the LAPACK incompatibility issue.  It might be fixed in
> later versions of ESSL on POWER, but as far as I know, it remains present
> on Blue Gene.

PETSc attempts to detect essl usage - deal with this incompatibility -
so the recommended usage is different for us..

> 
> Jeff
> (who wrote this documentation as an Argonne employee but now works for
> Intel and hasn't used Blue Gene much lately)
> 
> From https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q#Linking_ESSL:
> 
> Linking ESSL
> 
> ESSL does not provide full LAPACK so you need to link Netlib in first (from
> the left) if you rely upon proper LAPACK behavior. It is not just a
> missing-symbol issue but rather than one or more LAPACK symbols are
> implemented differently in ESSL in such a way as to break codes that assume
> Netlib LAPACK calling conventions.

with PETSc - one can use "-lesslbg" - and configure will check and
disable missing lapack features - and enable the use of alternate
LAPACK API.

However if one specifies "-llapack -lesslbg" - the alternate LAPACK
API will still get enabled - resulting in buggy runs. So the
recommendation is to use:

"-lesslbg -llapack -lesslbg" - so that -llapack is used only for
LAPACK functions that are missing from -lesslbg

Satish


> Single-threaded
> 
> I avoid the use of -lxlomp_ser since I recall that it will break in some
> cases.
> 
> export IBMCMP_ROOT=${IBM_MAIN_DIR}
> export BLAS_LIB=/soft/libraries/alcf/current/xl/BLAS/lib
> export LAPACK_LIB=/soft/libraries/alcf/current/xl/LAPACK/lib
> export ESSL_LIB=/soft/libraries/essl/current/essl/5.1/lib64
> export XLF_LIB=${IBMCMP_ROOT}/xlf/bg/14.1/bglib64
> export XLSMP_LIB=${IBMCMP_ROOT}/xlsmp/bg/3.1/bglib64
> export XLMASS_LIB=${IBMCMP_ROOT}/xlmass/bg/7.3/bglib64
> export MATH_LIBS="-L${XLMASS_LIB} -lmassv -lmass -L${LAPACK_LIB} -llapack \
> -L${ESSL_LIB} -lesslbg -L${XLF_LIB} -lxlf90_r \
> -L${XLSMP_LIB} -lxlsmp -lxlopt -lxlfmath -lxl \
> -Wl,--allow-multiple-definition"
> 
> Multi-threaded
> 
> The key difference is is -lesslsmpbg.
> 
> export IBMCMP_ROOT=${IBM_MAIN_DIR}
> export BLAS_LIB=/soft/libraries/alcf/current/xl/BLAS/lib
> export LAPACK_LIB=/soft/libraries/alcf/current/xl/LAPACK/lib
> export ESSL_LIB=/soft/libraries/essl/current/essl/5.1/lib64
> export XLF_LIB=${IBMCMP_ROOT}/xlf/bg/14.1/bglib64
> export XLSMP_LIB=${IBMCMP_ROOT}/xlsmp/bg/3.1/bglib64
> export XLMASS_LIB=${IBMCMP_ROOT}/xlmass/bg/7.3/bglib64
> export MATH_LIBS="-L${XLMASS_LIB} -lmassv -lmass -L${LAPACK_LIB} -llapack \
> -L${ESSL_LIB} -lesslsmpbg -L${XLF_LIB} -lxlf90_r \
> -L${XLSMP_LIB} -lxlsmp -lxlopt -lxlfmath -lxl \
> -Wl,--allow-multiple-definition"
> 
> 
> On Mon, Feb 20, 2017 at 8:26 AM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> > On Mon, 20 Feb 2017, Barry Smith wrote:
> >
> > >
> > > > On Feb 20, 2017, at 5:13 AM, Pierre Jolivet <
> > pierre.jolivet at enseeiht.fr> wrote:
> > > >
> > > > Hello,
> > > > It looks like the configure does not directly handle libesslsmpbg
> > instead of libesslbg. On the other hand, it handles both
> > libmkl_intel_thread and libmkl_sequential.
> > > > Is there a reason for such a limitation? (I'd like to have direct
> > solvers linked with threaded BLAS).
> > >
> > >    There is no reason. We wrote the essl interface many years ago and
> > actually don't even have access to it any more. If you can make the
> > additions and provide a pull request we'd be happy to include it.
> > >
> >
> > The defaults are just that - defaults.
> >
> > One should use whatever blaslapack that suitable with the option
> > --with-blas-lapack-lib option.
> >
> > Wrt libmkl_intel_thread vs libmkl_sequential - configure defaults to
> > libmkl_sequential for reqular use.
> >
> > libmkl_intel_thread is picked up only when MKL pardiso is requested
> > [so the assumption here is aware of pardiso requirements wrt
> > threads].
> >
> > I see mkl check for windows is also looking for the thread version -
> > don't remember the reason..
> >
> > Satish
> >
> >
> 
> 
>