[petsc-dev] Compilation with ESSL SMP

[Jeff Hammond]

The following ALCF documentation may be relevant to folks who want to link
ESSL (note the need for path modifications for installations with different
filesystem organization).

Linking ESSL with GCC is a bit more complicated because the compiler
runtime libraries will not be added automatically.

Please also note the LAPACK incompatibility issue.  It might be fixed in
later versions of ESSL on POWER, but as far as I know, it remains present
on Blue Gene.

Jeff
(who wrote this documentation as an Argonne employee but now works for
Intel and hasn't used Blue Gene much lately)

>From https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q#Linking_ESSL:

Linking ESSL

ESSL does not provide full LAPACK so you need to link Netlib in first (from
the left) if you rely upon proper LAPACK behavior. It is not just a
missing-symbol issue but rather than one or more LAPACK symbols are
implemented differently in ESSL in such a way as to break codes that assume
Netlib LAPACK calling conventions.
Single-threaded

I avoid the use of -lxlomp_ser since I recall that it will break in some
cases.

export IBMCMP_ROOT=${IBM_MAIN_DIR}
export BLAS_LIB=/soft/libraries/alcf/current/xl/BLAS/lib
export LAPACK_LIB=/soft/libraries/alcf/current/xl/LAPACK/lib
export ESSL_LIB=/soft/libraries/essl/current/essl/5.1/lib64
export XLF_LIB=${IBMCMP_ROOT}/xlf/bg/14.1/bglib64
export XLSMP_LIB=${IBMCMP_ROOT}/xlsmp/bg/3.1/bglib64
export XLMASS_LIB=${IBMCMP_ROOT}/xlmass/bg/7.3/bglib64
export MATH_LIBS="-L${XLMASS_LIB} -lmassv -lmass -L${LAPACK_LIB} -llapack \
-L${ESSL_LIB} -lesslbg -L${XLF_LIB} -lxlf90_r \
-L${XLSMP_LIB} -lxlsmp -lxlopt -lxlfmath -lxl \
-Wl,--allow-multiple-definition"

Multi-threaded

The key difference is is -lesslsmpbg.

export IBMCMP_ROOT=${IBM_MAIN_DIR}
export BLAS_LIB=/soft/libraries/alcf/current/xl/BLAS/lib
export LAPACK_LIB=/soft/libraries/alcf/current/xl/LAPACK/lib
export ESSL_LIB=/soft/libraries/essl/current/essl/5.1/lib64
export XLF_LIB=${IBMCMP_ROOT}/xlf/bg/14.1/bglib64
export XLSMP_LIB=${IBMCMP_ROOT}/xlsmp/bg/3.1/bglib64
export XLMASS_LIB=${IBMCMP_ROOT}/xlmass/bg/7.3/bglib64
export MATH_LIBS="-L${XLMASS_LIB} -lmassv -lmass -L${LAPACK_LIB} -llapack \
-L${ESSL_LIB} -lesslsmpbg -L${XLF_LIB} -lxlf90_r \
-L${XLSMP_LIB} -lxlsmp -lxlopt -lxlfmath -lxl \
-Wl,--allow-multiple-definition"


On Mon, Feb 20, 2017 at 8:26 AM, Satish Balay <balay at mcs.anl.gov> wrote:

> On Mon, 20 Feb 2017, Barry Smith wrote:
>
> >
> > > On Feb 20, 2017, at 5:13 AM, Pierre Jolivet <
> pierre.jolivet at enseeiht.fr> wrote:
> > >
> > > Hello,
> > > It looks like the configure does not directly handle libesslsmpbg
> instead of libesslbg. On the other hand, it handles both
> libmkl_intel_thread and libmkl_sequential.
> > > Is there a reason for such a limitation? (I'd like to have direct
> solvers linked with threaded BLAS).
> >
> >    There is no reason. We wrote the essl interface many years ago and
> actually don't even have access to it any more. If you can make the
> additions and provide a pull request we'd be happy to include it.
> >
>
> The defaults are just that - defaults.
>
> One should use whatever blaslapack that suitable with the option
> --with-blas-lapack-lib option.
>
> Wrt libmkl_intel_thread vs libmkl_sequential - configure defaults to
> libmkl_sequential for reqular use.
>
> libmkl_intel_thread is picked up only when MKL pardiso is requested
> [so the assumption here is aware of pardiso requirements wrt
> threads].
>
> I see mkl check for windows is also looking for the thread version -
> don't remember the reason..
>
> Satish
>
>


-- 
Jeff Hammond
jeff.science at gmail.com
http://jeffhammond.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20170220/2d5d49ba/attachment.html>