[petsc-dev] Compiling Petsc with Intel mpi safe thread library

[Barry Smith]

   You will need to have mkl-cpardiso.py    create and run a simple test program to make sure that the MPI supports funneled, see for example

http://www.mcs.anl.gov/research/projects/mpi/mpi-standard/mpi-report-2.0/node165.htm

Based on my reading this can only be determined by actually running a MPI program and checking the provided flag to see if the MPI is providing the funneled.

  unfortunately in the batch world this is a problem …..


  Barry




On Oct 19, 2013, at 11:13 PM, Jose David Bermeol <jbermeol at purdue.edu> wrote:

> many thakns, I didn't know that using -mt_mpi half way wouldn't work. 
> 
> Yes cpardiso need this flag if not, it compiles, but when I run it sends the following error:
> MPI_THREAD_FUNNELED level is not supported!
> Exit...
> 
> So there is a way to check in mkl-cpardiso.py that I'm using the flag -mt_mpi
> 
> Thanks
> ----- Original Message -----
> From: "Satish Balay" <balay at mcs.anl.gov>
> To: "Jose David Bermeol" <jbermeol at purdue.edu>
> Cc: petsc-dev at mcs.anl.gov
> Sent: Sunday, October 20, 2013 12:04:19 AM
> Subject: Re: [petsc-dev] Compiling Petsc with Intel mpi safe thread library
> 
> sounds like you are inserting -mt_mpi half way through the configure
> process via mkl-cpardiso.py.
> 
> This won't work.
> 
> is mkl-cpardiso tied to [thread safe variant of] intel mpi?
> i.e it won't work with regular intel-mpi or other MPI like mpich?
> 
> To use thread safe mpi - you specify thread safe mpi compilers to
> petsc configure.
> 
> i.e something like:
> 
> --with-cc='mpicc -mt_mpi' or --with-cc='mpicc' CFALGS='-mt_mpi'.
> 
> Satish
> 
> On Sat, 19 Oct 2013, Jose David Bermeol wrote:
> 
>> Hi I'm working again with the solver mkl-cpardiso. I did my implementation, the configuration works well, the problem is during compilation. The linking flags petsc is using are the followings:
>> 
>>    -Wl,-rpath,/home/jbermeol/software/test/arch-linux2-c-opt/lib -L/home/jbermeol/software/test/arch-linux2-c-opt/lib  -lpetsc -Wl,--start-group -L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -lpthread -lm -liomp5 -mt_mpi -Wl,--start-group -L/home/jbermeol/testPetscSolvers/intel_mkl_cpardiso/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -lcpardiso_lp64 -lcpardiso_mpi_lp64 -lpthread -lm -liomp5 -lifcore -Wl,-rpath,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 -L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -lX11 -lpthread -Wl,-rpath,/apps/rhel6/intel/impi/4.1.0.030/intel64/lib -L/apps/rhel6/intel/impi/4.1.0.030/intel64/lib -Wl,-rpath,/apps/rhel6/intel/composer_xe_2013.3.163/compiler/lib/intel64 -L/apps/rhel6/intel/composer_xe_2013.3.163/compiler/lib/intel64 -Wl,-rpath,/apps/rhel6/intel/compos
> 
> er_xe_2013.3.163/ipp/lib/intel64 -L/apps/rhel6/intel/composer_xe_2013.3.163/ipp/lib/intel64 -Wl,-rpath,/apps/rhel6/intel/composer_xe_2013.3.163/tbb/lib/intel64 -L/apps/rhel6/intel/composer_xe_2013.3.163/tbb/lib/intel64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -Wl,-rpath,/home/jbermeol/software/test/-Xlinker -Wl,-rpath,/opt/intel/mpi-rt/4.1 -lifport -lifcore -lm -lm -lmpigc4 -ldl -lmpigf -lmpi -lmpigi -lrt -lpthread -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lirc_s -ldl
>> 
>> The problem is with flag "-mt_mpi", this flag is telling to the Intel compiler, please link with the library libmpi_mt.so(this is a thread safe library for intel mpi). However petsc adds the flag "-lmpi" that links with libmpi.so. Because of this I'm gettig the following error:
>>   ld: MPIR_Thread: TLS definition in /apps/rhel6/intel/impi/4.1.0.030/intel64/lib/libmpi_mt.so section .tbss mismatches non-TLS definition in /apps/rhel6/intel/impi/4.1.0.030/intel64/lib/libmpi.so section .bss
>> 
>> So the first question would be where is this "-lmpi" library flag added??
>> Second how can I compile using the thread save mpi library from intel??
>> Is there a way to set a dependency such that the solver cpardiso is not installed if petsc is compiled without this thread safe library??
>> I don't know yet how to check if petsc is been configure with mkl blas/lapack
>> 
>> Thanks
>>