[mpich-discuss] how to configure mpich2 v1.1 for hybrid mpi/openmp code

[Susan A. Schwarz]

I have a question about how to configure MPICH2 v1.1 to run my hybrid Fortran
MPI/OpenMP code. I have been running it with MPICH1 and it has been working
correctly but when I tried to use it with MPICH2, the OpenMP section uses only 
one thread instead of the 8 that I request. Here is a description of how the 
code works.

In the MPI section of the code, the master processor sends out work to each of
the slave processors, the slave sends back the results and receives more work
from the master until all the work is done. After this point the slave processes 
wait while the master processor does the OpenMP section of the code. The master 
calls the OpenMP function omp_set_num_threads to set the number of  threads to 
be 8 which is the number of cores on a single node. The master processor then 
calls the OpenMP MKL pardiso function to invert the sparse matrix that contains 
the results from the MPI portion of the code. When the pardiso calculation is 
completed, the MPI section of the code goes onto the next iteration. We run this 
code on multiple dual quad-core nodes and use only one of the nodes for the 
OpenMP pardiso calculation.  Typically, I run this on 24 processors (3 8-core 
nodes). I would like to be able to use all 24 processors in the MPI section and 
then the 8 processors on the master node for the OpenMP section.

Up until recently, I have been using MPICH1 v1.2.7p1, the intel v11.1 fortran
compiler and the mkl v10.2 library for this calculation and it has been working
correctly. I recently tried to build this code to use MPICH2 compiled with the 
v11.1 intel compilers for the ch3:nemesis device which allows for 
multithreading. However, despite a call to set the number of openmp threads to 
8, it only uses 1 thread for the pardiso calculation.

Here is how I configured MPICH2:

./configure --prefix=/opt/mpich2/mpich2-1.1-intel-11.1 --with-device=ch3:nemesis 
--enable-debuginfo  --enable-f90 --enable-cxx --enable-romio

Here is how I initialize MPI:

  call MPI_INIT_THREAD(MPI_THREAD_MULTIPLE,provided,ierr)

I am running this code on 64-bit RHEL v5.3.

My question is, how do I configure my MPICH2 environment to run this code.

Susan Schwarz
Dartmouth College