[mpich-discuss] how to configure mpich2 v1.1 for hybrid mpi/openmp code

[chan at mcs.anl.gov]

If you don't make any MPI call within your OpenMP segment,
you don't need MPI_THREAD_MULTIPLE, i.e. MPI and OpenMP don't know
about each other.  MPICH2's configure does not know about OpenMP(yet),
so you need to compile your fortran code with intel OpenMP flag, -openmp,
with mpif77/mpif90. For portability/convenience, you may want to use
environment variable OMP_NUM_THREADS to set the number of OMP threads
instead of calling omp_set_num_threads() inside your code (mpiexec has
option to set environment variable in every process).

A.Chan

----- "Susan A. Schwarz" <Susan.A.Schwarz at dartmouth.edu> wrote:

> I have a question about how to configure MPICH2 v1.1 to run my hybrid
> Fortran
> MPI/OpenMP code. I have been running it with MPICH1 and it has been
> working
> correctly but when I tried to use it with MPICH2, the OpenMP section
> uses only 
> one thread instead of the 8 that I request. Here is a description of
> how the 
> code works.
> 
> In the MPI section of the code, the master processor sends out work to
> each of
> the slave processors, the slave sends back the results and receives
> more work
> from the master until all the work is done. After this point the slave
> processes 
> wait while the master processor does the OpenMP section of the code.
> The master 
> calls the OpenMP function omp_set_num_threads to set the number of 
> threads to 
> be 8 which is the number of cores on a single node. The master
> processor then 
> calls the OpenMP MKL pardiso function to invert the sparse matrix that
> contains 
> the results from the MPI portion of the code. When the pardiso
> calculation is 
> completed, the MPI section of the code goes onto the next iteration.
> We run this 
> code on multiple dual quad-core nodes and use only one of the nodes
> for the 
> OpenMP pardiso calculation.  Typically, I run this on 24 processors (3
> 8-core 
> nodes). I would like to be able to use all 24 processors in the MPI
> section and 
> then the 8 processors on the master node for the OpenMP section.
> 
> Up until recently, I have been using MPICH1 v1.2.7p1, the intel v11.1
> fortran
> compiler and the mkl v10.2 library for this calculation and it has
> been working
> correctly. I recently tried to build this code to use MPICH2 compiled
> with the 
> v11.1 intel compilers for the ch3:nemesis device which allows for 
> multithreading. However, despite a call to set the number of openmp
> threads to 
> 8, it only uses 1 thread for the pardiso calculation.
> 
> Here is how I configured MPICH2:
> 
> ./configure --prefix=/opt/mpich2/mpich2-1.1-intel-11.1
> --with-device=ch3:nemesis 
> --enable-debuginfo  --enable-f90 --enable-cxx --enable-romio
> 
> Here is how I initialize MPI:
> 
>   call MPI_INIT_THREAD(MPI_THREAD_MULTIPLE,provided,ierr)
> 
> I am running this code on 64-bit RHEL v5.3.
> 
> My question is, how do I configure my MPICH2 environment to run this
> code.
> 
> Susan Schwarz
> Dartmouth College
> 
> _______________________________________________
> mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss