[mpich-discuss] Poor scaling of MPI_WIN_CREATE?

[Timothy Stitt]

Hi all,

I am currently trying to improve the scaling of a CFD code on some Cray machines at NERSC (I believe Cray systems leverage mpich2 for their MPI communications, hence the posting to this list) and I am running into some scalability issues with the MPI_WIN_CREATE() routine.

To cut a long story short, the CFD code requires each process to receive values from some neighborhood processes. Unfortunately, each process doesn't know who its neighbors should be in advance. To overcome this we exploit the one-sided MPI_PUT() routine to communicate data from neighbors directly.

Recent profiling at 256, 512 and 1024 processes shows that the MPI_WIN_CREATE routine is starting to dominate the walltime and reduce our scalability quite rapidly. For instance the %walltime for MPI_WIN_CREATE over various process sizes increases as follows:

256 cores - 4.0%
512 cores - 9.8%
1024 cores - 24.3%

I was wondering if anyone in the MPICH2 community had any advice on how one can improve the performance of MPI_WIN_CREATE? Or maybe someone has a better strategy for communicating the data that bypasses the (poorly scaling?) MPI_WIN_CREATE routine.

Thanks in advance for any help you can provide.

Regards,

Tim.