[MPICH] slowdown in replica exchange dynamics calculation

[baxa at uchicago.edu]

Hello, I was referred to this mailing list by the system
administrator of the cluster on which running simulations. 
Please let me know if this is not the appropriate forum for my
issue.  

I am running replica exchange combined with langevin dynamics
using a single fortran program.  For example, on a set of 10
nodes, a langevin dynamics trajectory is calculated on each
node for 10,000 steps.  After all the nodes have completed
their number of steps, the nodes communicate with one another
via MPI and determine whether to swap temperatures.  Once
temperatures have been swapped, the pattern repeats.

It turns out though, that the dynamics calculation portion of
the above program takes longer than it would if I were to just
run a langevin dynamics calculation on a single node for 10000
steps.  For 10 nodes, it is approximately 5-10X slower.  While
no MPI commands are being executed during this time, the
duration of the dynamics portion appears to depend on the
number of nodes.  For this reason, it was suspected that this
slowdown may be due to a suboptimal distribution of tasks,
unnecessary barriers, or IO waits.

Finally, in comparing my "fast" dynamics calculation on one
node, to my "slow" replica exchange dynamics on 10 nodes, I
observed the following in the respective output files.

> single dynamics trajectory:
> Limits: neednodes=1,nodes=1,walltime=01:00:00
> Resources:
> cput=00:09:18,mem=7664kb,vmem=129348kb,walltime=00:09:25
>
> "slow" replica exchange:
> Limits:         neednodes=10,nodes=10,walltime=01:00:00
> Resources:
> cput=00:01:45,mem=14824kb,vmem=216924kb,walltime=00:53:00

 
For the "slow" replica exchange simulation, there was only one
swap period.  Looking at the .slog2 file, all of the MPI
commands are condensed into a small time window compared to
the entire program time.

I understand that this is a lot of information at once. 
Please let me know if I can provide any more information.

Thank you,

Michael Baxa

****************************************
Michael C. Baxa
Department of Physics
Sosnick Lab (773) 834-0658
Department of Biochemistry and Molecular Biology
Institute for Biophysical Dynamics
University of Chicago, CIS W107E
929 E 57th St, Chicago, IL 60637
baxa at uchicago.edu
****************************************