[MPICH] MPICH2 performance tuning and characterising

[stephen mulcahy]

Hi,

We're currently using MPICH2 (compiled with pgi) on a 20 node cluster of 
dual core opterons connected with a gigabit network to run an 
Oceanographic numerical model (http://www.myroms.org/). I've been 
reading the archives of this list and the all the documentation on the 
MPICH2 website but I haven't come across a lot of concrete performnce 
tuning information. I did come across the MPE/SLOG/Jumpshot 
documentation but I'm still not sure I know enough to use the 
information that provides :)

We're currently using ch3:sock but wondering if we should look at 
ch3:ssm, ch3:nemesis or ch3: sctp (the 1.0.5 notes suggest that sctp on 
Linux still has issues). What kind of performance increases should I 
expecting from these - percents or fractions of percents? Also, what 
kind of performance improvement on a cluster should --enable-fast bring?)

Are there other things I can do to tune the operation of MPICH2 at the 
OS level? I'm not a Fortran programmer and we're using a 3rd party model 
so I'm not sure how much customisation is feasible (I'm assuming the 
developers of the model have done a certain amount of tuning of the 
basic model ... but any suggestions on how to verify this are also 
welcome). Is the speed at which MPI operates largely dependent on the 
interconnect latency or are there other factors involved?

MPI message sizes are estimated to be about 40k and 200k, does the size 
of the MPI message influence behaviour significantly? Do different sized 
messages have better affinity to different kinds of interconnect?

What have peoples experiences been in moving from Gigabit to Infiniband? 
Should one always realise significant performance benefits or does it 
depend on the load. What kind of MPI loads lend themselves to faster 
interconnects? How can I see if our MPI load has those characteristics?

I've sent a similar query to the beowulf mailing list, apologies if 
you're on both :)

Thanks,

-stephen

-- 
Stephen Mulcahy, Applepie Solutions Ltd, Innovation in Business Center,
    GMIT, Dublin Rd, Galway, Ireland.      http://www.aplpi.com