[MPICH] MPICH2 performance tuning and characterising

[stephen mulcahy]

Hi Anthony,

Thanks for your response.

Anthony Chan wrote:
> If you have any question on MPE2/SLOG2/Jumpshot, you can always ask us.

I guess I've read the docs on MPE that I could find and I'm still not 
clear what information it will give me and whether it's what I'm looking 
for. I guess to best way to is to go ahead and try it.

It is also a little unclear as to how to build in "mpe support" - or 
whether it is included by default.

I've just rebuilt mpich2 1.0.5p3 with

configure --enable-mpe --with-mpe

Will that do the trick or is that unneccesary?

> Does your MPI job requires thread support ?  If not, you should give
> ch3:nemesis a try and compare the result with ch3:sock's.  The simplest
> way to time your job is:
> 
> date ; mpiexec -np 40 myroms ... ; date

We're not using threads so I'll try nemesis. I might just try it with 
logging first and get some statistics that I can start comparing. Does 
--fast-enable conflict with mpe logging or can I have both?

For real-world performance analysis - our model creates history files 
every X steps. We're using the time to create these history files as a 
rough indicator of performance since the model is long-running.

> I would guess that you could tune the tcp frame size based on the typical
> message size used in your apps.  A google search on the subject should
> help.

I've been considering enabling jumbo frames and went ahead with this 
last night. At a first pass it looks like we're seeing about a 2-3% 
performance improvement but I'll need to do more detailed timing to be 
sure ... I'm not sure if an intermediate frame size between 1500 bytes 
and 9000 bytes could be useful here also .. I guess I'll know more when 
I have some logging output analysed.

> To verify if the code is fairly optimized, you could enable MPE logging
> on your app and see if the logfile makes sense to you.  If you are using
> mpich2-1.0.5, you can relink your code with "mpif90 -mpe=mpilog", it will
> generate a clog2 file which can be "viewed" by jumpshot (which will
> convert it to the native format, slog2).  If your slog2 file is small
> enough, you could send it to us when you have questons (We always like to
> look at logfile, :))

I'm going to give this a shot this morning - I may need further 
assistance from the list to get logging going!

>> What have peoples experiences been in moving from Gigabit to Infiniband?
>> Should one always realise significant performance benefits or does it
>> depend on the load. What kind of MPI loads lend themselves to faster
>> interconnects? How can I see if our MPI load has those characteristics?
> 
> If your app is well-written MPI code and is tighly coupled parallel job, a
> better interconect will significantly improve performance.  I have no
> experience on infiniband, so can't comment on it.  For example, an
> astrophysical code, FLASH2, performed much better with myrinet than with
> ethernet.

I'll need to investigate alternate interconnects further - I'll need 
some more evidence of a concrete benefit before I can justify the 
investment in Infiniband or Myrinet but it does sound promising.

Thanks,

-stephen

-- 
Stephen Mulcahy, Applepie Solutions Ltd, Innovation in Business Center,
    GMIT, Dublin Rd, Galway, Ireland.      http://www.aplpi.com