[MPICH] MPICH2 performance issue with dual core

[Rajeev Thakur]

The collectives in MPICH2 are not optimized for multicore, but it's at the
top of our list to do.

Rajeev 


> -----Original Message-----
> From: owner-mpich-discuss at mcs.anl.gov 
> [mailto:owner-mpich-discuss at mcs.anl.gov] On Behalf Of Tony Ladd
> Sent: Wednesday, December 26, 2007 12:00 PM
> To: mpich-discuss at mcs.anl.gov
> Subject: [MPICH] MPICH2 performance issue with dual core
> 
> I am using MPICH2 over Gigabit ethernet (Intel PRO 1000 + Extreme 
> Networks x450a-s48t switches). For a single process per node 
> MPICH2 is 
> very fast; typical throughput on edge exchange is ~100MBytes/sec both 
> ways. MPICH2 has more uniform throughput than LAM, is much 
> faster than 
> OpenMPI and almost as good throughput as MPIGAMMA (using 1MB TCP 
> buffers). Latency is 24 microsecs with tuned NIC drivers. So far so 
> (very) good.
> 
> Collective communications are excellent for 1 process as well, but 
> terrible with 2 processes per node. For example, an AlltoAll with 16 
> processes has average 1-way throughput of 56MBytes/sec when 
> distributed 
> over 16 nodes but only 6MBytes per sec when using 8 nodes and 2 
> processes per node. This is of course the reverse of what one would 
> expect. I also see the latency goes up more with 2 processes 
> per node. 
> So a 4 process Barrier call takes about 58 microsecs on 4 
> nodes and 68 
> microsecs on 2 nodes. I checked with a single node and two 
> processes and 
> that was very fast (over 400MBytes/sec) so perhaps the issue is the 
> interaction of shared memory and TCP. I compiled ch3:ssm and 
> ch3:nemesis 
> with the same result. Also with and without --enable-fast. 
> This also did 
> little.
> 
> Finally I notice the cpu utilization is 100%; can this be part of the 
> problem?
> 
> I apologize if this has been gone over before, but I am new to MPICH2.
> 
> Thanks
> 
> Tony
> 
> -- 
> Tony Ladd
> 
> Chemical Engineering Department
> University of Florida
> Gainesville, Florida 32611-6005
> USA
> 
> Email: tladd-"(AT)"-che.ufl.edu
> WebL   http://ladd.che.ufl.edu
> 
> Tel:   (352)-392-6509
> FAX:   (352)-392-9514
> 
>