<html><head><style type="text/css"><!-- DIV {margin:0px;} --></style></head><body><div style="font-family:times new roman, new york, times, serif;font-size:12pt"><DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif">tony, You are good at lookig at the code. When I run into MPICH2 performance issues, I changed my</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif">algorithm to work around it. There is no MPI collective call in my code anymore.</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"> </DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif">tan</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"><BR><BR> </DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif">----- Original Message ----<BR>From: Tony Ladd <tladd@che.ufl.edu><BR>To: Rajeev Thakur <thakur@mcs.anl.gov><BR>Cc: mpich-discuss@mcs.anl.gov<BR>Sent: Thursday, December 27, 2007 10:37:35 AM<BR>Subject: Re: [MPICH] MPICH2 performance issue with dual core<BR><BR>Rajeev Thakur wrote:<BR>> The collectives in MPICH2 are not optimized for multicore, but it's at the<BR>> top of our list to do.<BR>><BR>> Rajeev <BR>><BR>><BR>> <BR>>> -----Original Message-----<BR>>> From: <A href="mailto:owner-mpich-discuss@mcs.anl.gov" ymailto="mailto:owner-mpich-discuss@mcs.anl.gov">owner-mpich-discuss@mcs.anl.gov</A> <BR>>> [mailto:<A href="mailto:owner-mpich-discuss@mcs.anl.gov" ymailto="mailto:owner-mpich-discuss@mcs.anl.gov">owner-mpich-discuss@mcs.anl.gov</A>] On Behalf Of Tony Ladd<BR>>> Sent: Wednesday, December 26, 2007
12:00 PM<BR>>> To: <A href="mailto:mpich-discuss@mcs.anl.gov" ymailto="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</A><BR>>> Subject: [MPICH] MPICH2 performance issue with dual core<BR>>><BR>>> I am using MPICH2 over Gigabit ethernet (Intel PRO 1000 + Extreme <BR>>> Networks x450a-s48t switches). For a single process per node <BR>>> MPICH2 is <BR>>> very fast; typical throughput on edge exchange is ~100MBytes/sec both <BR>>> ways. MPICH2 has more uniform throughput than LAM, is much <BR>>> faster than <BR>>> OpenMPI and almost as good throughput as MPIGAMMA (using 1MB TCP <BR>>> buffers). Latency is 24 microsecs with tuned NIC drivers. So far so <BR>>> (very) good.<BR>>><BR>>> Collective communications are excellent for 1 process as well, but <BR>>> terrible with 2 processes per node. For example, an AlltoAll with 16 <BR>>> processes has
average 1-way throughput of 56MBytes/sec when <BR>>> distributed <BR>>> over 16 nodes but only 6MBytes per sec when using 8 nodes and 2 <BR>>> processes per node. This is of course the reverse of what one would <BR>>> expect. I also see the latency goes up more with 2 processes <BR>>> per node. <BR>>> So a 4 process Barrier call takes about 58 microsecs on 4 <BR>>> nodes and 68 <BR>>> microsecs on 2 nodes. I checked with a single node and two <BR>>> processes and <BR>>> that was very fast (over 400MBytes/sec) so perhaps the issue is the <BR>>> interaction of shared memory and TCP. I compiled ch3:ssm and <BR>>> ch3:nemesis <BR>>> with the same result. Also with and without --enable-fast. <BR>>> This also did <BR>>> little.<BR>>><BR>>> Finally I notice the cpu utilization is 100%; can this be part of the <BR>>> problem?<BR>>><BR>>> I
apologize if this has been gone over before, but I am new to MPICH2.<BR>>><BR>>> Thanks<BR>>><BR>>> Tony<BR>>><BR>>> -- <BR>>> Tony Ladd<BR>>><BR>>> Chemical Engineering Department<BR>>> University of Florida<BR>>> Gainesville, Florida 32611-6005<BR>>> USA<BR>>><BR>>> Email: tladd-"(AT)"-che.ufl.edu<BR>>> WebL <A href="http://ladd.che.ufl.edu/" target=_blank>http://ladd.che.ufl.edu</A><BR>>><BR>>> Tel: (352)-392-6509<BR>>> FAX: (352)-392-9514<BR>>><BR>>><BR>>> <BR>><BR>> <BR>Rajeev<BR><BR>Is there a time frame for the release of optimized multi-core collectives?<BR><BR>Since there is going to be development work, could I also make a couple <BR>of observations about where the collectives in MPICH1 (and perhaps <BR>MPICH2) could be improved.<BR><BR>1) I think Rabenseifner's AllReduce can
be done in 1/2 the time by <BR>avoiding the copy to a single head node. For a message length N the <BR>reduce to multiple nodes takes order N (M messages of length N/M) and <BR>the copy to the head node is also N (M * N/M). Similarly for the B'cast. <BR>If you use all the nodes as sources for the B'cast you can avoid the <BR>copies to and from the head node. So then the time is 2N rather than 4N. <BR>My hand-coded ALlReduce is typically about twice as fast at <BR>MPICH1/MPIGAMMA. Of course you need a special AllReduce function not a <BR>combination of Reduce + Bcast but would not the extra speed be worth it <BR>in such an important collective?<BR><BR>2) In the Alltoall I noticed that MPICH1 posts all the receives and then <BR>uses non-blocking sends + Msg_Wait. This can lead to oversubscription <BR>and packet loss in my experience. Last year, when working with Guiseppe <BR>Ciaccio to test MPIGAMMA on our cluster I found truly horrible <BR>performance on
Alltoall for large numbers of processes (hours instead <BR>of seconds for M ~ 100). The problem was exacerbated by GAMMA's <BR>rudimentary flow control which does not expect oversubscription. However <BR>in my opinion a scalable collective algorithm should not oversubscribe <BR>if at all possible. In the case of Alltoall an additional loop enables <BR>an arbitrary number of receives to be posted at once. I found about 4 <BR>was optimum for MPIGAMMA.<BR><BR>One other thing. Is is possible to tune the eager-rendezvous transition. <BR>Its good for pairwise exchanges, but for rings I think something a bit <BR>larger would be best for our set up.<BR><BR>I really like what I have seen of MPICH2 so far (apart from the <BR>multicore collectives). I think it will easily be the best MPI for TCP.<BR><BR>Tony<BR><BR>-- <BR>Tony Ladd<BR><BR>Chemical Engineering Department<BR>University of Florida<BR>Gainesville, Florida 32611-6005<BR>USA<BR><BR>Email:
tladd-"(AT)"-che.ufl.edu<BR>Web <A href="http://ladd.che.ufl.edu/" target=_blank>http://ladd.che.ufl.edu</A><BR><BR>Tel: (352)-392-6509<BR>FAX: (352)-392-9514<BR><BR></DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"><BR></DIV></div><br>
<hr size=1>Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. <a href="http://us.rd.yahoo.com/evt=51733/*http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ "> Try it now.</a></body></html>