[MPICH] MPICH2 1.05 MPI_Send & MPI_Recv dropping packages randomly

[chong tan]

all the messages are short messages, the shortest being 3 integer (32 bits), the longest 9 integers.

I can't send you the code per company policy.  There are about 3 million lines of C, C++ and Tcl.  MPI is used in an isolated part of the code.

I will try sock, sock runs almost 11X slower on this prtiticular machine.  On 2.1.04p1, overhead by ssm was 50 sec, and sock's overhead was 520 sec on the failed test.

tan



----- Original Message ----
From: William Gropp <gropp at mcs.anl.gov>
To: chong tan <chong_guan_tan at yahoo.com>
Cc: mpich-discuss at mcs.anl.gov
Sent: Wednesday, January 17, 2007 6:37:19 PM
Subject: Re: [MPICH] MPICH2 1.05 MPI_Send & MPI_Recv dropping packages randomly

Can you send us the test case?  Does it fail with the ch3:sock device?  Are the messages short or long?  


Bill


On Jan 17, 2007, at 7:06 PM, chong tan wrote:


OS : RedHat Enterprise 4, 2.6.9-42.ELsmp
CPU   4 dual core Intel
 
the package was built with :
setenv CFLAGS "-m32 -O2"
setenv CC         gcc
./configure -prefix=/u/cgtan/my_release_dir --with-device=ch3:ssm --enable-fast |& tee configure.log

-----
the test programs run 5 processes, one master and 4 slaves.  Master always recv from slaves and them send to all of them.  Randomly, an MPI_Send performed in the master will complete, but the corresponidng MPI_Recv in the targeted slave would not complete, and the who thing hangs. 
 
I have a debugging mechanism that attachs a sequence id to all packages sent.  The packages are dumped before and after sent, and recv.  a message is also dumped on the the pending recv.  The sequence id traced OK all the way to the lost package.
 
The same code work fine with 2.1.04p1.  it has been tested on test cases longer than 100 million send/recv sequences.  any suggestions ?
 
tan
 



Bored stiff? Loosen up...
Download and play hundreds of games for free on Yahoo! Games.


 
____________________________________________________________________________________
8:00? 8:25? 8:40? Find a flick in no time 
with the Yahoo! Search movie showtime shortcut.
http://tools.search.yahoo.com/shortcuts/#news
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/mpich-discuss/attachments/20070118/53b3d6a5/attachment.htm>