<html><head><style type="text/css"><!-- DIV {margin:0px;} --></style></head><body><div style="font-family:times new roman, new york, times, serif;font-size:12pt"><DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif">I don't have a simple program to show the behavior. I will see what I can do</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif">for that. </DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"> </DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif">My tracing shows that one of the slaves has completed MPI_Send, while</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif">the master who is waiting for the package never sees it. The sync is like this:</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"> </DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"> master slave</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif">------------------------------------------</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"> loop { loop { </DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"> for all slaves</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"> MPI_recv MPI_Send to master</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"> compute status </DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"> create package+status</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"> for all slaves</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"> MPI_send MPI_Recv from master</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"> if status== end if status == END</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"> break break </DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"> } }<BR></DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif">the system will be stucked if any package is lost.</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"> </DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif">MPI_Send and MPI_Recv are the only 2 MPI functions called within the loop, which is</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif">deeply nested in another loop. When the lost happens, the code is usually calling</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif">more than 50,000 MPI_Send/MPI_Recv per second.</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"> </DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif">Hope this info help some.</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"> </DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif">thanks</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif">tan</DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"><BR> </DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif">----- Original Message ----<BR>From: Darius Buntinas <buntinas@mcs.anl.gov><BR>To: chong tan <chong_guan_tan@yahoo.com><BR>Cc: mpich-discuss@mcs.anl.gov<BR>Sent: Thursday, April 12, 2007 7:58:17 AM<BR>Subject: Re: [MPICH] nemesis in a threaded code<BR><BR>
<DIV>Can you send us a simple sample program that shows this behavior? If <BR>only one thread is making calls to MPI functions, I can't see that other <BR>threads should be interfering with this. I'd like to get a better idea <BR>of what's going on, so any more info you can give me would help.<BR><BR>Thanks,<BR>Darius<BR><BR>chong tan wrote:<BR>> randomly, a package sent by MPI_Send disappear, the reciever ended up<BR>> waiting for a package that never show up. This does not happen when the<BR>> apllication is not threaded.<BR>> <BR>> I have completed the test using --enable-threads=funnel. The main thread is<BR>> the only one calling MPI. (actually, only the master of all MPI <BR>> processes are<BR>> multi-threaded). It works much better, the horror is, it drops packages <BR>> once<BR>> in many runs. So, nemesis is pretty much useless for any threaded
<BR>> application.<BR>> <BR>> BTW, the package dropping usually happen after more than 30,000,000 calls<BR>> to MPI_Send and MPI_Recv.<BR>> <BR>> Any suggestion to debug this ?<BR>> <BR>> thanks<BR>> tan<BR>> <BR>> <BR>> <BR>> ----- Original Message ----<BR>> From: Darius Buntinas <buntinas@mcs.anl.gov><BR>> To: chong tan <chong_guan_tan@yahoo.com><BR>> Cc: mpich-discuss@mcs.anl.gov<BR>> Sent: Wednesday, April 11, 2007 2:22:39 PM<BR>> Subject: Re: [MPICH] nemesis in a threaded code<BR>> <BR>> Hmm, what do you mean by "dropping packages"? What symptoms are you seeing?<BR>> <BR>> Darius<BR>> <BR>> chong tan wrote:<BR>> > that is what I thought too. But without using --enable-threads, <BR>> nemesis is<BR>> > dropping packages randomly in a threaded env. I will rebuild
with<BR>> > --enable-threads=funnel to see what happen.<BR>> > <BR>> > tan<BR>> ><BR>> ><BR>> > <BR>> > ----- Original Message ----<BR>> > From: Darius Buntinas <buntinas@mcs.anl.gov><BR>> > To: chong tan <chong_guan_tan@yahoo.com><BR>> > Cc: mpich-discuss@mcs.anl.gov<BR>> > Sent: Wednesday, April 11, 2007 1:07:04 PM<BR>> > Subject: Re: [MPICH] nemesis in a threaded code<BR>> ><BR>> > Nemesis only supports thread_single. However, as long as you ensure<BR>> > that only one thread is making MPI calls, it should work fine.<BR>> ><BR>> > Darius<BR>> ><BR>> > chong tan wrote:<BR>> > >
with MPI calls done by the main thread, nemesis is dropping <BR>> packages. <BR>> > > Should I build MPICH<BR>> > > using --enable-threads=funneled ?<BR>> > > <BR>> > > thanks<BR>> > > tan<BR>> > > <BR>> > ><BR>> > > <BR>> ------------------------------------------------------------------------<BR>> > > 8:00? 8:25? 8:40? Find a flick <<BR>> > > <A href="http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news" target=_blank>http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news</A> <BR>> <<A href="http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news"
target=_blank>http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news</A>><BR>> > <<A href="http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news" target=_blank>http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news</A> <BR>> <<A href="http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news" target=_blank>http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news</A>>>> in no time<BR>> > > with theYahoo! Search movie showtime shortcut. <<BR>> > > <A href="http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news" target=_blank>http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news</A> <BR>> <<A href="http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news" target=_blank>http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news</A>><BR>> > <<A
href="http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news" target=_blank>http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news</A> <BR>> <<A href="http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news" target=_blank>http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news</A>>>><BR>> ><BR>> ><BR>> > ------------------------------------------------------------------------<BR>> > TV dinner still cooling?<BR>> > Check out "Tonight's Picks"<BR>> > <<A href="http://us.rd.yahoo.com/evt=49979/*http://tv.yahoo.com/" target=_blank>http://us.rd.yahoo.com/evt=49979/*http://tv.yahoo.com/</A>> on Yahoo! TV.<BR>> <BR>> <BR>> ------------------------------------------------------------------------<BR>> We won't tell. Get more on shows you hate to love <BR>> <<A
href="http://us.rd.yahoo.com/evt=49980/*http://tv.yahoo.com/collections/265" target=_blank>http://us.rd.yahoo.com/evt=49980/*http://tv.yahoo.com/collections/265</A> ><BR>> (and love to hate): Yahoo! TV's Guilty Pleasures list. <BR>> <<A href="http://us.rd.yahoo.com/evt=49980/*http://tv.yahoo.com/collections/265" target=_blank>http://us.rd.yahoo.com/evt=49980/*http://tv.yahoo.com/collections/265</A> ></DIV></DIV>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif"><BR></DIV></div><br>
<hr size=1>Don't be flakey. <a href="http://us.rd.yahoo.com/evt=43909/*http://mobile.yahoo.com/mail">Get Yahoo! Mail for Mobile</a> and <br><a href="http://us.rd.yahoo.com/evt=43909/*http://mobile.yahoo.com/mail">always stay connected</a> to friends.</body></html>