[MPICH] nemesis in a threaded code

[chong tan]

I don't have a simple program  to show the behavior.   I will see what I can do
for that.  

My tracing shows that one of the slaves has completed MPI_Send, while
the master who is waiting for the package never sees it.  The sync is like this:

                master                             slave
------------------------------------------
           loop {                                    loop {    
               for all slaves
                    MPI_recv                        MPI_Send to master
               compute status  
               create package+status
               for all slaves
                 MPI_send                           MPI_Recv from master
               if status== end                       if status == END
                   break                                     break                                                 
           }                                           }

the system will be stucked if any package is lost.

MPI_Send and MPI_Recv are the only 2 MPI functions called within the loop, which is
deeply nested in another loop.   When the lost happens, the code is usually calling
more than 50,000 MPI_Send/MPI_Recv per second.

Hope this info help some.

thanks
tan


----- Original Message ----
From: Darius Buntinas <buntinas at mcs.anl.gov>
To: chong tan <chong_guan_tan at yahoo.com>
Cc: mpich-discuss at mcs.anl.gov
Sent: Thursday, April 12, 2007 7:58:17 AM
Subject: Re: [MPICH] nemesis in a threaded code


Can you send us a simple sample program that shows this behavior?  If 
only one thread is making calls to MPI functions, I can't see that other 
threads should be interfering with this.  I'd like to get a better idea 
of what's going on, so any more info you can give me would help.

Thanks,
Darius

chong tan wrote:
> randomly, a package sent by MPI_Send disappear, the reciever ended up
> waiting for a package that never show up.  This does not happen when the
> apllication is not threaded.
>  
> I have completed the test using --enable-threads=funnel.  The main thread is
> the only one calling MPI.  (actually, only the master of all MPI 
> processes are
> multi-threaded).  It works much better, the horror is, it drops packages 
> once
> in many runs.  So, nemesis is pretty much useless for any threaded 
> application.
>  
> BTW, the package dropping usually happen after more than 30,000,000 calls
> to MPI_Send and MPI_Recv.
>  
> Any suggestion to debug this ?
>  
> thanks
> tan
> 
> 
>  
> ----- Original Message ----
> From: Darius Buntinas <buntinas at mcs.anl.gov>
> To: chong tan <chong_guan_tan at yahoo.com>
> Cc: mpich-discuss at mcs.anl.gov
> Sent: Wednesday, April 11, 2007 2:22:39 PM
> Subject: Re: [MPICH] nemesis in a threaded code
> 
> Hmm, what do you mean by "dropping packages"?  What symptoms are you seeing?
> 
> Darius
> 
> chong tan wrote:
>  > that is what I thought too.  But without using --enable-threads, 
> nemesis is
>  > dropping packages randomly in a threaded env.  I will rebuild with
>  > --enable-threads=funnel to see what happen.
>  >  
>  > tan
>  >
>  >
>  >  
>  > ----- Original Message ----
>  > From: Darius Buntinas <buntinas at mcs.anl.gov>
>  > To: chong tan <chong_guan_tan at yahoo.com>
>  > Cc: mpich-discuss at mcs.anl.gov
>  > Sent: Wednesday, April 11, 2007 1:07:04 PM
>  > Subject: Re: [MPICH] nemesis in a threaded code
>  >
>  > Nemesis only supports thread_single.  However, as long as you ensure
>  > that only one thread is making MPI calls, it should work fine.
>  >
>  > Darius
>  >
>  > chong tan wrote:
>  >  > with MPI calls done by the main thread, nemesis is dropping 
> packages.  
>  >  > Should I build MPICH
>  >  > using --enable-threads=funneled ?
>  >  >  
>  >  > thanks
>  >  > tan
>  >  >  
>  >  >
>  >  > 
> ------------------------------------------------------------------------
>  >  > 8:00? 8:25? 8:40? Find a flick <
>  >  > http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news 
> <http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news>
>  > <http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news 
> <http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news>>> in no time
>  >  > with theYahoo! Search movie showtime shortcut. <
>  >  > http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news 
> <http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news>
>  > <http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news 
> <http://tools.search.yahoo.com/shortcuts/?fr=oni_on_mail&#news>>>
>  >
>  >
>  > ------------------------------------------------------------------------
>  > TV dinner still cooling?
>  > Check out "Tonight's Picks"
>  > <http://us.rd.yahoo.com/evt=49979/*http://tv.yahoo.com/> on Yahoo! TV.
> 
> 
> ------------------------------------------------------------------------
> We won't tell. Get more on shows you hate to love 
> <http://us.rd.yahoo.com/evt=49980/*http://tv.yahoo.com/collections/265 >
> (and love to hate): Yahoo! TV's Guilty Pleasures list. 
> <http://us.rd.yahoo.com/evt=49980/*http://tv.yahoo.com/collections/265 >


       
____________________________________________________________________________________
It's here! Your new message!  
Get new email alerts with the free Yahoo! Toolbar.
http://tools.search.yahoo.com/toolbar/features/mail/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/mpich-discuss/attachments/20070412/0dbb3c5d/attachment.htm>