<html><head><style type="text/css"><!-- DIV {margin:0px;} --></style></head><body><div style="font-family:times new roman, new york, times, serif;font-size:12pt"><DIV><BR></DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">sorry that i ovr cleaned my inbox, this is the only one I have left to continue with</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">this thread.</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> </DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">After cleaning up some over-looked situation and making sure that we are running with</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">MPI_Init_thread with THREAD_MULTIPLE, we rerun a few tests, we</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">are seeing up to 10% performance drop compared to un-thread code. We used affinity,</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">the main and the spawned thread are running on different core of the same physical CPU.</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> </DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">We use posix thread to create our thread, is that bad for MPI ? </DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> </DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">ta</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"><BR> </DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"><FONT size=2 face=Tahoma>
<HR SIZE=1>
<B><SPAN style="FONT-WEIGHT: bold">From:</SPAN></B> chong tan <chong_guan_tan@yahoo.com><BR><B><SPAN style="FONT-WEIGHT: bold">To:</SPAN></B> mpich-discuss@mcs.anl.gov<BR><B><SPAN style="FONT-WEIGHT: bold">Sent:</SPAN></B> Thursday, July 23, 2009 3:40:51 PM<BR><B><SPAN style="FONT-WEIGHT: bold">Subject:</SPAN></B> [mpich-discuss] thread MPI calls<BR></FONT><BR>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">
<DIV>box X86 box running Linux. MPICH2 1.1 configured with </DIV>
<DIV> </DIV>
<DIV> --disable-f77 --disable-f90 --with-device=ch3:nemesis --enable-fast</DIV>
<DIV> </DIV>
<DIV>machine : 16 cores, load of mem. running with mpiexec -n 4<BR></DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">The master process (MPICH rank 0) has 2 thread, main and recv, the locks are constructed using pthread. The master's</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">recv function wait for the recv thread to be done with all recieving, then process the data, and reenable the recv thread,</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">like this:</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> </DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> main recv thread</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> </DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> recvFunc() { wait for 'run' </DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> wait for read-done for n processes</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> apply data call MPI_IRecv</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> signal run MPI_waitall</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> } signal read-done</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> <BR>recvFunc is repeatedly called during the life of my applications. (In a few tests I have, it is called 100+ billion times)</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> </DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">compared to the same application with no thread and using IRecv or Recv, the threaded version can run 20+% </DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">slower. From the process monitoring, it looks like the MPI Irecv calls maybe thread. This performance degration</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">is inline with 2.0.6 configured using --enable-threads=funneled. The application ran for many hours, so 20% is</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">significant.</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> </DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">Questions :</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">- is MPICH2 1.1 self-sensing ? that is, does it know it can decide if it has to do thread-multiple ?</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">- IRecv and MPI wait all, is that also threaded ? </DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">- anyone experimented this before ?</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> </DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">thanks</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">tan</DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> </DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"><FONT size=2 face=Tahoma>
<HR SIZE=1>
<B><SPAN style="FONT-WEIGHT: bold">From:</SPAN></B> Rajeev Thakur <thakur@mcs.anl.gov><BR><B><SPAN style="FONT-WEIGHT: bold">To:</SPAN></B> mpich-discuss@mcs.anl.gov<BR><B><SPAN style="FONT-WEIGHT: bold">Sent:</SPAN></B> Friday, June 26, 2009 12:12:18 PM<BR><B><SPAN style="FONT-WEIGHT: bold">Subject:</SPAN></B> Re: [mpich-discuss] does 1.1 support real threading of MPI call ?<BR></FONT><BR>
<STYLE type=text/css>DIV {
MARGIN:0px;}
</STYLE>
<DIV dir=ltr align=left><SPAN class=087571119-26062009><FONT color=#0000ff size=2 face=Arial>Yes, it supports MPI_THREAD_MULTIPLE as defined by the MPI standard.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=087571119-26062009><FONT color=#0000ff size=2 face=Arial></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=087571119-26062009><FONT color=#0000ff size=2 face=Arial>Rajeev</FONT></SPAN></DIV><BR>
<BLOCKQUOTE style="BORDER-LEFT: #0000ff 2px solid; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; MARGIN-RIGHT: 0px">
<DIV dir=ltr lang=en-us class=OutlookMessageHeader align=left>
<HR tabIndex=-1>
<FONT size=2 face=Tahoma><B>From:</B> mpich-discuss-bounces@mcs.anl.gov [mailto:mpich-discuss-bounces@mcs.anl.gov] <B>On Behalf Of </B>chong tan<BR><B>Sent:</B> Friday, June 26, 2009 2:04 PM<BR><B>To:</B> mpich-discuss@mcs.anl.gov<BR><B>Subject:</B> [mpich-discuss] does 1.1 support real threading of MPI call ?<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">
<DIV>Does anyone know if the 1.1 release supports real threaded MPI calls. That is,</DIV>
<DIV>a process, say 1, may contain n threads, and each calling MPI_Send/Recv to</DIV>
<DIV>other processes, but still think they are all process 1 ?</DIV>
<DIV> </DIV>
<DIV>I am not looking at the funneled solution, that does not help me.</DIV>
<DIV> </DIV>
<DIV>thanks</DIV>
<DIV>tan</DIV>
<DIV><BR> </DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"><BR>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">
<P><FONT size=2></FONT></P></DIV></DIV></DIV><BR></BLOCKQUOTE></DIV></DIV><BR></DIV></div><br>
</body></html>