<html><head><style type="text/css"><!-- DIV {margin:0px;} --></style></head><body><div style="font-family:times new roman, new york, times, serif;font-size:12pt"><DIV>pavan,</DIV>
<DIV>the behavior you described is the expected behavior. However, using your example, we are also seeing</DIV>
<DIV>a lot of system activity in process 1 in all of our experiments. That contributes significantly </DIV>
<DIV>to the negative gain.</DIV>
<DIV> </DIV>
<DIV>BTW, we did observed some gain in some tests, the slave processes of those tests were all </DIV>
<DIV>significanly light weighted compared to the master. </DIV>
<DIV> </DIV>
<DIV>Another data we saw, comparing non-thread MPI_Send/MPI_Recv against threaded MPI_Send/MPI_IRecv/Wait,</DIV>
<DIV>was that there is litle 'system activity' for the non-threaded case, while the threaded case was</DIV>
<DIV>overwhelmed with system activity. Are we talking about different ways of 'polling' to get the status between</DIV>
<DIV>Recv and Wait ? Or even Irecv is doing its share of polling ?</DIV>
<DIV> </DIV>
<DIV>tan</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV><BR> </DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"><BR>
<DIV style="FONT-FAMILY: arial, helvetica, sans-serif; FONT-SIZE: 13px"><FONT size=2 face=Tahoma>
<HR SIZE=1>
<B><SPAN style="FONT-WEIGHT: bold">From:</SPAN></B> Pavan Balaji <balaji@mcs.anl.gov><BR><B><SPAN style="FONT-WEIGHT: bold">To:</SPAN></B> mpich-discuss@mcs.anl.gov<BR><B><SPAN style="FONT-WEIGHT: bold">Sent:</SPAN></B> Wednesday, July 29, 2009 11:57:39 PM<BR><B><SPAN style="FONT-WEIGHT: bold">Subject:</SPAN></B> Re: [mpich-discuss] thread MPI calls<BR></FONT><BR><BR>If a process is idle for a long time, the Wait call with keep calling poll() to check if anything has arrived. For example, try this experiment:<BR><BR>Process 0:<BR> MPI_Irecv();<BR> MPI_Wait();<BR><BR>Process 1:<BR> sleep(10);<BR> MPI_Send();<BR><BR>Process 0 will see a lot of "system activity" as it's busily waiting for data to arrive.<BR><BR>-- Pavan<BR><BR>On 07/29/2009 03:19 PM, chong tan wrote:<BR>> I like to provide futher info on what we have experiment, hopefully this<BR>> can of some use, even with
my future competitor (they suscribe this email too).<BR>> 1. We replace Wait_all for Wait_any, and there is no different in performance<BR>> 2. We experimented with affining the recv thread to the same physcial CPU-same core<BR>> pair, different core pair, and same core as main thread (core pair done on AMD boxes). . This experiment is done on :<BR>> - Irecv versus Recv<BR>> - wait_all and wait_any<BR>> - early versus late wait_all, and wait_any, wait<BR>> - wait in recv thread versus wait in main thread<BR>> Amazigly, running the recv thread on the same core as main thread is the fastest one, it almost<BR>> has the same performance as non-threaded implementation with some code combo.<BR>>
Early wait, both all and any, is the worst performer.<BR>> Given that we know one partitcular test between more than 5% of time is spent in master<BR>> proc (id ==0) completing the already sent data via MPI_Recv (from MPE and our monitor, and<BR>> the fact that master proc got the biggest chunk of work in the test), we expect to see some positive<BR>> sign of live using threaded MPI. or at least not the negative gain as we have experience.<BR>> one issue we observed what that Wait*, is causing significant sys activities in other processes.<BR>> Maybe this is the problem, maybe not.<BR>> tan<BR>> ------------------------------------------------------------------------<BR>> *From:* Pavan Balaji <<A href="mailto:balaji@mcs.anl.gov" ymailto="mailto:balaji@mcs.anl.gov">balaji@mcs.anl.gov</A>><BR>> *To:* <A href="mailto:mpich-discuss@mcs.anl.gov"
ymailto="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</A><BR>> *Sent:* Tuesday, July 28, 2009 7:24:49 PM<BR>> *Subject:* Re: [mpich-discuss] thread MPI calls<BR>> <BR>> <BR>> > we just completed 1 partiticular tests using SERIALIZED, and that make no difference (compared to<BR>> > MULTIPLE).<BR>> <BR>> In that case, the problem is likely with the algorithm itself. In SERIALIZED mode, MPI does not add any locks and will not add any additional overhead. But it looks like your algorithm is blocking waiting for data from all slave processes before proceeding to the next iteration -- this will cause a lot of idle time. Is there someway you can optimize your algorithm?<BR>> <BR>> -- Pavan<BR>> <BR>> -- Pavan Balaji<BR>> http://www.mcs.anl.gov/~balaji<BR>> <BR><BR>-- Pavan Balaji<BR><A href="http://www.mcs.anl.gov/~balaji"
target=_blank>http://www.mcs.anl.gov/~balaji</A><BR></DIV></DIV></div><br>
</body></html>