<html><head><style type="text/css"><!-- DIV {margin:0px;} --></style></head><body><div style="font-family:times new roman, new york, times, serif;font-size:12pt"><DIV>I like to provide futher info on what we have experiment, hopefully this</DIV>
<DIV>can of some use, even with my future competitor (they suscribe this email too).</DIV>
<DIV> </DIV>
<DIV>1. We replace Wait_all for Wait_any, and there is no different in performance</DIV>
<DIV>2. We experimented with affining the recv thread to the same physcial CPU-same core </DIV>
<DIV> pair, different core pair, and same core as main thread (core pair done on AMD boxes). . </DIV>
<DIV> This experiment is done on :</DIV>
<DIV> - Irecv versus Recv</DIV>
<DIV> - wait_all and wait_any</DIV>
<DIV> - early versus late wait_all, and wait_any, wait</DIV>
<DIV> - wait in recv thread versus wait in main thread</DIV>
<DIV> Amazigly, running the recv thread on the same core as main thread is the fastest one, it almost
<DIV> has the same performance as non-threaded implementation with some code combo.</DIV>
<DIV> </DIV>
<DIV> Early wait, both all and any, is the worst performer.</DIV>
<DIV> </DIV>
<DIV> Given that we know one partitcular test between more than 5% of time is spent in master</DIV>
<DIV> proc (id ==0) completing the already sent data via MPI_Recv (from MPE and our monitor, and</DIV>
<DIV> the fact that master proc got the biggest chunk of work in the test), we expect to see some positive </DIV>
<DIV> sign of live using threaded MPI. or at least not the negative gain as we have experience.</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>one issue we observed what that Wait*, is causing significant sys activities in other processes.<BR>Maybe this is the problem, maybe not.</DIV>
<DIV> </DIV>
<DIV>tan</DIV>
<DIV> </DIV></DIV>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">
<DIV style="FONT-FAMILY: arial, helvetica, sans-serif; FONT-SIZE: 13px"><FONT size=2 face=Tahoma>
<HR SIZE=1>
<B><SPAN style="FONT-WEIGHT: bold">From:</SPAN></B> Pavan Balaji <balaji@mcs.anl.gov><BR><B><SPAN style="FONT-WEIGHT: bold">To:</SPAN></B> mpich-discuss@mcs.anl.gov<BR><B><SPAN style="FONT-WEIGHT: bold">Sent:</SPAN></B> Tuesday, July 28, 2009 7:24:49 PM<BR><B><SPAN style="FONT-WEIGHT: bold">Subject:</SPAN></B> Re: [mpich-discuss] thread MPI calls<BR></FONT><BR><BR>> we just completed 1 partiticular tests using SERIALIZED, and that make no difference (compared to<BR>> MULTIPLE). <BR><BR>In that case, the problem is likely with the algorithm itself. In SERIALIZED mode, MPI does not add any locks and will not add any additional overhead. But it looks like your algorithm is blocking waiting for data from all slave processes before proceeding to the next iteration -- this will cause a lot of idle time. Is there someway you can optimize your algorithm?<BR><BR>-- Pavan<BR><BR>-- Pavan Balaji<BR>http://www.mcs.anl.gov/~balaji<BR></DIV></DIV></div><br>
</body></html>