Thanks, I did consider that option too but some of the benchmarks I ran took roughly 1.5x the time with sock instead of shm. I was wondering if I could get the best of both (nice behavior of sock during waits vs. performance of shm) somehow, given that I need the slow but well-behaved barrier only in certain isolated places. I'm not highly familiar with the way the channels work, but it looked like it would be highly non-trivial if not impossible to interchange channels at runtime.
<br><br>Regards,<br>Sudarshan<br><br><div><span class="gmail_quote">On 26/02/07, <b class="gmail_sendername">Rajeev Thakur</b> <<a href="mailto:thakur@mcs.anl.gov">thakur@mcs.anl.gov</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2">You could directly use the ch3:sock channel instead of
ch3:shm. It would behave better in this case because it waits on a poll() as you
point out, but the rest of the communication would be slower with ch3:sock than
ch3:shm.</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2"></font></span> </div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2">Rajeev</font></span></div><br>
<blockquote style="border-left: 2px solid rgb(0, 0, 255); padding-left: 5px; margin-left: 5px; margin-right: 0px;">
<div dir="ltr" align="left" lang="en-us">
<hr>
<font face="Tahoma" size="2"><b>From:</b> Sudarshan Raghunathan
[mailto:<a href="mailto:rdarshan@gmail.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">rdarshan@gmail.com</a>] <br><b>Sent:</b> Monday, February 26, 2007 5:58
PM<br><b>To:</b> Rajeev Thakur<br><b>Cc:</b>
<a href="mailto:mpich-discuss@mcs.anl.gov" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">mpich-discuss@mcs.anl.gov</a><br><b>Subject:</b> Re: [MPICH] MPI_Barrier on
ch3:shm and sched_yield<br></font><br></div><div><span class="e" id="q_1110087e6b3c9203_1">
<div></div>Rajeev,<br><br>Thank you for the reply.<br><br>I was running four
MPI processes and then started another computationally intensive task (running
another MPI program on 2 processes, for example). The other task gets around
50% of the CPU, but seems to be constantly competing with the first MPI
program which isn't really doing any heavy lifting except waiting on the
barrier. <br><br>I looked into a few options, all of which seem to have
problems:<br>(a) put a sleep in the MPIDI_CH3I_Progress code so that the
processes waiting on the barrier sleep for a while instead of spinning in a
sched_yield loop and compete with other processes which might actually need to
use the CPU. The problem with this approach is that it might slow down
portions of the application where a fast barrier _is_ required <br>(b) use
semaphores. It looks like it might touch a lot of places in the implementation
and again might cause performance problems where a fast barrier is
needed<br>(c) use sockets. The sock device in MPICH seems to be calling poll
with a time-out and that seems to be behaving much better than the shm device.
I'm not sure how much of the existing MPICH code can be reused. <br><br>Any
other saner/more feasible options or advice on which of the above three
options is simplest to implement would be most appreciated.<br><br>Thanks
again,<br>Sudarshan<br><br>
<div><span class="gmail_quote">On 26/02/07, <b class="gmail_sendername">Rajeev
Thakur</b> <<a href="mailto:thakur@mcs.anl.gov" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">thakur@mcs.anl.gov </a>>
wrote:</span>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2">Sudarshan,</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2">
How many MPI processes are you running on the 4-way SMP and how many
processes of the other application? Does the other application get any CPU
at all? sched_yield just gives up the current time slice for the process and
moves it to the end of the run queue. It will get run again when its turn
comes. In other words, the process doesn't sleep until some event happens.
It shares the CPU with other processes. </font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2"></font></span> </div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2">Rajeev</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Arial" size="2"></font></span><br></div></div></blockquote></div><br></span></div></blockquote></div>
</blockquote></div><br><br clear="all"><br>-- <br>When the impossible has been eliminated, whatever remains, however improbable must be the Truth