<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2658.34">
<TITLE>RE: [MPICH] Impact of changing recvTimeout</TITLE>
</HEAD>
<BODY>
<P><FONT SIZE=2>Interesting thought -- I will check this out. </FONT>
</P>
<P><FONT SIZE=2>Our job submission software chooses a master node for each submission, so it isn't always the same node; this behaviour is consistant regardless of the starting node. </FONT></P>
<P><FONT SIZE=2>Is there a number I should worry about? Some reasonable maximum? :)</FONT>
</P>
<P><FONT SIZE=2>I'm curious: how many cpus are other folks successfully using?</FONT>
</P>
<P><FONT SIZE=2>Simon</FONT>
</P>
<P><FONT SIZE=2> </FONT>
</P>
<P><FONT SIZE=2>-----Original Message-----</FONT>
<BR><FONT SIZE=2>From: owner-mpich-discuss@mcs.anl.gov [<A HREF="mailto:owner-mpich-discuss@mcs.anl.gov">mailto:owner-mpich-discuss@mcs.anl.gov</A>] On Behalf Of Yusong Wang</FONT>
<BR><FONT SIZE=2>Sent: May 23, 2006 11:10 AM</FONT>
<BR><FONT SIZE=2>To: Galton, Simon</FONT>
<BR><FONT SIZE=2>Cc: 'mpich-discuss@mcs.anl.gov'</FONT>
<BR><FONT SIZE=2>Subject: Re: [MPICH] Impact of changing recvTimeout</FONT>
</P>
<P><FONT SIZE=2>I suspect this is something related with socket operations. You could try the command netstat | grep tcp | wc -l on the master working node (not the head node you submitted the job) to see the number of ports used before, during and after execution of your application. If you run the application several times, I would expect the returned number to increase.</FONT></P>
<P><FONT SIZE=2>Yusong</FONT>
</P>
<P><FONT SIZE=2> </FONT>
<BR><FONT SIZE=2>On Tue, 2006-05-23 at 08:48 -0400, Galton, Simon wrote:</FONT>
<BR><FONT SIZE=2>> I finally found the "timeout" variable in mpdrun.py.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> It's called recvTimeout, and it's normally set to 20 (which turns out </FONT>
<BR><FONT SIZE=2>> to be 20 seconds).</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> I found that I had to set it to 45 to reliably run on 54 cpus, and 110 </FONT>
<BR><FONT SIZE=2>> to reliably run on 122 cpus.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Is there anything "wrong" with changing this variable? Is there any </FONT>
<BR><FONT SIZE=2>> other impact?</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Simon</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> -----Original Message-----</FONT>
<BR><FONT SIZE=2>> From: Galton, Simon</FONT>
<BR><FONT SIZE=2>> Sent: May 18, 2006 11:08 AM</FONT>
<BR><FONT SIZE=2>> To: mpich-discuss@mcs.anl.gov</FONT>
<BR><FONT SIZE=2>> Subject: Serious MPICH2 problem with many CPUs</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Folks,</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> We're running into a problem when dispatching to "many" CPUs. It </FONT>
<BR><FONT SIZE=2>> seems when we hit ~50 CPUs on our dual-Xeon GigE-connected cluster we </FONT>
<BR><FONT SIZE=2>> start to get the following error, and the job fails:</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> mpdrun_12429 (mpd_recv 386): other error after recv </FONT>
<BR><FONT SIZE=2>> __main__.mpdrunInterrupted :SIGALRM:</FONT>
<BR><FONT SIZE=2>> mpdrun failed: no msg recvd from mpd when expecting ack of request </FONT>
<BR><FONT SIZE=2>> traceback: [('/usr/local/mpich2/bin/mpdrun.py', '256', 'mpdrun'), </FONT>
<BR><FONT SIZE=2>> ('/usr/local/mpich2/bin/mpdrun.py', '978', '?')]</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> It happens 100% of the time at 54 CPUs or more.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> It looks like a setup timeout. Can we increase/fix this? I'd rather </FONT>
<BR><FONT SIZE=2>> not recompile (validation issue)...</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Simon Galton</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> CONFIDENTIAL AND PRIVILEGED INFORMATION NOTICE</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> This e-mail, and any attachments, may contain information that is </FONT>
<BR><FONT SIZE=2>> confidential, subject to copyright, or exempt from disclosure.</FONT>
<BR><FONT SIZE=2>> Any unauthorized review, disclosure, retransmission, dissemination or </FONT>
<BR><FONT SIZE=2>> other use of or reliance on this information may be unlawful and is </FONT>
<BR><FONT SIZE=2>> strictly prohibited.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> AVIS D'INFORMATION CONFIDENTIELLE ET PRIVILGIE</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Le prsent courriel, et toute pice jointe, peut contenir de </FONT>
<BR><FONT SIZE=2>> l'information qui est confidentielle, rgie par les droits d'auteur, ou </FONT>
<BR><FONT SIZE=2>> interdite de divulgation. Tout examen, divulgation, retransmission, </FONT>
<BR><FONT SIZE=2>> diffusion ou autres utilisations non autorises de l'information ou </FONT>
<BR><FONT SIZE=2>> dpendance non autorise envers celle-ci peut tre illgale et est </FONT>
<BR><FONT SIZE=2>> strictement interdite.</FONT>
</P>
<br><br><table bgcolor=white style="color:black"><tr><td><br>CONFIDENTIAL AND PRIVILEGED INFORMATION NOTICE<br>
<br>
This e-mail, and any attachments, may contain information that<br>
is confidential, subject to copyright, or exempt from disclosure.<br>
Any unauthorized review, disclosure, retransmission, <br>
dissemination or other use of or reliance on this information <br>
may be unlawful and is strictly prohibited. <br>
<br>
AVIS D'INFORMATION CONFIDENTIELLE ET PRIVILÉGIÉE<br>
<br>
Le présent courriel, et toute pièce jointe, peut contenir de <br>
l'information qui est confidentielle, régie par les droits <br>
d'auteur, ou interdite de divulgation. Tout examen, <br>
divulgation, retransmission, diffusion ou autres utilisations <br>
non autorisées de l'information ou dépendance non autorisée <br>
envers celle-ci peut être illégale et est strictement interdite.</td></tr></table></BODY>
</HTML>