I could reproduce the bug going from bridled to mcs with the same configuration. I am seeing 2 timeouts: one is the HEARTBEAT and other similar timeout messages and second is the register timeout message when trying to start a worker after about a gap of 5 minutes. This is a very similar scenario to OSG since the workers will only start after a delay (often long). The exact message is:<br>
<br>Failed to register (timeout)<br><br>So, Mihael, if you try the catsn example that I sent you from any machine to mcs workstations, you should be able to see the symptoms. Following are the config etc files that you could use:<br>
<br>====config======<br>wrapperlog.always.transfer=false<br>sitedir.keep=true<br>execution.retries=0<br>lazy.errors=false<br>status.mode=provider<br>use.provider.staging=true<br>provider.staging.pin.swiftfiles=false<br>foreach.max.threads=200<br>
==========<br><br>=====sites.xml=====<br> <config><br> <pool handle="grid"><br> <execution provider="coaster-persistent" url="<a href="http://localhost:50000">http://localhost:50000</a>" jobmanager="local:local"/><br>
<profile namespace="globus" key="workerManager">passive</profile><br> <profile namespace="globus" key="jobsPerNode">1</profile><br> <profile key="jobThrottle" namespace="karajan">0.02</profile><br>
<profile namespace="karajan" key="initialScore">10000</profile><br> <!-- <filesystem provider="local" url="none" /> --><br> <profile namespace="swift" key="stagingMethod">proxy</profile><br>
<profile namespace="globus" key="workerLoggingLevel">DEBUG</profile><br> <workdirectory>/tmp/ketan</workdirectory><br> </pool><br></config><br>==============<br>
<br>====tc======<br>grid cat /bin/cat null null null<br>======<br><br>The catsn example tarball is here: <a href="http://ci.uchicago.edu/~ketan/catsn-exp-formihael.tgz">http://ci.uchicago.edu/~ketan/catsn-exp-formihael.tgz</a><br>
<br><br>Regards,<br>Ketan<br><br><div class="gmail_quote">On Wed, Jan 25, 2012 at 1:15 PM, Mihael Hategan <span dir="ltr"><<a href="mailto:hategan@mcs.anl.gov">hategan@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Sorry. I was with the sshcl provider and the merging. I'll have to look<br>
at it this weekend.<br>
<div><div></div><div class="h5"><br>
On Wed, 2012-01-25 at 08:33 -0600, Michael Wilde wrote:<br>
> Mihael, Ketan, can you send an update on this, and escalate the priority of resolving this problem?<br>
><br>
> A resolution is needed rather urgently for the ExTENCI project.<br>
><br>
> Mihael, do you know where the problem lies, and have a strategy for a fix?<br>
><br>
> Thanks,<br>
><br>
> - Mike<br>
><br>
> ----- Original Message -----<br>
> > From: "Ketan Maheshwari" <<a href="mailto:ketancmaheshwari@gmail.com">ketancmaheshwari@gmail.com</a>><br>
> > To: "Mihael Hategan" <<a href="mailto:hategan@mcs.anl.gov">hategan@mcs.anl.gov</a>><br>
> > Cc: "Swift Devel" <<a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>><br>
> > Sent: Thursday, January 19, 2012 5:22:19 PM<br>
> > Subject: Re: [Swift-devel] timeout on OSG with coasters provider staging<br>
> > Here is another worker log this one is for a real SCEC run:<br>
> ><br>
> ><br>
> > <a href="http://ci.uchicago.edu/%7Eketan/timeout_worker_log_scec.txt" target="_blank">ci.uchicago.edu/~ketan/timeout_worker_log_scec.txt</a><br>
> ><br>
> ><br>
> > On Thu, Jan 19, 2012 at 1:54 PM, Ketan Maheshwari <<br>
> > <a href="mailto:ketancmaheshwari@gmail.com">ketancmaheshwari@gmail.com</a> > wrote:<br>
> ><br>
> ><br>
> > Mihael,<br>
> ><br>
> ><br>
> > I have the logs now. Filed as bug 690:<br>
> ><br>
> ><br>
> > <a href="https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=690" target="_blank">https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=690</a><br>
> ><br>
> > Regards,<br>
> > Ketan<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On Mon, Jan 16, 2012 at 2:24 PM, Ketan Maheshwari <<br>
> > <a href="mailto:ketancmaheshwari@gmail.com">ketancmaheshwari@gmail.com</a> > wrote:<br>
> ><br>
> ><br>
> > Mihael,<br>
> ><br>
> ><br>
> > Please find service log here:<br>
> > <a href="http://ci.uchicago.edu/%7Eketan/swift.log.tar.gz" target="_blank">http://ci.uchicago.edu/~ketan/swift.log.tar.gz</a><br>
> ><br>
> > worker logs seems to have lost. I'll see if I can find'em.<br>
> ><br>
> > Regards,<br>
> > Ketan<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On Mon, Jan 16, 2012 at 1:38 PM, Mihael Hategan < <a href="mailto:hategan@mcs.anl.gov">hategan@mcs.anl.gov</a><br>
> > > wrote:<br>
> ><br>
> ><br>
> > Nothing interesting there. Do you also happen to have the service and<br>
> > worker logs?<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On Mon, 2012-01-16 at 11:05 -0600, Ketan Maheshwari wrote:<br>
> > > Hi Mihael,<br>
> > ><br>
> > ><br>
> > > I could reproduce this timeout exception on OSG with catsn Swift<br>
> > > jobs.<br>
> > ><br>
> > ><br>
> > > These are 100 jobs with a data size of 10MB each. So, 2000MB of data<br>
> > > movement in all.<br>
> > ><br>
> > ><br>
> > > I tried with 1 worker running on a single OSG site. I tried three<br>
> > > different OSG sites: Nebraska, UChicago and RENCI.<br>
> > ><br>
> > ><br>
> > > In each of these cases, I run into the following timeout after ~4<br>
> > > minutes of run (15-70 jobs complete during this period) . :<br>
> > ><br>
> > ><br>
> > > Timeout<br>
> > > org.globus.cog.karajan.workflow.service.TimeoutException:<br>
> > > Handler(562,<br>
> > > PUT): timed out receiving request. Last time 940817-011255.807, now:<br>
> > > 120115-194100.072<br>
> > > at<br>
> > > org.globus.cog.karajan.workflow.service.handlers.RequestHandler.handleTimeout(RequestHandler.java:124)<br>
> > > at<br>
> > > org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.checkTimeouts(AbstractKarajanChannel.java:131)<br>
> > > at<br>
> > > org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.checkTimeouts(AbstractKarajanChannel.java:123)<br>
> > > at<br>
> > > org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel$1.run(AbstractKarajanChannel.java:116)<br>
> > > at java.util.TimerThread.mainLoop(Timer.java:512)<br>
> > > at java.util.TimerThread.run(Timer.java:462)<br>
> > > Command(168, SUBMITJOB): handling reply timeout;<br>
> > > sendReqTime=120115-193900.255, sendTime=120115-193900.255,<br>
> > > now=120115-194100.416, channel=SC-null<br>
> > ><br>
> > ><br>
> > > This is followed by messages similar to the above last line but the<br>
> > > progress of workflow halts.<br>
> > ><br>
> > ><br>
> > > Here is the tarball of the<br>
> > > experiment: <a href="http://ci.uchicago.edu/%7Eketan/catsn-exp-formihael.tgz" target="_blank">http://ci.uchicago.edu/~ketan/catsn-exp-formihael.tgz</a><br>
> > ><br>
> > ><br>
> > > It contains a README which has the steps to run: basically<br>
> > > start-service on localhost -> start worker on OSG site -> run swift<br>
> > ><br>
> > ><br>
> > > Regards,<br>
> > > --<br>
> > > Ketan<br>
> > ><br>
> > ><br>
> > ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Ketan<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Ketan<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Ketan<br>
> ><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Swift-devel mailing list<br>
> > <a href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>
> > <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>
><br>
<br>
<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Ketan<br><br><br>