Mihael,<div><br></div><div>That is likely. The walltime is 20 mins and most jobs as far as I know are less than 10 mins. However, there could be outliers. These are about 120k jobs.<br><br></div><div>Ketan</div><div><br><div class="gmail_quote">
On Thu, Sep 1, 2011 at 1:43 PM, Mihael Hategan <span dir="ltr"><<a href="mailto:hategan@mcs.anl.gov">hategan@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Is there any chance that some of your jobs run longer than their<br>
requested walltime?<br>
<div><div></div><div class="h5"><br>
On Wed, 2011-08-31 at 09:04 -0500, Ketan Maheshwari wrote:<br>
> Mihael,<br>
><br>
><br>
> I did the run with the debug enabled on coasters. Please find the logs<br>
> etc, for this run here:<br>
><br>
><br>
> <a href="http://www.ci.uchicago.edu/~ketan/run25.tgz" target="_blank">http://www.ci.uchicago.edu/~ketan/run25.tgz</a><br>
><br>
><br>
><br>
><br>
> Note that the run went well and ran upto 20k jobs without issues.<br>
> After that I did not get nodes so I stopped it and resumed it this<br>
> morning. It ran for about 1000+ jobs and crashed with the same error<br>
> message.<br>
><br>
><br>
><br>
><br>
> Regards,<br>
> Ketan<br>
><br>
> On Tue, Aug 30, 2011 at 3:05 PM, Mihael Hategan <<a href="mailto:hategan@mcs.anl.gov">hategan@mcs.anl.gov</a>><br>
> wrote:<br>
> Any chance you can re-run this with debug enabled on coasters<br>
> (log4j.logger.org.globus.cog.abstraction.coaster=DEBUG)?<br>
><br>
><br>
> On Mon, 2011-08-29 at 20:55 -0700, Mihael Hategan wrote:<br>
> > My bad. The info is in the swift log.<br>
> ><br>
> > On Mon, 2011-08-29 at 20:59 -0500, Ketan Maheshwari wrote:<br>
> > > This is on Beagle. I am running local:pbs from /lustre.<br>
> > ><br>
> > > On Mon, Aug 29, 2011 at 8:30 PM, Mihael Hategan<br>
> <<a href="mailto:hategan@mcs.anl.gov">hategan@mcs.anl.gov</a>><br>
> > > wrote:<br>
> > > On Mon, 2011-08-29 at 19:52 -0500, Ketan<br>
> Maheshwari wrote:<br>
> > > > Mihael,<br>
> > > ><br>
> > > ><br>
> > > > This run was with automatic coasters. I do not<br>
> see any<br>
> > > specific<br>
> > > > coasters.log file written during this run<br>
> in .globus/coaster<br>
> > > nor in<br>
> > > > the run's work dir.<br>
> > ><br>
> > ><br>
> > > It's on the remote site in .globus/coasters.<br>
> > ><br>
> > > ><br>
> > > ><br>
> > > > Ketan<br>
> > > ><br>
> > > > On Mon, Aug 29, 2011 at 7:16 PM, Mihael Hategan<br>
> > > <<a href="mailto:hategan@mcs.anl.gov">hategan@mcs.anl.gov</a>><br>
> > > > wrote:<br>
> > > > Can I have the coasters log please?<br>
> > > ><br>
> > > ><br>
> > > > On Sun, 2011-08-28 at 16:47 -0500, Ketan<br>
> Maheshwari<br>
> > > wrote:<br>
> > > > > Hello,<br>
> > > > ><br>
> > > > ><br>
> > > > > I remember this error happened in the<br>
> past with<br>
> > > Glen's and<br>
> > > > Sheri's<br>
> > > > > runs. I saw this today again on Beagle<br>
> with 0.93<br>
> > > while<br>
> > > > running the<br>
> > > > > DSSAT run.<br>
> > > > ><br>
> > > > ><br>
> > > > > The run stops with the following<br>
> complete message:<br>
> > > > ><br>
> > > > ><br>
> > > > > queuedsize > 0 but no job dequeued.<br>
> Queued: {}<br>
> > > > > java.lang.Throwable<br>
> > > > > at<br>
> > > > ><br>
> > > ><br>
> > ><br>
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.requeueNonFitting(BlockQueueProcessor.java:269)<br>
> > > > > at<br>
> > > > ><br>
> > > ><br>
> > ><br>
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.updatePlan(BlockQueueProcessor.java:539)<br>
> > > > > at<br>
> > > > ><br>
> > > ><br>
> > ><br>
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.run(BlockQueueProcessor.java:110)<br>
> > > > > queuedsize > 0 but no job dequeued.<br>
> Queued: {}<br>
> > > > > java.lang.Throwable<br>
> > > > > at<br>
> > > > ><br>
> > > ><br>
> > ><br>
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.requeueNonFitting(BlockQueueProcessor.java:269)<br>
> > > > > at<br>
> > > > ><br>
> > > ><br>
> > ><br>
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.updatePlan(BlockQueueProcessor.java:539)<br>
> > > > > at<br>
> > > > ><br>
> > > ><br>
> > ><br>
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.run(BlockQueueProcessor.java:110)<br>
> > > > > Progress: time: Sun, 28 Aug 2011<br>
> 13:34:26 -0600<br>
> > > > Submitted:76<br>
> > > > > Active:23 Checking status:1<br>
> Finished<br>
> > > successfully:597<br>
> > > > ><br>
> > > > ><br>
> > > > ><br>
> > > > ><br>
> > > > > The logs, properties and sources for<br>
> this run are:<br>
> > > > ><br>
> <a href="http://www.ci.uchicago.edu/~ketan/run23.tgz" target="_blank">http://www.ci.uchicago.edu/~ketan/run23.tgz</a><br>
> > > > ><br>
> > > > ><br>
> > > > > Regards,<br>
> > > > > --<br>
> > > > > Ketan<br>
> > > > ><br>
> > > > ><br>
> > > > ><br>
> > > ><br>
> > > > ><br>
> _______________________________________________<br>
> > > > > Swift-devel mailing list<br>
> > > > > <a href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>
> > > > ><br>
> > > ><br>
> > ><br>
> <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > --<br>
> > > > Ketan<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > Ketan<br>
> > ><br>
> > ><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Swift-devel mailing list<br>
> > <a href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>
> ><br>
> <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> --<br>
> Ketan<br>
><br>
><br>
><br>
<br>
<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Ketan<br><br><br>
</div>