Hi Mike,<div><br></div><div>I can do passwordless-ssh to beagle and uc3 from midway. </div><div><br></div><div>I am attaching a log from the failures I'm seeing, and David just confirmed that he is also</div><div>seeing the same.</div>
<div><br></div><div>-Yadu<br><br><div class="gmail_quote">On Thu, Mar 21, 2013 at 1:35 AM, Michael Wilde <span dir="ltr"><<a href="mailto:wilde@mcs.anl.gov" target="_blank">wilde@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Yadu, I dont think the beagle test requires you to have any directories on beagle.<br>
<br>
But it does expect that you have set up your ssh keys so that from the midway login host you can do a password-less ssh to beagle. Test that manually before trying test.beagle.<br>
<div class="im HOEnZb"><br>
- Mike<br>
<br>
----- Original Message -----<br>
</div><div class="im HOEnZb">> From: "Yadu Nand" <<a href="mailto:yadudoc1729@gmail.com">yadudoc1729@gmail.com</a>><br>
> To: "Michael Wilde" <<a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a>><br>
</div><div class="HOEnZb"><div class="h5">> Cc: "David Kelly" <<a href="mailto:davidk@ci.uchicago.edu">davidk@ci.uchicago.edu</a>>, "swift-devel" <<a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>><br>
> Sent: Wednesday, March 20, 2013 2:46:03 PM<br>
> Subject: Re: [Swift-devel] MODIS freezes on Midway<br>
><br>
><br>
><br>
> Quick update.<br>
><br>
> The sites.xml file for modis.midway was messed up while copying. Now<br>
> the tests for local and midway are running fine from the test suite.<br>
><br>
> I'm seeing the test.beagle fail because there isn't a folder in my<br>
> name on lustre... It fails with<br>
> Could not submit job<br>
> Could not start coaster service<br>
> Task ended before registration was received. Failed to download<br>
> bootstrap jar.<br>
><br>
> On test.uc3, I can see jobs getting submitted and completed, but it<br>
> looks like an exception is thrown in perl, halting the test.<br>
><br>
><br>
> Sorry about the formatting, I'm mailing from my phone.<br>
><br>
> -yadu<br>
> On Mar 21, 2013 12:43 AM, "Michael Wilde" < <a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a> ><br>
> wrote:<br>
><br>
><br>
> Also the code is now in svn:<br>
><br>
> <a href="https://svn.ci.uchicago.edu/svn/vdl2/SwiftTutorials/OSG_2013-03-11/MODIS" target="_blank">https://svn.ci.uchicago.edu/svn/vdl2/SwiftTutorials/OSG_2013-03-11/MODIS</a><br>
><br>
> - Mike<br>
><br>
> ----- Original Message -----<br>
> > From: "David Kelly" < <a href="mailto:davidk@ci.uchicago.edu">davidk@ci.uchicago.edu</a> ><br>
> > To: "Yadu Nand" < <a href="mailto:yadudoc1729@gmail.com">yadudoc1729@gmail.com</a> ><br>
> > Cc: "swift-devel" < <a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a> >, "Michael Wilde"<br>
> > < <a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a> ><br>
> > Sent: Wednesday, March 20, 2013 2:05:56 PM<br>
> > Subject: Re: [Swift-devel] MODIS freezes on Midway<br>
> ><br>
> ><br>
> > Yadu,<br>
> ><br>
> ><br>
> > As a test, I just untarred swiftdemo.v04.tgz, removed the<br>
> > SBATCH_RESERVATION line from setup.sh and was able to run on<br>
> > midway.<br>
> > Send me a message on Skype when you have a few minutes and we can<br>
> > take a closer look at this.<br>
> ><br>
> > David<br>
> ><br>
> ><br>
> > ----- Original Message -----<br>
> ><br>
> ><br>
> > From: "Yadu Nand" < <a href="mailto:yadudoc1729@gmail.com">yadudoc1729@gmail.com</a> ><br>
> > To: "Michael Wilde" < <a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a> ><br>
> > Cc: "swift-devel" < <a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a> ><br>
> > Sent: Wednesday, March 20, 2013 12:19:01 PM<br>
> > Subject: Re: [Swift-devel] MODIS freezes on Midway<br>
> ><br>
> > Hi everyone,<br>
> ><br>
> ><br>
> > David, please find the submit files attached to the mail.<br>
> > I am running the 5 different variants of modis,<br>
> > (local,midway,beagle,uc3, multiple)<br>
> > from the test system we have. I am not setting the<br>
> > SBATCH_RESERVATION<br>
> > variable<br>
> > in the setup scripts.<br>
> ><br>
> ><br>
> > Ketan, for modis_local and modis_midway, I am setting <filesystem<br>
> > provider="local"/><br>
> > and interestingly, modis_local works fine from the stress test apps<br>
> > group now, the rest<br>
> > fail though. I think there is a different issue here now.<br>
> ><br>
> ><br>
> > It looks like most failures I'm seeing now is from perl and tc.data<br>
> > issues. I've attached the<br>
> > modis.stdout from the 5 testcases, if you'd like to take a look.<br>
> > The<br>
> > tc.data supplied is the<br>
> > same as the ones that came with swiftdemo.<br>
> ><br>
> ><br>
> > -Yadu<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On Wed, Mar 20, 2013 at 5:26 AM, Michael Wilde < <a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a><br>
> > ><br>
> > wrote:<br>
> ><br>
> ><br>
> > Likely not needed if its using provider staging.<br>
> ><br>
> ><br>
> > ----- Original Message -----<br>
> > > From: "Ketan Maheshwari" < <a href="mailto:ketancmaheshwari@gmail.com">ketancmaheshwari@gmail.com</a> ><br>
> > > To: "David Kelly" < <a href="mailto:davidk@ci.uchicago.edu">davidk@ci.uchicago.edu</a> ><br>
> > > Cc: "swift-devel" < <a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a> ><br>
> > > Sent: Tuesday, March 19, 2013 6:04:44 PM<br>
> > > Subject: Re: [Swift-devel] MODIS freezes on Midway<br>
> > ><br>
> > ><br>
> > ><br>
> > > In addition to what David mentioned, from logs, it seems that<br>
> > > your<br>
> > > sites file is missing this line:<br>
> > ><br>
> > ><br>
> > > <filesystem provider="local"/><br>
> > ><br>
> > ><br>
> > ><br>
> > > Or David, correct me if this is not required in this<br>
> > > configuration.<br>
> > ><br>
> > ><br>
> > ><br>
> > > On Tue, Mar 19, 2013 at 6:00 PM, David Kelly <<br>
> > > <a href="mailto:davidk@ci.uchicago.edu">davidk@ci.uchicago.edu</a><br>
> > > > wrote:<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > Yadu,<br>
> > ><br>
> > ><br>
> > > The setup.sh script sets this environment variable:<br>
> > ><br>
> > ><br>
> > > export SBATCH_RESERVATION=osg<br>
> > ><br>
> > ><br>
> > > I believe sbatch is picking up on this and trying to run with<br>
> > > this<br>
> > > reservation, which is likely expired. Can you try unsetting<br>
> > > SBATCH_RESERVATION, commenting out that line in setup.sh and<br>
> > > trying<br>
> > > again?<br>
> > ><br>
> > ><br>
> > > Thanks,<br>
> > > David<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> ><br>
> ><br>
> > > From: "Yadu Nand" < <a href="mailto:yadudoc1729@gmail.com">yadudoc1729@gmail.com</a> ><br>
> > > To: "swift-devel" < <a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a> ><br>
> > > Sent: Tuesday, March 19, 2013 5:35:41 PM<br>
> > > Subject: [Swift-devel] MODIS freezes on Midway<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > Hi,<br>
> > ><br>
> > > I've been running the modis tests on Midway, from the demo that<br>
> > > mike<br>
> > > had shared:<br>
> > ><br>
> > > /home/wilde/osgdemo/modis/svn/swiftdemo.v04.tgz<br>
> > ><br>
> > ><br>
> > > In the logs(please see attachment) I see a fail message : "<br>
> > > sbatch:<br>
> > > error: Batch job<br>
> > > submission failed: Requested reservation is i nvalid"<br>
> > ><br>
> > ><br>
> > > The fact that no error messages are shown on stdout doesn't help,<br>
> > > plus, swift just<br>
> > > seems to hang forever. * Please help! *<br>
> > ><br>
> > ><br>
> > > I see test.midway show no progress, with just the same status for<br>
> > > about 20mins:<br>
> > ><br>
> > > Progress: time: Tue, 19 Mar 2013 20:21:18 +0000 Selecting site:35<br>
> > > Submitted:65<br>
> > ><br>
> > ><br>
> > > After this, I tried to kill by Ctrl+C and then I get a few error<br>
> > > messages :<br>
> > ><br>
> > > Failed to shut down block: Block 0319-5807480-000000<br>
> > > (16x3540.000s)<br>
> > > org.globus.cog.abstraction.impl.common.task.TaskSubmissionException:<br>
> > > Can only cancel an active task<br>
> > > at<br>
> > > org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.cancel(AbstractExecutor.java:196)<br>
> > > at<br>
> > > org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.cancel(AbstractJobSubmissionTaskHandler.java:85)<br>
> > > at<br>
> > > org.globus.cog.abstraction.impl.common.AbstractTaskHandler.cancel(AbstractTaskHandler.java:69)<br>
> > > at<br>
> > > org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:106)<br>
> > > at<br>
> > > org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:95)<br>
> > > at<br>
> > > org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.cancel(BlockTaskSubmitter.java:46)<br>
> > > at<br>
> > > org.globus.cog.abstraction.coaster.service.job.manager.Block.forceShutdown(Block.java:332)<br>
> > > at<br>
> > > org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown(Block.java:312)<br>
> > > at<br>
> > > org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdownBlocks(BlockQueueProcessor.java:800)<br>
> > > at<br>
> > > org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdown(BlockQueueProcessor.java:789)<br>
> > > at<br>
> > > org.globus.cog.abstraction.coaster.service.job.manager.JobQueue.shutdown(JobQueue.java:119)<br>
> > > at<br>
> > > org.globus.cog.abstraction.coaster.service.CoasterService.shutdown(CoasterService.java:271)<br>
> > > at<br>
> > > org.globus.cog.abstraction.coaster.service.ServiceShutdownHandler.requestComplete(ServiceShutdownHandler.java:28)<br>
> > > at<br>
> > > org.globus.cog.karajan.workflow.service.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)<br>
> > > at<br>
> > > org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.handleRequest(AbstractKarajanChannel.java:519)<br>
> > > at<br>
> > > org.globus.cog.karajan.workflow.service.channels.AbstractPipedChannel.actualSend(AbstractPipedChannel.java:86)<br>
> > > at<br>
> > > org.globus.cog.karajan.workflow.service.channels.AbstractPipedChannel$Sender.run(AbstractPipedChannel.java:115)<br>
> > > /home/wilde/osgdemo/modis/svn/swiftdemo.v04.tgz<br>
> > ><br>
> > > --<br>
> > > Yadu Nand B<br>
> > > _______________________________________________<br>
> > > Swift-devel mailing list<br>
> > > <a href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>
> > > <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>
> > ><br>
> > ><br>
> > > _______________________________________________<br>
> > > Swift-devel mailing list<br>
> > > <a href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>
> > > <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > Ketan<br>
> > ><br>
> > ><br>
> > > _______________________________________________<br>
> > > Swift-devel mailing list<br>
> > > <a href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>
> > > <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>
> > ><br>
> > _______________________________________________<br>
> > Swift-devel mailing list<br>
> > <a href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>
> > <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Yadu Nand B<br>
> > _______________________________________________<br>
> > Swift-devel mailing list<br>
> > <a href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>
> > <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>
> ><br>
> ><br>
><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Yadu Nand B
</div>