[Swift-devel] coaster problem with jobmanager=gt2:pbs

[Mihael Hategan]

On Mon, 2009-04-06 at 16:25 -0500, Michael Wilde wrote:
> We just tested that rev, and now it seems as if the jobs are getting 
> submitted to the fork JM instead of to PBS.
> 
> Need a log for that, or is the cause obvious?

No. I'll debug and see.

> 
> 
> On 4/6/09 3:25 PM, Mihael Hategan wrote:
> > Oops. cog r2367 should fix that.
> > 
> > On Mon, 2009-04-06 at 15:17 -0500, Michael Wilde wrote:
> >> Mihael, I just updated our test swift+cog source and rebuilt.
> >>
> >> Glen is now getting:
> >>
> >> Caused by:
> >>          Invalid GSSCredentials
> >> org.globus.cog.abstraction.impl.common.task.InvalidSecurityContextException: 
> >> Invalid GSSCredentials
> >>          at 
> >> org.globus.cog.abstraction.impl.execution.gt2.JobSubmissionTaskHandler.submitSingleJob(JobSubmissionTaskHandler.java:149)
> >>          at 
> >> org.globus.cog.abstraction.impl.execution.gt2.JobSubmissionTaskHandler.submit(JobSubmissionTaskHandler.java:99)
> >>          at 
> >> org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:46)
> >>          at 
> >> org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:50)
> >>          at 
> >> org.globus.cog.abstraction.coaster.service.job.manager.WorkerManager.startWorker(WorkerManager.java:222)
> >>          at 
> >> org.globus.cog.abstraction.coaster.service.job.manager.WorkerManager.run(WorkerManager.java:145)
> >> Caused by: GSSException: Invalid name provided [Caused by: [JGLOBUS-112] 
> >> Malformed name, "=" missing in "38356/jobmanager-pbs"]
> >>          at 
> >> org.globus.gsi.gssapi.GlobusGSSName.<init>(GlobusGSSName.java:137)
> >>          at 
> >> org.globus.gsi.gssapi.GlobusGSSManagerImpl.createName(GlobusGSSManagerImpl.java:304)
> >>          at 
> >> org.globus.gsi.gssapi.auth.IdentityAuthorization.getExpectedName(IdentityAuthorization.java:82)
> >>          at org.globus.gram.Gram.gatekeeperConnect(Gram.java:85)
> >>          at org.globus.gram.Gram.request(Gram.java:310)
> >>          at org.globus.gram.GramJob.request(GramJob.java:262)
> >>          at 
> >> org.globus.cog.abstraction.impl.execution.gt2.JobSubmissionTaskHandler.submitSingleJob(JobSubmissionTaskHandler.java:133)
> >>          ... 5 more
> >> what's up here
> >>
> >>
> >> Any chance I picked up code in transition, or a new problem in recent 
> >> commits?
> >>
> >> - Mike
> >>
> >>
> >>
> >> On 4/6/09 1:02 PM, Mihael Hategan wrote:
> >>> Yes. This is one of those "can't find executable unless run through
> >>> 'bash -l' or maybe not" which we saw using wget and md5sum. I'm thinking
> >>> how to deal with the situation.
> >>>
> >>> On Mon, 2009-04-06 at 12:10 -0500, Michael Wilde wrote:
> >>>> With this sites entry:
> >>>>
> >>>> <pool handle="qb" >
> >>>>    <profile namespace="globus" key="project">TG-CDA070002T</profile>
> >>>>    <execution provider="coaster" url="queenbee.loni-lsu.teragrid.org"
> >>>>               jobManager="gt2:pbs" />
> >>>>    <gridftp url="gsiftp://qb1.loni.org"/>
> >>>>    <workdirectory>/home/ux454325/swiftwork</workdirectory>
> >>>> </pool>
> >>>>
> >>>> I get the error below. Files are on CI net at /home/wilde/swift/lab.
> >>>>
> >>>> I will try to copy coaster boot logs and gram logs to same place when I 
> >>>> find them, in subdirs named by $RunID.logs.
> >>>>
> >>>> --
> >>>>
> >>>> com$ swift -tc.file tc.data -sites.file qb.coasters.xml cat.swift
> >>>> Swift svn swift-r2809 cog-r2350
> >>>>
> >>>> RunID: 20090406-1155-pgc5nj00
> >>>> Progress:
> >>>> Progress:  Stage in:1
> >>>> Progress:  Submitted:1
> >>>> Failed to transfer wrapper log from cat-20090406-1155-pgc5nj00/info/m on qb
> >>>> Progress:  Failed:1
> >>>> Execution failed:
> >>>>          Exception in cat:
> >>>> Arguments: [data.txt]
> >>>> Host: qb
> >>>> Directory: cat-20090406-1155-pgc5nj00/jobs/m/cat-m91ku09j
> >>>> stderr.txt:
> >>>>
> >>>> stdout.txt:
> >>>>
> >>>> ----
> >>>>
> >>>> Caused by:
> >>>>          Cannot submit job: Cannot run program "qsub": 
> >>>> java.io.IOException: error=2, No such file or directory
> >>>> org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: 
> >>>> Cannot submit job: Cannot run program "qsub": java.io.IOException: 
> >>>> error=2, No such file or directory
> >>>>          at 
> >>>> org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:63)
> >>>>          at 
> >>>> org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:46)
> >>>>          at 
> >>>> org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:50)
> >>>>          at 
> >>>> org.globus.cog.abstraction.coaster.service.job.manager.WorkerManager.startWorker(WorkerManager.java:221)
> >>>>          at 
> >>>> org.globus.cog.abstraction.coaster.service.job.manager.WorkerManager.run(WorkerManager.java:145)
> >>>> Caused by: java.io.IOException: Cannot run program "qsub": 
> >>>> java.io.IOException: error=2, No such file or directory
> >>>>          at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
> >>>>          at java.lang.Runtime.exec(Runtime.java:593)
> >>>>          at 
> >>>> org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.start(AbstractExecutor.java:73)
> >>>>          at 
> >>>> org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:53)
> >>>>          ... 4 more
> >>>> Caused by: java.io.IOException: java.io.IOException: error=2, No such 
> >>>> file or directory
> >>>>          at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> >>>>          at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> >>>>          at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
> >>>>          ... 7 more
> >>>>
> >>>> Cleaning up...
> >>>> Shutting down service at https://208.100.92.21:44166
> >>>> Got channel MetaChannel: 24235184 -> GSSSChannel-null(1)
> >>>> - Done
> >>>> com$ pwd
> >>>> _______________________________________________
> >>>> Swift-devel mailing list
> >>>> Swift-devel at ci.uchicago.edu
> >>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> >