[Swift-devel] coaster problem with jobmanager=gt2:pbs

[Michael Wilde]

We just tested that rev, and now it seems as if the jobs are getting 
submitted to the fork JM instead of to PBS.

Need a log for that, or is the cause obvious?


On 4/6/09 3:25 PM, Mihael Hategan wrote:
> Oops. cog r2367 should fix that.
> 
> On Mon, 2009-04-06 at 15:17 -0500, Michael Wilde wrote:
>> Mihael, I just updated our test swift+cog source and rebuilt.
>>
>> Glen is now getting:
>>
>> Caused by:
>>          Invalid GSSCredentials
>> org.globus.cog.abstraction.impl.common.task.InvalidSecurityContextException: 
>> Invalid GSSCredentials
>>          at 
>> org.globus.cog.abstraction.impl.execution.gt2.JobSubmissionTaskHandler.submitSingleJob(JobSubmissionTaskHandler.java:149)
>>          at 
>> org.globus.cog.abstraction.impl.execution.gt2.JobSubmissionTaskHandler.submit(JobSubmissionTaskHandler.java:99)
>>          at 
>> org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:46)
>>          at 
>> org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:50)
>>          at 
>> org.globus.cog.abstraction.coaster.service.job.manager.WorkerManager.startWorker(WorkerManager.java:222)
>>          at 
>> org.globus.cog.abstraction.coaster.service.job.manager.WorkerManager.run(WorkerManager.java:145)
>> Caused by: GSSException: Invalid name provided [Caused by: [JGLOBUS-112] 
>> Malformed name, "=" missing in "38356/jobmanager-pbs"]
>>          at 
>> org.globus.gsi.gssapi.GlobusGSSName.<init>(GlobusGSSName.java:137)
>>          at 
>> org.globus.gsi.gssapi.GlobusGSSManagerImpl.createName(GlobusGSSManagerImpl.java:304)
>>          at 
>> org.globus.gsi.gssapi.auth.IdentityAuthorization.getExpectedName(IdentityAuthorization.java:82)
>>          at org.globus.gram.Gram.gatekeeperConnect(Gram.java:85)
>>          at org.globus.gram.Gram.request(Gram.java:310)
>>          at org.globus.gram.GramJob.request(GramJob.java:262)
>>          at 
>> org.globus.cog.abstraction.impl.execution.gt2.JobSubmissionTaskHandler.submitSingleJob(JobSubmissionTaskHandler.java:133)
>>          ... 5 more
>> what's up here
>>
>>
>> Any chance I picked up code in transition, or a new problem in recent 
>> commits?
>>
>> - Mike
>>
>>
>>
>> On 4/6/09 1:02 PM, Mihael Hategan wrote:
>>> Yes. This is one of those "can't find executable unless run through
>>> 'bash -l' or maybe not" which we saw using wget and md5sum. I'm thinking
>>> how to deal with the situation.
>>>
>>> On Mon, 2009-04-06 at 12:10 -0500, Michael Wilde wrote:
>>>> With this sites entry:
>>>>
>>>> <pool handle="qb" >
>>>>    <profile namespace="globus" key="project">TG-CDA070002T</profile>
>>>>    <execution provider="coaster" url="queenbee.loni-lsu.teragrid.org"
>>>>               jobManager="gt2:pbs" />
>>>>    <gridftp url="gsiftp://qb1.loni.org"/>
>>>>    <workdirectory>/home/ux454325/swiftwork</workdirectory>
>>>> </pool>
>>>>
>>>> I get the error below. Files are on CI net at /home/wilde/swift/lab.
>>>>
>>>> I will try to copy coaster boot logs and gram logs to same place when I 
>>>> find them, in subdirs named by $RunID.logs.
>>>>
>>>> --
>>>>
>>>> com$ swift -tc.file tc.data -sites.file qb.coasters.xml cat.swift
>>>> Swift svn swift-r2809 cog-r2350
>>>>
>>>> RunID: 20090406-1155-pgc5nj00
>>>> Progress:
>>>> Progress:  Stage in:1
>>>> Progress:  Submitted:1
>>>> Failed to transfer wrapper log from cat-20090406-1155-pgc5nj00/info/m on qb
>>>> Progress:  Failed:1
>>>> Execution failed:
>>>>          Exception in cat:
>>>> Arguments: [data.txt]
>>>> Host: qb
>>>> Directory: cat-20090406-1155-pgc5nj00/jobs/m/cat-m91ku09j
>>>> stderr.txt:
>>>>
>>>> stdout.txt:
>>>>
>>>> ----
>>>>
>>>> Caused by:
>>>>          Cannot submit job: Cannot run program "qsub": 
>>>> java.io.IOException: error=2, No such file or directory
>>>> org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: 
>>>> Cannot submit job: Cannot run program "qsub": java.io.IOException: 
>>>> error=2, No such file or directory
>>>>          at 
>>>> org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:63)
>>>>          at 
>>>> org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:46)
>>>>          at 
>>>> org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:50)
>>>>          at 
>>>> org.globus.cog.abstraction.coaster.service.job.manager.WorkerManager.startWorker(WorkerManager.java:221)
>>>>          at 
>>>> org.globus.cog.abstraction.coaster.service.job.manager.WorkerManager.run(WorkerManager.java:145)
>>>> Caused by: java.io.IOException: Cannot run program "qsub": 
>>>> java.io.IOException: error=2, No such file or directory
>>>>          at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
>>>>          at java.lang.Runtime.exec(Runtime.java:593)
>>>>          at 
>>>> org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.start(AbstractExecutor.java:73)
>>>>          at 
>>>> org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:53)
>>>>          ... 4 more
>>>> Caused by: java.io.IOException: java.io.IOException: error=2, No such 
>>>> file or directory
>>>>          at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>>>>          at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>>>>          at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
>>>>          ... 7 more
>>>>
>>>> Cleaning up...
>>>> Shutting down service at https://208.100.92.21:44166
>>>> Got channel MetaChannel: 24235184 -> GSSSChannel-null(1)
>>>> - Done
>>>> com$ pwd
>>>> _______________________________________________
>>>> Swift-devel mailing list
>>>> Swift-devel at ci.uchicago.edu
>>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>