[Swift-devel] coaster problem with jobmanager=gt2:pbs

[Michael Wilde]

Mihael, when I svn updated our test swift+cog source and rebuilt, Glen
Glen gets the errors below.

When I reverted back to last Tuesday Mar 31, this new error does not occur.

Does "Caused by: GSSException: Invalid name provided [Caused by: 
[JGLOBUS-112] Malformed name, "=" missing in "38356/jobmanager-pbs"]"

suggest a new error introduced in the commits since Tuesday?

This is with coasters and gt2:gt2:pbs.

- Mike



Caused by:
         Invalid GSSCredentials
org.globus.cog.abstraction.impl.common.task.InvalidSecurityContextException: 
Invalid GSSCredentials
         at 
org.globus.cog.abstraction.impl.execution.gt2.JobSubmissionTaskHandler.submitSingleJob(JobSubmissionTaskHandler.java:149)
         at 
org.globus.cog.abstraction.impl.execution.gt2.JobSubmissionTaskHandler.submit(JobSubmissionTaskHandler.java:99)
         at 
org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:46)
         at 
org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:50)
         at 
org.globus.cog.abstraction.coaster.service.job.manager.WorkerManager.startWorker(WorkerManager.java:222)
         at 
org.globus.cog.abstraction.coaster.service.job.manager.WorkerManager.run(WorkerManager.java:145)
Caused by: GSSException: Invalid name provided [Caused by: [JGLOBUS-112] 
Malformed name, "=" missing in "38356/jobmanager-pbs"]
         at 
org.globus.gsi.gssapi.GlobusGSSName.<init>(GlobusGSSName.java:137)
         at 
org.globus.gsi.gssapi.GlobusGSSManagerImpl.createName(GlobusGSSManagerImpl.java:304)
         at 
org.globus.gsi.gssapi.auth.IdentityAuthorization.getExpectedName(IdentityAuthorization.java:82)
         at org.globus.gram.Gram.gatekeeperConnect(Gram.java:85)
         at org.globus.gram.Gram.request(Gram.java:310)
         at org.globus.gram.GramJob.request(GramJob.java:262)
         at 
org.globus.cog.abstraction.impl.execution.gt2.JobSubmissionTaskHandler.submitSingleJob(JobSubmissionTaskHandler.java:133)
         ... 5 more
what's up here


Any chance I picked up code in transition, or a new problem in recent 
commits?

- Mike



On 4/6/09 1:02 PM, Mihael Hategan wrote:
> Yes. This is one of those "can't find executable unless run through
> 'bash -l' or maybe not" which we saw using wget and md5sum. I'm thinking
> how to deal with the situation.
> 
> On Mon, 2009-04-06 at 12:10 -0500, Michael Wilde wrote:
>> With this sites entry:
>>
>> <pool handle="qb" >
>>    <profile namespace="globus" key="project">TG-CDA070002T</profile>
>>    <execution provider="coaster" url="queenbee.loni-lsu.teragrid.org"
>>               jobManager="gt2:pbs" />
>>    <gridftp url="gsiftp://qb1.loni.org"/>
>>    <workdirectory>/home/ux454325/swiftwork</workdirectory>
>> </pool>
>>
>> I get the error below. Files are on CI net at /home/wilde/swift/lab.
>>
>> I will try to copy coaster boot logs and gram logs to same place when I 
>> find them, in subdirs named by $RunID.logs.
>>
>> --
>>
>> com$ swift -tc.file tc.data -sites.file qb.coasters.xml cat.swift
>> Swift svn swift-r2809 cog-r2350
>>
>> RunID: 20090406-1155-pgc5nj00
>> Progress:
>> Progress:  Stage in:1
>> Progress:  Submitted:1
>> Failed to transfer wrapper log from cat-20090406-1155-pgc5nj00/info/m on qb
>> Progress:  Failed:1
>> Execution failed:
>>          Exception in cat:
>> Arguments: [data.txt]
>> Host: qb
>> Directory: cat-20090406-1155-pgc5nj00/jobs/m/cat-m91ku09j
>> stderr.txt:
>>
>> stdout.txt:
>>
>> ----
>>
>> Caused by:
>>          Cannot submit job: Cannot run program "qsub": 
>> java.io.IOException: error=2, No such file or directory
>> org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: 
>> Cannot submit job: Cannot run program "qsub": java.io.IOException: 
>> error=2, No such file or directory
>>          at 
>> org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:63)
>>          at 
>> org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:46)
>>          at 
>> org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:50)
>>          at 
>> org.globus.cog.abstraction.coaster.service.job.manager.WorkerManager.startWorker(WorkerManager.java:221)
>>          at 
>> org.globus.cog.abstraction.coaster.service.job.manager.WorkerManager.run(WorkerManager.java:145)
>> Caused by: java.io.IOException: Cannot run program "qsub": 
>> java.io.IOException: error=2, No such file or directory
>>          at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
>>          at java.lang.Runtime.exec(Runtime.java:593)
>>          at 
>> org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.start(AbstractExecutor.java:73)
>>          at 
>> org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:53)
>>          ... 4 more
>> Caused by: java.io.IOException: java.io.IOException: error=2, No such 
>> file or directory
>>          at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>>          at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>>          at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
>>          ... 7 more
>>
>> Cleaning up...
>> Shutting down service at https://208.100.92.21:44166
>> Got channel MetaChannel: 24235184 -> GSSSChannel-null(1)
>> - Done
>> com$ pwd
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>