[Swift-devel] Provider staging is failing

wilde at mcs.anl.gov wilde at mcs.anl.gov
Mon Aug 30 20:26:27 CDT 2010 

I turned on the TRACE output level in worker.pl. I need to dig deeper but it looks to me that the pathnames its trying to fetch are getting mangled/confused with the file:// portion of the URI:

org.globus.cog.karajan.workflow.service.ProtocolException: java.io.FileNotFoundException: /autonfs/home/wilde/./file:/localhost/home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging (No such file or directory)

The file "/home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging" does exist on the client side.

I can't yet tell if its really trying to open a pathname of the form "/autonfs/home/wilde/./file:/localhost/home/wilde/etcetc..."

Full log is below.

- Mike

com$ cat worker-0830-070800-000000.log
1283216820.360 INFO  - 0830-070800-000000 Logging started: Mon Aug 30 20:07:00 2010
1283216820.362 INFO  - Running on node communicado.ci.uchicago.edu
1283216820.362 DEBUG - uri=http://128.135.125.17:50001
1283216820.362 DEBUG - scheme=http
1283216820.362 DEBUG - host=128.135.125.17
1283216820.362 DEBUG - port=50001
1283216820.362 DEBUG - blockid=0830-070800-000000
1283216820.362 INFO  - Connecting (0)...
1283216820.362 DEBUG - Trying 128.135.125.17:50001...
1283216820.363 INFO  - Connected
1283216820.363 DEBUG - Replies: {}
1283216820.363 DEBUG - OUT: len=8, tag=0, flags=0
1283216820.363 TRACE - REGISTER
1283216820.363 DEBUG - OUT: len=18, tag=0, flags=0
1283216820.363 TRACE - 0830-070800-000000
1283216820.363 DEBUG - OUT: len=0, tag=0, flags=2
1283216820.363 TRACE - 
1283216820.363 DEBUG - done sending frags for 0
1283216820.409 TRACE -  IN: len=6, actuallen=6, tag=0, flags=3, 000000
1283216820.409 DEBUG - Fin flag set
1283216820.409 INFO  000000 Registration successful. ID=000000
1283216820.410 TRACE 000000  IN: len=9, actuallen=9, tag=1, flags=0, SUBMITJOB
1283216820.410 DEBUG 000000 New request (1)
1283216820.410 TRACE 000000  IN: len=759, actuallen=759, tag=1, flags=2, identity=1283216820263
executable=/bin/bash
directory=/home/wilde/swiftwork/catsn-20100830-2006-f0dhgma1-f-cat-fn4k12yj
batch=false
arg=_swiftwrap.staging
arg=-e
arg=/bin/cat
arg=-out
arg=outdir/f.0001.out
arg=-err
arg=stderr.txt
arg=-i
arg=-d
arg=|outdir
arg=-if
arg=data.txt
arg=-of
arg=outdir/f.0001.out
arg=-k
arg=-cdmfile
arg=
arg=-status
arg=provider
arg=-a
arg=data.txt
stagein=file://localhost//home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging\n_swiftwrap.staging
stagein=file://localhost/data.txt\n./data.txt
stageout=wrapper.log\nfile://localhost/catsn-20100830-2006-f0dhgma1.d/cat-fn4k12yj.info
stageout=./outdir/f.0001.out\nfile://localhost/outdir/f.0001.out
cleanup=.
contact=communicado.ci.uchicago.edu
provider=coaster
jm=ssh:local

1283216820.410 DEBUG 000000 Fin flag set
1283216820.410 DEBUG 000000 Processing request
1283216820.410 DEBUG 000000 Cmd is SUBMITJOB
1283216820.410 INFO  000000 1283216820263 Job info received (tag=1)
1283216820.411 DEBUG 000000 1283216820263 Job check ok (dir: /home/wilde/swiftwork/catsn-20100830-2006-f0dhgma1-f-cat-fn4k12yj)
1283216820.411 INFO  000000 1283216820263 Sending submit reply (tag=1)
1283216820.411 DEBUG 000000 OUT: len=2, tag=1, flags=3
1283216820.411 TRACE 000000 OK
1283216820.411 DEBUG 000000 done sending frags for 1
1283216820.411 INFO  000000 1283216820263 Submit reply sent (tag=1)
1283216820.411 DEBUG 000000 Replies: {}
1283216820.411 DEBUG 000000 OUT: len=9, tag=1, flags=0
1283216820.411 TRACE 000000 JOBSTATUS
1283216820.411 DEBUG 000000 OUT: len=13, tag=1, flags=0
1283216820.411 TRACE 000000 1283216820263
1283216820.411 DEBUG 000000 OUT: len=2, tag=1, flags=0
1283216820.411 TRACE 000000 16
1283216820.411 DEBUG 000000 OUT: len=1, tag=1, flags=0
1283216820.411 TRACE 000000 0
1283216820.411 DEBUG 000000 OUT: len=15, tag=1, flags=2
1283216820.411 TRACE 000000 workerid=000000
1283216820.411 DEBUG 000000 done sending frags for 1
1283216820.411 DEBUG 000000 1283216820263 Staging in file://localhost//home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging
1283216820.411 DEBUG 000000 1283216820263 src: file://localhost//home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging, protocol: file, path: localhost//home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging
1283216820.413 DEBUG 000000 Opening /home/wilde/swiftwork/catsn-20100830-2006-f0dhgma1-f-cat-fn4k12yj/_swiftwrap.staging in cwd /
...
1283216820.414 DEBUG 000000 1283216820263 Opened /home/wilde/swiftwork/catsn-20100830-2006-f0dhgma1-f-cat-fn4k12yj/_swiftwrap.staging
1283216820.414 DEBUG 000000 Replies: {1 = ARRAY(0xe3c4060)}
1283216820.414 DEBUG 000000 OUT: len=3, tag=2, flags=0
1283216820.414 TRACE 000000 GET
1283216820.414 DEBUG 000000 OUT: len=78, tag=2, flags=0
1283216820.414 TRACE 000000 file://localhost//home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging
1283216820.414 DEBUG 000000 OUT: len=84, tag=2, flags=2
1283216820.414 TRACE 000000 /home/wilde/swiftwork/catsn-20100830-2006-f0dhgma1-f-cat-fn4k12yj/_swiftwrap.staging
1283216820.414 DEBUG 000000 done sending frags for 2
1283216820.433 TRACE 000000  IN: len=2, actuallen=2, tag=1, flags=3, OK
1283216820.433 DEBUG 000000 Fin flag set
1283216820.438 TRACE 000000  IN: len=216, actuallen=216, tag=2, flags=5, org.globus.cog.karajan.workflow.service.ProtocolException: java.io.FileNotFoundException: /autonfs/home/wilde/./file:/localhost/home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging (No such file or directory)
1283216820.438 DEBUG 000000 1283216820263 getFileCBDataIn jobid: 1283216820263, state: 0, tag: 2, err: 4, fin: 0
1283216820.438 DEBUG 000000 Replies: {2 = ARRAY(0xe3c40f0)}
1283216820.438 DEBUG 000000 OUT: len=9, tag=3, flags=0
1283216820.438 TRACE 000000 JOBSTATUS
1283216820.438 DEBUG 000000 OUT: len=13, tag=3, flags=0
1283216820.438 TRACE 000000 1283216820263
1283216820.438 DEBUG 000000 OUT: len=1, tag=3, flags=0
1283216820.438 TRACE 000000 5
1283216820.438 DEBUG 000000 OUT: len=3, tag=3, flags=0
1283216820.438 TRACE 000000 520
1283216820.438 DEBUG 000000 OUT: len=239, tag=3, flags=2
1283216820.438 TRACE 000000 Error staging in file: org.globus.cog.karajan.workflow.service.ProtocolException: java.io.FileNotFoundException: /autonfs/home/wilde/./file:/localhost/home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging (No such file or directory)
1283216820.438 DEBUG 000000 done sending frags for 3
1283216820.438 TRACE 000000  IN: len=2200, actuallen=2200, tag=2, flags=7, org.globus.cog.karajan.workflow.service.ProtocolException: org.globus.cog.karajan.workflow.service.ProtocolException: java.io.FileNotFoundException: /autonfs/home/wilde/./file:/localhost/home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging (No such file or directory)
        at org.globus.cog.abstraction.impl.file.coaster.handlers.GetFileHandler.requestComplete(GetFileHandler.java:41)
        at org.globus.cog.karajan.workflow.service.handlers.RequestHandler.receiveCompleted(RequestHandler.java:84)
        at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.handleRequest(AbstractKarajanChannel.java:387)
        at org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel.step(AbstractStreamKarajanChannel.java:159)
        at org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel$Multiplexer.run(AbstractStreamKarajanChannel.java:368)
Caused by: org.globus.cog.karajan.workflow.service.ProtocolException: java.io.FileNotFoundException: /autonfs/home/wilde/./file:/localhost/home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging (No such file or directory)
        at org.globus.cog.abstraction.impl.file.coaster.handlers.GetFileHandler.send(GetFileHandler.java:64)
        at org.globus.cog.karajan.workflow.service.handlers.RequestHandler.sendReply(RequestHandler.java:37)
        at org.globus.cog.abstraction.impl.file.coaster.handlers.CoasterFileRequestHandler.sendReply(CoasterFileRequestHandler.java:45)
        at org.globus.cog.abstraction.impl.file.coaster.handlers.GetFileHandler.requestComplete(GetFileHandler.java:38)
        ... 4 more
Caused by: java.io.FileNotFoundException: /autonfs/home/wilde/./file:/localhost/home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging (No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:106)
        at org.globus.cog.abstraction.impl.file.coaster.handlers.providers.LocalIOProvider$Reader.<init>(LocalIOProvider.java:120)
        at org.globus.cog.abstraction.impl.file.coaster.handlers.providers.LocalIOProvider.pull(LocalIOProvider.java:35)
        at org.globus.cog.abstraction.impl.file.coaster.handlers.GetFileHandler.send(GetFileHandler.java:60)
        ... 7 more

1283216820.438 DEBUG 000000 Fin flag set
1283216820.438 DEBUG 000000 1283216820263 getFileCBDataIn jobid: 1283216820263, state: 0, tag: 2, err: 4, fin: 2
1283216820.438 DEBUG 000000 Replies: {3 = ARRAY(0xe3c4040)}
1283216820.438 DEBUG 000000 OUT: len=9, tag=4, flags=0
1283216820.438 TRACE 000000 JOBSTATUS
1283216820.438 DEBUG 000000 OUT: len=13, tag=4, flags=0
1283216820.438 TRACE 000000 1283216820263
1283216820.438 DEBUG 000000 OUT: len=1, tag=4, flags=0
1283216820.438 TRACE 000000 5
1283216820.438 DEBUG 000000 OUT: len=3, tag=4, flags=0
1283216820.438 TRACE 000000 520
1283216820.438 DEBUG 000000 OUT: len=2223, tag=4, flags=2
1283216820.438 TRACE 000000 Error staging in file: org.globus.cog.karajan.workflow.service.ProtocolException: org.globus.cog.karajan.workflow.service.ProtocolException: java.io.FileNotFoundException: /autonfs/home/wilde/./file:/localhost/home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging (No such file or directory)
        at org.globus.cog.abstraction.impl.file.coaster.handlers.GetFileHandler.requestComplete(GetFileHandler.java:41)
        at org.globus.cog.karajan.workflow.service.handlers.RequestHandler.receiveCompleted(RequestHandler.java:84)
        at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.handleRequest(AbstractKarajanChannel.java:387)
        at org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel.step(AbstractStreamKarajanChannel.java:159)
        at org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel$Multiplexer.run(AbstractStreamKarajanChannel.java:368)
Caused by: org.globus.cog.karajan.workflow.service.ProtocolException: java.io.FileNotFoundException: /autonfs/home/wilde/./file:/localhost/home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging (No such file or directory)
        at org.globus.cog.abstraction.impl.file.coaster.handlers.GetFileHandler.send(GetFileHandler.java:64)
        at org.globus.cog.karajan.workflow.service.handlers.RequestHandler.sendReply(RequestHandler.java:37)
        at org.globus.cog.abstraction.impl.file.coaster.handlers.CoasterFileRequestHandler.sendReply(CoasterFileRequestHandler.java:45)
        at org.globus.cog.abstraction.impl.file.coaster.handlers.GetFileHandler.requestComplete(GetFileHandler.java:38)
        ... 4 more
Caused by: java.io.FileNotFoundException: /autonfs/home/wilde/./file:/localhost/home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging (No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:106)
        at org.globus.cog.abstraction.impl.file.coaster.handlers.providers.LocalIOProvider$Reader.<init>(LocalIOProvider.java:120)
        at org.globus.cog.abstraction.impl.file.coaster.handlers.providers.LocalIOProvider.pull(LocalIOProvider.java:35)
        at org.globus.cog.abstraction.impl.file.coaster.handlers.GetFileHandler.send(GetFileHandler.java:60)
        ... 7 more

1283216820.438 DEBUG 000000 done sending frags for 4
1283216820.482 TRACE 000000  IN: len=2, actuallen=2, tag=3, flags=3, OK
1283216820.482 DEBUG 000000 Fin flag set
1283216820.482 TRACE 000000  IN: len=2, actuallen=2, tag=4, flags=3, OK
1283216820.482 DEBUG 000000 Fin flag set
1283216820.687 TRACE 000000  IN: len=8, actuallen=8, tag=2, flags=2, SHUTDOWN
1283216820.687 DEBUG 000000 New request (2)
1283216820.687 DEBUG 000000 Fin flag set
1283216820.687 DEBUG 000000 Processing request
1283216820.687 DEBUG 000000 Cmd is SHUTDOWN
1283216820.687 DEBUG 000000 Shutdown command received
1283216820.687 DEBUG 000000 OUT: len=2, tag=2, flags=3
1283216820.687 TRACE 000000 OK
1283216820.687 DEBUG 000000 done sending frags for 2
com$  
----- wilde at mcs.anl.gov wrote:

> OK, I see now that it is honoring the workdirectory tag. (I thought
> that was not used with provider staging, but seems that it is).
> 
> WHen mkdir was failing I was getting an  error code 524; now Im
> getting an error code 520 - seems to be failing now in the actual
> transfer of swiftwrap.
> 
> worker log is pasted below.
> 
> - Mike
> 
> com$ cat worker-0830-560709-000000.log
> 1283216169.574 INFO  - 0830-560709-000000 Logging started: Mon Aug 30
> 19:56:09 2010
> 1283216169.576 INFO  - Running on node communicado.ci.uchicago.edu
> 1283216169.576 DEBUG - uri=http://128.135.125.17:50001
> 1283216169.576 DEBUG - scheme=http
> 1283216169.576 DEBUG - host=128.135.125.17
> 1283216169.576 DEBUG - port=50001
> 1283216169.576 DEBUG - blockid=0830-560709-000000
> 1283216169.576 INFO  - Connecting (0)...
> 1283216169.576 DEBUG - Trying 128.135.125.17:50001...
> 1283216169.578 INFO  - Connected
> 1283216169.578 DEBUG - Replies: {}
> 1283216169.578 DEBUG - OUT: len=8, tag=0, flags=0
> 1283216169.578 DEBUG - OUT: len=18, tag=0, flags=0
> 1283216169.578 DEBUG - OUT: len=0, tag=0, flags=2
> 1283216169.578 DEBUG - done sending frags for 0
> 1283216169.623 DEBUG - Fin flag set
> 1283216169.624 INFO  000000 Registration successful. ID=000000
> 1283216169.624 DEBUG 000000 New request (1)
> 1283216169.624 DEBUG 000000 Fin flag set
> 1283216169.624 DEBUG 000000 Processing request
> 1283216169.625 DEBUG 000000 Cmd is SUBMITJOB
> 1283216169.625 INFO  000000 1283216169479 Job info received (tag=1)
> 1283216169.625 DEBUG 000000 1283216169479 Job check ok (dir:
> /home/wilde/swiftwork/catsn-20100830-1956-z61dqk05-d-cat-duz412yj)
> 1283216169.625 INFO  000000 1283216169479 Sending submit reply
> (tag=1)
> 1283216169.625 DEBUG 000000 OUT: len=2, tag=1, flags=3
> 1283216169.625 DEBUG 000000 done sending frags for 1
> 1283216169.625 INFO  000000 1283216169479 Submit reply sent (tag=1)
> 1283216169.625 DEBUG 000000 Replies: {}
> 1283216169.625 DEBUG 000000 OUT: len=9, tag=1, flags=0
> 1283216169.625 DEBUG 000000 OUT: len=13, tag=1, flags=0
> 1283216169.625 DEBUG 000000 OUT: len=2, tag=1, flags=0
> 1283216169.625 DEBUG 000000 OUT: len=1, tag=1, flags=0
> 1283216169.625 DEBUG 000000 OUT: len=15, tag=1, flags=2
> 1283216169.626 DEBUG 000000 done sending frags for 1
> 1283216169.626 DEBUG 000000 1283216169479 Staging in
> file://localhost//home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging
> 1283216169.626 DEBUG 000000 1283216169479 src:
> file://localhost//home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging,
> protocol: file, path:
> localhost//home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging
> 1283216169.627 DEBUG 000000 Opening
> /home/wilde/swiftwork/catsn-20100830-1956-z61dqk05-d-cat-duz412yj/_swiftwrap.staging
> in cwd /
> ...
> 1283216169.628 DEBUG 000000 1283216169479 Opened
> /home/wilde/swiftwork/catsn-20100830-1956-z61dqk05-d-cat-duz412yj/_swiftwrap.staging
> 1283216169.628 DEBUG 000000 Replies: {1 = ARRAY(0x93ce8f0)}
> 1283216169.628 DEBUG 000000 OUT: len=3, tag=2, flags=0
> 1283216169.628 DEBUG 000000 OUT: len=78, tag=2, flags=0
> 1283216169.628 DEBUG 000000 OUT: len=84, tag=2, flags=2
> 1283216169.628 DEBUG 000000 done sending frags for 2
> 1283216169.647 DEBUG 000000 Fin flag set
> 1283216169.653 DEBUG 000000 1283216169479 getFileCBDataIn jobid:
> 1283216169479, state: 0, tag: 2, err: 4, fin: 0
> 1283216169.653 DEBUG 000000 Replies: {2 = ARRAY(0x93ce980)}
> 1283216169.653 DEBUG 000000 OUT: len=9, tag=3, flags=0
> 1283216169.653 DEBUG 000000 OUT: len=13, tag=3, flags=0
> 1283216169.653 DEBUG 000000 OUT: len=1, tag=3, flags=0
> 1283216169.653 DEBUG 000000 OUT: len=3, tag=3, flags=0
> 1283216169.653 DEBUG 000000 OUT: len=239, tag=3, flags=2
> 1283216169.653 DEBUG 000000 done sending frags for 3
> 1283216169.653 DEBUG 000000 Fin flag set
> 1283216169.653 DEBUG 000000 1283216169479 getFileCBDataIn jobid:
> 1283216169479, state: 0, tag: 2, err: 4, fin: 2
> 1283216169.653 DEBUG 000000 Replies: {3 = ARRAY(0x93ce8d0)}
> 1283216169.653 DEBUG 000000 OUT: len=9, tag=4, flags=0
> 1283216169.653 DEBUG 000000 OUT: len=13, tag=4, flags=0
> 1283216169.653 DEBUG 000000 OUT: len=1, tag=4, flags=0
> 1283216169.653 DEBUG 000000 OUT: len=3, tag=4, flags=0
> 1283216169.653 DEBUG 000000 OUT: len=2223, tag=4, flags=2
> 1283216169.653 DEBUG 000000 done sending frags for 4
> 1283216169.698 DEBUG 000000 Fin flag set
> 1283216169.698 DEBUG 000000 Fin flag set
> 1283216169.902 DEBUG 000000 New request (2)
> 1283216169.902 DEBUG 000000 Fin flag set
> 1283216169.902 DEBUG 000000 Processing request
> 1283216169.902 DEBUG 000000 Cmd is SHUTDOWN
> 1283216169.902 DEBUG 000000 Shutdown command received
> 1283216169.902 DEBUG 000000 OUT: len=2, tag=2, flags=3
> 1283216169.902 DEBUG 000000 done sending frags for 2
> com$ 
> 
> ----- wilde at mcs.anl.gov wrote:
> 
> > Mihael, Justin,
> > 
> > Im trying to use provider-staging for the first time. It seems to
> be
> > starting the worker in /, and hence staging in fails right away (on
> > _swiftwrap).
> > 
> > Where is the worker supposed to start when using provider staging?
> > 
> > Ive tried to set the jobdir to /tmp using the <scratch> tag but
> that
> > doesnt seem to be honored.
> > 
> > Ive tried a few different sites configurations; Im running from
> > bridled to communicado using ssh. My most recent is:
> > 
> >   <pool handle="localhost">
> > 
> >     <execution provider="coaster" url="communicado.ci.uchicago.edu"
> > jobmanager="ssh:local"/>
> >     <!-- <profile namespace="globus"
> > key="workerManager">passive</profile> -->
> > 
> >     <profile namespace="globus" key="workersPerNode">8</profile>
> >     <profile key="jobThrottle" namespace="karajan">.07</profile>
> >     <profile namespace="karajan" key="initialScore">10000</profile>
> >     
> >     <profile namespace="swift" key="stagingMethod">file</profile>
> > 
> >     <filesystem provider="local" url="none" />
> > <!--
> >     <workdirectory>/home/wilde/swiftwork</workdirectory>
> > -->
> >     <scratch>/tmp/wilde/scratch</scratch> 
> >   </pool>
> > 
> > 
> > - Mike
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> 
> -- 
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory

More information about the Swift-devel mailing list