[Swift-devel] Provider staging is failing

Mon Aug 30 20:32:57 CDT 2010

coaster.log says same as the java error from the worker log:

2010-08-30 20:07:00,430-0500 DEBUG TaskImpl Task(type=JOB_SUBMISSION, identity=urn:1283216814307-1\
283216820262-1283216820263) setting status to Submitted
2010-08-30 20:07:00,430-0500 DEBUG TaskImpl Task(type=JOB_SUBMISSION, identity=urn:1283216814307-1\
283216820262-1283216820263) setting status to Stagein workerid=000000
2010-08-30 20:07:00,480-0500 DEBUG TaskImpl Task(type=JOB_SUBMISSION, identity=urn:1283216814307-1\
283216820262-1283216820263) setting status to Active
2010-08-30 20:07:00,480-0500 DEBUG TaskImpl Task(type=JOB_SUBMISSION, identity=urn:1283216814307-1\
283216820262-1283216820263) setting status to Failed Error staging in file: org.globus.cog.karajan\
.workflow.service.ProtocolException: java.io.FileNotFoundException: /autonfs/home/wilde/./file:/lo\
calhost/home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging (No such file or directory)
2010-08-30 20:07:00,480-0500 INFO  Cpu 0830-070800-000000:0 jobTerminated
2010-08-30 20:07:00,480-0500 INFO  Cpu 0830-070800-000000:0 pull
2010-08-30 20:07:00,686-0500 INFO  BlockQueueProcessor Shutting down blocks


----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:

> It's getting some error from the coaster service. I wonder why it
> isn't
> being printed. But the coaster/swift log will probably have it.
> 
> Mihael
> 
> On Mon, 2010-08-30 at 19:02 -0600, wilde at mcs.anl.gov wrote:
> > OK, I see now that it is honoring the workdirectory tag. (I thought
> that was not used with provider staging, but seems that it is).
> > 
> > WHen mkdir was failing I was getting an  error code 524; now Im
> getting an error code 520 - seems to be failing now in the actual
> transfer of swiftwrap.
> > 
> > worker log is pasted below.
> > 
> > - Mike
> > 
> > com$ cat worker-0830-560709-000000.log
> > 1283216169.574 INFO  - 0830-560709-000000 Logging started: Mon Aug
> 30 19:56:09 2010
> > 1283216169.576 INFO  - Running on node communicado.ci.uchicago.edu
> > 1283216169.576 DEBUG - uri=http://128.135.125.17:50001
> > 1283216169.576 DEBUG - scheme=http
> > 1283216169.576 DEBUG - host=128.135.125.17
> > 1283216169.576 DEBUG - port=50001
> > 1283216169.576 DEBUG - blockid=0830-560709-000000
> > 1283216169.576 INFO  - Connecting (0)...
> > 1283216169.576 DEBUG - Trying 128.135.125.17:50001...
> > 1283216169.578 INFO  - Connected
> > 1283216169.578 DEBUG - Replies: {}
> > 1283216169.578 DEBUG - OUT: len=8, tag=0, flags=0
> > 1283216169.578 DEBUG - OUT: len=18, tag=0, flags=0
> > 1283216169.578 DEBUG - OUT: len=0, tag=0, flags=2
> > 1283216169.578 DEBUG - done sending frags for 0
> > 1283216169.623 DEBUG - Fin flag set
> > 1283216169.624 INFO  000000 Registration successful. ID=000000
> > 1283216169.624 DEBUG 000000 New request (1)
> > 1283216169.624 DEBUG 000000 Fin flag set
> > 1283216169.624 DEBUG 000000 Processing request
> > 1283216169.625 DEBUG 000000 Cmd is SUBMITJOB
> > 1283216169.625 INFO  000000 1283216169479 Job info received (tag=1)
> > 1283216169.625 DEBUG 000000 1283216169479 Job check ok (dir:
> /home/wilde/swiftwork/catsn-20100830-1956-z61dqk05-d-cat-duz412yj)
> > 1283216169.625 INFO  000000 1283216169479 Sending submit reply
> (tag=1)
> > 1283216169.625 DEBUG 000000 OUT: len=2, tag=1, flags=3
> > 1283216169.625 DEBUG 000000 done sending frags for 1
> > 1283216169.625 INFO  000000 1283216169479 Submit reply sent (tag=1)
> > 1283216169.625 DEBUG 000000 Replies: {}
> > 1283216169.625 DEBUG 000000 OUT: len=9, tag=1, flags=0
> > 1283216169.625 DEBUG 000000 OUT: len=13, tag=1, flags=0
> > 1283216169.625 DEBUG 000000 OUT: len=2, tag=1, flags=0
> > 1283216169.625 DEBUG 000000 OUT: len=1, tag=1, flags=0
> > 1283216169.625 DEBUG 000000 OUT: len=15, tag=1, flags=2
> > 1283216169.626 DEBUG 000000 done sending frags for 1
> > 1283216169.626 DEBUG 000000 1283216169479 Staging in
> file://localhost//home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging
> > 1283216169.626 DEBUG 000000 1283216169479 src:
> file://localhost//home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging,
> protocol: file, path:
> localhost//home/wilde/swift/rev/trunk/bin/../libexec/_swiftwrap.staging
> > 1283216169.627 DEBUG 000000 Opening
> /home/wilde/swiftwork/catsn-20100830-1956-z61dqk05-d-cat-duz412yj/_swiftwrap.staging
> in cwd /
> > ...
> > 1283216169.628 DEBUG 000000 1283216169479 Opened
> /home/wilde/swiftwork/catsn-20100830-1956-z61dqk05-d-cat-duz412yj/_swiftwrap.staging
> > 1283216169.628 DEBUG 000000 Replies: {1 = ARRAY(0x93ce8f0)}
> > 1283216169.628 DEBUG 000000 OUT: len=3, tag=2, flags=0
> > 1283216169.628 DEBUG 000000 OUT: len=78, tag=2, flags=0
> > 1283216169.628 DEBUG 000000 OUT: len=84, tag=2, flags=2
> > 1283216169.628 DEBUG 000000 done sending frags for 2
> > 1283216169.647 DEBUG 000000 Fin flag set
> > 1283216169.653 DEBUG 000000 1283216169479 getFileCBDataIn jobid:
> 1283216169479, state: 0, tag: 2, err: 4, fin: 0
> > 1283216169.653 DEBUG 000000 Replies: {2 = ARRAY(0x93ce980)}
> > 1283216169.653 DEBUG 000000 OUT: len=9, tag=3, flags=0
> > 1283216169.653 DEBUG 000000 OUT: len=13, tag=3, flags=0
> > 1283216169.653 DEBUG 000000 OUT: len=1, tag=3, flags=0
> > 1283216169.653 DEBUG 000000 OUT: len=3, tag=3, flags=0
> > 1283216169.653 DEBUG 000000 OUT: len=239, tag=3, flags=2
> > 1283216169.653 DEBUG 000000 done sending frags for 3
> > 1283216169.653 DEBUG 000000 Fin flag set
> > 1283216169.653 DEBUG 000000 1283216169479 getFileCBDataIn jobid:
> 1283216169479, state: 0, tag: 2, err: 4, fin: 2
> > 1283216169.653 DEBUG 000000 Replies: {3 = ARRAY(0x93ce8d0)}
> > 1283216169.653 DEBUG 000000 OUT: len=9, tag=4, flags=0
> > 1283216169.653 DEBUG 000000 OUT: len=13, tag=4, flags=0
> > 1283216169.653 DEBUG 000000 OUT: len=1, tag=4, flags=0
> > 1283216169.653 DEBUG 000000 OUT: len=3, tag=4, flags=0
> > 1283216169.653 DEBUG 000000 OUT: len=2223, tag=4, flags=2
> > 1283216169.653 DEBUG 000000 done sending frags for 4
> > 1283216169.698 DEBUG 000000 Fin flag set
> > 1283216169.698 DEBUG 000000 Fin flag set
> > 1283216169.902 DEBUG 000000 New request (2)
> > 1283216169.902 DEBUG 000000 Fin flag set
> > 1283216169.902 DEBUG 000000 Processing request
> > 1283216169.902 DEBUG 000000 Cmd is SHUTDOWN
> > 1283216169.902 DEBUG 000000 Shutdown command received
> > 1283216169.902 DEBUG 000000 OUT: len=2, tag=2, flags=3
> > 1283216169.902 DEBUG 000000 done sending frags for 2
> > com$ 
> > 
> > ----- wilde at mcs.anl.gov wrote:
> > 
> > > Mihael, Justin,
> > > 
> > > Im trying to use provider-staging for the first time. It seems to
> be
> > > starting the worker in /, and hence staging in fails right away
> (on
> > > _swiftwrap).
> > > 
> > > Where is the worker supposed to start when using provider
> staging?
> > > 
> > > Ive tried to set the jobdir to /tmp using the <scratch> tag but
> that
> > > doesnt seem to be honored.
> > > 
> > > Ive tried a few different sites configurations; Im running from
> > > bridled to communicado using ssh. My most recent is:
> > > 
> > >   <pool handle="localhost">
> > > 
> > >     <execution provider="coaster"
> url="communicado.ci.uchicago.edu"
> > > jobmanager="ssh:local"/>
> > >     <!-- <profile namespace="globus"
> > > key="workerManager">passive</profile> -->
> > > 
> > >     <profile namespace="globus" key="workersPerNode">8</profile>
> > >     <profile key="jobThrottle" namespace="karajan">.07</profile>
> > >     <profile namespace="karajan"
> key="initialScore">10000</profile>
> > >     
> > >     <profile namespace="swift" key="stagingMethod">file</profile>
> > > 
> > >     <filesystem provider="local" url="none" />
> > > <!--
> > >     <workdirectory>/home/wilde/swiftwork</workdirectory>
> > > -->
> > >     <scratch>/tmp/wilde/scratch</scratch> 
> > >   </pool>
> > > 
> > > 
> > > - Mike
> > > _______________________________________________
> > > Swift-devel mailing list
> > > Swift-devel at ci.uchicago.edu
> > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> >

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory