[Swift-user] Questions about concurrent_mapper and parameter lists
Jorgen Brandt
joergen.brandt at onlinehome.de
Tue Apr 30 03:31:20 CDT 2013
Dear Sir or Madam,
I just started using Swift. There is several things, though, I cannot
figure out.
- I want to perform a computation in parallel, using the filesys_mapper
to parse my input directory and the concurrent_mapper to create the
corresponding output files. The output files are created temporarily but
they are not there any more, when the execution finishes. What do I do
wrong? (See listings below.)
- I want to append an array of filenames (of which I do not know its
length at compile-time) to the parameter list of an app. I tried several
things but had no luck so far. I'd expected that something like
type Text;
app( Text o )mycatReduce( Text[] inputset ) {
cat @inputset stdout=@o;
}
would work and indeed the above snippet survives compilation. But the
stage-in does not seem to work. Cat complains about its input not being
there. (Listings below.)
I've been googling around but the only thing I've found that may relate
to this problem is your article in Computer 42 (11) from 2009 in which
it says
<< Swift's foreach construct performs a simple map operation, and the
act of passing a multimember dataset to a procedure provides a simple
and natural way to implement reduction operations. >>
No example was given though.
I appreciate any help and hope not to steal too much of your time.
Best regards
Jorgen
The rest of this email explains in detail what I have done: I am
currently working with Swift 0.94 (RC4). I have a working directory
containing a folder 'infileset'. this folder contains 2 files: 'a.txt'
and 'b.txt'.
--- Start Listing -------------------------------------------
jorgen at turing:~/workspace_set/swift/test$ ls infileset
a.txt b.txt
jorgen at turing:~/workspace_set/swift/test$ cat infileset/a.txt
Hello
jorgen at turing:~/workspace_set/swift/test$ cat infileset/b.txt
World
--- End Listing ---------------------------------------------
In this working directory I also store my swift-script.
--- Start Listing -------------------------------------------
jorgen at turing:~/workspace_set/swift/test$ cat test_simple.swift
type Text;
app( Text o )mycat( Text input ) {
cat @input stdout=@o;
}
( Text[] oset )mycatMap( Text[] inputset ) {
foreach input, i in inputset {
oset[ i ] = mycat( input );
trace( @filename( oset[ i ] ) );
}
}
app( Text o )mycatReduce( Text[] inputset ) {
cat @inputset stdout=@o;
}
Text[] p <filesys_mapper; location="infileset", suffix=".txt">;
Text[] q <concurrent_mapper; location="outfileset", prefix="map_output",
suffix=".txt">;
Text r <"outfileset/reduce_output.txt">;
q = mycatMap( p ); // So far so good, but the output q is not persistent.
r = mycatReduce( p ); // Drops out with 'infileset/a.txt
infileset/b.txt: No such
// file or directory'
--- End Listing ---------------------------------------------
Running this script yields the following:
--- Start Listing -------------------------------------------
jorgen at turing:~/workspace_set/swift/test$
~/software/swift-0.94/bin/swift test_simple.swift
Swift 0.94 swift-r6320 cog-r3615
RunID: 20130430-1017-sfans555
Progress: time: Tue, 30 Apr 2013 10:17:56 +0200
SwiftScript trace: outfileset/_concurrent/map_output--array//elt-0.txt
SwiftScript trace: outfileset/_concurrent/map_output--array//elt-1.txt
Progress: time: Tue, 30 Apr 2013 10:17:57 +0200 Selecting site:1
Checking status:1 Finished successfully:1
Execution failed:
Exception in cat:
Arguments: [infileset/a.txt infileset/b.txt]
Host: localhost
Directory: test_simple-20130430-1017-sfans555/jobs/c/cat-ceb95s8l
stderr.txt: /bin/cat: infileset/a.txt infileset/b.txt: No such file
or directory
stdout.txt:
Caused by:
File not found:
/var/tmp/test_simple-20130430-1017-sfans555/shared/outfileset/reduce_output.txt
mycatReduce, test_simple.swift, line 25
--- End Listing ---------------------------------------------
The script created an .rlog file called
test_simple-20130430-1017-sfans555.0.rlog.
--- Start Listing -------------------------------------------
jorgen at turing:~/workspace_set/swift/test$ cat
test_simple-20130430-1017-sfans555.0.rlog
# Log file created Tue Apr 30 10:17:56 CEST 2013
:q.[0]!file://localhost/outfileset/_concurrent/map_output--array//elt-0.txt
--- End Listing ---------------------------------------------
Also, the script created a swift.log file with the following content:
--- Start Listing -------------------------------------------
jorgen at turing:~/workspace_set/swift/test$ cat swift.log
2013-04-30 10:17:53,338+0200 DEBUG Loader Swift started
--- End Listing ---------------------------------------------
Eventually, the script created a log file called
test_simple-20130430-1017-sfans555.log.
--- Start Listing -------------------------------------------
jorgen at turing:~/workspace_set/swift/test$ cat
test_simple-20130430-1017-sfans555.log
2013-04-30 10:17:53,377+0200 DEBUG Loader arguments: [test_simple.swift]
2013-04-30 10:17:53,381+0200 DEBUG Loader Max heap: 954466304
2013-04-30 10:17:53,383+0200 DEBUG textfiles BEGIN SWIFTSCRIPT:
type Text;
app( Text o )mycat( Text input ) {
cat @input stdout=@o;
}
( Text[] oset )mycatMap( Text[] inputset ) {
foreach input, i in inputset {
oset[ i ] = mycat( input );
trace( @filename( oset[ i ] ) );
}
}
app( Text o )mycatReduce( Text[] inputset ) {
cat @inputset stdout=@o;
}
Text[] p <filesys_mapper; location="infileset", suffix=".txt">;
Text[] q <concurrent_mapper; location="outfileset", prefix="map_output",
suffix=".txt">;
Text r <"outfileset/reduce_output.txt">;
q = mycatMap( p ); // So far so good, but the output q is not persistent.
r = mycatReduce( p ); // Drops out with 'infileset/a.txt
infileset/b.txt: No such
// file or directory'
2013-04-30 10:17:53,383+0200 DEBUG textfiles END SWIFTSCRIPT:
2013-04-30 10:17:53,384+0200 INFO Loader test_simple.swift: source file
is new. Recompiling.
2013-04-30 10:17:54,647+0200 DEBUG Loader Swift configuration ([]):
{pgraph.node.options=color="seagreen", style="filled",
clustering.min.time=60, clustering.queue.delay=4,
sites.file=/home/jorgen/software/swift-0.94/bin/../etc/sites.xml,
ticker.prefix=Progress: time:, provenance.log=false,
kickstart.enabled=maybe,
tc.file=/home/jorgen/software/swift-0.94/bin/../etc/tc.data,
foreach.max.threads=16384, lazy.errors=false, ticker.date.format=,
sitedir.keep=false, throttle.score.job.factor=0.2,
kickstart.always.transfer=false, throttle.host.submit=2,
replication.limit=3, clustering.enabled=false, pgraph=false,
file.gc.enabled=true, replication.min.queue.time=60,
wrapper.parameter.mode=args, replication.enabled=false,
throttle.transfers=4, wrapperlog.always.transfer=false,
throttle.submit=4, pgraph.graph.options=splines="compound",
rankdir="TB", caching.algorithm=LRU,
logfile=test_simple-20130430-1017-sfans555.log,
use.wrapper.staging=false, tracing.enabled=true,
provider.staging.pin.swiftfiles=false, wrapper.invocation.mode=absolute,
mapping.checker=on, execution.retries=0, cdm.broadcast.mode=file,
status.mode=files, use.provider.staging=false, throttle.file.operations=8}
2013-04-30 10:17:54,647+0200 DEBUG textfiles using default sites file
2013-04-30 10:17:54,655+0200 DEBUG VDL2ExecutionContext Stack dump:
Level 1
[iA = 0, iB = 0, bA = false, bB = false]
vdl:instanceconfig = Swift configuration ([]):
{pgraph.node.options=color="seagreen", style="filled",
clustering.min.time=60, clustering.queue.delay=4,
sites.file=/home/jorgen/software/swift-0.94/bin/../etc/sites.xml,
ticker.prefix=Progress: time:, provenance.log=false,
kickstart.enabled=maybe,
tc.file=/home/jorgen/software/swift-0.94/bin/../etc/tc.data,
foreach.max.threads=16384, lazy.errors=false, ticker.date.format=,
sitedir.keep=false, throttle.score.job.factor=0.2,
kickstart.always.transfer=false, throttle.host.submit=2,
replication.limit=3, clustering.enabled=false, pgraph=false,
file.gc.enabled=true, replication.min.queue.time=60,
wrapper.parameter.mode=args, replication.enabled=false,
throttle.transfers=4, wrapperlog.always.transfer=false,
throttle.submit=4, pgraph.graph.options=splines="compound",
rankdir="TB", caching.algorithm=LRU,
logfile=test_simple-20130430-1017-sfans555.log,
use.wrapper.staging=false, tracing.enabled=true,
provider.staging.pin.swiftfiles=false, wrapper.invocation.mode=absolute,
mapping.checker=on, execution.retries=0, cdm.broadcast.mode=file,
status.mode=files, use.provider.staging=false, throttle.file.operations=8}
vdl:operation = run
swift.home = /home/jorgen/software/swift-0.94/bin/..
PATH_SEPARATOR = /
2013-04-30 10:17:54,657+0200 INFO VDL2ExecutionContext swift.home =
/home/jorgen/software/swift-0.94/bin/..
2013-04-30 10:17:55,636+0200 INFO swift Using sites file:
/home/jorgen/software/swift-0.94/bin/../etc/sites.xml
2013-04-30 10:17:55,702+0200 INFO swift Using tc.data:
/home/jorgen/software/swift-0.94/bin/../etc/tc.data
2013-04-30 10:17:55,830+0200 INFO AbstractScheduler Setting resources
to: {localhost=localhost}
2013-04-30 10:17:56,660+0200 INFO swift Swift 0.94 swift-r6320 cog-r3615
2013-04-30 10:17:56,661+0200 INFO swift RUNID id=run:20130430-1017-sfans555
2013-04-30 10:17:56,746+0200 INFO TRACE DECLARE, line 19, thread main, p
2013-04-30 10:17:56,760+0200 INFO TRACE VARIABLE, line 19, thread main,
p INITIALIZED <filesys_mapper; input = true, location = "infileset",
suffix = ".txt">
2013-04-30 10:17:56,777+0200 INFO TRACE VARIABLE, line 19, thread main,
p MAPPING [0], file://localhost/infileset/a.txt
2013-04-30 10:17:56,778+0200 INFO TRACE VARIABLE, line 19, thread main,
p MAPPING [1], file://localhost/infileset/b.txt
2013-04-30 10:17:56,784+0200 INFO TRACE DECLARE, line 20, thread main, q
2013-04-30 10:17:56,785+0200 INFO TRACE VARIABLE, line 20, thread main,
q INITIALIZED <concurrent_mapper; input = null, location = "outfileset",
prefix = _concurrent/map_output-, thread_prefix = , suffix = ".txt">
2013-04-30 10:17:56,786+0200 INFO TRACE VARIABLE, line 21, thread main,
r INITIALIZED <single_file_mapper; input = null, file =
outfileset/reduce_output.txt>
2013-04-30 10:17:56,787+0200 INFO TRACE DECLARE, line 21, thread main,
r <single_file_mapper; input = null, file = outfileset/reduce_output.txt>
2013-04-30 10:17:56,809+0200 INFO TRACE CALL, line 7, thread 0-5,
mycatMap(inputset = p)
2013-04-30 10:17:56,810+0200 INFO TRACE APPCALL, line 15, thread 0-6,
mycatReduce(inputset = p)
2013-04-30 10:17:56,811+0200 DEBUG swift PROCEDURE line=15 thread=0-6
name=mycat_reduce
2013-04-30 10:17:56,812+0200 INFO swift STARTCOMPOUND thread=0-5
name=mycat_map
2013-04-30 10:17:56,830+0200 INFO RuntimeStats$ProgressTicker
2013-04-30 10:17:56,833+0200 DEBUG swift FOREACH_IT_START line=9
thread=0-5-1-0
2013-04-30 10:17:56,836+0200 DEBUG swift SCOPE thread=0-5-1-0
2013-04-30 10:17:56,837+0200 DEBUG swift INTERNALPROC_START
thread=0-5-1-0-1 name=trace
2013-04-30 10:17:56,840+0200 WARN trace SwiftScript trace:
outfileset/_concurrent/map_output--array//elt-0.txt
2013-04-30 10:17:56,840+0200 DEBUG swift INTERNALPROC_END
thread=0-5-1-0-1 name=trace
2013-04-30 10:17:56,836+0200 DEBUG swift FOREACH_IT_START line=9
thread=0-5-1-1
2013-04-30 10:17:56,841+0200 DEBUG swift SCOPE thread=0-5-1-1
2013-04-30 10:17:56,841+0200 DEBUG swift INTERNALPROC_START
thread=0-5-1-1-1 name=trace
2013-04-30 10:17:56,842+0200 INFO TRACE APPCALL, line 3, thread
0-5-1-1-0, mycat(input = ?p[1])
2013-04-30 10:17:56,842+0200 DEBUG swift PROCEDURE line=3
thread=0-5-1-1-0 name=mycat
2013-04-30 10:17:56,843+0200 WARN trace SwiftScript trace:
outfileset/_concurrent/map_output--array//elt-1.txt
2013-04-30 10:17:56,838+0200 INFO TRACE APPCALL, line 3, thread
0-5-1-0-0, mycat(input = ?p[0])
2013-04-30 10:17:56,835+0200 INFO swift START thread=0-6 tr=cat
2013-04-30 10:17:56,849+0200 DEBUG swift PROCEDURE line=3
thread=0-5-1-0-0 name=mycat
2013-04-30 10:17:56,851+0200 INFO swift START thread=0-5-1-1-0 tr=cat
2013-04-30 10:17:56,859+0200 DEBUG swift INTERNALPROC_END
thread=0-5-1-1-1 name=trace
2013-04-30 10:17:56,855+0200 INFO swift START thread=0-5-1-0-0 tr=cat
2013-04-30 10:17:56,882+0200 INFO WeightedHostScoreScheduler Sorted:
[localhost:0.000(1.000):0/1 overload: 0]
2013-04-30 10:17:56,882+0200 INFO WeightedHostScoreScheduler
CONTACT_SELECTED host=localhost, score=1.000
2013-04-30 10:17:56,887+0200 INFO GlobalSubmitQueue No global submit
throttle set. Using default (1024)
2013-04-30 10:17:56,891+0200 INFO LateBindingScheduler jobs queued: 2
2013-04-30 10:17:56,893+0200 INFO LateBindingScheduler jobs queued: 2
2013-04-30 10:17:56,896+0200 INFO swift START host=localhost -
Initializing shared directory
2013-04-30 10:17:56,909+0200 INFO LateBindingScheduler jobs queued: 2
2013-04-30 10:17:56,931+0200 INFO LateBindingScheduler jobs queued: 2
2013-04-30 10:17:56,935+0200 INFO LateBindingScheduler jobs queued: 2
2013-04-30 10:17:56,936+0200 INFO LateBindingScheduler jobs queued: 2
2013-04-30 10:17:56,940+0200 INFO LateBindingScheduler jobs queued: 2
2013-04-30 10:17:56,943+0200 INFO LateBindingScheduler jobs queued: 2
2013-04-30 10:17:56,945+0200 INFO swift END host=localhost - Done
initializing shared directory
2013-04-30 10:17:56,947+0200 DEBUG swift THREAD_ASSOCIATION
jobid=cat-beb95s8l thread=0-5-1-0-0-1 host=localhost
replicationGroup=aeb95s8l
2013-04-30 10:17:56,961+0200 INFO swift START jobid=cat-beb95s8l
host=localhost - Initializing directory structure
2013-04-30 10:17:56,964+0200 INFO swift START path=infileset
dir=test_simple-20130430-1017-sfans555/shared - Creating directory structure
2013-04-30 10:17:56,965+0200 INFO LateBindingScheduler jobs queued: 2
2013-04-30 10:17:56,966+0200 INFO swift START
path=outfileset/_concurrent/map_output--array/
dir=test_simple-20130430-1017-sfans555/shared - Creating directory structure
2013-04-30 10:17:56,967+0200 INFO LateBindingScheduler jobs queued: 2
2013-04-30 10:17:56,968+0200 INFO swift END jobid=cat-beb95s8l - Done
initializing directory structure
2013-04-30 10:17:56,969+0200 INFO swift START jobid=cat-beb95s8l -
Staging in files
2013-04-30 10:17:56,981+0200 INFO LateBindingScheduler jobs queued: 2
2013-04-30 10:17:56,986+0200 DEBUG swift CDM:
file://localhost/infileset/a.txt : DEFAULT
2013-04-30 10:17:56,997+0200 DEBUG swift FILE_STAGE_IN_START file=a.txt
srchost=localhost srcdir=infileset srcname=a.txt desthost=localhost
destdir=test_simple-20130430-1017-sfans555/shared/infileset
provider=file policy=DEFAULT
2013-04-30 10:17:57,013+0200 INFO LateBindingScheduler jobs queued: 2
2013-04-30 10:17:57,014+0200 DEBUG swift FILE_STAGE_IN_END file=a.txt
srchost=localhost srcdir=infileset srcname=a.txt desthost=localhost
destdir=test_simple-20130430-1017-sfans555/shared/infileset provider=file
2013-04-30 10:17:57,015+0200 INFO swift END jobid=cat-beb95s8l -
Staging in finished
2013-04-30 10:17:57,017+0200 DEBUG swift JOB_START jobid=cat-beb95s8l
tr=cat arguments=[infileset/a.txt]
tmpdir=test_simple-20130430-1017-sfans555/jobs/b/cat-beb95s8l host=localhost
2013-04-30 10:17:57,022+0200 DEBUG textfiles BEGIN TC:
#This is the transformation catalog.
#
#It comes pre-configured with a number of simple transformations with
#paths that are likely to work on a linux box. However, on some systems,
#the paths to these executables will be different (for example, sometimes
#some of these programs are found in /usr/bin rather than in /bin)
#
#NOTE WELL: fields in this file must be separated by tabs, not spaces; and
#there must be no trailing whitespace at the end of each line.
#
# sitename transformation path INSTALLED platform profiles
localhost echo /bin/echo INSTALLED INTEL32::LINUX null
localhost cat /bin/cat INSTALLED INTEL32::LINUX null
localhost ls /bin/ls INSTALLED INTEL32::LINUX null
localhost grep /bin/grep INSTALLED INTEL32::LINUX null
localhost sort /bin/sort INSTALLED INTEL32::LINUX null
localhost paste /bin/paste INSTALLED INTEL32::LINUX null
localhost cp /bin/cp INSTALLED INTEL32::LINUX null
localhost touch /bin/touch INSTALLED INTEL32::LINUX null
localhost wc /usr/bin/wc INSTALLED INTEL32::LINUX null
localhost sleep /bin/sleep null null null
2013-04-30 10:17:57,022+0200 DEBUG textfiles END TC:
2013-04-30 10:17:57,048+0200 INFO GridExec TASK_DEFINITION:
Task(type=JOB_SUBMISSION, identity=urn:0-5-1-0-0-1-1367309875676) is
/bin/bash /var/tmp/test_simple-20130430-1017-sfans555/shared/_swiftwrap
cat-beb95s8l -jobdir b -scratch -e /bin/cat -out
outfileset/_concurrent/map_output--array//elt-0.txt -err stderr.txt -i
-d infileset|outfileset/_concurrent/map_output--array/ -if
infileset/a.txt -of outfileset/_concurrent/map_output--array//elt-0.txt
-k -cdmfile -status files -a infileset/a.txt
2013-04-30 10:17:57,054+0200 INFO JobSubmissionTaskHandler Submit: in:
/var/tmp/test_simple-20130430-1017-sfans555 command: /bin/bash
/var/tmp/test_simple-20130430-1017-sfans555/shared/_swiftwrap
cat-beb95s8l -jobdir b -scratch -e /bin/cat -out
outfileset/_concurrent/map_output--array//elt-0.txt -err stderr.txt -i
-d infileset|outfileset/_concurrent/map_output--array/ -if
infileset/a.txt -of outfileset/_concurrent/map_output--array//elt-0.txt
-k -cdmfile -status files -a infileset/a.txt
2013-04-30 10:17:57,165+0200 INFO LateBindingScheduler jobs queued: 2
2013-04-30 10:17:57,168+0200 DEBUG swift START jobid=cat-beb95s8l
2013-04-30 10:17:57,170+0200 INFO LateBindingScheduler jobs queued: 2
2013-04-30 10:17:57,171+0200 INFO swift SUCCESS jobid=cat-beb95s8l -
Success file found
2013-04-30 10:17:57,172+0200 DEBUG swift STAGING_OUT jobid=cat-beb95s8l
2013-04-30 10:17:57,174+0200 INFO swift START jobid=cat-beb95s8l -
Staging out files
2013-04-30 10:17:57,175+0200 DEBUG swift stageouts: [[$, q[0]:Text - Open]]
2013-04-30 10:17:57,183+0200 DEBUG swift FILE_STAGE_OUT_START
srcname=elt-0.txt
srcdir=test_simple-20130430-1017-sfans555/shared/outfileset/_concurrent/map_output--array/
srchost=localhost destdir=outfileset/_concurrent/map_output--array/
desthost=localhost provider=file
2013-04-30 10:17:57,187+0200 INFO LateBindingScheduler jobs queued: 2
2013-04-30 10:17:57,189+0200 DEBUG swift CDM:
file://localhost/outfileset/_concurrent/map_output--array//elt-0.txt :
DEFAULT
2013-04-30 10:17:57,193+0200 INFO LateBindingScheduler jobs queued: 2
2013-04-30 10:17:57,195+0200 DEBUG swift FILE_STAGE_OUT_END
srcname=elt-0.txt
srcdir=test_simple-20130430-1017-sfans555/shared/outfileset/_concurrent/map_output--array/
srchost=localhost destdir=outfileset/_concurrent/map_output--array/
desthost=localhost provider=file
2013-04-30 10:17:57,197+0200 INFO LateBindingScheduler jobs queued: 2
2013-04-30 10:17:57,201+0200 INFO swift END jobid=cat-beb95s8l -
Staging out finished
2013-04-30 10:17:57,206+0200 DEBUG swift JOB_END jobid=cat-beb95s8l
2013-04-30 10:17:57,207+0200 INFO swift END_SUCCESS thread=0-5-1-0-0 tr=cat
2013-04-30 10:17:57,209+0200 DEBUG swift PROCEDURE_END line=3
thread=0-5-1-0-0 name=mycat
2013-04-30 10:17:57,210+0200 DEBUG swift FOREACH_IT_END line=9
thread=0-5-1-0
2013-04-30 10:17:57,698+0200 INFO WeightedHostScoreScheduler Sorted:
[localhost:1.300(2.109):0/1 overload: 0]
2013-04-30 10:17:57,698+0200 INFO WeightedHostScoreScheduler
CONTACT_SELECTED host=localhost, score=2.109
2013-04-30 10:17:57,704+0200 DEBUG swift THREAD_ASSOCIATION
jobid=cat-ceb95s8l thread=0-6-1 host=localhost replicationGroup=8eb95s8l
2013-04-30 10:17:57,712+0200 INFO swift START jobid=cat-ceb95s8l
host=localhost - Initializing directory structure
2013-04-30 10:17:57,716+0200 INFO swift START path=outfileset
dir=test_simple-20130430-1017-sfans555/shared - Creating directory structure
2013-04-30 10:17:57,717+0200 INFO LateBindingScheduler jobs queued: 1
2013-04-30 10:17:57,719+0200 INFO swift END jobid=cat-ceb95s8l - Done
initializing directory structure
2013-04-30 10:17:57,720+0200 INFO swift START jobid=cat-ceb95s8l -
Staging in files
2013-04-30 10:17:57,725+0200 INFO LateBindingScheduler jobs queued: 1
2013-04-30 10:17:57,726+0200 INFO LateBindingScheduler jobs queued: 1
2013-04-30 10:17:57,728+0200 DEBUG swift CDM:
file://localhost/infileset/b.txt : DEFAULT
2013-04-30 10:17:57,728+0200 DEBUG swift CDM:
file://localhost/infileset/a.txt : DEFAULT
2013-04-30 10:17:57,730+0200 DEBUG swift FILE_STAGE_IN_START file=b.txt
srchost=localhost srcdir=infileset srcname=b.txt desthost=localhost
destdir=test_simple-20130430-1017-sfans555/shared/infileset
provider=file policy=DEFAULT
2013-04-30 10:17:57,734+0200 INFO LateBindingScheduler jobs queued: 1
2013-04-30 10:17:57,735+0200 DEBUG swift FILE_STAGE_IN_END file=b.txt
srchost=localhost srcdir=infileset srcname=b.txt desthost=localhost
destdir=test_simple-20130430-1017-sfans555/shared/infileset provider=file
2013-04-30 10:17:57,736+0200 INFO swift END jobid=cat-ceb95s8l -
Staging in finished
2013-04-30 10:17:57,737+0200 DEBUG swift JOB_START jobid=cat-ceb95s8l
tr=cat arguments=[infileset/a.txt infileset/b.txt]
tmpdir=test_simple-20130430-1017-sfans555/jobs/c/cat-ceb95s8l host=localhost
2013-04-30 10:17:57,742+0200 INFO GridExec TASK_DEFINITION:
Task(type=JOB_SUBMISSION, identity=urn:0-6-1-1367309875696) is /bin/bash
shared/_swiftwrap cat-ceb95s8l -jobdir c -scratch -e /bin/cat -out
outfileset/reduce_output.txt -err stderr.txt -i -d infileset|outfileset
-if infileset/a.txt|infileset/b.txt -of outfileset/reduce_output.txt -k
-cdmfile -status files -a infileset/a.txt infileset/b.txt
2013-04-30 10:17:57,743+0200 INFO JobSubmissionTaskHandler Submit: in:
/var/tmp/test_simple-20130430-1017-sfans555 command: /bin/bash
/var/tmp/test_simple-20130430-1017-sfans555/shared/_swiftwrap
cat-ceb95s8l -jobdir c -scratch -e /bin/cat -out
outfileset/reduce_output.txt -err stderr.txt -i -d infileset|outfileset
-if infileset/a.txt|infileset/b.txt -of outfileset/reduce_output.txt -k
-cdmfile -status files -a infileset/a.txt infileset/b.txt
2013-04-30 10:17:57,865+0200 INFO LateBindingScheduler jobs queued: 1
2013-04-30 10:17:57,867+0200 INFO RuntimeStats$ProgressTicker
Selecting site:1 Checking status:1 Finished successfully:1
2013-04-30 10:17:57,870+0200 DEBUG swift START jobid=cat-ceb95s8l
2013-04-30 10:17:57,879+0200 INFO LateBindingScheduler jobs queued: 1
2013-04-30 10:17:57,880+0200 INFO swift FAILURE jobid=cat-ceb95s8l -
Failure file found
2013-04-30 10:17:57,884+0200 INFO LateBindingScheduler jobs queued: 1
2013-04-30 10:17:57,889+0200 INFO LateBindingScheduler jobs queued: 1
2013-04-30 10:17:57,892+0200 INFO LateBindingScheduler jobs queued: 1
2013-04-30 10:17:57,895+0200 DEBUG swift STAGING_OUT jobid=cat-ceb95s8l
2013-04-30 10:17:57,896+0200 INFO swift START jobid=cat-ceb95s8l -
Staging out files
2013-04-30 10:17:57,897+0200 DEBUG swift stageouts: [[$, r:Text - Open]]
2013-04-30 10:17:57,907+0200 DEBUG swift FILE_STAGE_OUT_START
srcname=reduce_output.txt
srcdir=test_simple-20130430-1017-sfans555/shared/outfileset
srchost=localhost destdir=outfileset desthost=localhost provider=file
2013-04-30 10:17:57,909+0200 INFO LateBindingScheduler jobs queued: 1
2013-04-30 10:17:57,910+0200 DEBUG swift CDM:
file://localhost/outfileset/reduce_output.txt : DEFAULT
2013-04-30 10:17:57,927+0200 INFO LateBindingScheduler jobs queued: 1
2013-04-30 10:17:57,928+0200 INFO swift NO_STATUS_FILE
jobid=cat-ceb95s8l - Error file missing
2013-04-30 10:17:57,929+0200 DEBUG swift APPLICATION_EXCEPTION
jobid=cat-ceb95s8l - Application exception: Caused by: File not found:
/var/tmp/test_simple-20130430-1017-sfans555/shared/outfileset/reduce_output.txt
2013-04-30 10:17:57,941+0200 INFO LateBindingScheduler jobs queued: 1
2013-04-30 10:17:57,948+0200 INFO LateBindingScheduler jobs queued: 1
2013-04-30 10:17:57,952+0200 INFO LateBindingScheduler jobs queued: 1
2013-04-30 10:17:57,955+0200 INFO LateBindingScheduler jobs queued: 1
2013-04-30 10:17:57,958+0200 INFO LateBindingScheduler jobs queued: 1
2013-04-30 10:17:57,962+0200 INFO swift END_FAILURE thread=0-6 tr=cat
2013-04-30 10:17:57,965+0200 DEBUG VDL2ExecutionContext Karajan level
error: Exception in cat:
Arguments: [infileset/a.txt infileset/b.txt]
Host: localhost
Directory: test_simple-20130430-1017-sfans555/jobs/c/cat-ceb95s8l
stderr.txt: /bin/cat: infileset/a.txt infileset/b.txt: No such file
or directory
stdout.txt: at
sys:exception @ vdl-int.k, line: 599
sys:throw @ vdl-int.k, line: 598
sys:catch @ vdl-int.k, line: 565
sys:try @ vdl-int.k, line: 467
task:allocatehost @ vdl-int.k, line: 417
vdl:execute2 @ execute-default.k, line: 23
sys:ignoreerrors @ execute-default.k, line: 21
sys:parallelfor @ execute-default.k, line: 20
sys:restartonerror @ execute-default.k, line: 16
vdl:throttled @ execute-default.k, line: 14
sys:try @ execute-default.k, line: 13
sys:if @ execute-default.k, line: 12
sys:then @ execute-default.k, line: 11
sys:if @ execute-default.k, line: 10
vdl:execute @ test_simple.kml, line: 109
mycat_reduce @ test_simple.kml, line: 223
sys:uparallel @ test_simple.kml, line: 179
vdl:mainp @ test_simple.kml, line: 178
mainp @ vdl.k, line: 118
vdl:mains @ test_simple.kml, line: 176
vdl:mains @ test_simple.kml, line: 176
rlog:restartlog @ test_simple.kml, line: 175
kernel:project @ test_simple.kml, line: 2
test_simple-20130430-1017-sfans555
caused by: null at
task:transfer @ vdl-int.k, line: 310
sys:restartonerror @ vdl-int.k, line: 309
sys:then @ vdl-int.k, line: 309
sys:if @ vdl-int.k, line: 307
sys:uparallelfor @ vdl-int.k, line: 290
list:list @ vdl-int.k, line: 290
sys:set @ vdl-int.k, line: 289
vdl:dostageout @ vdl-int.k, line: 545
sys:sequential @ vdl-int.k, line: 468
sys:try @ vdl-int.k, line: 467
task:allocatehost @ vdl-int.k, line: 417
vdl:execute2 @ execute-default.k, line: 23
sys:ignoreerrors @ execute-default.k, line: 21
sys:parallelfor @ execute-default.k, line: 20
sys:restartonerror @ execute-default.k, line: 16
vdl:throttled @ execute-default.k, line: 14
sys:try @ execute-default.k, line: 13
sys:if @ execute-default.k, line: 12
sys:then @ execute-default.k, line: 11
sys:if @ execute-default.k, line: 10
vdl:execute @ test_simple.kml, line: 109
mycat_reduce @ test_simple.kml, line: 223
sys:uparallel @ test_simple.kml, line: 179
vdl:mainp @ test_simple.kml, line: 178
mainp @ vdl.k, line: 118
vdl:mains @ test_simple.kml, line: 176
vdl:mains @ test_simple.kml, line: 176
rlog:restartlog @ test_simple.kml, line: 175
kernel:project @ test_simple.kml, line: 2
test_simple-20130430-1017-sfans555
2013-04-30 10:17:58,207+0200 INFO ExecutionContext Detailed exception:
Exception in cat:
Arguments: [infileset/a.txt infileset/b.txt]
Host: localhost
Directory: test_simple-20130430-1017-sfans555/jobs/c/cat-ceb95s8l
stderr.txt: /bin/cat: infileset/a.txt infileset/b.txt: No such file
or directory
stdout.txt:
Caused by: File not found:
/var/tmp/test_simple-20130430-1017-sfans555/shared/outfileset/reduce_output.txt
at
org.globus.cog.karajan.workflow.nodes.functions.KException.function(KException.java:29)
at
org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27)
at
org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
at
org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
at
org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
at
org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
at
org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29)
at
org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20)
at
org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
at
org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139)
at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197)
at
org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:227)
at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:98)
at
org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Caused by: Caused by: File not found:
/var/tmp/test_simple-20130430-1017-sfans555/shared/outfileset/reduce_output.txt
at
org.globus.cog.karajan.workflow.nodes.FlowNode.failImmediately(FlowNode.java:101)
at
org.globus.cog.karajan.workflow.nodes.grid.AbstractGridNode$1.run(AbstractGridNode.java:308)
... 6 more
Caused by: org.globus.cog.abstraction.impl.file.FileNotFoundException:
File not found:
/var/tmp/test_simple-20130430-1017-sfans555/shared/outfileset/reduce_output.txt
at
org.globus.cog.abstraction.impl.file.local.FileResourceImpl.getFile(FileResourceImpl.java:225)
at
org.globus.cog.abstraction.impl.file.local.FileResourceImpl.putFile(FileResourceImpl.java:286)
at
org.globus.cog.abstraction.impl.file.AbstractFileResource.putFile(AbstractFileResource.java:210)
at
org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(DelegatedFileTransferHandler.java:307)
at
org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestination(CachingDelegatedFileTransferHandler.java:46)
at
org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTransferHandler.java:482)
... 1 more
2013-04-30 10:17:58,211+0200 INFO Loader Swift finished with errors
--- End Listing ---------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 899 bytes
Desc: OpenPGP digital signature
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20130430/41b14646/attachment.pgp>
More information about the Swift-user
mailing list