From joergen.brandt at onlinehome.de Tue Apr 30 03:31:20 2013 From: joergen.brandt at onlinehome.de (Jorgen Brandt) Date: Tue, 30 Apr 2013 08:31:20 -0000 Subject: [Swift-user] Questions about concurrent_mapper and parameter lists Message-ID: <517F8145.6090706@onlinehome.de> Dear Sir or Madam, I just started using Swift. There is several things, though, I cannot figure out. - I want to perform a computation in parallel, using the filesys_mapper to parse my input directory and the concurrent_mapper to create the corresponding output files. The output files are created temporarily but they are not there any more, when the execution finishes. What do I do wrong? (See listings below.) - I want to append an array of filenames (of which I do not know its length at compile-time) to the parameter list of an app. I tried several things but had no luck so far. I'd expected that something like type Text; app( Text o )mycatReduce( Text[] inputset ) { cat @inputset stdout=@o; } would work and indeed the above snippet survives compilation. But the stage-in does not seem to work. Cat complains about its input not being there. (Listings below.) I've been googling around but the only thing I've found that may relate to this problem is your article in Computer 42 (11) from 2009 in which it says << Swift's foreach construct performs a simple map operation, and the act of passing a multimember dataset to a procedure provides a simple and natural way to implement reduction operations. >> No example was given though. I appreciate any help and hope not to steal too much of your time. Best regards Jorgen The rest of this email explains in detail what I have done: I am currently working with Swift 0.94 (RC4). I have a working directory containing a folder 'infileset'. this folder contains 2 files: 'a.txt' and 'b.txt'. --- Start Listing ------------------------------------------- jorgen at turing:~/workspace_set/swift/test$ ls infileset a.txt b.txt jorgen at turing:~/workspace_set/swift/test$ cat infileset/a.txt Hello jorgen at turing:~/workspace_set/swift/test$ cat infileset/b.txt World --- End Listing --------------------------------------------- In this working directory I also store my swift-script. --- Start Listing ------------------------------------------- jorgen at turing:~/workspace_set/swift/test$ cat test_simple.swift type Text; app( Text o )mycat( Text input ) { cat @input stdout=@o; } ( Text[] oset )mycatMap( Text[] inputset ) { foreach input, i in inputset { oset[ i ] = mycat( input ); trace( @filename( oset[ i ] ) ); } } app( Text o )mycatReduce( Text[] inputset ) { cat @inputset stdout=@o; } Text[] p ; Text[] q ; Text r <"outfileset/reduce_output.txt">; q = mycatMap( p ); // So far so good, but the output q is not persistent. r = mycatReduce( p ); // Drops out with 'infileset/a.txt infileset/b.txt: No such // file or directory' --- End Listing --------------------------------------------- Running this script yields the following: --- Start Listing ------------------------------------------- jorgen at turing:~/workspace_set/swift/test$ ~/software/swift-0.94/bin/swift test_simple.swift Swift 0.94 swift-r6320 cog-r3615 RunID: 20130430-1017-sfans555 Progress: time: Tue, 30 Apr 2013 10:17:56 +0200 SwiftScript trace: outfileset/_concurrent/map_output--array//elt-0.txt SwiftScript trace: outfileset/_concurrent/map_output--array//elt-1.txt Progress: time: Tue, 30 Apr 2013 10:17:57 +0200 Selecting site:1 Checking status:1 Finished successfully:1 Execution failed: Exception in cat: Arguments: [infileset/a.txt infileset/b.txt] Host: localhost Directory: test_simple-20130430-1017-sfans555/jobs/c/cat-ceb95s8l stderr.txt: /bin/cat: infileset/a.txt infileset/b.txt: No such file or directory stdout.txt: Caused by: File not found: /var/tmp/test_simple-20130430-1017-sfans555/shared/outfileset/reduce_output.txt mycatReduce, test_simple.swift, line 25 --- End Listing --------------------------------------------- The script created an .rlog file called test_simple-20130430-1017-sfans555.0.rlog. --- Start Listing ------------------------------------------- jorgen at turing:~/workspace_set/swift/test$ cat test_simple-20130430-1017-sfans555.0.rlog # Log file created Tue Apr 30 10:17:56 CEST 2013 :q.[0]!file://localhost/outfileset/_concurrent/map_output--array//elt-0.txt --- End Listing --------------------------------------------- Also, the script created a swift.log file with the following content: --- Start Listing ------------------------------------------- jorgen at turing:~/workspace_set/swift/test$ cat swift.log 2013-04-30 10:17:53,338+0200 DEBUG Loader Swift started --- End Listing --------------------------------------------- Eventually, the script created a log file called test_simple-20130430-1017-sfans555.log. --- Start Listing ------------------------------------------- jorgen at turing:~/workspace_set/swift/test$ cat test_simple-20130430-1017-sfans555.log 2013-04-30 10:17:53,377+0200 DEBUG Loader arguments: [test_simple.swift] 2013-04-30 10:17:53,381+0200 DEBUG Loader Max heap: 954466304 2013-04-30 10:17:53,383+0200 DEBUG textfiles BEGIN SWIFTSCRIPT: type Text; app( Text o )mycat( Text input ) { cat @input stdout=@o; } ( Text[] oset )mycatMap( Text[] inputset ) { foreach input, i in inputset { oset[ i ] = mycat( input ); trace( @filename( oset[ i ] ) ); } } app( Text o )mycatReduce( Text[] inputset ) { cat @inputset stdout=@o; } Text[] p ; Text[] q ; Text r <"outfileset/reduce_output.txt">; q = mycatMap( p ); // So far so good, but the output q is not persistent. r = mycatReduce( p ); // Drops out with 'infileset/a.txt infileset/b.txt: No such // file or directory' 2013-04-30 10:17:53,383+0200 DEBUG textfiles END SWIFTSCRIPT: 2013-04-30 10:17:53,384+0200 INFO Loader test_simple.swift: source file is new. Recompiling. 2013-04-30 10:17:54,647+0200 DEBUG Loader Swift configuration ([]): {pgraph.node.options=color="seagreen", style="filled", clustering.min.time=60, clustering.queue.delay=4, sites.file=/home/jorgen/software/swift-0.94/bin/../etc/sites.xml, ticker.prefix=Progress: time:, provenance.log=false, kickstart.enabled=maybe, tc.file=/home/jorgen/software/swift-0.94/bin/../etc/tc.data, foreach.max.threads=16384, lazy.errors=false, ticker.date.format=, sitedir.keep=false, throttle.score.job.factor=0.2, kickstart.always.transfer=false, throttle.host.submit=2, replication.limit=3, clustering.enabled=false, pgraph=false, file.gc.enabled=true, replication.min.queue.time=60, wrapper.parameter.mode=args, replication.enabled=false, throttle.transfers=4, wrapperlog.always.transfer=false, throttle.submit=4, pgraph.graph.options=splines="compound", rankdir="TB", caching.algorithm=LRU, logfile=test_simple-20130430-1017-sfans555.log, use.wrapper.staging=false, tracing.enabled=true, provider.staging.pin.swiftfiles=false, wrapper.invocation.mode=absolute, mapping.checker=on, execution.retries=0, cdm.broadcast.mode=file, status.mode=files, use.provider.staging=false, throttle.file.operations=8} 2013-04-30 10:17:54,647+0200 DEBUG textfiles using default sites file 2013-04-30 10:17:54,655+0200 DEBUG VDL2ExecutionContext Stack dump: Level 1 [iA = 0, iB = 0, bA = false, bB = false] vdl:instanceconfig = Swift configuration ([]): {pgraph.node.options=color="seagreen", style="filled", clustering.min.time=60, clustering.queue.delay=4, sites.file=/home/jorgen/software/swift-0.94/bin/../etc/sites.xml, ticker.prefix=Progress: time:, provenance.log=false, kickstart.enabled=maybe, tc.file=/home/jorgen/software/swift-0.94/bin/../etc/tc.data, foreach.max.threads=16384, lazy.errors=false, ticker.date.format=, sitedir.keep=false, throttle.score.job.factor=0.2, kickstart.always.transfer=false, throttle.host.submit=2, replication.limit=3, clustering.enabled=false, pgraph=false, file.gc.enabled=true, replication.min.queue.time=60, wrapper.parameter.mode=args, replication.enabled=false, throttle.transfers=4, wrapperlog.always.transfer=false, throttle.submit=4, pgraph.graph.options=splines="compound", rankdir="TB", caching.algorithm=LRU, logfile=test_simple-20130430-1017-sfans555.log, use.wrapper.staging=false, tracing.enabled=true, provider.staging.pin.swiftfiles=false, wrapper.invocation.mode=absolute, mapping.checker=on, execution.retries=0, cdm.broadcast.mode=file, status.mode=files, use.provider.staging=false, throttle.file.operations=8} vdl:operation = run swift.home = /home/jorgen/software/swift-0.94/bin/.. PATH_SEPARATOR = / 2013-04-30 10:17:54,657+0200 INFO VDL2ExecutionContext swift.home = /home/jorgen/software/swift-0.94/bin/.. 2013-04-30 10:17:55,636+0200 INFO swift Using sites file: /home/jorgen/software/swift-0.94/bin/../etc/sites.xml 2013-04-30 10:17:55,702+0200 INFO swift Using tc.data: /home/jorgen/software/swift-0.94/bin/../etc/tc.data 2013-04-30 10:17:55,830+0200 INFO AbstractScheduler Setting resources to: {localhost=localhost} 2013-04-30 10:17:56,660+0200 INFO swift Swift 0.94 swift-r6320 cog-r3615 2013-04-30 10:17:56,661+0200 INFO swift RUNID id=run:20130430-1017-sfans555 2013-04-30 10:17:56,746+0200 INFO TRACE DECLARE, line 19, thread main, p 2013-04-30 10:17:56,760+0200 INFO TRACE VARIABLE, line 19, thread main, p INITIALIZED 2013-04-30 10:17:56,777+0200 INFO TRACE VARIABLE, line 19, thread main, p MAPPING [0], file://localhost/infileset/a.txt 2013-04-30 10:17:56,778+0200 INFO TRACE VARIABLE, line 19, thread main, p MAPPING [1], file://localhost/infileset/b.txt 2013-04-30 10:17:56,784+0200 INFO TRACE DECLARE, line 20, thread main, q 2013-04-30 10:17:56,785+0200 INFO TRACE VARIABLE, line 20, thread main, q INITIALIZED 2013-04-30 10:17:56,786+0200 INFO TRACE VARIABLE, line 21, thread main, r INITIALIZED 2013-04-30 10:17:56,787+0200 INFO TRACE DECLARE, line 21, thread main, r 2013-04-30 10:17:56,809+0200 INFO TRACE CALL, line 7, thread 0-5, mycatMap(inputset = p) 2013-04-30 10:17:56,810+0200 INFO TRACE APPCALL, line 15, thread 0-6, mycatReduce(inputset = p) 2013-04-30 10:17:56,811+0200 DEBUG swift PROCEDURE line=15 thread=0-6 name=mycat_reduce 2013-04-30 10:17:56,812+0200 INFO swift STARTCOMPOUND thread=0-5 name=mycat_map 2013-04-30 10:17:56,830+0200 INFO RuntimeStats$ProgressTicker 2013-04-30 10:17:56,833+0200 DEBUG swift FOREACH_IT_START line=9 thread=0-5-1-0 2013-04-30 10:17:56,836+0200 DEBUG swift SCOPE thread=0-5-1-0 2013-04-30 10:17:56,837+0200 DEBUG swift INTERNALPROC_START thread=0-5-1-0-1 name=trace 2013-04-30 10:17:56,840+0200 WARN trace SwiftScript trace: outfileset/_concurrent/map_output--array//elt-0.txt 2013-04-30 10:17:56,840+0200 DEBUG swift INTERNALPROC_END thread=0-5-1-0-1 name=trace 2013-04-30 10:17:56,836+0200 DEBUG swift FOREACH_IT_START line=9 thread=0-5-1-1 2013-04-30 10:17:56,841+0200 DEBUG swift SCOPE thread=0-5-1-1 2013-04-30 10:17:56,841+0200 DEBUG swift INTERNALPROC_START thread=0-5-1-1-1 name=trace 2013-04-30 10:17:56,842+0200 INFO TRACE APPCALL, line 3, thread 0-5-1-1-0, mycat(input = ?p[1]) 2013-04-30 10:17:56,842+0200 DEBUG swift PROCEDURE line=3 thread=0-5-1-1-0 name=mycat 2013-04-30 10:17:56,843+0200 WARN trace SwiftScript trace: outfileset/_concurrent/map_output--array//elt-1.txt 2013-04-30 10:17:56,838+0200 INFO TRACE APPCALL, line 3, thread 0-5-1-0-0, mycat(input = ?p[0]) 2013-04-30 10:17:56,835+0200 INFO swift START thread=0-6 tr=cat 2013-04-30 10:17:56,849+0200 DEBUG swift PROCEDURE line=3 thread=0-5-1-0-0 name=mycat 2013-04-30 10:17:56,851+0200 INFO swift START thread=0-5-1-1-0 tr=cat 2013-04-30 10:17:56,859+0200 DEBUG swift INTERNALPROC_END thread=0-5-1-1-1 name=trace 2013-04-30 10:17:56,855+0200 INFO swift START thread=0-5-1-0-0 tr=cat 2013-04-30 10:17:56,882+0200 INFO WeightedHostScoreScheduler Sorted: [localhost:0.000(1.000):0/1 overload: 0] 2013-04-30 10:17:56,882+0200 INFO WeightedHostScoreScheduler CONTACT_SELECTED host=localhost, score=1.000 2013-04-30 10:17:56,887+0200 INFO GlobalSubmitQueue No global submit throttle set. Using default (1024) 2013-04-30 10:17:56,891+0200 INFO LateBindingScheduler jobs queued: 2 2013-04-30 10:17:56,893+0200 INFO LateBindingScheduler jobs queued: 2 2013-04-30 10:17:56,896+0200 INFO swift START host=localhost - Initializing shared directory 2013-04-30 10:17:56,909+0200 INFO LateBindingScheduler jobs queued: 2 2013-04-30 10:17:56,931+0200 INFO LateBindingScheduler jobs queued: 2 2013-04-30 10:17:56,935+0200 INFO LateBindingScheduler jobs queued: 2 2013-04-30 10:17:56,936+0200 INFO LateBindingScheduler jobs queued: 2 2013-04-30 10:17:56,940+0200 INFO LateBindingScheduler jobs queued: 2 2013-04-30 10:17:56,943+0200 INFO LateBindingScheduler jobs queued: 2 2013-04-30 10:17:56,945+0200 INFO swift END host=localhost - Done initializing shared directory 2013-04-30 10:17:56,947+0200 DEBUG swift THREAD_ASSOCIATION jobid=cat-beb95s8l thread=0-5-1-0-0-1 host=localhost replicationGroup=aeb95s8l 2013-04-30 10:17:56,961+0200 INFO swift START jobid=cat-beb95s8l host=localhost - Initializing directory structure 2013-04-30 10:17:56,964+0200 INFO swift START path=infileset dir=test_simple-20130430-1017-sfans555/shared - Creating directory structure 2013-04-30 10:17:56,965+0200 INFO LateBindingScheduler jobs queued: 2 2013-04-30 10:17:56,966+0200 INFO swift START path=outfileset/_concurrent/map_output--array/ dir=test_simple-20130430-1017-sfans555/shared - Creating directory structure 2013-04-30 10:17:56,967+0200 INFO LateBindingScheduler jobs queued: 2 2013-04-30 10:17:56,968+0200 INFO swift END jobid=cat-beb95s8l - Done initializing directory structure 2013-04-30 10:17:56,969+0200 INFO swift START jobid=cat-beb95s8l - Staging in files 2013-04-30 10:17:56,981+0200 INFO LateBindingScheduler jobs queued: 2 2013-04-30 10:17:56,986+0200 DEBUG swift CDM: file://localhost/infileset/a.txt : DEFAULT 2013-04-30 10:17:56,997+0200 DEBUG swift FILE_STAGE_IN_START file=a.txt srchost=localhost srcdir=infileset srcname=a.txt desthost=localhost destdir=test_simple-20130430-1017-sfans555/shared/infileset provider=file policy=DEFAULT 2013-04-30 10:17:57,013+0200 INFO LateBindingScheduler jobs queued: 2 2013-04-30 10:17:57,014+0200 DEBUG swift FILE_STAGE_IN_END file=a.txt srchost=localhost srcdir=infileset srcname=a.txt desthost=localhost destdir=test_simple-20130430-1017-sfans555/shared/infileset provider=file 2013-04-30 10:17:57,015+0200 INFO swift END jobid=cat-beb95s8l - Staging in finished 2013-04-30 10:17:57,017+0200 DEBUG swift JOB_START jobid=cat-beb95s8l tr=cat arguments=[infileset/a.txt] tmpdir=test_simple-20130430-1017-sfans555/jobs/b/cat-beb95s8l host=localhost 2013-04-30 10:17:57,022+0200 DEBUG textfiles BEGIN TC: #This is the transformation catalog. # #It comes pre-configured with a number of simple transformations with #paths that are likely to work on a linux box. However, on some systems, #the paths to these executables will be different (for example, sometimes #some of these programs are found in /usr/bin rather than in /bin) # #NOTE WELL: fields in this file must be separated by tabs, not spaces; and #there must be no trailing whitespace at the end of each line. # # sitename transformation path INSTALLED platform profiles localhost echo /bin/echo INSTALLED INTEL32::LINUX null localhost cat /bin/cat INSTALLED INTEL32::LINUX null localhost ls /bin/ls INSTALLED INTEL32::LINUX null localhost grep /bin/grep INSTALLED INTEL32::LINUX null localhost sort /bin/sort INSTALLED INTEL32::LINUX null localhost paste /bin/paste INSTALLED INTEL32::LINUX null localhost cp /bin/cp INSTALLED INTEL32::LINUX null localhost touch /bin/touch INSTALLED INTEL32::LINUX null localhost wc /usr/bin/wc INSTALLED INTEL32::LINUX null localhost sleep /bin/sleep null null null 2013-04-30 10:17:57,022+0200 DEBUG textfiles END TC: 2013-04-30 10:17:57,048+0200 INFO GridExec TASK_DEFINITION: Task(type=JOB_SUBMISSION, identity=urn:0-5-1-0-0-1-1367309875676) is /bin/bash /var/tmp/test_simple-20130430-1017-sfans555/shared/_swiftwrap cat-beb95s8l -jobdir b -scratch -e /bin/cat -out outfileset/_concurrent/map_output--array//elt-0.txt -err stderr.txt -i -d infileset|outfileset/_concurrent/map_output--array/ -if infileset/a.txt -of outfileset/_concurrent/map_output--array//elt-0.txt -k -cdmfile -status files -a infileset/a.txt 2013-04-30 10:17:57,054+0200 INFO JobSubmissionTaskHandler Submit: in: /var/tmp/test_simple-20130430-1017-sfans555 command: /bin/bash /var/tmp/test_simple-20130430-1017-sfans555/shared/_swiftwrap cat-beb95s8l -jobdir b -scratch -e /bin/cat -out outfileset/_concurrent/map_output--array//elt-0.txt -err stderr.txt -i -d infileset|outfileset/_concurrent/map_output--array/ -if infileset/a.txt -of outfileset/_concurrent/map_output--array//elt-0.txt -k -cdmfile -status files -a infileset/a.txt 2013-04-30 10:17:57,165+0200 INFO LateBindingScheduler jobs queued: 2 2013-04-30 10:17:57,168+0200 DEBUG swift START jobid=cat-beb95s8l 2013-04-30 10:17:57,170+0200 INFO LateBindingScheduler jobs queued: 2 2013-04-30 10:17:57,171+0200 INFO swift SUCCESS jobid=cat-beb95s8l - Success file found 2013-04-30 10:17:57,172+0200 DEBUG swift STAGING_OUT jobid=cat-beb95s8l 2013-04-30 10:17:57,174+0200 INFO swift START jobid=cat-beb95s8l - Staging out files 2013-04-30 10:17:57,175+0200 DEBUG swift stageouts: [[$, q[0]:Text - Open]] 2013-04-30 10:17:57,183+0200 DEBUG swift FILE_STAGE_OUT_START srcname=elt-0.txt srcdir=test_simple-20130430-1017-sfans555/shared/outfileset/_concurrent/map_output--array/ srchost=localhost destdir=outfileset/_concurrent/map_output--array/ desthost=localhost provider=file 2013-04-30 10:17:57,187+0200 INFO LateBindingScheduler jobs queued: 2 2013-04-30 10:17:57,189+0200 DEBUG swift CDM: file://localhost/outfileset/_concurrent/map_output--array//elt-0.txt : DEFAULT 2013-04-30 10:17:57,193+0200 INFO LateBindingScheduler jobs queued: 2 2013-04-30 10:17:57,195+0200 DEBUG swift FILE_STAGE_OUT_END srcname=elt-0.txt srcdir=test_simple-20130430-1017-sfans555/shared/outfileset/_concurrent/map_output--array/ srchost=localhost destdir=outfileset/_concurrent/map_output--array/ desthost=localhost provider=file 2013-04-30 10:17:57,197+0200 INFO LateBindingScheduler jobs queued: 2 2013-04-30 10:17:57,201+0200 INFO swift END jobid=cat-beb95s8l - Staging out finished 2013-04-30 10:17:57,206+0200 DEBUG swift JOB_END jobid=cat-beb95s8l 2013-04-30 10:17:57,207+0200 INFO swift END_SUCCESS thread=0-5-1-0-0 tr=cat 2013-04-30 10:17:57,209+0200 DEBUG swift PROCEDURE_END line=3 thread=0-5-1-0-0 name=mycat 2013-04-30 10:17:57,210+0200 DEBUG swift FOREACH_IT_END line=9 thread=0-5-1-0 2013-04-30 10:17:57,698+0200 INFO WeightedHostScoreScheduler Sorted: [localhost:1.300(2.109):0/1 overload: 0] 2013-04-30 10:17:57,698+0200 INFO WeightedHostScoreScheduler CONTACT_SELECTED host=localhost, score=2.109 2013-04-30 10:17:57,704+0200 DEBUG swift THREAD_ASSOCIATION jobid=cat-ceb95s8l thread=0-6-1 host=localhost replicationGroup=8eb95s8l 2013-04-30 10:17:57,712+0200 INFO swift START jobid=cat-ceb95s8l host=localhost - Initializing directory structure 2013-04-30 10:17:57,716+0200 INFO swift START path=outfileset dir=test_simple-20130430-1017-sfans555/shared - Creating directory structure 2013-04-30 10:17:57,717+0200 INFO LateBindingScheduler jobs queued: 1 2013-04-30 10:17:57,719+0200 INFO swift END jobid=cat-ceb95s8l - Done initializing directory structure 2013-04-30 10:17:57,720+0200 INFO swift START jobid=cat-ceb95s8l - Staging in files 2013-04-30 10:17:57,725+0200 INFO LateBindingScheduler jobs queued: 1 2013-04-30 10:17:57,726+0200 INFO LateBindingScheduler jobs queued: 1 2013-04-30 10:17:57,728+0200 DEBUG swift CDM: file://localhost/infileset/b.txt : DEFAULT 2013-04-30 10:17:57,728+0200 DEBUG swift CDM: file://localhost/infileset/a.txt : DEFAULT 2013-04-30 10:17:57,730+0200 DEBUG swift FILE_STAGE_IN_START file=b.txt srchost=localhost srcdir=infileset srcname=b.txt desthost=localhost destdir=test_simple-20130430-1017-sfans555/shared/infileset provider=file policy=DEFAULT 2013-04-30 10:17:57,734+0200 INFO LateBindingScheduler jobs queued: 1 2013-04-30 10:17:57,735+0200 DEBUG swift FILE_STAGE_IN_END file=b.txt srchost=localhost srcdir=infileset srcname=b.txt desthost=localhost destdir=test_simple-20130430-1017-sfans555/shared/infileset provider=file 2013-04-30 10:17:57,736+0200 INFO swift END jobid=cat-ceb95s8l - Staging in finished 2013-04-30 10:17:57,737+0200 DEBUG swift JOB_START jobid=cat-ceb95s8l tr=cat arguments=[infileset/a.txt infileset/b.txt] tmpdir=test_simple-20130430-1017-sfans555/jobs/c/cat-ceb95s8l host=localhost 2013-04-30 10:17:57,742+0200 INFO GridExec TASK_DEFINITION: Task(type=JOB_SUBMISSION, identity=urn:0-6-1-1367309875696) is /bin/bash shared/_swiftwrap cat-ceb95s8l -jobdir c -scratch -e /bin/cat -out outfileset/reduce_output.txt -err stderr.txt -i -d infileset|outfileset -if infileset/a.txt|infileset/b.txt -of outfileset/reduce_output.txt -k -cdmfile -status files -a infileset/a.txt infileset/b.txt 2013-04-30 10:17:57,743+0200 INFO JobSubmissionTaskHandler Submit: in: /var/tmp/test_simple-20130430-1017-sfans555 command: /bin/bash /var/tmp/test_simple-20130430-1017-sfans555/shared/_swiftwrap cat-ceb95s8l -jobdir c -scratch -e /bin/cat -out outfileset/reduce_output.txt -err stderr.txt -i -d infileset|outfileset -if infileset/a.txt|infileset/b.txt -of outfileset/reduce_output.txt -k -cdmfile -status files -a infileset/a.txt infileset/b.txt 2013-04-30 10:17:57,865+0200 INFO LateBindingScheduler jobs queued: 1 2013-04-30 10:17:57,867+0200 INFO RuntimeStats$ProgressTicker Selecting site:1 Checking status:1 Finished successfully:1 2013-04-30 10:17:57,870+0200 DEBUG swift START jobid=cat-ceb95s8l 2013-04-30 10:17:57,879+0200 INFO LateBindingScheduler jobs queued: 1 2013-04-30 10:17:57,880+0200 INFO swift FAILURE jobid=cat-ceb95s8l - Failure file found 2013-04-30 10:17:57,884+0200 INFO LateBindingScheduler jobs queued: 1 2013-04-30 10:17:57,889+0200 INFO LateBindingScheduler jobs queued: 1 2013-04-30 10:17:57,892+0200 INFO LateBindingScheduler jobs queued: 1 2013-04-30 10:17:57,895+0200 DEBUG swift STAGING_OUT jobid=cat-ceb95s8l 2013-04-30 10:17:57,896+0200 INFO swift START jobid=cat-ceb95s8l - Staging out files 2013-04-30 10:17:57,897+0200 DEBUG swift stageouts: [[$, r:Text - Open]] 2013-04-30 10:17:57,907+0200 DEBUG swift FILE_STAGE_OUT_START srcname=reduce_output.txt srcdir=test_simple-20130430-1017-sfans555/shared/outfileset srchost=localhost destdir=outfileset desthost=localhost provider=file 2013-04-30 10:17:57,909+0200 INFO LateBindingScheduler jobs queued: 1 2013-04-30 10:17:57,910+0200 DEBUG swift CDM: file://localhost/outfileset/reduce_output.txt : DEFAULT 2013-04-30 10:17:57,927+0200 INFO LateBindingScheduler jobs queued: 1 2013-04-30 10:17:57,928+0200 INFO swift NO_STATUS_FILE jobid=cat-ceb95s8l - Error file missing 2013-04-30 10:17:57,929+0200 DEBUG swift APPLICATION_EXCEPTION jobid=cat-ceb95s8l - Application exception: Caused by: File not found: /var/tmp/test_simple-20130430-1017-sfans555/shared/outfileset/reduce_output.txt 2013-04-30 10:17:57,941+0200 INFO LateBindingScheduler jobs queued: 1 2013-04-30 10:17:57,948+0200 INFO LateBindingScheduler jobs queued: 1 2013-04-30 10:17:57,952+0200 INFO LateBindingScheduler jobs queued: 1 2013-04-30 10:17:57,955+0200 INFO LateBindingScheduler jobs queued: 1 2013-04-30 10:17:57,958+0200 INFO LateBindingScheduler jobs queued: 1 2013-04-30 10:17:57,962+0200 INFO swift END_FAILURE thread=0-6 tr=cat 2013-04-30 10:17:57,965+0200 DEBUG VDL2ExecutionContext Karajan level error: Exception in cat: Arguments: [infileset/a.txt infileset/b.txt] Host: localhost Directory: test_simple-20130430-1017-sfans555/jobs/c/cat-ceb95s8l stderr.txt: /bin/cat: infileset/a.txt infileset/b.txt: No such file or directory stdout.txt: at sys:exception @ vdl-int.k, line: 599 sys:throw @ vdl-int.k, line: 598 sys:catch @ vdl-int.k, line: 565 sys:try @ vdl-int.k, line: 467 task:allocatehost @ vdl-int.k, line: 417 vdl:execute2 @ execute-default.k, line: 23 sys:ignoreerrors @ execute-default.k, line: 21 sys:parallelfor @ execute-default.k, line: 20 sys:restartonerror @ execute-default.k, line: 16 vdl:throttled @ execute-default.k, line: 14 sys:try @ execute-default.k, line: 13 sys:if @ execute-default.k, line: 12 sys:then @ execute-default.k, line: 11 sys:if @ execute-default.k, line: 10 vdl:execute @ test_simple.kml, line: 109 mycat_reduce @ test_simple.kml, line: 223 sys:uparallel @ test_simple.kml, line: 179 vdl:mainp @ test_simple.kml, line: 178 mainp @ vdl.k, line: 118 vdl:mains @ test_simple.kml, line: 176 vdl:mains @ test_simple.kml, line: 176 rlog:restartlog @ test_simple.kml, line: 175 kernel:project @ test_simple.kml, line: 2 test_simple-20130430-1017-sfans555 caused by: null at task:transfer @ vdl-int.k, line: 310 sys:restartonerror @ vdl-int.k, line: 309 sys:then @ vdl-int.k, line: 309 sys:if @ vdl-int.k, line: 307 sys:uparallelfor @ vdl-int.k, line: 290 list:list @ vdl-int.k, line: 290 sys:set @ vdl-int.k, line: 289 vdl:dostageout @ vdl-int.k, line: 545 sys:sequential @ vdl-int.k, line: 468 sys:try @ vdl-int.k, line: 467 task:allocatehost @ vdl-int.k, line: 417 vdl:execute2 @ execute-default.k, line: 23 sys:ignoreerrors @ execute-default.k, line: 21 sys:parallelfor @ execute-default.k, line: 20 sys:restartonerror @ execute-default.k, line: 16 vdl:throttled @ execute-default.k, line: 14 sys:try @ execute-default.k, line: 13 sys:if @ execute-default.k, line: 12 sys:then @ execute-default.k, line: 11 sys:if @ execute-default.k, line: 10 vdl:execute @ test_simple.kml, line: 109 mycat_reduce @ test_simple.kml, line: 223 sys:uparallel @ test_simple.kml, line: 179 vdl:mainp @ test_simple.kml, line: 178 mainp @ vdl.k, line: 118 vdl:mains @ test_simple.kml, line: 176 vdl:mains @ test_simple.kml, line: 176 rlog:restartlog @ test_simple.kml, line: 175 kernel:project @ test_simple.kml, line: 2 test_simple-20130430-1017-sfans555 2013-04-30 10:17:58,207+0200 INFO ExecutionContext Detailed exception: Exception in cat: Arguments: [infileset/a.txt infileset/b.txt] Host: localhost Directory: test_simple-20130430-1017-sfans555/jobs/c/cat-ceb95s8l stderr.txt: /bin/cat: infileset/a.txt infileset/b.txt: No such file or directory stdout.txt: Caused by: File not found: /var/tmp/test_simple-20130430-1017-sfans555/shared/outfileset/reduce_output.txt at org.globus.cog.karajan.workflow.nodes.functions.KException.function(KException.java:29) at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) at org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:227) at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:98) at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) Caused by: Caused by: File not found: /var/tmp/test_simple-20130430-1017-sfans555/shared/outfileset/reduce_output.txt at org.globus.cog.karajan.workflow.nodes.FlowNode.failImmediately(FlowNode.java:101) at org.globus.cog.karajan.workflow.nodes.grid.AbstractGridNode$1.run(AbstractGridNode.java:308) ... 6 more Caused by: org.globus.cog.abstraction.impl.file.FileNotFoundException: File not found: /var/tmp/test_simple-20130430-1017-sfans555/shared/outfileset/reduce_output.txt at org.globus.cog.abstraction.impl.file.local.FileResourceImpl.getFile(FileResourceImpl.java:225) at org.globus.cog.abstraction.impl.file.local.FileResourceImpl.putFile(FileResourceImpl.java:286) at org.globus.cog.abstraction.impl.file.AbstractFileResource.putFile(AbstractFileResource.java:210) at org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(DelegatedFileTransferHandler.java:307) at org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestination(CachingDelegatedFileTransferHandler.java:46) at org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTransferHandler.java:482) ... 1 more 2013-04-30 10:17:58,211+0200 INFO Loader Swift finished with errors --- End Listing --------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 899 bytes Desc: OpenPGP digital signature URL: