[Swift-commit] r7229 - in SwiftApps/Swift-MapRed/mapred_combiner_V4: . examples examples/sort examples/wc test
yadunandb at ci.uchicago.edu
yadunandb at ci.uchicago.edu
Thu Oct 24 15:38:19 CDT 2013
Author: yadunandb
Date: 2013-10-24 15:38:19 -0500 (Thu, 24 Oct 2013)
New Revision: 7229
Added:
SwiftApps/Swift-MapRed/mapred_combiner_V4/compile
SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/
SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/sort/
SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/sort/map_genrands.sh
SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/sort/reduce_sort.sh
SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/t.sh
SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/wc/
SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/wc/map_genrands.sh
SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/wc/reduce_sort.sh
SwiftApps/Swift-MapRed/mapred_combiner_V4/test/
SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.0.26283aa
SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.0.26283ab
SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.0.26283ac
SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.0.26283ad
SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.22871aa
SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.22871ab
SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.22871ac
SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.26087aa
SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.26087ab
SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.26087ac
SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.32009aa
SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.32009ab
SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.32009ac
SwiftApps/Swift-MapRed/mapred_combiner_V4/test/k_way.sh
SwiftApps/Swift-MapRed/mapred_combiner_V4/test/pointers
Removed:
SwiftApps/Swift-MapRed/mapred_combiner_V4/beagle.xml
SwiftApps/Swift-MapRed/mapred_combiner_V4/dereference.sh
SwiftApps/Swift-MapRed/mapred_combiner_V4/midway.xml
SwiftApps/Swift-MapRed/mapred_combiner_V4/node_uniq.sh
Modified:
SwiftApps/Swift-MapRed/mapred_combiner_V4/LAB_RECORDS
SwiftApps/Swift-MapRed/mapred_combiner_V4/README
SwiftApps/Swift-MapRed/mapred_combiner_V4/map_core.sh
SwiftApps/Swift-MapRed/mapred_combiner_V4/multiple.xml
SwiftApps/Swift-MapRed/mapred_combiner_V4/reduce_core.sh
SwiftApps/Swift-MapRed/mapred_combiner_V4/run.sh
SwiftApps/Swift-MapRed/mapred_combiner_V4/setup.sh
SwiftApps/Swift-MapRed/mapred_combiner_V4/swiftreduce.swift
Log:
Major changes to scripts:
Changes to the map, reduce interface expected from the user supplied scripts.
Added example folders, with sorting as main example.
Documented to README
Modified: SwiftApps/Swift-MapRed/mapred_combiner_V4/LAB_RECORDS
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/LAB_RECORDS 2013-10-24 20:14:30 UTC (rev 7228)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/LAB_RECORDS 2013-10-24 20:38:19 UTC (rev 7229)
@@ -4427,13 +4427,29 @@
EXPECTED COUNT: 100000
---------------------------------------------
================================================================
-RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple, CHUNKS:20 X FILESIZE:10
-Warning: Parameter reducer, on line 47, shadows variable of same name on line 30
-Warning: Parameter all_map, on line 47, shadows variable of same name on line 43
-Warning: Parameter reducer, on line 51, shadows variable of same name on line 30
-Warning: Parameter all_map, on line 51, shadows variable of same name on line 43
-Warning: Parameter reducer, on line 55, shadows variable of same name on line 30
-chunks 20 filesize 10 site multiple type Swift+Chirp
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:beagle, CHUNKS:20 X FILESIZE:10
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131015-2256-yts3ga1c
+Progress: time: Tue, 15 Oct 2013 22:57:00 +0000
+Progress: time: Tue, 15 Oct 2013 22:57:30 +0000 Submitted:20
+Progress: time: Tue, 15 Oct 2013 22:58:00 +0000 Submitted:20
+Progress: time: Tue, 15 Oct 2013 22:58:30 +0000 Submitted:20
+Progress: time: Tue, 15 Oct 2013 22:59:00 +0000 Submitted:20
+Progress: time: Tue, 15 Oct 2013 22:59:30 +0000 Submitted:20
+Progress: time: Tue, 15 Oct 2013 23:00:00 +0000 Submitted:20
+Progress: time: Tue, 15 Oct 2013 23:00:30 +0000 Submitted:20
+Progress: time: Tue, 15 Oct 2013 23:01:00 +0000 Submitted:20
+Progress: time: Tue, 15 Oct 2013 23:01:30 +0000 Submitted:20
+Progress: time: Tue, 15 Oct 2013 23:02:00 +0000 Submitted:20
+Progress: time: Tue, 15 Oct 2013 23:02:30 +0000 Submitted:20
+Progress: time: Tue, 15 Oct 2013 23:03:00 +0000 Submitted:20
+Progress: time: Tue, 15 Oct 2013 23:03:30 +0000 Submitted:20
+Progress: time: Tue, 15 Oct 2013 23:04:00 +0000 Submitted:20
+Progress: time: Tue, 15 Oct 2013 23:04:30 +0000 Submitted:20
+chunks 20 filesize 10 site beagle type Swift+Chirp
---------------------------------------------
COUNT does *NOT* match expected records
ACTUAL COUNT :
@@ -4441,18 +4457,17 @@
---------------------------------------------
================================================================
RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple, CHUNKS:10 X FILESIZE:1
-Warning: Parameter reducer, on line 47, shadows variable of same name on line 30
-Warning: Parameter all_map, on line 47, shadows variable of same name on line 43
-Warning: Parameter reducer, on line 51, shadows variable of same name on line 30
-Warning: Parameter all_map, on line 51, shadows variable of same name on line 43
-Warning: Parameter reducer, on line 55, shadows variable of same name on line 30
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
Swift 0.94.1 RC4 swift-r7109 cog-r3802
-RunID: 20131016-1515-psg21p04
-Progress: time: Wed, 16 Oct 2013 15:15:06 +0000
-Progress: time: Wed, 16 Oct 2013 15:15:23 +0000 Submitting:4 Submitted:1 Finished successfully:5
-Progress: time: Wed, 16 Oct 2013 15:15:36 +0000 Submitted:5 Finished successfully:5
-Progress: time: Wed, 16 Oct 2013 15:16:06 +0000 Submitted:5 Finished successfully:5
+RunID: 20131015-2304-73oflrug
+Progress: time: Tue, 15 Oct 2013 23:04:56 +0000
+Progress: time: Tue, 15 Oct 2013 23:05:18 +0000 Stage in:1 Submitted:9
+Final status: Tue, 15 Oct 2013 23:05:18 +0000 Finished successfully:11
+real 24.38
+user 43.16
+sys 9.60
chunks 10 filesize 1 site multiple type Swift+Chirp
---------------------------------------------
COUNT does *NOT* match expected records
@@ -4461,46 +4476,848 @@
---------------------------------------------
================================================================
RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple, CHUNKS:10 X FILESIZE:1
-Warning: Parameter reducer, on line 47, shadows variable of same name on line 30
-Warning: Parameter all_map, on line 47, shadows variable of same name on line 43
-Warning: Parameter reducer, on line 51, shadows variable of same name on line 30
-Warning: Parameter all_map, on line 51, shadows variable of same name on line 43
-Warning: Parameter reducer, on line 55, shadows variable of same name on line 30
+Swift trunk swift-r7138 cog-r3806
+RunID: 20131015-2305-yxcu8to7
+Warning: The @ syntax for function invocation is deprecated
+Progress: time: Tue, 15 Oct 2013 23:05:36 +0000
+Progress: time: Tue, 15 Oct 2013 23:05:37 +0000 Submitted:10
+Final status: Tue, 15 Oct 2013 23:05:51 +0000 Finished successfully:11
+real 18.04
+user 3.54
+sys 0.24
+chunks 10 filesize 1 site multiple type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 100000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple, CHUNKS:5 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Could not start execution
+ Compile error in procedure invocation at line 61
+ Wrong number of procedure input arguments: specified 4 and should be 3
+real 2.42
+user 1.64
+sys 0.11
+chunks 5 filesize 1 site multiple type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 50000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple, CHUNKS:5 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Could not start execution
+ Compile error in procedure invocation at line 61
+ Wrong type for parameter number 3, expected file[int], got fileptr[int]
+real 1.02
+user 1.66
+sys 0.09
+chunks 5 filesize 1 site multiple type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 50000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple, CHUNKS:5 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Could not start execution
+ Compile error in procedure invocation at line 61
+ Variable final_result was not declared in this scope.
+real 1.02
+user 1.67
+sys 0.08
+chunks 5 filesize 1 site multiple type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 50000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple, CHUNKS:5 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
Swift 0.94.1 RC4 swift-r7109 cog-r3802
-RunID: 20131016-1516-3xtve80d
-Progress: time: Wed, 16 Oct 2013 15:16:44 +0000
-Progress: time: Wed, 16 Oct 2013 15:16:50 +0000 Stage out:1 Finished successfully:11
-Final status: Wed, 16 Oct 2013 15:16:50 +0000 Finished successfully:13
-real 9.12
-user 16.47
-sys 2.72
-chunks 10 filesize 1 site multiple type Swift+Chirp
+RunID: 20131016-1402-6rltevy6
+Progress: time: Wed, 16 Oct 2013 14:02:59 +0000
+Progress: time: Wed, 16 Oct 2013 14:03:00 +0000 Stage in:1 Finished successfully:5
+Final status: Wed, 16 Oct 2013 14:03:00 +0000 Finished successfully:7
+real 3.96
+user 6.72
+sys 0.55
+chunks 5 filesize 1 site multiple type Swift+Chirp
---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 50000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple, CHUNKS:5 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-1404-uc3icre3
+Progress: time: Wed, 16 Oct 2013 14:04:11 +0000
+Final status: Wed, 16 Oct 2013 14:04:11 +0000 Finished successfully:7
+real 3.37
+user 5.82
+sys 0.37
+chunks 5 filesize 1 site multiple type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT : 0
+EXPECTED COUNT: 50000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple, CHUNKS:5 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-1409-itnftan8
+Progress: time: Wed, 16 Oct 2013 14:09:57 +0000
+Progress: time: Wed, 16 Oct 2013 14:10:27 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 14:10:57 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 14:11:27 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 14:11:57 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 14:12:27 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 14:12:57 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 14:13:27 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 14:13:57 +0000 Active:1 Finished successfully:6
+chunks 5 filesize 1 site multiple type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 50000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple, CHUNKS:5 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-1638-ke09zcf2
+Progress: time: Wed, 16 Oct 2013 16:38:40 +0000
+Progress: time: Wed, 16 Oct 2013 16:39:10 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 16:39:40 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 16:40:10 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 16:40:40 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 16:41:10 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 16:41:40 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 16:42:10 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 16:42:40 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 16:43:10 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 16:43:40 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 16:44:10 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 16:44:40 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 16:45:10 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 16:45:40 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 16:46:10 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 16:46:40 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 16:47:10 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 16:47:40 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 16:48:10 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 16:48:40 +0000 Active:1 Finished successfully:6
+Progress: time: Wed, 16 Oct 2013 16:48:41 +0000 Stage out:1 Finished successfully:6
+Execution failed:
+ Exception in mid_bash:
+ Arguments: [reduce_wrapper.sh, reduce_core.sh, plain, all_map.out]
+ Host: midway
+ Directory: swiftreduce-20131016-1638-ke09zcf2/jobs/l/mid_bash-lb6mpqgl
+
+Caused by:
+ Walltime exceeded
+org.globus.cog.abstraction.impl.common.execution.JobException: Walltime exceeded (exit code: 513)
+ at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:40)
+ at org.globus.cog.karajan.workflow.service.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
+ at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.handleRequest(AbstractKarajanChannel.java:527)
+ at org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel.stepNIO(AbstractStreamKarajanChannel.java:238)
+ at org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
+ at org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
+ (exit code: 513)
+ reduce_mid, swiftreduce.swift, line 61
+real 604.05
+user 6.76
+sys 0.56
+chunks 5 filesize 1 site multiple type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 50000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:2 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Could not open: multiple.xml.xml
+Execution failed:
+
+ Could not find sites file: multiple.xml.xml
+
+real 2.15
+user 3.08
+sys 0.15
+chunks 2 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 20000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:2 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-1915-gtjty6j9
+Progress: time: Wed, 16 Oct 2013 19:15:52 +0000
+Final status: Wed, 16 Oct 2013 19:15:53 +0000 Finished successfully:4
+real 5.48
+user 5.89
+sys 0.46
+chunks 2 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT : =
+EXPECTED COUNT: 20000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:2 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-1920-lu3aold6
+Progress: time: Wed, 16 Oct 2013 19:20:50 +0000
+Progress: time: Wed, 16 Oct 2013 19:20:52 +0000 Stage in:1 Submitted:1
+Final status: Wed, 16 Oct 2013 19:20:52 +0000 Finished successfully:4
+real 4.85
+user 8.42
+sys 0.96
+chunks 2 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT : =
+EXPECTED COUNT: 20000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:2 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-1923-oexsdz9c
+Progress: time: Wed, 16 Oct 2013 19:23:25 +0000
+Final status: Wed, 16 Oct 2013 19:23:26 +0000 Finished successfully:4
+real 3.28
+user 5.57
+sys 0.32
+chunks 2 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT : =
+EXPECTED COUNT: 20000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:2 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-1924-3bk83atb
+Progress: time: Wed, 16 Oct 2013 19:24:26 +0000
+Final status: Wed, 16 Oct 2013 19:24:27 +0000 Finished successfully:4
+real 3.29
+user 5.44
+sys 0.37
+chunks 2 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT : =
+EXPECTED COUNT: 20000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:2 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-1932-6uos5of5
+Progress: time: Wed, 16 Oct 2013 19:32:52 +0000
+Progress: time: Wed, 16 Oct 2013 19:33:22 +0000 Active:1 Finished successfully:3
+Progress: time: Wed, 16 Oct 2013 19:33:52 +0000 Active:1 Finished successfully:3
+Progress: time: Wed, 16 Oct 2013 19:34:22 +0000 Active:1 Finished successfully:3
+chunks 2 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 20000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:2 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-1935-zkom0ro7
+Progress: time: Wed, 16 Oct 2013 19:35:34 +0000
+Final status: Wed, 16 Oct 2013 19:35:34 +0000 Finished successfully:4
+real 3.32
+user 5.52
+sys 0.34
+chunks 2 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 20000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:2 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-1937-e3j6rlqf
+Progress: time: Wed, 16 Oct 2013 19:37:10 +0000
+Progress: time: Wed, 16 Oct 2013 19:37:23 +0000 Stage in:1 Submitted:1
+Final status: Wed, 16 Oct 2013 19:37:23 +0000 Finished successfully:4
+real 15.57
+user 26.75
+sys 5.79
+chunks 2 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 20000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:2 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-1938-41onhc36
+Progress: time: Wed, 16 Oct 2013 19:38:33 +0000
+Progress: time: Wed, 16 Oct 2013 19:38:57 +0000 Stage in:1 Submitted:1
+Final status: Wed, 16 Oct 2013 19:38:57 +0000 Finished successfully:4
+real 26.83
+user 45.94
+sys 11.60
+chunks 2 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 20000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:2 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-1947-qocdj5r8
+Progress: time: Wed, 16 Oct 2013 19:47:06 +0000
+Final status: Wed, 16 Oct 2013 19:47:07 +0000 Finished successfully:4
+real 3.60
+user 6.01
+sys 0.50
+chunks 2 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 20000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:2 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-1948-ag23q2r9
+Progress: time: Wed, 16 Oct 2013 19:48:10 +0000
+Final status: Wed, 16 Oct 2013 19:48:10 +0000 Finished successfully:4
+real 3.35
+user 5.56
+sys 0.40
+chunks 2 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 20000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:2 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-1949-yzvzkppg
+Progress: time: Wed, 16 Oct 2013 19:49:11 +0000
+Final status: Wed, 16 Oct 2013 19:49:11 +0000 Finished successfully:4
+real 3.36
+user 5.77
+sys 0.35
+chunks 2 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 20000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:2 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-1950-h1vb0l48
+Progress: time: Wed, 16 Oct 2013 19:50:35 +0000
+Final status: Wed, 16 Oct 2013 19:50:35 +0000 Finished successfully:4
+real 3.34
+user 5.59
+sys 0.35
+chunks 2 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 20000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:5 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-2049-4pboij8a
+Progress: time: Wed, 16 Oct 2013 20:49:22 +0000
+Progress: time: Wed, 16 Oct 2013 20:49:28 +0000 Stage out:1 Finished successfully:6
+Final status: Wed, 16 Oct 2013 20:49:28 +0000 Finished successfully:7
+real 9.15
+user 5.70
+sys 0.41
+chunks 5 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 50000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:5 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-2055-2r77pks7
+Progress: time: Wed, 16 Oct 2013 20:55:47 +0000
+Progress: time: Wed, 16 Oct 2013 20:55:53 +0000 Stage out:1 Finished successfully:6
+Final status: Wed, 16 Oct 2013 20:55:53 +0000 Finished successfully:7
+real 8.34
+user 5.71
+sys 0.31
+chunks 5 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 50000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:5 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-2057-nvpgtjuf
+Progress: time: Wed, 16 Oct 2013 20:57:40 +0000
+Progress: time: Wed, 16 Oct 2013 20:57:45 +0000 Stage out:1 Finished successfully:6
+Final status: Wed, 16 Oct 2013 20:57:45 +0000 Finished successfully:7
+real 8.42
+user 5.80
+sys 0.35
+chunks 5 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
COUNT matches expected records
-ACTUAL COUNT : 100000
+ACTUAL COUNT : 50000
+EXPECTED COUNT: 50000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:10 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-2110-i6sxj5b4
+Progress: time: Wed, 16 Oct 2013 21:10:30 +0000
+Progress: time: Wed, 16 Oct 2013 21:10:37 +0000 Stage in:1 Submitted:9
+Progress: time: Wed, 16 Oct 2013 21:10:42 +0000 Stage out:1 Finished successfully:11
+Final status: Wed, 16 Oct 2013 21:10:42 +0000 Finished successfully:13
+real 14.78
+user 31.20
+sys 5.20
+chunks 10 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT : 0
EXPECTED COUNT: 100000
---------------------------------------------
================================================================
RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:10 X FILESIZE:1
-Warning: Parameter reducer, on line 44, shadows variable of same name on line 28
-Warning: Parameter all_map, on line 44, shadows variable of same name on line 40
-Warning: Parameter reducer, on line 48, shadows variable of same name on line 28
-Warning: Parameter all_map, on line 48, shadows variable of same name on line 40
-Warning: Parameter reducer, on line 52, shadows variable of same name on line 28
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
Swift 0.94.1 RC4 swift-r7109 cog-r3802
-RunID: 20131016-1616-egnuduk7
-Progress: time: Wed, 16 Oct 2013 16:16:54 +0000
-Progress: time: Wed, 16 Oct 2013 16:17:02 +0000 Stage in:1 Submitted:9
-Progress: time: Wed, 16 Oct 2013 16:17:07 +0000 Stage out:1 Finished successfully:11
-Final status: Wed, 16 Oct 2013 16:17:07 +0000 Finished successfully:13
-real 16.97
-user 33.06
-sys 5.51
+RunID: 20131016-2116-k8zlv2k9
+Progress: time: Wed, 16 Oct 2013 21:16:04 +0000
+Progress: time: Wed, 16 Oct 2013 21:16:15 +0000 Stage in:1 Submitted:9
+Progress: time: Wed, 16 Oct 2013 21:16:20 +0000 Stage out:1 Finished successfully:11
+Final status: Wed, 16 Oct 2013 21:16:20 +0000 Finished successfully:13
+real 19.36
+user 42.54
+sys 7.91
chunks 10 filesize 1 site multiple.xml type Swift+Chirp
---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 100000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:10 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-2130-0yzig722
+Progress: time: Wed, 16 Oct 2013 21:30:45 +0000
+Progress: time: Wed, 16 Oct 2013 21:31:02 +0000 Stage in:1 Submitted:9
+Final status: Wed, 16 Oct 2013 21:31:02 +0000 Finished successfully:13
+real 19.53
+user 46.09
+sys 7.77
+chunks 10 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT : 0
+EXPECTED COUNT: 100000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:10 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-2138-7nk1y3f0
+Progress: time: Wed, 16 Oct 2013 21:38:37 +0000
+Progress: time: Wed, 16 Oct 2013 21:38:48 +0000 Stage in:1 Submitted:9
+Final status: Wed, 16 Oct 2013 21:38:48 +0000 Finished successfully:13
+real 14.11
+user 33.90
+sys 5.25
+chunks 10 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT : 0
+EXPECTED COUNT: 100000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:10 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-2141-bk2qe4be
+Progress: time: Wed, 16 Oct 2013 21:41:49 +0000
+Progress: time: Wed, 16 Oct 2013 21:42:06 +0000 Stage in:1 Submitted:9
+Final status: Wed, 16 Oct 2013 21:42:06 +0000 Finished successfully:13
+real 20.05
+user 48.15
+sys 7.91
+chunks 10 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT : 0
+EXPECTED COUNT: 100000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:10 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-2145-dp3z46id
+Progress: time: Wed, 16 Oct 2013 21:45:04 +0000
+Progress: time: Wed, 16 Oct 2013 21:45:24 +0000 Stage in:1 Submitted:9
+Final status: Wed, 16 Oct 2013 21:45:24 +0000 Finished successfully:13
+real 23.23
+user 56.75
+sys 9.11
+chunks 10 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT : 0
+EXPECTED COUNT: 100000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:multiple.xml, CHUNKS:10 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131016-2146-yj5luboc
+Progress: time: Wed, 16 Oct 2013 21:46:33 +0000
+Progress: time: Wed, 16 Oct 2013 21:47:03 +0000 Stage in:1 Submitted:9
+Progress: time: Wed, 16 Oct 2013 21:47:08 +0000 Stage out:1 Finished successfully:11
+Final status: Wed, 16 Oct 2013 21:47:08 +0000 Finished successfully:13
+real 37.40
+user 88.78
+sys 15.50
+chunks 10 filesize 1 site multiple.xml type Swift+Chirp
+---------------------------------------------
COUNT matches expected records
ACTUAL COUNT : 100000
EXPECTED COUNT: 100000
---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:, CHUNKS:10 X FILESIZE:1
+SwiftScript program does not exist: -files=10
+
+For usage information: swift -help
+
+real 0.78
+user 0.26
+sys 0.03
+chunks 10 filesize 1 site type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 100000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:, CHUNKS:10 X FILESIZE:1
+SwiftScript program does not exist: -files=10
+
+For usage information: swift -help
+
+real 0.23
+user 0.26
+sys 0.03
+chunks 10 filesize 1 site type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 100000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:, CHUNKS:multiple.xml X FILESIZE:10
+SwiftScript program does not exist: -files=multiple.xml
+
+For usage information: swift -help
+
+real 0.22
+user 0.26
+sys 0.03
+chunks multiple.xml filesize 10 site type Swift+Chirp
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:, CHUNKS:10 X FILESIZE:1
+Warning: Parameter all_map, on line 23, shadows variable of same name on line 56
+Warning: Parameter all_map, on line 27, shadows variable of same name on line 56
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131017-1841-jpe11yb5
+Progress: time: Thu, 17 Oct 2013 18:41:20 +0000
+Progress: time: Thu, 17 Oct 2013 18:41:31 +0000 Stage in:1 Submitted:9
+Progress: time: Thu, 17 Oct 2013 18:41:37 +0000 Stage out:1 Finished successfully:11
+Final status: Thu, 17 Oct 2013 18:41:37 +0000 Finished successfully:13
+real 20.00
+user 43.11
+sys 8.33
+chunks 10 filesize 1 site type Swift+Chirp
+---------------------------------------------
+COUNT matches expected records
+ACTUAL COUNT : 100000
+EXPECTED COUNT: 100000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:, CHUNKS:10 X FILESIZE:1
+Warning: Parameter all_map, on line 26, shadows variable of same name on line 59
+Warning: Parameter all_map, on line 30, shadows variable of same name on line 59
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131017-1844-iepl730d
+Progress: time: Thu, 17 Oct 2013 18:44:43 +0000
+Progress: time: Thu, 17 Oct 2013 18:45:01 +0000 Stage in:1 Submitted:9
+Progress: time: Thu, 17 Oct 2013 18:45:06 +0000 Stage out:1 Finished successfully:11
+Final status: Thu, 17 Oct 2013 18:45:06 +0000 Finished successfully:13
+real 26.06
+user 59.10
+sys 10.44
+chunks 10 filesize 1 site type Swift+Chirp
+---------------------------------------------
+COUNT matches expected records
+ACTUAL COUNT : 100000
+EXPECTED COUNT: 100000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:, CHUNKS:20 X FILESIZE:10
+Warning: Parameter all_map, on line 26, shadows variable of same name on line 59
+Warning: Parameter all_map, on line 30, shadows variable of same name on line 59
+Could not start execution
+ Compile error in foreach statement at line 53
+ Compile error in procedure invocation at line 54
+ Wrong type for parameter number 2, expected file, got int
+real 1.02
+user 1.73
+sys 0.10
+chunks 20 filesize 10 site type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 2000000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:, CHUNKS:10 X FILESIZE:10
+Warning: Parameter all_map, on line 24, shadows variable of same name on line 59
+Warning: Parameter all_map, on line 28, shadows variable of same name on line 59
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131017-1921-ncuct83e
+Progress: time: Thu, 17 Oct 2013 19:21:39 +0000
+Progress: time: Thu, 17 Oct 2013 19:22:01 +0000 Stage in:1 Submitted:9
+Progress: time: Thu, 17 Oct 2013 19:22:08 +0000 Stage out:1 Finished successfully:11
+Progress: time: Thu, 17 Oct 2013 19:22:10 +0000 Active:1 Finished successfully:12
+Progress: time: Thu, 17 Oct 2013 19:22:11 +0000 Stage out:1 Finished successfully:12
+Final status: Thu, 17 Oct 2013 19:22:11 +0000 Finished successfully:13
+real 35.40
+user 74.21
+sys 14.31
+chunks 10 filesize 10 site type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 1000000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:, CHUNKS:100 X FILESIZE:10
+Warning: Parameter all_map, on line 24, shadows variable of same name on line 59
+Warning: Parameter all_map, on line 28, shadows variable of same name on line 59
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131017-1927-m19gavx1
+Progress: time: Thu, 17 Oct 2013 19:27:04 +0000
+Progress: time: Thu, 17 Oct 2013 19:27:31 +0000 Selecting site:35 Stage in:1 Submitted:64
+Progress: time: Thu, 17 Oct 2013 19:27:34 +0000 Active:1 Finished successfully:101
+Progress: time: Thu, 17 Oct 2013 19:28:04 +0000 Active:1 Finished successfully:101
+Progress: time: Thu, 17 Oct 2013 19:28:06 +0000 Stage out:1 Finished successfully:101
+Progress: time: Thu, 17 Oct 2013 19:28:15 +0000 Finished successfully:102
+Progress: time: Thu, 17 Oct 2013 19:28:23 +0000 Active:1 Finished successfully:102
+Progress: time: Thu, 17 Oct 2013 19:28:34 +0000 Active:1 Finished successfully:102
+Progress: time: Thu, 17 Oct 2013 19:28:40 +0000 Stage out:1 Finished successfully:102
+Progress: time: Thu, 17 Oct 2013 19:28:49 +0000 Finished successfully:103
+Final status: Thu, 17 Oct 2013 19:28:49 +0000 Finished successfully:103
+real 107.87
+user 155.31
+sys 37.07
+chunks 100 filesize 10 site type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 10000000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:, CHUNKS:10 X FILESIZE:20
+Warning: Parameter all_map, on line 24, shadows variable of same name on line 59
+Warning: Parameter all_map, on line 28, shadows variable of same name on line 59
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131017-2134-2z229bh0
+Progress: time: Thu, 17 Oct 2013 21:34:08 +0000
+Progress: time: Thu, 17 Oct 2013 21:34:18 +0000 Stage in:1 Submitted:9
+Progress: time: Thu, 17 Oct 2013 21:34:27 +0000 Stage out:1 Finished successfully:11
+Progress: time: Thu, 17 Oct 2013 21:34:28 +0000 Finished successfully:12
+Progress: time: Thu, 17 Oct 2013 21:34:30 +0000 Active:1 Finished successfully:12
+Progress: time: Thu, 17 Oct 2013 21:34:33 +0000 Stage out:1 Finished successfully:12
+Progress: time: Thu, 17 Oct 2013 21:34:35 +0000 Finished successfully:13
+Final status: Thu, 17 Oct 2013 21:34:35 +0000 Finished successfully:13
+real 30.40
+user 51.53
+sys 10.72
+chunks 10 filesize 20 site type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 2000000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:, CHUNKS:5 X FILESIZE:1
+Warning: Parameter all_map, on line 24, shadows variable of same name on line 59
+Warning: Parameter all_map, on line 28, shadows variable of same name on line 59
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131017-2141-da4k2nk5
+Progress: time: Thu, 17 Oct 2013 21:41:39 +0000
+Progress: time: Thu, 17 Oct 2013 21:41:45 +0000 Stage out:1 Finished successfully:6
+Final status: Thu, 17 Oct 2013 21:41:45 +0000 Finished successfully:8
+real 8.85
+user 16.10
+sys 2.70
+chunks 5 filesize 1 site type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 50000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:, CHUNKS:100 X FILESIZE:10
+Warning: Parameter all_map, on line 24, shadows variable of same name on line 59
+Warning: Parameter all_map, on line 28, shadows variable of same name on line 59
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131017-2230-287bzeo0
+Progress: time: Thu, 17 Oct 2013 22:30:48 +0000
+Progress: time: Thu, 17 Oct 2013 22:30:49 +0000 Selecting site:19 Stage in:16 Submitting:1 Submitted:48 Finished successfully:16
+Progress: time: Thu, 17 Oct 2013 22:30:50 +0000 Stage out:1 Finished successfully:100
+Progress: time: Thu, 17 Oct 2013 22:31:18 +0000 Active:1 Finished successfully:101
+Progress: time: Thu, 17 Oct 2013 22:31:22 +0000 Stage out:1 Finished successfully:101
+Progress: time: Thu, 17 Oct 2013 22:31:30 +0000 Finished successfully:102
+Progress: time: Thu, 17 Oct 2013 22:31:39 +0000 Active:1 Finished successfully:102
+Progress: time: Thu, 17 Oct 2013 22:31:48 +0000 Active:1 Finished successfully:102
+Progress: time: Thu, 17 Oct 2013 22:31:56 +0000 Stage out:1 Finished successfully:102
+Progress: time: Thu, 17 Oct 2013 22:32:05 +0000 Finished successfully:103
+Final status: Thu, 17 Oct 2013 22:32:05 +0000 Finished successfully:103
+real 78.98
+user 87.76
+sys 20.89
+chunks 100 filesize 10 site type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 10000000
+---------------------------------------------
+================================================================
+RUNTYPE :SWIFT+Chirp (With local combiners), SITE:, CHUNKS:100 X FILESIZE:10
+Warning: Parameter all_map, on line 24, shadows variable of same name on line 59
+Warning: Parameter all_map, on line 28, shadows variable of same name on line 59
+Swift 0.94.1 RC4 swift-r7109 cog-r3802
+
+RunID: 20131017-2234-t20i0o06
+Progress: time: Thu, 17 Oct 2013 22:34:29 +0000
+Progress: time: Thu, 17 Oct 2013 22:34:30 +0000 Submitted:36 Active:15 Stage out:1 Finished successfully:48
+Progress: time: Thu, 17 Oct 2013 22:34:32 +0000 Submitted:4 Active:13 Stage out:3 Finished successfully:80
+Progress: time: Thu, 17 Oct 2013 22:34:59 +0000 Active:1 Finished successfully:101
+Progress: time: Thu, 17 Oct 2013 22:35:01 +0000 Stage out:1 Finished successfully:101
+Progress: time: Thu, 17 Oct 2013 22:35:10 +0000 Finished successfully:102
+Final status: Thu, 17 Oct 2013 22:35:10 +0000 Finished successfully:102
+real 43.70
+user 16.21
+sys 2.96
+chunks 100 filesize 10 site type Swift+Chirp
+---------------------------------------------
+COUNT does *NOT* match expected records
+ACTUAL COUNT :
+EXPECTED COUNT: 10000000
+---------------------------------------------
Modified: SwiftApps/Swift-MapRed/mapred_combiner_V4/README
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/README 2013-10-24 20:14:30 UTC (rev 7228)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/README 2013-10-24 20:38:19 UTC (rev 7229)
@@ -1,34 +1,43 @@
Swift-Reduce
============
+REQUIREMENTS:
+1. CCTools - Install cctools on every site.
+ http://www3.nd.edu/~ccl/software/manuals/install.html
+
+
Run MapReduce style applications with Swift. Since swift is a file based
model as opposed to the Key-Value record based approach of MapReduce, there
are differences in the way the user supplied Map and Reduce functions
-should work.
+are required to work.
setup.sh
--------
-Run the setup.sh script to embed the map_core.sh and reduce_core.sh scripts
-in the wrapper scripts used by swift in runtime. Usernames to use across
-midway and beagle should be defined here.
+Modify the USERNAME variables in setup.sh to condifure scripts appropriately
+Setup.sh also compiles are copies the exec server and client to the remote sites.
-map_core.sh
------------
+Map application
+---------------
+The user supplied Map application could be any executable that accepts the
+arguments defined in the swift script and should return its result to stdout.
+You can use any number of arguments/files in the swiftscript and expect them
+to be passed transparently to your map application.
-The user supplied Map function should be a bash function named mapper().
-It would take any number of arguments from swift and would output it's
-results to stdout. Swift would redirect the output to appropriate files.
-The mapper function should be placed in a file map_core.sh.
+Note: Only results sent to stdout from map will be accepted by the Swiftreduce
+framework.
-reduce_core.sh
---------------
+Reduce application
+------------------
-The Reduce function should be named reduce() and placed in reduce_core.sh.
-This function would be passed intermediate files generated from the map
-stage and output its results to a file of its choosing. The filename of
-the result file is output to stdout.
+The reduce application can by any executable that accepts one or more files
+from the Map application. Results should be sent to stdout and Swiftreduce
+will route them. If the application is distributive (commutative and associative)
+you may use different reduce styles, such as local combine, k-way reduction tree.
+#TODO : Explain different reduce options
+#TODO : Case reducer != combiner ?
+
run.sh
------
@@ -47,11 +56,24 @@
Each record consists of 1 random number and count, padded to 100 bytes.
So, in the usual run a file of 10^8 records generated would have 10^6 bytes.
+Example - Gigaasort
+===================
+Map - Generates 100 byte records containing 1 random number.
+Reduce - Accepts one or more files, sorts each file and merge sorts the
+ sorted files.
+
+
Describe workflow
=================
teragen will split the task of generating files containing 10^8 records (~100 MB)
over 10000 tasks, thus generating in total 1 Tera records.
-./teragen_wrap.sh 10 # Generates 10000000
\ No newline at end of file
+./teragen_wrap.sh 10 # Generates 10000000
+
+
+TODOs and Future Work
+=====================
+1. Builds for exec_{server,client} on different architectures.
+2.
\ No newline at end of file
Deleted: SwiftApps/Swift-MapRed/mapred_combiner_V4/beagle.xml
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/beagle.xml 2013-10-24 20:14:30 UTC (rev 7228)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/beagle.xml 2013-10-24 20:38:19 UTC (rev 7229)
@@ -1,24 +0,0 @@
-<config>
- <pool handle="beagle">
- <execution provider="coaster" jobmanager="local:pbs"/>
- <profile namespace="globus" key="jobsPerNode">1</profile>
- <profile namespace="globus" key="lowOverAllocation">100</profile>
- <profile namespace="globus" key="highOverAllocation">100</profile>
- <profile namespace="globus" key="providerAttributes">pbs.aprun;pbs.mpp;depth=24</profile>
- <!-- to use a beage reservation, eg:
- <profile namespace="globus" key="providerAttributes">pbs.aprun;pbs.mpp;depth=24;pbs.resource_list=advres=wilde.1768</profile>
- -->
- <profile namespace="globus" key="maxtime">2400</profile>
- <profile namespace="globus" key="maxWalltime">00:20:00</profile>
- <profile namespace="globus" key="queue">batch</profile>
- <profile namespace="globus" key="userHomeOverride">/lustre/beagle/BEAGLE_USERNAME/swiftwork</profile>
- <profile namespace="globus" key="slots">10</profile>
- <profile namespace="globus" key="maxnodes">1</profile>
- <profile namespace="globus" key="nodeGranularity">1</profile>
- <profile namespace="karajan" key="jobThrottle">1.00</profile>
- <profile namespace="karajan" key="initialScore">10000</profile>
- <!-- <filesystem provider="local"/> -->
- <!-- <workdirectory>/lustre/beagle/yadunandb/swiftwork</workdirectory> -->
- <workdirectory>/tmp/BEAGLE_USERNAME/swiftwork</workdirectory>
- </pool>
-</config>
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/compile
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/compile (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/compile 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,20 @@
+#!/bin/bash
+
+gcc server.c -o exec_server
+gcc client.c -o exec_client
+
+chmod a+x exec_server
+chmod a+x exec_client
+
+if [[ "$HOSTNAME" == *midway* ]]; then
+ cp exec_server /scratch/midway/yadunand/bin/
+ cp exec_client /scratch/midway/yadunand/bin/
+ echo "Copied to /scratch/midway/yadunandb/bin/ on Midway"
+ echo "
+
+
+elif [[ "$HOSTNAME" == *beagle* ]]; then
+ cp exec_server /lustre/beagle/yadunandb/bin/
+ cp exec_client /lustre/beagle/yadunandb/bin/
+ echo "Copied to /lustre/beagle/yadunandb/bin/ on Beagle"
+fi
Property changes on: SwiftApps/Swift-MapRed/mapred_combiner_V4/compile
___________________________________________________________________
Added: svn:executable
+ *
Deleted: SwiftApps/Swift-MapRed/mapred_combiner_V4/dereference.sh
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/dereference.sh 2013-10-24 20:14:30 UTC (rev 7228)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/dereference.sh 2013-10-24 20:38:19 UTC (rev 7229)
@@ -1,20 +0,0 @@
-#!/bin/bash
-# The derefernce script takes one fileptr as input
-# and dumps the content of the file to stdout.
-TOKEN=($(cat $1))
-REMOTE=${TOKEN[0]}
-FILE=${TOKEN[1]}
-TARGETFILE=$(basename $FILE)
-REPODIR=$(dirname $FILE)
-echo "$REMOTE $HOSTNAME" 1>&2
-if [ "$REMOTE" == "$(hostname -f)" ]
-then # Move to an identifiable file if local
- cat $FILE
-else # Fetch to an identifiable file if remote
- echo "chirp_get $REMOTE $TARGETFILE $FILE" 1>&2
- chirp_get $REMOTE $TARGETFILE $FILE
- cat $FILE
- rm $FILE
-fi
-
-
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/sort/map_genrands.sh
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/sort/map_genrands.sh (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/sort/map_genrands.sh 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,12 @@
+#!/bin/bash
+ARG1=$1;
+LOWERLIMIT=0
+UPPERLIMIT=1000000
+SLICESIZE=10000 # 10^4 records padded to 100B would result in 1MB file
+#Generate $SLICESIZE number of 100B records of format: RANDOM_NUMBER 1
+shuf -i $LOWERLIMIT-$UPPERLIMIT -n $SLICESIZE | awk '{printf "%-99s\n", $0}' > tmp_shuf
+for i in $(seq 1 1 $ARG1)
+do
+ cat tmp_shuf
+done
+rm tmp_shuf
Property changes on: SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/sort/map_genrands.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/sort/reduce_sort.sh
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/sort/reduce_sort.sh (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/sort/reduce_sort.sh 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,8 @@
+#!/bin/bash
+
+for f in $*
+do
+ sort -n $f -o $f
+done
+sort -n -m $*
+rm $*
Property changes on: SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/sort/reduce_sort.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/t.sh
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/t.sh (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/t.sh 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,11 @@
+#!/bin/bash
+
+LOGS=OFF
+log()
+{
+ [ "$LOGS" == "ON" ] && echo "[$(date +'%Y-%m-%d %H:%M:%S.%N')] : $*" 1>&2
+}
+
+log "start do"
+sleep 1
+log "stop do"
Property changes on: SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/t.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/wc/map_genrands.sh
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/wc/map_genrands.sh (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/wc/map_genrands.sh 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,12 @@
+#!/bin/bash
+ARG1=$1;
+LOWERLIMIT=0
+UPPERLIMIT=1000000
+SLICESIZE=10000 # 10^4 records padded to 100B would result in 1MB file
+#Generate $SLICESIZE number of 100B records of format: RANDOM_NUMBER 1
+shuf -i $LOWERLIMIT-$UPPERLIMIT -n $SLICESIZE | awk '{printf "%-99s\n", $0}' > tmp_shuf
+for i in $(seq 1 1 $ARG1)
+do
+ cat tmp_shuf
+done
+rm tmp_shuf
Property changes on: SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/wc/map_genrands.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/wc/reduce_sort.sh
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/wc/reduce_sort.sh (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/wc/reduce_sort.sh 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,8 @@
+#!/bin/bash
+
+for f in $*
+do
+ sort -n $f -o $f
+done
+sort -n -m $*
+rm $*
Property changes on: SwiftApps/Swift-MapRed/mapred_combiner_V4/examples/wc/reduce_sort.sh
___________________________________________________________________
Added: svn:executable
+ *
Modified: SwiftApps/Swift-MapRed/mapred_combiner_V4/map_core.sh
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/map_core.sh 2013-10-24 20:14:30 UTC (rev 7228)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/map_core.sh 2013-10-24 20:38:19 UTC (rev 7229)
@@ -1,13 +1,12 @@
-mapper (){
- ARG1=$1;
- LOWERLIMIT=0
- UPPERLIMIT=1000000
- SLICESIZE=10000 # 10^4 records padded to 100B would result in 1MB file
- #Generate $SLICESIZE number of 100B records of format: RANDOM_NUMBER 1
- shuf -i $LOWERLIMIT-$UPPERLIMIT -n $SLICESIZE | awk '{printf "%-97s %s\n", $0, 1}' > tmp_shuf
- for i in $(seq 1 1 $ARG1)
- do
- cat tmp_shuf
- done
- rm tmp_shuf
-}
+#!/bin/bash
+ARG1=$1;
+LOWERLIMIT=0
+UPPERLIMIT=1000000
+SLICESIZE=10000 # 10^4 records padded to 100B would result in 1MB file
+#Generate $SLICESIZE number of 100B records of format: RANDOM_NUMBER 1
+shuf -i $LOWERLIMIT-$UPPERLIMIT -n $SLICESIZE | awk '{printf "%-97s %s\n", $0, 1}' > tmp_shuf
+for i in $(seq 1 1 $ARG1)
+do
+ cat tmp_shuf
+done
+rm tmp_shuf
Deleted: SwiftApps/Swift-MapRed/mapred_combiner_V4/midway.xml
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/midway.xml 2013-10-24 20:14:30 UTC (rev 7228)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/midway.xml 2013-10-24 20:38:19 UTC (rev 7229)
@@ -1,17 +0,0 @@
-<config>
- <pool handle="midway">
- <execution provider="coaster" jobmanager="local:slurm"/>
- <profile namespace="globus" key="queue">sandyb</profile>
- <profile namespace="globus" key="jobsPerNode">16</profile>
- <profile namespace="globus" key="maxWalltime">00:10:00</profile>
- <profile namespace="globus" key="maxTime">3600</profile>
- <profile namespace="globus" key="highOverAllocation">100</profile>
- <profile namespace="globus" key="lowOverAllocation">100</profile>
- <profile namespace="globus" key="slots">10</profile>
- <profile namespace="globus" key="maxNodes">1</profile>
- <profile namespace="globus" key="nodeGranularity">1</profile>
- <profile namespace="karajan" key="jobThrottle">1.50</profile>
- <profile namespace="karajan" key="initialScore">10000</profile>
- <workdirectory>/tmp/yadunand/work</workdirectory>
- </pool>
-</config>
\ No newline at end of file
Modified: SwiftApps/Swift-MapRed/mapred_combiner_V4/multiple.xml
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/multiple.xml 2013-10-24 20:14:30 UTC (rev 7228)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/multiple.xml 2013-10-24 20:38:19 UTC (rev 7229)
@@ -32,10 +32,52 @@
<profile namespace="karajan" key="initialScore">10000</profile>
<!-- <profile namespace="karajan" key="workerLoggingLevel">trace</profile> -->
<profile namespace="globus" key="workerLoggingLevel">DEBUG</profile>
- <!-- <workdirectory>/lustre/beagle/BEAGLE_USERNAME/swiftwork</workdirectory> -->
+ <!-- <workdirectory>/lustre/beagle/MIDWAY_USERNAMEb/swiftwork</workdirectory> -->
<workdirectory>/tmp/BEAGLE_USERNAME/swiftwork</workdirectory>
</pool>
+
+ <pool handle="beagle_local">
+ <execution provider="coaster" jobmanager="local:pbs"/>
+ <profile namespace="globus" key="jobsPerNode">1</profile>
+ <profile namespace="globus" key="lowOverAllocation">100</profile>
+ <profile namespace="globus" key="highOverAllocation">100</profile>
+ <profile namespace="globus" key="providerAttributes">pbs.aprun;pbs.mpp;depth=24</profile>
+ <!-- to use a beage reservation, eg:
+ <profile namespace="globus" key="providerAttributes">pbs.aprun;pbs.mpp;depth=24;pbs.resource_list=advres=wilde.1768</profile>
+ -->
+ <profile namespace="globus" key="maxtime">2400</profile>
+ <profile namespace="globus" key="maxWalltime">00:20:00</profile>
+ <profile namespace="globus" key="queue">batch</profile>
+ <profile namespace="globus" key="userHomeOverride">/lustre/beagle/BEAGLE_USERNAME/swiftwork</profile>
+ <profile namespace="globus" key="slots">1</profile>
+ <profile namespace="globus" key="maxnodes">4</profile>
+ <profile namespace="globus" key="nodeGranularity">1</profile>
+ <profile namespace="karajan" key="jobThrottle">1.00</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ <!-- <filesystem provider="local"/> -->
+ <!-- <workdirectory>/lustre/beagle/BEAGLE_USERNAME/swiftwork</workdirectory> -->
+ <workdirectory>/tmp/BEAGLE_USERNAME/swiftwork</workdirectory>
+ </pool>
+
+ <pool handle="beagle_remote">
+ <execution provider="coaster" jobmanager="ssh-cl:pbs" url="login4.beagle.ci.uchicago.edu"/>
+ <profile namespace="globus" key="jobsPerNode">4</profile>
+ <profile namespace="globus" key="lowOverAllocation">100</profile>
+ <profile namespace="globus" key="highOverAllocation">100</profile>
+ <profile namespace="globus" key="providerAttributes">pbs.aprun;pbs.mpp;depth=24;pbs.resource_list=advres=wilde.1768</profile>
+ <profile namespace="globus" key="maxtime">1200</profile>
+ <profile namespace="globus" key="maxWalltime">00:10:00</profile>
+ <profile namespace="globus" key="userHomeOverride">/lustre/beagle/BEAGLE_USERNAME/swiftwork</profile>
+ <profile namespace="globus" key="slots">1</profile>
+ <profile namespace="globus" key="maxnodes">1</profile>
+ <profile namespace="globus" key="nodeGranularity">1</profile>
+ <profile namespace="karajan" key="jobThrottle">1.00</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ <!-- <workdirectory>/tmp/BEAGLE_USERNAME/swiftwork</workdirectory> -->
+ <workdirectory>/lustre/beagle/BEAGLE_USERNAME/swiftwork</workdirectory>
+ </pool>
+
<pool handle="midway">
<execution provider="coaster" jobmanager="local:slurm"/>
<profile namespace="globus" key="queue">westmere</profile>
Deleted: SwiftApps/Swift-MapRed/mapred_combiner_V4/node_uniq.sh
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/node_uniq.sh 2013-10-24 20:14:30 UTC (rev 7228)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/node_uniq.sh 2013-10-24 20:38:19 UTC (rev 7229)
@@ -1,17 +0,0 @@
-#!/bin/bash
-
-ARGS=$*
-echo ${ARGS[*]} 1>&2
-
-SITE=${ARGS[0]}
-cat $(grep -l $SITE) > tmp
-
-LOCATIONS=($(awk '{ print $1 }' tmp | sort -u))
-
-for LOCATION in ${LOCATIONS[*]}
-do
- MATCH=$(grep $LOCATION tmp | awk '{print $2}')
- L=($(echo $LOCATION | sed 's/_/\ /'))
- echo ${L[0]} ${L[1]} ${MATCH[*]}
-done
-rm tmp;
Modified: SwiftApps/Swift-MapRed/mapred_combiner_V4/reduce_core.sh
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/reduce_core.sh 2013-10-24 20:14:30 UTC (rev 7228)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/reduce_core.sh 2013-10-24 20:38:19 UTC (rev 7229)
@@ -1,8 +1,3 @@
-reduce(){
- FILES=($*)
- DIRNAME=$(dirname ${FILES[0]})
- RESULT=$DIRNAME/$(hostname -f).$RANDOM.imd
- awk '{ sum += $1; count += $2 } END { print sum,count }' ${FILES[*]} > $RESULT
- echo $RESULT
- rm -rf ${FILES[*]} &> /dev/null
-}
+#!/bin/bash
+awk '{ sum += $1; count += $2 } END { print sum,count }' $*
+
Modified: SwiftApps/Swift-MapRed/mapred_combiner_V4/run.sh
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/run.sh 2013-10-24 20:14:30 UTC (rev 7228)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/run.sh 2013-10-24 20:38:19 UTC (rev 7229)
@@ -1,21 +1,8 @@
#!/bin/bash
-clear;
-#swift -tc.file tc.data -sites.file sites.xml motoko.swift
-#echo "Running run_catsn.swift"
-#swift -tc.file tc.data -sites.file sites.xml run_catsn.swift
-
-
-if [ "$1" != "" ]
-then
- SITE="$1"
-else
- SITE="beagle"
-fi
-
+clear;
./clean.sh
-
#Call as check_error $? <ERR_CODE> <Error message>
check_error()
{
@@ -26,6 +13,7 @@
echo "Run completed with code:$1"
fi
}
+
#export COG_OPTS="-Dtcp.channel.log.io.performance=true"
verify()
@@ -34,7 +22,7 @@
RECSIZE=$2
ACTUAL_RESULT=($(tail -n 1 final_result))
EXPECTED_RESULT=$(($LOOPS * $RECSIZE * 10000 ))
- if [ "$EXPECTED_RESULT" == "${ACTUAL_RESULT[1]}" ]
+ if [ "$EXPECTED_RESULT" == "${ACTUAL_RESULT[1]}" ]
then
echo "---------------------------------------------"
echo "COUNT matches expected records"
@@ -53,12 +41,12 @@
run_swift_chirp_combiner()
{
- LOOPS=$1
- RECSIZE=$2
- #./clean.sh &> /dev/null
- echo "================================================================" >> LAB_RECORDS
+ SITESXML=$1
+ LOOPS=$2
+ RECSIZE=$3
+ echo "================================================================" >> LAB_RECORDS
echo "RUNTYPE :SWIFT+Chirp (With local combiners), SITE:$SITE, CHUNKS:$LOOPS X FILESIZE:$RECSIZE " | tee -a LAB_RECORDS
- ( time -p swift -tc.file tc.data -config swift.properties -sites.file $SITE swiftreduce.swift -files=$LOOPS -filesize=$RECSIZE ) 2>&1 | tee -a LAB_RECORDS
+ ( time -p swift -tc.file tc.data -config swift.properties -sites.file $SITESXML swiftreduce.swift -files=$LOOPS -filesize=$RECSIZE ) 2>&1 | tee -a LAB_RECORDS
echo "chunks $LOOPS filesize $RECSIZE site $SITE type Swift+Chirp" >> LAB_RECORDS
verify $LOOPS $RECSIZE | tee -a LAB_RECORDS
echo "================================================================"
@@ -66,10 +54,8 @@
if [ "$2" != "" ] && [ "$3" != "" ]
then
- run_swift_chirp_combiner $2 $3
+ run_swift_chirp_combiner $*
else
- run_swift_chirp_combiner 20 10
+ run_swift_chirp_combiner multiple.xml 20 10
fi
-
-#run_swift_chirp_combiner 10 10;
exit 0
Modified: SwiftApps/Swift-MapRed/mapred_combiner_V4/setup.sh
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/setup.sh 2013-10-24 20:14:30 UTC (rev 7228)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/setup.sh 2013-10-24 20:38:19 UTC (rev 7229)
@@ -1,46 +1,28 @@
#!/bin/bash
+# Configure the following before proceeding the run the scripts
+
[ ! -z $BEAGLE_USERNAME ] || BEAGLE_USERNAME="yadunandb"
[ ! -z $MIDWAY_USERNAME ] || MIDWAY_USERNAME="yadunand"
+
+# Do not change
BEAGLE_SHARE_FOLDER=/lustre/beagle/$BEAGLE_USERNAME/bin/
MIDWAY_SHARE_FOLDER=/scratch/midway/$MIDWAY_USERNAME/bin/
+echo "$BEAGLE_SHARE_FOLDER is in PATH on BEAGLE"
+echo "MIDWAY_SHARE_FOLDER is in PATH on MIDWAY"
-if [ ! -f "map_core.sh" ]
-then
- echo "map_core.sh missing"
- exit 0;
-fi
-
-if [ ! -f "mapper_template.sh" ];
-then
- echo "mapper_template missing"
- exit 0;
-else
- sed -e "/\#USER_MAP/r map_core.sh" mapper_template.sh \
- > mapper.sh
- sed -i "s/BEAGLE_USERNAME/$BEAGLE_USERNAME/g" mapper.sh
- sed -i "s/MIDWAY_USERNAME/$MIDWAY_USERNAME/g" mapper.sh
- chmod 777 mapper.sh
+if [ -f "map_wrapper.sh" ];then
+ sed -i "s/BEAGLE\_USERNAME/$BEAGLE_USERNAME/g" map_wrapper.sh
+ sed -i "s/MIDWAY\_USERNAME/$MIDWAY_USERNAME/g" map_wrapper.sh
+ chmod 777 map_wrapper.sh
fi;
-if [ ! -f "reduce_core.sh" ]
-then
- echo "reduce_core.sh missing"
- exit 0;
+if [ ! -f "reduce_wrapper.sh" ];
+ sed -i "s/BEAGLE\_USERNAME/$BEAGLE_USERNAME/g" reduce_wrapper.sh
+ sed -i "s/MIDWAY\_USERNAME/$MIDWAY_USERNAME/g" reduce_wrapper.sh
+ chmod 777 reduce_wrapper.sh
fi;
-if [ ! -f "reducer_template.sh" ];
-then
- echo "reducer_template missing"
- exit 0;
-else
- sed -e "/\#USER_REDUCE/r reduce_core.sh" reducer_template.sh \
- > reducer.sh
- sed -i "s/BEAGLE_USERNAME/$BEAGLE_USERNAME/g" reducer.sh
- sed -i "s/MIDWAY_USERNAME/$MIDWAY_USERNAME/g" reducer.sh
- chmod 777 reducer.sh
-fi;
-
gcc server.c -o exec_server
gcc client.c -o exec_client
@@ -52,9 +34,17 @@
scp exec_server login4.beagle.ci.uchicago.edu:$BEAGLE_SHARE_FOLDER
scp exec_client login4.beagle.ci.uchicago.edu:$BEAGLE_SHARE_FOLDER
+sed -i "s/BEAGLE\_USERNAME/$BEAGLE_USERNAME/g" *xml
+sed -i "s/MIDWAY\_USERNAME/$MIDWAY_USERNAME/g" *xml
+# Todo extend chirp check to remote machines ?
+which chirp_server &>/dev/null
+if [ "$?" != "0" ];then
+ echo "ERROR: Chirp not available in system path"
+fi
+;
Modified: SwiftApps/Swift-MapRed/mapred_combiner_V4/swiftreduce.swift
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/swiftreduce.swift 2013-10-24 20:14:30 UTC (rev 7228)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/swiftreduce.swift 2013-10-24 20:38:19 UTC (rev 7229)
@@ -1,6 +1,7 @@
+//import "Swiftreduce_lib";
type file;
type script;
-// type fileptr will be a regular file to swift, but the contents of
+// type fileptr will be a regular file to swift, but the contents of
// which, will be
// interpreted as globally addressable paths to one or more files.
// The current format of a fileptr is:
@@ -9,9 +10,9 @@
// SiteY_Noden /path/to/file0 ...
type fileptr;
-app (fileptr out, file err) map_data (script run, int recsize)
+app (fileptr out, file err) map_data (script wrapper,script map, int recsize)
{
- bash @run recsize stdout=@out stderr=@err;
+ bash @wrapper @map recsize stdout=@out stderr=@err;
}
app (fileptr out) aggregate (script a, fileptr map_results[])
@@ -19,67 +20,56 @@
bash @a @map_results stdout=@out;
}
+// There must be one reduce function per site, and one for local
+app (file out, file err) reduce_bgl (script wrapper, script reducer, string cmd, fileptr all_map){
+ bgl_bash @wrapper @reducer cmd @all_map stdout=@out stderr=@err;
+}
+
+app (file out, file err) reduce_mid (script wrapper, script reducer, string cmd, fileptr all_map){
+ mid_bash @wrapper @reducer cmd @all_map stdout=@out stderr=@err;
+}
+
+app (file out, file err) reduce_local (script wrapper, script reducer, string cmd, file site_results[]){
+ loc_bash @wrapper @reducer cmd @site_results stdout=@out stderr=@err;
+}
+
+
int files = @toInt(@arg("files","0"));
int fsize = @toInt(@arg("filesize","10")); // This would make 10^4 records per file
+// List of sites:
string sites[] = ["beagle", "midway"];
+// Defaults:
+script map_wrapper <"map_wrapper.sh">;
+script red_wrapper <"reduce_wrapper.sh">;
+script aggr <"aggregate.sh">;
+// User scripts:
+//script map_func <"map_core.sh">;
+script map_func <"examples/map_genrands.sh">;
+//script red_func <"reduce_core.sh">;'
+script red_func <"examples/reduce_sort.sh">;
-script mapper <"mapper.sh">;
-script reducer <"reducer.sh">;
-script aggr <"aggregate.sh">;
fileptr map_out[] <simple_mapper; prefix="map", suffix=".out">;
file map_err[] <simple_mapper; prefix="map", suffix=".err">;
// Map stage. File pointers map_out[]
foreach item,i in [0:files-1] {
- (map_out[i], map_err[i]) = map_data (mapper, fsize);
+ (map_out[i], map_err[i]) = map_data (map_wrapper, map_func, fsize);
}
-// We don't want to send 100s of files to different sites.
-// One fileptr would do
+// Aggregate all fileptrs into one fileptr
fileptr all_map <"all_map.out">;
all_map = aggregate(aggr, map_out);
-
-app (file out, file err) reduce_bgl (script reducer, string cmd, fileptr all_map){
- bgl_bash @reducer cmd @all_map stdout=@out stderr=@err;
-}
-
-app (file out, file err) reduce_mid (script reducer, string cmd, fileptr all_map){
- mid_bash @reducer cmd @all_map stdout=@out stderr=@err;
-}
-
-app (file out, file err) reduce_local (script reducer, string cmd, file site_results[]){
- loc_bash @reducer cmd @site_results stdout=@out stderr=@err;
-}
-
+// Reduce/Local combine at node level
+// TODO: Fix/file bug for foreach loop with array inside issue
file redsite_out[] <simple_mapper; prefix="reduce", suffix=".out">;
file redsite_err[] <simple_mapper; prefix="reduce", suffix=".err">;
+(redsite_out[0], redsite_err[0]) = reduce_bgl (red_wrapper, red_func, "local", all_map);
+(redsite_out[1], redsite_err[1]) = reduce_mid (red_wrapper, red_func, "local", all_map);
-// Types :
-// Naive : Takes files and gives reduced result, No fileptr hanlding
-// L1/plain : Reduces all results within a site at an internal node
-// L2 : Node -> Site -> Global
-// L3_K_way : Node -> Inter-node -> Site -> Global
-
-/*
-// This is most likely a bug.
-// TODO: Report bug
-foreach site,s in sites{
- switch (s){
- case 0: // Beagle
- (redsite_out[s], redsite_err[s]) = reduce_bgl (reducer, "plain", all_map);
- case 1: // Midway
- (redsite_out[s], redsite_err[s]) = reduce_mid (reducer, "plain", all_map);
- default:
- tracef("Unknown site: ERROR\n");
- }
-}
-*/
-
-//(redsite_out[0], redsite_err[0]) = reduce_bgl (reducer, "local", all_map);
-(redsite_out[1], redsite_err[1]) = reduce_mid (reducer, "local", all_map);
+// Final global reduce
file final_out <"final_result">;
file final_err <"final_errors">;
-(final_out, final_err) = reduce_local (reducer, "naive", redsite_out);
+(final_out, final_err) = reduce_local (red_wrapper, red_func, "naive", redsite_out);
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.0.26283aa
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.0.26283aa (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.0.26283aa 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,6 @@
+0
+1
+2
+3
+4
+5
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.0.26283ab
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.0.26283ab (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.0.26283ab 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,6 @@
+6
+7
+8
+9
+10
+11
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.0.26283ac
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.0.26283ac (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.0.26283ac 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,6 @@
+12
+13
+14
+15
+16
+17
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.0.26283ad
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.0.26283ad (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.0.26283ad 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,3 @@
+18
+19
+20
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.22871aa
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.22871aa (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.22871aa 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,2 @@
+0
+1
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.22871ab
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.22871ab (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.22871ab 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,2 @@
+2
+3
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.22871ac
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.22871ac (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.22871ac 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,2 @@
+4
+5
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.26087aa
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.26087aa (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.26087aa 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,2 @@
+12
+13
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.26087ab
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.26087ab (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.26087ab 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,2 @@
+14
+15
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.26087ac
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.26087ac (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.26087ac 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,2 @@
+16
+17
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.32009aa
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.32009aa (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.32009aa 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,2 @@
+6
+7
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.32009ab
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.32009ab (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.32009ab 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,2 @@
+8
+9
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.32009ac
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.32009ac (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/test/K.1.32009ac 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,2 @@
+10
+11
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/test/k_way.sh
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/test/k_way.sh (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/test/k_way.sh 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,46 @@
+#!/bin/bash
+
+
+# We don't fetch because we don't know where files reside ?
+# actually we do know where files reside.
+
+leaf ()
+{
+ filepointers=($(cat $*))
+ center=${filepointers[0]}
+ echo "Accumulating ${filepointers[*]} at $center"
+ #exec_client
+}
+
+# Divide into K and Conquer
+foo()
+{
+ #echo "Foo() called with args : [$*]"
+ LEVEL=$1 ; shift
+ KWAY=$1 ; shift
+ FILEPTR=$1 ; shift
+ NODE_COUNT=`wc -l < $FILEPTR`
+ # In case what we have is a few nodes in the start case
+ if (( "$NODE_COUNT" <= "$KWAY" ))
+ then
+ leaf $FILEPTR
+ return
+ else
+ LINES_PER_SPLIT=$(($(( $NODE_COUNT + $KWAY - 1)) / $KWAY ))
+ #if (( "$LINES_PER_SPLIT" > "$KWAY" ))
+ ID=$RANDOM
+ #echo "Lines per split $NODE_COUNT > $KWAY"
+ split --lines=$LINES_PER_SPLIT $FILEPTR "K.$LEVEL.$ID"
+ for fptr in $(ls K.$LEVEL.$ID*)
+ do
+ {
+ #echo "Split[$LEVEL] : $fptr"
+ foo $(($LEVEL+1)) $KWAY $fptr
+ } &
+ done
+ wait
+ fi
+}
+
+foo 0 4 pointers
+#rm K*
Property changes on: SwiftApps/Swift-MapRed/mapred_combiner_V4/test/k_way.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: SwiftApps/Swift-MapRed/mapred_combiner_V4/test/pointers
===================================================================
--- SwiftApps/Swift-MapRed/mapred_combiner_V4/test/pointers (rev 0)
+++ SwiftApps/Swift-MapRed/mapred_combiner_V4/test/pointers 2013-10-24 20:38:19 UTC (rev 7229)
@@ -0,0 +1,21 @@
+0
+1
+2
+3
+4
+5
+6
+7
+8
+9
+10
+11
+12
+13
+14
+15
+16
+17
+18
+19
+20
More information about the Swift-commit
mailing list