[Swift-commit] r7180 - in SwiftTutorials/OSDC-2013-10-17: doc part06
davidk at ci.uchicago.edu
davidk at ci.uchicago.edu
Fri Oct 18 13:41:11 CDT 2013
Author: davidk
Date: 2013-10-18 13:41:11 -0500 (Fri, 18 Oct 2013)
New Revision: 7180
Added:
SwiftTutorials/OSDC-2013-10-17/part06/simulate.sh
SwiftTutorials/OSDC-2013-10-17/part06/stats.sh
Modified:
SwiftTutorials/OSDC-2013-10-17/doc/README
SwiftTutorials/OSDC-2013-10-17/part06/p6.swift
Log:
Fix issue with part06, temporarily remove info about part05
Modified: SwiftTutorials/OSDC-2013-10-17/doc/README
===================================================================
--- SwiftTutorials/OSDC-2013-10-17/doc/README 2013-10-18 13:41:38 UTC (rev 7179)
+++ SwiftTutorials/OSDC-2013-10-17/doc/README 2013-10-18 18:41:11 UTC (rev 7180)
@@ -800,118 +800,8 @@
Part 5: Controlling the compute-node pools where applications run
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-In this section we'll use the script `p5.swift`, very similar to
-`p4.swift` of the prior section, to show how we can route apps to
-specific sites, and also how we can make multiple pools of resources
-(on the same or on different computer systems) available to run a
-single Swift script.
+This section is under development.
-////
-image::part05.png[align="center"]
-
-.p5.swift
-----
-sys::[cat ../part05/p5.swift]
-----
-////
-
-First, lets specify that the analysis app `stats.sh` should be run on
-the local login node instead of on the cluster. This is done simply by
-change the site name field of the analyze app in the apps file:
-
------
-$ cat apps
-raven simulate simulate.sh
-localhost stats stats.sh
------
-
-Running this with `swift p5.swift` we see:
------
-$ grep "on node:" output/*.log
-output/average.log:Running on node: raven
-output/sim_0.log:Running on node: nid00029
-output/sim_1.log:Running on node: nid00029
-output/sim_2.log:Running on node: nid00029
-output/sim_3.log:Running on node: nid00029
-output/sim_4.log:Running on node: nid00029
-output/sim_5.log:Running on node: nid00029
-output/sim_6.log:Running on node: nid00029
-output/sim_7.log:Running on node: nid00029
-output/sim_8.log:Running on node: nid00029
-output/sim_9.log:Running on node: nid00029
------
-
-
-FIXME: Adjust to OSDC from Cray (show two pools of VMs):
-
-Now lets make further use of Swift's ability to route specific apps to
-specific pools of resources. The Cray Raven system has two node types,
-XE6 32-core 2 x IL-16, and XK7 16-core 1 x IL-16 plus one GPU. Each
-"pool" of nodes has different queue characteristics. We can define
-these differences to Swift as two separate pools, and then spread the
-load of executing a large ensemble of simulations across all the
-pools. (And we'll continue to run the analysis script on a third pool,
-comprising the single login host.)
-
-We use the following `apps` file:
------
-$ cat multipools
-raven simulate simulate.sh
-ravenGPU simulate simulate.sh
-localhost stats stats.sh
-----
-and we adjust the sites file to specify 4-node jobs in the Raven pool:
------
- <profile namespace="globus" key="maxNodes">4</profile>
- <profile namespace="globus" key="nodeGranularity">4</profile>
------
-This results in these PBS jobs:
------
-p01532 at raven:~> qstat -u $USER
-
-Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
---------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
-288687.sdb p01532 small B0827-3406 9919 4 128 -- 00:59 R 00:00
-288688.sdb p01532 gpu_node B0827-3406 9931 6 96 -- 00:59 R 00:00
------
-... and achieves the following parallelism (of about 224 concurrent app tasks):
-
------
-$ swift -tc.file multipools p5.swift -nsim=1000 -steps=3
-Swift 0.94.1 RC2 swift-r6895 cog-r3765
-
-RunID: 20130827-1829-o3h6mht5
-Progress: time: Tue, 27 Aug 2013 18:29:31 -0500
-Progress: time: Tue, 27 Aug 2013 18:29:32 -0500 Initializing:997
-Progress: time: Tue, 27 Aug 2013 18:29:34 -0500 Selecting site:774 Submitting:225 Submitted:1
-Progress: time: Tue, 27 Aug 2013 18:29:35 -0500 Selecting site:774 Stage in:1 Submitted:225
-Progress: time: Tue, 27 Aug 2013 18:29:36 -0500 Selecting site:774 Stage in:1 Submitted:37 Active:188
-Progress: time: Tue, 27 Aug 2013 18:29:39 -0500 Selecting site:774 Submitted:2 Active:223 Stage out:1
-Progress: time: Tue, 27 Aug 2013 18:29:40 -0500 Selecting site:750 Submitted:17 Active:208 Stage out:1 Finished successfully:24
-Progress: time: Tue, 27 Aug 2013 18:29:41 -0500 Selecting site:640 Stage in:1 Submitted:51 Active:174 Finished successfully:134
-Progress: time: Tue, 27 Aug 2013 18:29:42 -0500 Selecting site:551 Submitted:11 Active:214 Stage out:1 Finished successfully:223
-Progress: time: Tue, 27 Aug 2013 18:29:43 -0500 Selecting site:542 Submitted:2 Active:223 Stage out:1 Finished successfully:232
-Progress: time: Tue, 27 Aug 2013 18:29:44 -0500 Selecting site:511 Submitting:1 Submitted:19 Active:206 Finished successfully:263
-Progress: time: Tue, 27 Aug 2013 18:29:45 -0500 Selecting site:463 Stage in:1 Submitted:43 Active:182 Finished successfully:311
-Progress: time: Tue, 27 Aug 2013 18:29:46 -0500 Selecting site:367 Submitting:1 Submitted:38 Active:186 Stage out:1 Finished successfully:407
-Progress: time: Tue, 27 Aug 2013 18:29:47 -0500 Selecting site:309 Submitted:2 Active:223 Stage out:1 Finished successfully:465
-Progress: time: Tue, 27 Aug 2013 18:29:48 -0500 Selecting site:300 Submitted:2 Active:223 Stage out:1 Finished successfully:474
-Progress: time: Tue, 27 Aug 2013 18:29:50 -0500 Selecting site:259 Submitted:11 Active:214 Stage out:1 Finished successfully:515
-Progress: time: Tue, 27 Aug 2013 18:29:51 -0500 Selecting site:201 Stage in:1 Submitted:39 Active:186 Finished successfully:573
-Progress: time: Tue, 27 Aug 2013 18:29:52 -0500 Selecting site:80 Submitted:42 Active:184 Finished successfully:694
-Progress: time: Tue, 27 Aug 2013 18:29:53 -0500 Selecting site:54 Submitted:2 Active:223 Stage out:1 Finished successfully:720
-Progress: time: Tue, 27 Aug 2013 18:29:54 -0500 Selecting site:32 Submitted:4 Active:220 Stage out:1 Finished successfully:743
-Progress: time: Tue, 27 Aug 2013 18:29:55 -0500 Submitted:3 Active:216 Stage out:1 Finished successfully:780
-Progress: time: Tue, 27 Aug 2013 18:29:56 -0500 Stage in:1 Active:143 Finished successfully:856
-Progress: time: Tue, 27 Aug 2013 18:29:57 -0500 Active:38 Stage out:1 Finished successfully:961
-Progress: time: Tue, 27 Aug 2013 18:29:58 -0500 Active:8 Stage out:1 Finished successfully:991
-Progress: time: Tue, 27 Aug 2013 18:29:59 -0500 Stage out:1 Finished successfully:999
-Progress: time: Tue, 27 Aug 2013 18:30:01 -0500 Stage in:1 Finished successfully:1000
-Progress: time: Tue, 27 Aug 2013 18:30:02 -0500 Active:1 Finished successfully:1000
-Progress: time: Tue, 27 Aug 2013 18:30:06 -0500 Stage out:1 Finished successfully:1000
-Final status: Tue, 27 Aug 2013 18:30:07 -0500 Finished successfully:1001
------
-
Part 6: Specifying more complex workflow patterns
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -928,15 +818,14 @@
sys::[cat ../part06/p6.swift]
----
-Note that the workflow is based on data flow dependencies: each simulation depends on the seed value, calculated in these two dependent statements:
+Note that the workflow is based on data flow dependencies: each simulation depends on the seed value, calculated in this statement:
-----
seedfile = genseed(1);
-int seedval = readData(seedfile);
-----
and on the bias file, computed and then consumed in these two dependent statements:
-----
- biasfile = genbias(1000, 20);
- (simout,simlog) = simulation(steps, range, biasfile, 1000000, values);
+ biasfile = genbias(1000, 20, simulate_script);
+ (simout,simlog) = simulation(steps, range, biasfile, 1000000, values, simulate_script, seedfile);
-----
To run:
Modified: SwiftTutorials/OSDC-2013-10-17/part06/p6.swift
===================================================================
--- SwiftTutorials/OSDC-2013-10-17/part06/p6.swift 2013-10-18 13:41:38 UTC (rev 7179)
+++ SwiftTutorials/OSDC-2013-10-17/part06/p6.swift 2013-10-18 18:41:11 UTC (rev 7180)
@@ -13,10 +13,11 @@
}
app (file out, file log) simulation (int timesteps, int sim_range,
- file bias_file, int scale, int sim_count, file sim_script)
+ file bias_file, int scale, int sim_count,
+ file sim_script, file seed_file)
{
sh "simulate.sh" "-t" timesteps "-r" sim_range "-B" @bias_file "-x" scale
- "-n" sim_count stdout=@out stderr=@log;
+ "-n" sim_count "-S" @seed_file stdout=@out stderr=@log;
}
app (file out, file log) analyze (file s[], file stat_script)
@@ -35,15 +36,11 @@
file simulate_script <"simulate.sh">;
file stats_script <"stats.sh">;
+file seedfile <"output/seed.dat">; # Dynamically generated bias for simulation ensemble
tracef("\n*** Script parameters: nsim=%i range=%i num values=%i\n\n", nsim, range, values);
-
-file seedfile<"output/seed.dat">; # Dynamically generated bias for simulation ensemble
seedfile = genseed(1,simulate_script);
-int seedval = readData(seedfile);
-tracef("Generated seed=%i\n", seedval);
-
file sims[]; # Array of files to hold each simulation output
foreach i in [0:nsim-1] {
@@ -51,7 +48,7 @@
file simout <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
file simlog <single_file_mapper; file=@strcat("output/sim_",i,".log")>;
biasfile = genbias(1000, 20, simulate_script);
- (simout,simlog) = simulation(steps, range, biasfile, 1000000, values, simulate_script);
+ (simout,simlog) = simulation(steps, range, biasfile, 1000000, values, simulate_script, seedfile);
sims[i] = simout;
}
Added: SwiftTutorials/OSDC-2013-10-17/part06/simulate.sh
===================================================================
--- SwiftTutorials/OSDC-2013-10-17/part06/simulate.sh (rev 0)
+++ SwiftTutorials/OSDC-2013-10-17/part06/simulate.sh 2013-10-18 18:41:11 UTC (rev 7180)
@@ -0,0 +1 @@
+link ../app/simulate.sh
\ No newline at end of file
Property changes on: SwiftTutorials/OSDC-2013-10-17/part06/simulate.sh
___________________________________________________________________
Added: svn:special
+ *
Added: SwiftTutorials/OSDC-2013-10-17/part06/stats.sh
===================================================================
--- SwiftTutorials/OSDC-2013-10-17/part06/stats.sh (rev 0)
+++ SwiftTutorials/OSDC-2013-10-17/part06/stats.sh 2013-10-18 18:41:11 UTC (rev 7180)
@@ -0,0 +1 @@
+link ../app/stats.sh
\ No newline at end of file
Property changes on: SwiftTutorials/OSDC-2013-10-17/part06/stats.sh
___________________________________________________________________
Added: svn:special
+ *
More information about the Swift-commit
mailing list