[Swift-commit] r7180 - in SwiftTutorials/OSDC-2013-10-17: doc part06

davidk at ci.uchicago.edu davidk at ci.uchicago.edu
Fri Oct 18 13:41:11 CDT 2013


Author: davidk
Date: 2013-10-18 13:41:11 -0500 (Fri, 18 Oct 2013)
New Revision: 7180

Added:
   SwiftTutorials/OSDC-2013-10-17/part06/simulate.sh
   SwiftTutorials/OSDC-2013-10-17/part06/stats.sh
Modified:
   SwiftTutorials/OSDC-2013-10-17/doc/README
   SwiftTutorials/OSDC-2013-10-17/part06/p6.swift
Log:
Fix issue with part06, temporarily remove info about part05


Modified: SwiftTutorials/OSDC-2013-10-17/doc/README
===================================================================
--- SwiftTutorials/OSDC-2013-10-17/doc/README	2013-10-18 13:41:38 UTC (rev 7179)
+++ SwiftTutorials/OSDC-2013-10-17/doc/README	2013-10-18 18:41:11 UTC (rev 7180)
@@ -800,118 +800,8 @@
 Part 5: Controlling the compute-node pools where applications run
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-In this section we'll use the script `p5.swift`, very similar to
-`p4.swift` of the prior section, to show how we can route apps to
-specific sites, and also how we can make multiple pools of resources
-(on the same or on different computer systems) available to run a
-single Swift script.
+This section is under development.
 
-////
-image::part05.png[align="center"]
-
-.p5.swift
-----
-sys::[cat ../part05/p5.swift]
-----
-////
-
-First, lets specify that the analysis app `stats.sh` should be run on
-the local login node instead of on the cluster. This is done simply by
-change the site name field of the analyze app in the apps file:
-
------
-$ cat apps
-raven     simulate  simulate.sh
-localhost stats     stats.sh
------
-
-Running this with `swift p5.swift` we see:
------
-$ grep "on node:" output/*.log
-output/average.log:Running on node: raven
-output/sim_0.log:Running on node: nid00029
-output/sim_1.log:Running on node: nid00029
-output/sim_2.log:Running on node: nid00029
-output/sim_3.log:Running on node: nid00029
-output/sim_4.log:Running on node: nid00029
-output/sim_5.log:Running on node: nid00029
-output/sim_6.log:Running on node: nid00029
-output/sim_7.log:Running on node: nid00029
-output/sim_8.log:Running on node: nid00029
-output/sim_9.log:Running on node: nid00029
------
-
-
-FIXME:  Adjust to OSDC from Cray (show two pools of VMs):
-
-Now lets make further use of Swift's ability to route specific apps to
-specific pools of resources. The Cray Raven system has two node types,
-XE6 32-core 2 x IL-16, and XK7 16-core 1 x IL-16 plus one GPU.  Each
-"pool" of nodes has different queue characteristics. We can define
-these differences to Swift as two separate pools, and then spread the
-load of executing a large ensemble of simulations across all the
-pools. (And we'll continue to run the analysis script on a third pool,
-comprising the single login host.)
-
-We use the following `apps` file:
------
-$ cat multipools
-raven     simulate simulate.sh
-ravenGPU  simulate simulate.sh
-localhost stats    stats.sh
-----
-and we adjust the sites file to specify 4-node jobs in the Raven pool:
------
-    <profile namespace="globus" key="maxNodes">4</profile>
-    <profile namespace="globus" key="nodeGranularity">4</profile>
------
-This results in these PBS jobs:
------
-p01532 at raven:~> qstat -u $USER
-
-Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
---------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
-288687.sdb      p01532   small    B0827-3406   9919   4 128    --  00:59 R 00:00
-288688.sdb      p01532   gpu_node B0827-3406   9931   6  96    --  00:59 R 00:00
------
-... and achieves the following parallelism (of about 224 concurrent app tasks):
-
------
-$ swift -tc.file multipools p5.swift -nsim=1000 -steps=3
-Swift 0.94.1 RC2 swift-r6895 cog-r3765
-
-RunID: 20130827-1829-o3h6mht5
-Progress:  time: Tue, 27 Aug 2013 18:29:31 -0500
-Progress:  time: Tue, 27 Aug 2013 18:29:32 -0500  Initializing:997
-Progress:  time: Tue, 27 Aug 2013 18:29:34 -0500  Selecting site:774  Submitting:225  Submitted:1
-Progress:  time: Tue, 27 Aug 2013 18:29:35 -0500  Selecting site:774  Stage in:1  Submitted:225
-Progress:  time: Tue, 27 Aug 2013 18:29:36 -0500  Selecting site:774  Stage in:1  Submitted:37  Active:188
-Progress:  time: Tue, 27 Aug 2013 18:29:39 -0500  Selecting site:774  Submitted:2  Active:223  Stage out:1
-Progress:  time: Tue, 27 Aug 2013 18:29:40 -0500  Selecting site:750  Submitted:17  Active:208  Stage out:1  Finished successfully:24
-Progress:  time: Tue, 27 Aug 2013 18:29:41 -0500  Selecting site:640  Stage in:1  Submitted:51  Active:174  Finished successfully:134
-Progress:  time: Tue, 27 Aug 2013 18:29:42 -0500  Selecting site:551  Submitted:11  Active:214  Stage out:1  Finished successfully:223
-Progress:  time: Tue, 27 Aug 2013 18:29:43 -0500  Selecting site:542  Submitted:2  Active:223  Stage out:1  Finished successfully:232
-Progress:  time: Tue, 27 Aug 2013 18:29:44 -0500  Selecting site:511  Submitting:1  Submitted:19  Active:206  Finished successfully:263
-Progress:  time: Tue, 27 Aug 2013 18:29:45 -0500  Selecting site:463  Stage in:1  Submitted:43  Active:182  Finished successfully:311
-Progress:  time: Tue, 27 Aug 2013 18:29:46 -0500  Selecting site:367  Submitting:1  Submitted:38  Active:186  Stage out:1  Finished successfully:407
-Progress:  time: Tue, 27 Aug 2013 18:29:47 -0500  Selecting site:309  Submitted:2  Active:223  Stage out:1  Finished successfully:465
-Progress:  time: Tue, 27 Aug 2013 18:29:48 -0500  Selecting site:300  Submitted:2  Active:223  Stage out:1  Finished successfully:474
-Progress:  time: Tue, 27 Aug 2013 18:29:50 -0500  Selecting site:259  Submitted:11  Active:214  Stage out:1  Finished successfully:515
-Progress:  time: Tue, 27 Aug 2013 18:29:51 -0500  Selecting site:201  Stage in:1  Submitted:39  Active:186  Finished successfully:573
-Progress:  time: Tue, 27 Aug 2013 18:29:52 -0500  Selecting site:80  Submitted:42  Active:184  Finished successfully:694
-Progress:  time: Tue, 27 Aug 2013 18:29:53 -0500  Selecting site:54  Submitted:2  Active:223  Stage out:1  Finished successfully:720
-Progress:  time: Tue, 27 Aug 2013 18:29:54 -0500  Selecting site:32  Submitted:4  Active:220  Stage out:1  Finished successfully:743
-Progress:  time: Tue, 27 Aug 2013 18:29:55 -0500  Submitted:3  Active:216  Stage out:1  Finished successfully:780
-Progress:  time: Tue, 27 Aug 2013 18:29:56 -0500  Stage in:1  Active:143  Finished successfully:856
-Progress:  time: Tue, 27 Aug 2013 18:29:57 -0500  Active:38  Stage out:1  Finished successfully:961
-Progress:  time: Tue, 27 Aug 2013 18:29:58 -0500  Active:8  Stage out:1  Finished successfully:991
-Progress:  time: Tue, 27 Aug 2013 18:29:59 -0500  Stage out:1  Finished successfully:999
-Progress:  time: Tue, 27 Aug 2013 18:30:01 -0500  Stage in:1  Finished successfully:1000
-Progress:  time: Tue, 27 Aug 2013 18:30:02 -0500  Active:1  Finished successfully:1000
-Progress:  time: Tue, 27 Aug 2013 18:30:06 -0500  Stage out:1  Finished successfully:1000
-Final status: Tue, 27 Aug 2013 18:30:07 -0500  Finished successfully:1001
------
-
 Part 6: Specifying more complex workflow patterns
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -928,15 +818,14 @@
 sys::[cat ../part06/p6.swift]
 ----
 
-Note that the workflow is based on data flow dependencies: each simulation depends on the seed value, calculated in these two dependent statements:
+Note that the workflow is based on data flow dependencies: each simulation depends on the seed value, calculated in this statement:
 -----
 seedfile = genseed(1);
-int seedval = readData(seedfile);
 -----
 and on the bias file, computed and then consumed in these two dependent statements:
 -----
-  biasfile = genbias(1000, 20);
-  (simout,simlog) = simulation(steps, range, biasfile, 1000000, values);
+  biasfile = genbias(1000, 20, simulate_script);
+  (simout,simlog) = simulation(steps, range, biasfile, 1000000, values, simulate_script, seedfile);
 -----
 
 To run:

Modified: SwiftTutorials/OSDC-2013-10-17/part06/p6.swift
===================================================================
--- SwiftTutorials/OSDC-2013-10-17/part06/p6.swift	2013-10-18 13:41:38 UTC (rev 7179)
+++ SwiftTutorials/OSDC-2013-10-17/part06/p6.swift	2013-10-18 18:41:11 UTC (rev 7180)
@@ -13,10 +13,11 @@
 }
 
 app (file out, file log) simulation (int timesteps, int sim_range,
-                                     file bias_file, int scale, int sim_count, file sim_script)
+                                     file bias_file, int scale, int sim_count, 
+                                     file sim_script, file seed_file)
 {
   sh "simulate.sh" "-t" timesteps "-r" sim_range "-B" @bias_file "-x" scale
-           "-n" sim_count stdout=@out stderr=@log;
+           "-n" sim_count "-S" @seed_file stdout=@out stderr=@log;
 }
 
 app (file out, file log) analyze (file s[], file stat_script)
@@ -35,15 +36,11 @@
 
 file simulate_script <"simulate.sh">;
 file stats_script <"stats.sh">;
+file seedfile <"output/seed.dat">;        # Dynamically generated bias for simulation ensemble
 
 tracef("\n*** Script parameters: nsim=%i range=%i num values=%i\n\n", nsim, range, values);
-
-file seedfile<"output/seed.dat">;        # Dynamically generated bias for simulation ensemble
 seedfile = genseed(1,simulate_script);
 
-int seedval = readData(seedfile);
-tracef("Generated seed=%i\n", seedval);
-
 file sims[];                      # Array of files to hold each simulation output
 
 foreach i in [0:nsim-1] {
@@ -51,7 +48,7 @@
   file simout   <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
   file simlog   <single_file_mapper; file=@strcat("output/sim_",i,".log")>;
   biasfile = genbias(1000, 20, simulate_script);
-  (simout,simlog) = simulation(steps, range, biasfile, 1000000, values, simulate_script);
+  (simout,simlog) = simulation(steps, range, biasfile, 1000000, values, simulate_script, seedfile);
   sims[i] = simout;
 }
 

Added: SwiftTutorials/OSDC-2013-10-17/part06/simulate.sh
===================================================================
--- SwiftTutorials/OSDC-2013-10-17/part06/simulate.sh	                        (rev 0)
+++ SwiftTutorials/OSDC-2013-10-17/part06/simulate.sh	2013-10-18 18:41:11 UTC (rev 7180)
@@ -0,0 +1 @@
+link ../app/simulate.sh
\ No newline at end of file


Property changes on: SwiftTutorials/OSDC-2013-10-17/part06/simulate.sh
___________________________________________________________________
Added: svn:special
   + *

Added: SwiftTutorials/OSDC-2013-10-17/part06/stats.sh
===================================================================
--- SwiftTutorials/OSDC-2013-10-17/part06/stats.sh	                        (rev 0)
+++ SwiftTutorials/OSDC-2013-10-17/part06/stats.sh	2013-10-18 18:41:11 UTC (rev 7180)
@@ -0,0 +1 @@
+link ../app/stats.sh
\ No newline at end of file


Property changes on: SwiftTutorials/OSDC-2013-10-17/part06/stats.sh
___________________________________________________________________
Added: svn:special
   + *




More information about the Swift-commit mailing list