[Swift-commit] r6691 - in SwiftTutorials/uc3: . part01 part02 part03 part04 part05 part06 part10
davidk at ci.uchicago.edu
davidk at ci.uchicago.edu
Tue Jul 30 13:38:04 CDT 2013
Author: davidk
Date: 2013-07-30 13:38:04 -0500 (Tue, 30 Jul 2013)
New Revision: 6691
Removed:
SwiftTutorials/uc3/part01/README
SwiftTutorials/uc3/part02/README
SwiftTutorials/uc3/part03/README
SwiftTutorials/uc3/part04/README
SwiftTutorials/uc3/part05/README
SwiftTutorials/uc3/part06/README
Modified:
SwiftTutorials/uc3/README
SwiftTutorials/uc3/part10/dynamic_bias.dat
Log:
Updated README in asciidoc
Modified: SwiftTutorials/uc3/README
===================================================================
--- SwiftTutorials/uc3/README 2013-07-30 18:35:01 UTC (rev 6690)
+++ SwiftTutorials/uc3/README 2013-07-30 18:38:04 UTC (rev 6691)
@@ -1,51 +1,439 @@
-UC3 Swift mini-tutorial
+Swift UC3 mini-tutorial
+=======================
-To set up:
+Installing UC3 tutorial
+-----------------------
- $ source setup.sh # sets swift config files in $HOME/.swift
+Check out scripts from SVN
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+To checkout the most recent UC3 tutorial scripts from SVN, run the following
+command:
- $ swift -version # verify that Swift 0.94 is in your $PATH and functional
+-----
+$ svn co https://svn.ci.uchicago.edu/svn/vdl2/SwiftTutorials/uc3
+-----
-There are 3 example Swift scripts, p00.swift, p01.swift, and p02.swift
+This will create a directory called uc3 which contains all of the scripts
+mentioned in this document.
+Run setup
+~~~~~~~~~
+Once the scripts are checked out, run the following commands to perform
+the initial setup.
+
+-----
+$ cd uc3 # change to the newly created uc3 directory
+$ source setup.sh # sets swift config files in $HOME/.swift
+$ swift -version # verify that Swift 0.94 is in your $PATH and functional
+-----
+
+Overview of the applications
+----------------------------
+There are two shell scripts included that act as a mock science application:
+simulation.sh and stats.sh
+
+simulation.sh
+~~~~~~~~~~~~~
+The simulation.sh script generates and prints a random number. It optionally
+takes the following arguments:
+
+.simulation.sh arguments
+[options="header"]
+|=======================
+|Argument number|Description
+|1 |runtime. Set how long simulation.sh should run, in seconds.
+|2 |range. Limit random numbers to a given range.
+|3 |biasfile. Look a number contained within this file to set bias.
+|4 |scale. Scale random number by this factor.
+|5 |n. Generate n number of random numbers.
+|=======================
+
+With no arguments, simulate.sh prints 1 number in the range of 1-100.
+-----
+$ ./simulate.sh
+96
+-----
+
+stats.sh
+~~~~~~~~
+The stats.sh script reads a file containing n numbers and prints the average
+of those numbers.
+
+Overview of the Swift scripts
+------------------------------
+Parts 1-6 run locally and serve as examples of the Swift language.
+Parts 7-9 submit jobs via Condor to UC3 resources
+
+part01
+~~~~~~
+The first swift script, p1.swift, runs simulate.sh to generate a single random
+number. It writes the number to a file.
+
+image:p1.png[]
+
+.p1.swift
+-----
+type file;
+
+app (file o) mysim ()
+{
+ simulate stdout=@filename(o);
+}
+
+file f = mysim();
+-----
+
+To run this script, run the following command:
+-----
+$ cd part01
+$ swift p1.swift
+-----
+
+The simulate application gets translated to simulate.sh within the 'apps' file.
+
+NOTE: Since the file you created is not named, swift will generate a random
+name for the file in a directory called _concurrent. To view the created
+output, run "cat _concurrent/*"
+
+To cleanup the directory and remove all outputs, run:
+-----
+$ ./cleanup.sh
+------
+
+part02
+~~~~~~
+The second swift script shows an example of naming the file. The output is now
+in a file called sim.out.
+
+image:p2.png[]
+
+.p2.swift
+-----
+type file;
+
+app (file o) mysim ()
+{
+ simulate stdout=@filename(o);
+}
+
+file f <"sim.out">;
+f = mysim();
+-----
+
+To run the script:
+-----
+$ cd part02
+$ swift p2.swift
+-----
+
+part03
+~~~~~~
+The p3.swift script introduces the foreach loop. This script runs many
+simulations. Output files are named here by Swift and will get created
+in the _concurrent directory.
+
+image:p3.png[]
+
+.p3.swift
+----
+type file;
+
+app (file o) mysim ()
+{
+ simulate stdout=@filename(o);
+}
+
+foreach i in [0:9] {
+ file f = mysim();
+}
+----
+
To run:
+----
+$ cd part03
+$ swift p3.swift
+----
- $ find data/ # Examine the initial data directory
- $ swift p00.swift # Runs 10 simulations
- $ find data/ # Examine the results in the data directory
+part04
+~~~~~~
+Part 4 gives an example of naming multiple files within a foreach loop.
- $ swift p01.swift # Simple workflow: Runs simulations, then analyzes results
- $ find data/ # Examine the input data directory
+image:p4.png[]
- $ swift p02.swift # More complex workflow with more dependencies
- $ find data/ # Examine the input data directory
+.p4.swift
+----
+type file;
+app (file o) mysim ()
+{
+ simulate stdout=@filename(o);
+}
-To experiment with the command line parameters of these scripts:
+foreach i in [0:9] {
+ file f <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
+ f = mysim();
+}
+----
- For p01.swift:
+To run:
+----
+$ swift p4.swift
+----
- $ swift p00.swift -nsim=100 -range=100000
+Output files will be named output/sim_N.out.
- # Note: you must use the -name=value convention!
-
- Parameters and their default values for p01.swift:
+part05
+~~~~~~
+Part 5 introduces a postprocessing step. After many simulations have run, the files
+created by simulation.sh will be sent to stats.sh for averaging.
- nsim 10 # number of simulation programs to run
- range 100 # range of the generated random numbers
+image:p5.png[]
- For p01.swift:
+.p5.swift
+----
+type file;
- nsim 10 # number of simulation programs to run
- steps 1 # number of "steps" each simulation (==seconds of runtime)
- range 100 # range of the generated random numbers
- count 10 # number of random numbers generated per simulation
+app (file o) mysim ()
+{
+ simulate stdout=@filename(o);
+}
- For p02.swift:
+app (file o) analyze (file s[])
+{
+ stats @filenames(s) stdout=@filename(o);
+}
- nsim 10 # number of simulation programs to run
- range 100 # range of the generated random numbers
- count 10 # number of random numbers generated per simulation
+file sims[];
- (steps value is determined dynamically in p02.swift script)
+int nsim = @toInt(@arg("nsim","10"));
+foreach i in [0:nsim-1] {
+ file simout <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
+ simout = mysim();
+ sims[i] = simout;
+}
+
+file stats<"output/average.out">;
+stats = analyze(sims);
+----
+
+To run:
+----
+$ swift p5.swift
+----
+
+part06
+~~~~~~
+Part 6 introduces command line arguments. The script sets a variable called
+"steps" here, which determines the length of time that the simulation.sh
+will run for. It also defines a variable called nsim, which determines the
+number of simulations to run.
+
+image:p6.png[]
+
+.p6.swift
+----
+type file;
+
+app (file o) mysim (int timesteps)
+{
+ simulate timesteps stdout=@filename(o);
+}
+
+app (file o) analyze (file s[])
+{
+ stats @filenames(s) stdout=@filename(o);
+}
+
+file sims[];
+int nsim = @toInt(@arg("nsim","10"));
+int steps = @toInt(@arg("steps","1"));
+
+foreach i in [0:nsim-1] {
+ file simout <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
+ simout = mysim(steps);
+ sims[i] = simout;
+}
+
+file stats<"output/average.out">;
+stats = analyze(sims);
+----
+
+Use the command below to specify the time for each simulation.
+----
+$ cd part06
+$ swift p6.swift -steps=3 # each simulation takes 3 seconds
+----
+
+part07
+~~~~~~
+Part 7 is the first script that will submit jobs to UC3 via Condor.
+It is similar to earlier scripts, with a few minor exceptions. Since
+there is not a shared filesystems when using OSG, the application simulate.sh
+will get transferred to the worker node by Swift.
+
+image:p7.png[]
+
+.p7.swift
+-----
+type file;
+
+# Application to be called by this script
+
+file simulation_script <"simulate.sh">;
+
+# app() functions for application programs to be called:
+
+app (file out) simulation (file script, int timesteps, int sim_range)
+{
+ sh @filename(script) timesteps sim_range stdout=@filename(out);
+}
+
+# Command line params to this script:
+
+int nsim = @toInt(@arg("nsim", "10")); # number of simulation programs to run
+int range = @toInt(@arg("range", "100")); # range of the generated random numbers
+
+# Main script and data
+
+int steps=3;
+
+tracef("\n*** Script parameters: nsim=%i steps=%i range=%i \n\n", nsim, steps, range);
+
+foreach i in [0:nsim-1] {
+ file simout <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
+ simout = simulation(simulation_script, steps, range);
+}
+-----
+
+To run:
+----
+$ cd part07
+$ swift p7.swift
+----
+
+part08
+~~~~~~
+Part 8 will also stage in and run stats.sh to calculate averages. It adds a
+trace statement so you can see the order in which things execute.
+
+image:p8.png[]
+
+.p8.swift
+-----
+type file;
+
+# Applications to be called by this script
+
+file simulation_script <"simulate.sh">;
+file analysis_script <"stats.sh">;
+
+# app() functions for application programs to be called:
+
+app (file out) simulation (file script, int timesteps, int sim_range, file bias_file, int scale, int sim_count)
+{
+ sh @filename(script) timesteps sim_range @filename(bias_file) scale sim_count stdout=@filename(out);
+}
+
+app (file out) analyze (file script, file s[])
+{
+ sh @script @filenames(s) stdout=@filename(out);
+}
+
+# Command line params to this script:
+
+int nsim = @toInt(@arg("nsim", "10")); # number of simulation programs to run
+int steps = @toInt(@arg("steps", "1")); # number of "steps" each simulation (==seconds of runtime)
+int range = @toInt(@arg("range", "100")); # range of the generated random numbers
+int count = @toInt(@arg("count", "10")); # number of random numbers generated per simulation
+
+# Main script and data
+
+tracef("\n*** Script parameters: nsim=%i steps=%i range=%i count=%i\n\n", nsim, steps, range, count);
+
+file sims[]; # Array of files to hold each simulation output
+file bias<"bias.dat">; # Input data file to "bias" the numbers:
+ # 1 line: scale offset ( N = n*scale + offset)
+foreach i in [0:nsim-1] {
+ file simout <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
+ simout = simulation(simulation_script, steps, range, bias, 100000, count);
+ sims[i] = simout;
+}
+
+file stats<"output/stats.out">; # Final output file: average of all "simulations"
+stats = analyze(analysis_script,sims);
+-----
+
+To run:
+----
+$ cd part08
+$ swift p8.swift
+----
+
+part09
+~~~~~~
+Part 9 adds another app function called genrand. Genrand will produce a random
+number that will be used to determine how long each simulation app will run.
+
+image:p9.png[]
+
+.p9.swift
+-----
+type file;
+
+# Applications to be called by this script
+
+file simulation_script <"simulate.sh">;
+file analysis_script <"stats.sh">;
+
+# app() functions for application programs to be called:
+
+app (file out) genrand (file script, int timesteps, int sim_range)
+{
+ sh @filename(script) timesteps sim_range stdout=@filename(out);
+}
+
+app (file out) simulation (file script, int timesteps, int sim_range, file bias_file, int scale, int sim_count)
+{
+ sh @filename(script) timesteps sim_range @filename(bias_file) scale sim_count stdout=@filename(out);
+}
+
+app (file out) analyze (file script, file s[])
+{
+ sh @script @filenames(s) stdout=@filename(out);
+}
+
+# Command line params to this script:
+int nsim = @toInt(@arg("nsim", "10")); # number of simulation programs to run
+int range = @toInt(@arg("range", "100")); # range of the generated random numbers
+int count = @toInt(@arg("count", "10")); # number of random numbers generated per simulation
+
+# Main script and data
+
+tracef("\n*** Script parameters: nsim=%i range=%i count=%i\n\n", nsim, range, count);
+
+file bias<"dynamic_bias.dat">; # Dynamically generated bias for simulation ensemble
+
+bias = genrand(simulation_script, 1, 1000);
+
+file sims[]; # Array of files to hold each simulation output
+
+foreach i in [0:nsim-1] {
+
+ int steps = readData(genrand(simulation_script, 1, 5));
+ tracef(" for simulation[%i] steps=%i\n", i, steps+1);
+
+ file simout <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
+ simout = simulation(simulation_script, steps+1, range, bias, 100000, count);
+ sims[i] = simout;
+}
+
+file stats<"output/stats.out">; # Final output file: average of all "simulations"
+stats = analyze(analysis_script,sims);
+-----
+
+To run:
+----
+$ cd part09
+$ swift p9.swift
+----
Deleted: SwiftTutorials/uc3/part01/README
===================================================================
--- SwiftTutorials/uc3/part01/README 2013-07-30 18:35:01 UTC (rev 6690)
+++ SwiftTutorials/uc3/part01/README 2013-07-30 18:38:04 UTC (rev 6691)
@@ -1,40 +0,0 @@
-
-TOPIC: First Swift script - run a "simulation" program, random.sh
-
-
-Verify that random.sh is in your path, and that it works:
-
-$ which random.sh
-$ random.sh
-
-Study the Swift program:
-
-$ cat p1.swift
-
-Examine the tool catalog file:
-
-$ cat tc
-
-Run the program:
-
-$ swift -tc.file tc p1.swift
-
-Look at the output:
-
-$ cat _concurrent/*
-
-Look at the other files generated:
-
-$ ls -l
-
-We'll talk more about these files later
-
-$ cleanup
-
-Then test the sript a few more times.
-
-$ swift -tc.file tc p1.swift; cat _concurrent/*
-
-$ swift -tc.file tc p1.swift; cat _concurrent/*
-
-$ swift -tc.file tc p1.swift; cat _concurrent/*
Deleted: SwiftTutorials/uc3/part02/README
===================================================================
--- SwiftTutorials/uc3/part02/README 2013-07-30 18:35:01 UTC (rev 6690)
+++ SwiftTutorials/uc3/part02/README 2013-07-30 18:38:04 UTC (rev 6691)
@@ -1,33 +0,0 @@
-
-TOPIC: specifying the output filenames
-
-Study the Swift program:
-
-$ cat p2.swift
-
-Examine the tool catalog file:
-
-$ cat tc
-
-Run the program:
-
-$ swift -tc.file tc p2.swift
-
-Look at the output:
-
-$ cat sim.out
-
-Then test the sript a few more times.
-
-$ swift -tc.file tc p1.swift; cat sim.out
-
-$ swift -tc.file tc p1.swift; cat sim.out
-
-$ swift -tc.file tc p1.swift; cat sim.out
-
-Things to try:
-
-- change the output filename
-
-- add a dir component to the output filename
-
Deleted: SwiftTutorials/uc3/part03/README
===================================================================
--- SwiftTutorials/uc3/part03/README 2013-07-30 18:35:01 UTC (rev 6690)
+++ SwiftTutorials/uc3/part03/README 2013-07-30 18:38:04 UTC (rev 6691)
@@ -1,20 +0,0 @@
-
-TOPIC: run many instances of a "simulation" program in a foreach() loop
-
-
-Run the program:
-
-$ swift -tc.file tc p3.swift
-
-Look at the output:
-
-$ cd _concurrent
-$ ls -l
-$ cat *
-
-Look at the order in which the files were written:
-
-$ ls -lt
-
-We'll come back to this...
-
Deleted: SwiftTutorials/uc3/part04/README
===================================================================
--- SwiftTutorials/uc3/part04/README 2013-07-30 18:35:01 UTC (rev 6690)
+++ SwiftTutorials/uc3/part04/README 2013-07-30 18:38:04 UTC (rev 6691)
@@ -1,42 +0,0 @@
-
-TOPIC: run a program in a foreach() loop with explicitly named output files.
-
-
-Study the program:
-
-- @strcat() is a Swift builtin function (the @name() form is used for many but not all builtins)
-
-- Note that @strcat() coerces numeric args to strings
-
-- <"filename"> was a shorthand for <single_file_mapper;...> but the latter is more flexible.
-
-Run the program:
-
-$ swift -tc.file tc p4.swift
-
-Look at the output:
-
-$ cd output
-$ ls -l
-$ cat *
-
-
-Try:
-
-- Adjust the script to take the number of simulations to do from the command line
-
- @arg("argname") returns arguments after the .swift script on the swift command line:
-
- swift -tc.file tc p4.swift -myarg1=value1 ...
-
- @arg("argname","default") returns "default" if the argname is not given on the command line
-
- @toInt() converts strings to integers:
-
- @toInt(@arg("myArg1"))
-
-
-- trace(expr1,...) traces expressions on stdout.
-
- Try inserting a few traces. But remember: Swift is *very* concurrent!
-
Deleted: SwiftTutorials/uc3/part05/README
===================================================================
--- SwiftTutorials/uc3/part05/README 2013-07-30 18:35:01 UTC (rev 6690)
+++ SwiftTutorials/uc3/part05/README 2013-07-30 18:38:04 UTC (rev 6691)
@@ -1,51 +0,0 @@
-
-
-TOPIC: Add a program that "analyzes" the output the parallel simulations
-
-
-Study the program:
-
-We added a new app to "analyze" all the simulation outputs:
-
-app (file o) analyze (file s[])
-{
- average @filenames(s) stdout=@filename(o);
-}
-
-Nots that it expects an array of files as its input, and that
-it uses @filenames() to place the names of all the files on
-the command line to the "average" program.
-
-Average does:
---------------------
-#! /bin/sh
-
-awk '
-
-{ sum += $1}
-
-END { print sum/NR }
-' $*
--------------------
-
-
-At the end of the script we insert:
-
-file stats<"output/average.out">;
-stats = analyze(sims);
-
-...which runs the analysis *after* all the simulations complete.
-
-
-Run the program:
-
-$ swift -tc.file tc p5.swift
-
-Look at the output:
-
-$ cd output
-$ ls -l
-$ cat sim*.out
-$ cat average.out
-
-
Deleted: SwiftTutorials/uc3/part06/README
===================================================================
--- SwiftTutorials/uc3/part06/README 2013-07-30 18:35:01 UTC (rev 6690)
+++ SwiftTutorials/uc3/part06/README 2013-07-30 18:38:04 UTC (rev 6691)
@@ -1,28 +0,0 @@
-
-TOPIC: add arguments to the "simulation" program
-(to control number of steps, range of output, and number of output lines)
-
-
-Test new versions of the "simulation" program:
-
- random.sh # original
- random2.sh [runtime] # runtime in seconds; defaults to 1 sec
- random3.sh [runtime] [range] # requests random numbers in {0:range-1}
- random4.sh [runtime] [range] [count] # requests "count" random numbers in {0:range}
-
-p6.swift changes to random2 for the simulation program and adds an argument.
-
-Run the program:
-
-$ swift -tc.file tc p6.swift -steps=3 # each simulation takes 3 seconds
-
-Next we need to use the parallel cluster for our simulations!
-
-Try:
-
-- change to random3.sh and add the extra argument "range"
-
-- change to random4.sh and add yet another argument "count"
-
-
-
Modified: SwiftTutorials/uc3/part10/dynamic_bias.dat
===================================================================
--- SwiftTutorials/uc3/part10/dynamic_bias.dat 2013-07-30 18:35:01 UTC (rev 6690)
+++ SwiftTutorials/uc3/part10/dynamic_bias.dat 2013-07-30 18:38:04 UTC (rev 6691)
@@ -1 +1 @@
-781
+208
More information about the Swift-commit
mailing list