[Swift-commit] r6681 - in SwiftTutorials/UofC_2013-07-29-uc3: . part01 part02 part03 part04 part05 part06 part07 part08 part09 part10
davidk at ci.uchicago.edu
davidk at ci.uchicago.edu
Mon Jul 29 15:22:35 CDT 2013
Author: davidk
Date: 2013-07-29 15:22:35 -0500 (Mon, 29 Jul 2013)
New Revision: 6681
Added:
SwiftTutorials/UofC_2013-07-29-uc3/README
SwiftTutorials/UofC_2013-07-29-uc3/part01/
SwiftTutorials/UofC_2013-07-29-uc3/part01/README
SwiftTutorials/UofC_2013-07-29-uc3/part01/p1.swift
SwiftTutorials/UofC_2013-07-29-uc3/part01/tc
SwiftTutorials/UofC_2013-07-29-uc3/part02/
SwiftTutorials/UofC_2013-07-29-uc3/part02/README
SwiftTutorials/UofC_2013-07-29-uc3/part02/p2.swift
SwiftTutorials/UofC_2013-07-29-uc3/part02/p2b.swift
SwiftTutorials/UofC_2013-07-29-uc3/part02/tc
SwiftTutorials/UofC_2013-07-29-uc3/part03/
SwiftTutorials/UofC_2013-07-29-uc3/part03/README
SwiftTutorials/UofC_2013-07-29-uc3/part03/p3.swift
SwiftTutorials/UofC_2013-07-29-uc3/part03/tc
SwiftTutorials/UofC_2013-07-29-uc3/part04/
SwiftTutorials/UofC_2013-07-29-uc3/part04/README
SwiftTutorials/UofC_2013-07-29-uc3/part04/p4.swift
SwiftTutorials/UofC_2013-07-29-uc3/part04/tc
SwiftTutorials/UofC_2013-07-29-uc3/part05/
SwiftTutorials/UofC_2013-07-29-uc3/part05/README
SwiftTutorials/UofC_2013-07-29-uc3/part05/p5.swift
SwiftTutorials/UofC_2013-07-29-uc3/part05/tc
SwiftTutorials/UofC_2013-07-29-uc3/part06/
SwiftTutorials/UofC_2013-07-29-uc3/part06/README
SwiftTutorials/UofC_2013-07-29-uc3/part06/p6.swift
SwiftTutorials/UofC_2013-07-29-uc3/part06/tc
SwiftTutorials/UofC_2013-07-29-uc3/part07/
SwiftTutorials/UofC_2013-07-29-uc3/part07/README
SwiftTutorials/UofC_2013-07-29-uc3/part07/p7.swift
SwiftTutorials/UofC_2013-07-29-uc3/part07/sites.xml
SwiftTutorials/UofC_2013-07-29-uc3/part07/tc
SwiftTutorials/UofC_2013-07-29-uc3/part08/
SwiftTutorials/UofC_2013-07-29-uc3/part08/apps
SwiftTutorials/UofC_2013-07-29-uc3/part08/p00.swift
SwiftTutorials/UofC_2013-07-29-uc3/part08/sites.xml
SwiftTutorials/UofC_2013-07-29-uc3/part09/
SwiftTutorials/UofC_2013-07-29-uc3/part09/apps
SwiftTutorials/UofC_2013-07-29-uc3/part09/p01.swift
SwiftTutorials/UofC_2013-07-29-uc3/part09/sites.xml
SwiftTutorials/UofC_2013-07-29-uc3/part10/
SwiftTutorials/UofC_2013-07-29-uc3/part10/apps
SwiftTutorials/UofC_2013-07-29-uc3/part10/p02.swift
SwiftTutorials/UofC_2013-07-29-uc3/part10/sites.xml
SwiftTutorials/UofC_2013-07-29-uc3/setup.sh
SwiftTutorials/UofC_2013-07-29-uc3/simulate.sh
SwiftTutorials/UofC_2013-07-29-uc3/stats.sh
Log:
Initial commit of scripts for uc3 demo
Added: SwiftTutorials/UofC_2013-07-29-uc3/README
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/README (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/README 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,51 @@
+UC3 Swift mini-tutorial
+
+To set up:
+
+ $ source setup.sh # sets swift config files in $HOME/.swift
+
+ $ swift -version # verify that Swift 0.94 is in your $PATH and functional
+
+There are 3 example Swift scripts, p00.swift, p01.swift, and p02.swift
+
+To run:
+
+ $ find data/ # Examine the initial data directory
+ $ swift p00.swift # Runs 10 simulations
+ $ find data/ # Examine the results in the data directory
+
+ $ swift p01.swift # Simple workflow: Runs simulations, then analyzes results
+ $ find data/ # Examine the input data directory
+
+ $ swift p02.swift # More complex workflow with more dependencies
+ $ find data/ # Examine the input data directory
+
+
+To experiment with the command line parameters of these scripts:
+
+ For p01.swift:
+
+ $ swift p00.swift -nsim=100 -range=100000
+
+ # Note: you must use the -name=value convention!
+
+ Parameters and their default values for p01.swift:
+
+ nsim 10 # number of simulation programs to run
+ range 100 # range of the generated random numbers
+
+ For p01.swift:
+
+ nsim 10 # number of simulation programs to run
+ steps 1 # number of "steps" each simulation (==seconds of runtime)
+ range 100 # range of the generated random numbers
+ count 10 # number of random numbers generated per simulation
+
+ For p02.swift:
+
+ nsim 10 # number of simulation programs to run
+ range 100 # range of the generated random numbers
+ count 10 # number of random numbers generated per simulation
+
+ (steps value is determined dynamically in p02.swift script)
+
Added: SwiftTutorials/UofC_2013-07-29-uc3/part01/README
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part01/README (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part01/README 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,40 @@
+
+TOPIC: First Swift script - run a "simulation" program, random.sh
+
+
+Verify that random.sh is in your path, and that it works:
+
+$ which random.sh
+$ random.sh
+
+Study the Swift program:
+
+$ cat p1.swift
+
+Examine the tool catalog file:
+
+$ cat tc
+
+Run the program:
+
+$ swift -tc.file tc p1.swift
+
+Look at the output:
+
+$ cat _concurrent/*
+
+Look at the other files generated:
+
+$ ls -l
+
+We'll talk more about these files later
+
+$ cleanup
+
+Then test the sript a few more times.
+
+$ swift -tc.file tc p1.swift; cat _concurrent/*
+
+$ swift -tc.file tc p1.swift; cat _concurrent/*
+
+$ swift -tc.file tc p1.swift; cat _concurrent/*
Added: SwiftTutorials/UofC_2013-07-29-uc3/part01/p1.swift
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part01/p1.swift (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part01/p1.swift 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,8 @@
+type file;
+
+app (file o) mysim ()
+{
+ random stdout=@filename(o);
+}
+
+file f = mysim();
Added: SwiftTutorials/UofC_2013-07-29-uc3/part01/tc
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part01/tc (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part01/tc 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1 @@
+localhost random random.sh
Added: SwiftTutorials/UofC_2013-07-29-uc3/part02/README
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part02/README (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part02/README 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,33 @@
+
+TOPIC: specifying the output filenames
+
+Study the Swift program:
+
+$ cat p2.swift
+
+Examine the tool catalog file:
+
+$ cat tc
+
+Run the program:
+
+$ swift -tc.file tc p2.swift
+
+Look at the output:
+
+$ cat sim.out
+
+Then test the sript a few more times.
+
+$ swift -tc.file tc p1.swift; cat sim.out
+
+$ swift -tc.file tc p1.swift; cat sim.out
+
+$ swift -tc.file tc p1.swift; cat sim.out
+
+Things to try:
+
+- change the output filename
+
+- add a dir component to the output filename
+
Added: SwiftTutorials/UofC_2013-07-29-uc3/part02/p2.swift
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part02/p2.swift (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part02/p2.swift 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,9 @@
+type file;
+
+app (file o) mysim ()
+{
+ random stdout=@filename(o);
+}
+
+file f<"sim.out">;
+f = mysim();
Added: SwiftTutorials/UofC_2013-07-29-uc3/part02/p2b.swift
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part02/p2b.swift (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part02/p2b.swift 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,9 @@
+type file;
+
+app (file o) mysim ()
+{
+ random stdout=@filename(o);
+}
+
+file f<"output/sim.out">;
+f = mysim();
Added: SwiftTutorials/UofC_2013-07-29-uc3/part02/tc
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part02/tc (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part02/tc 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1 @@
+localhost random random.sh
Added: SwiftTutorials/UofC_2013-07-29-uc3/part03/README
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part03/README (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part03/README 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,20 @@
+
+TOPIC: run many instances of a "simulation" program in a foreach() loop
+
+
+Run the program:
+
+$ swift -tc.file tc p3.swift
+
+Look at the output:
+
+$ cd _concurrent
+$ ls -l
+$ cat *
+
+Look at the order in which the files were written:
+
+$ ls -lt
+
+We'll come back to this...
+
Added: SwiftTutorials/UofC_2013-07-29-uc3/part03/p3.swift
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part03/p3.swift (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part03/p3.swift 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,11 @@
+type file;
+
+app (file o) mysim ()
+{
+ random stdout=@filename(o);
+}
+
+foreach i in [0:9] {
+ file f = mysim();
+}
+
Added: SwiftTutorials/UofC_2013-07-29-uc3/part03/tc
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part03/tc (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part03/tc 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1 @@
+localhost random random.sh
Added: SwiftTutorials/UofC_2013-07-29-uc3/part04/README
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part04/README (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part04/README 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,42 @@
+
+TOPIC: run a program in a foreach() loop with explicitly named output files.
+
+
+Study the program:
+
+- @strcat() is a Swift builtin function (the @name() form is used for many but not all builtins)
+
+- Note that @strcat() coerces numeric args to strings
+
+- <"filename"> was a shorthand for <single_file_mapper;...> but the latter is more flexible.
+
+Run the program:
+
+$ swift -tc.file tc p4.swift
+
+Look at the output:
+
+$ cd output
+$ ls -l
+$ cat *
+
+
+Try:
+
+- Adjust the script to take the number of simulations to do from the command line
+
+ @arg("argname") returns arguments after the .swift script on the swift command line:
+
+ swift -tc.file tc p4.swift -myarg1=value1 ...
+
+ @arg("argname","default") returns "default" if the argname is not given on the command line
+
+ @toInt() converts strings to integers:
+
+ @toInt(@arg("myArg1"))
+
+
+- trace(expr1,...) traces expressions on stdout.
+
+ Try inserting a few traces. But remember: Swift is *very* concurrent!
+
Added: SwiftTutorials/UofC_2013-07-29-uc3/part04/p4.swift
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part04/p4.swift (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part04/p4.swift 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,12 @@
+type file;
+
+app (file o) mysim ()
+{
+ random stdout=@filename(o);
+}
+
+foreach i in [0:9] {
+ file f <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
+ f = mysim();
+}
+
Added: SwiftTutorials/UofC_2013-07-29-uc3/part04/tc
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part04/tc (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part04/tc 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1 @@
+localhost random random.sh
Added: SwiftTutorials/UofC_2013-07-29-uc3/part05/README
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part05/README (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part05/README 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,51 @@
+
+
+TOPIC: Add a program that "analyzes" the output the parallel simulations
+
+
+Study the program:
+
+We added a new app to "analyze" all the simulation outputs:
+
+app (file o) analyze (file s[])
+{
+ average @filenames(s) stdout=@filename(o);
+}
+
+Nots that it expects an array of files as its input, and that
+it uses @filenames() to place the names of all the files on
+the command line to the "average" program.
+
+Average does:
+--------------------
+#! /bin/sh
+
+awk '
+
+{ sum += $1}
+
+END { print sum/NR }
+' $*
+-------------------
+
+
+At the end of the script we insert:
+
+file stats<"output/average.out">;
+stats = analyze(sims);
+
+...which runs the analysis *after* all the simulations complete.
+
+
+Run the program:
+
+$ swift -tc.file tc p5.swift
+
+Look at the output:
+
+$ cd output
+$ ls -l
+$ cat sim*.out
+$ cat average.out
+
+
Added: SwiftTutorials/UofC_2013-07-29-uc3/part05/p5.swift
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part05/p5.swift (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part05/p5.swift 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,24 @@
+type file;
+
+app (file o) mysim ()
+{
+ random stdout=@filename(o);
+}
+
+app (file o) analyze (file s[])
+{
+ average @filenames(s) stdout=@filename(o);
+}
+
+file sims[];
+
+int nsim = @toInt(@arg("nsim","10"));
+
+foreach i in [0:nsim-1] {
+ file simout <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
+ simout = mysim();
+ sims[i] = simout;
+}
+
+file stats<"output/average.out">;
+stats = analyze(sims);
Added: SwiftTutorials/UofC_2013-07-29-uc3/part05/tc
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part05/tc (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part05/tc 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,2 @@
+localhost random random.sh
+localhost average avg.sh
Added: SwiftTutorials/UofC_2013-07-29-uc3/part06/README
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part06/README (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part06/README 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,28 @@
+
+TOPIC: add arguments to the "simulation" program
+(to control number of steps, range of output, and number of output lines)
+
+
+Test new versions of the "simulation" program:
+
+ random.sh # original
+ random2.sh [runtime] # runtime in seconds; defaults to 1 sec
+ random3.sh [runtime] [range] # requests random numbers in {0:range-1}
+ random4.sh [runtime] [range] [count] # requests "count" random numbers in {0:range}
+
+p6.swift changes to random2 for the simulation program and adds an argument.
+
+Run the program:
+
+$ swift -tc.file tc p6.swift -steps=3 # each simulation takes 3 seconds
+
+Next we need to use the parallel cluster for our simulations!
+
+Try:
+
+- change to random3.sh and add the extra argument "range"
+
+- change to random4.sh and add yet another argument "count"
+
+
+
Added: SwiftTutorials/UofC_2013-07-29-uc3/part06/p6.swift
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part06/p6.swift (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part06/p6.swift 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,25 @@
+type file;
+
+app (file o) mysim2 (int timesteps)
+{
+ random2 timesteps stdout=@filename(o);
+}
+
+app (file o) analyze (file s[])
+{
+ average @filenames(s) stdout=@filename(o);
+}
+
+file sims[];
+int nsim = @toInt(@arg("nsim","10"));
+
+int steps = @toInt(@arg("steps","1"));
+
+foreach i in [0:nsim-1] {
+ file simout <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
+ simout = mysim2(steps);
+ sims[i] = simout;
+}
+
+file stats<"output/average.out">;
+stats = analyze(sims);
Added: SwiftTutorials/UofC_2013-07-29-uc3/part06/tc
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part06/tc (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part06/tc 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,2 @@
+localhost random2 random2.sh
+localhost average avg.sh
Added: SwiftTutorials/UofC_2013-07-29-uc3/part07/README
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part07/README (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part07/README 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,81 @@
+
+TOPIC: Run the "simulations" on the parallel cluster
+
+
+Til now we've been running our "simulations" on the local login host.
+
+Now we'll run them on the parallel cluster.
+
+The script will be the same as previous, but we change the tool catalog to
+indicate that our "simulation app" (random4.sh) should be run on the
+"westmere" partition of the Midway cluster using the "slurm" scheduler:
+
+We changed file "tc" from:
+
+localhost random4 random4.sh
+localhost average avg.sh
+
+to:
+
+westmere random4 random4.sh
+localhost average avg.sh
+
+And we added the "sites.xml" file which specifies two "pools" of resources,
+the "local host" pool (which we've been using till now) and the "westmere" pool.
+
+To run, we now add the sites.xml file to the swift command line:
+
+$ swift -tc.file tc -sites.file sites.xml p7.swift -steps=10 -range=1000 -count=6 -nsim=100
+
+Note that we're now asking for 100 simulations to be run, each of which
+is to run for 10 steps (ie 10 seconds).
+
+The sites file looks like this:
+
+<config>
+
+ <pool handle="localhost">
+ <execution provider="local"/>
+ <filesystem provider="local"/>
+ <workdirectory>/scratch/midway/{env.USER}/swiftwork</workdirectory>
+ </pool>
+
+ <pool handle="westmere">
+ <execution provider="coaster" jobmanager="local:slurm"/>
+
+ <!-- Set partition and account here: -->
+ <profile namespace="globus" key="queue">westmere</profile>
+ <profile namespace="globus" key="ppn">12</profile>
+ <!-- <profile namespace="globus" key="project">pi-wilde</profile> -->
+
+ <!-- Set number of jobs and nodes per job here: -->
+ <profile namespace="globus" key="slots">1</profile>
+ <profile namespace="globus" key="maxnodes">1</profile>
+ <profile namespace="globus" key="nodegranularity">1</profile>
+ <profile namespace="globus" key="jobsPerNode">12</profile> <!-- apps per node! -->
+ <profile namespace="karajan" key="jobThrottle">.11</profile> <!-- eg .11 -> 12 -->
+
+ <!-- Set estimated app time (maxwalltime) and requested job time (maxtime) here: -->
+ <profile namespace="globus" key="maxWalltime">00:15:00</profile>
+ <profile namespace="globus" key="maxtime">1800</profile> <!-- in seconds! -->
+
+ <!-- Set data staging model and work dir here: -->
+ <filesystem provider="local"/>
+ <workdirectory>/scratch/midway/{env.USER}/swiftwork</workdirectory>
+
+ <!-- Typically leave these constant: -->
+ <profile namespace="globus" key="slurm.exclusive">false</profile>
+ <profile namespace="globus" key="highOverAllocation">100</profile>
+ <profile namespace="globus" key="lowOverAllocation">100</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ </pool>
+
+</config>
+
+Try:
+
+- running more app() calls at once: change jobsPerNode and jobThrottle
+
+- requesting more than 1 slurm job at once: change slots
+
+- requesting nodes in units greater than 1: change maxnodes and nodeGranularity
Added: SwiftTutorials/UofC_2013-07-29-uc3/part07/p7.swift
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part07/p7.swift (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part07/p7.swift 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,27 @@
+type file;
+
+app (file o) mysim4 (int timesteps, int range, int count)
+{
+ random4 timesteps range count stdout=@filename(o);
+}
+
+app (file o) analyze (file s[])
+{
+ average @filenames(s) stdout=@filename(o);
+}
+
+file sims[];
+int nsim = @toInt(@arg("nsim", "10"));
+
+int steps = @toInt(@arg("steps", "1"));
+int range = @toInt(@arg("range", "100"));
+int count = @toInt(@arg("count", "10"));
+
+foreach i in [0:nsim-1] {
+ file simout <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
+ simout = mysim4(steps,range,count);
+ sims[i] = simout;
+}
+
+file stats<"output/average.out">;
+stats = analyze(sims);
Added: SwiftTutorials/UofC_2013-07-29-uc3/part07/sites.xml
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part07/sites.xml (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part07/sites.xml 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,39 @@
+<config>
+
+ <pool handle="localhost">
+ <execution provider="local"/>
+ <filesystem provider="local"/>
+ <workdirectory>/scratch/midway/{env.USER}/swiftwork</workdirectory>
+ </pool>
+
+ <pool handle="westmere">
+ <execution provider="coaster" jobmanager="local:slurm"/>
+
+ <!-- Set partition and account here: -->
+ <profile namespace="globus" key="queue">westmere</profile>
+ <profile namespace="globus" key="ppn">12</profile>
+ <!-- <profile namespace="globus" key="project">pi-wilde</profile> -->
+
+ <!-- Set number of jobs and nodes per job here: -->
+ <profile namespace="globus" key="slots">1</profile>
+ <profile namespace="globus" key="maxnodes">1</profile>
+ <profile namespace="globus" key="nodegranularity">1</profile>
+ <profile namespace="globus" key="jobsPerNode">12</profile> <!-- apps per node! -->
+ <profile namespace="karajan" key="jobThrottle">.11</profile> <!-- eg .11 -> 12 -->
+
+ <!-- Set estimated app time (maxwalltime) and requested job time (maxtime) here: -->
+ <profile namespace="globus" key="maxWalltime">00:15:00</profile>
+ <profile namespace="globus" key="maxtime">1800</profile> <!-- in seconds! -->
+
+ <!-- Set data staging model and work dir here: -->
+ <filesystem provider="local"/>
+ <workdirectory>/scratch/midway/{env.USER}/swiftwork</workdirectory>
+
+ <!-- Typically leave these constant: -->
+ <profile namespace="globus" key="slurm.exclusive">false</profile>
+ <profile namespace="globus" key="highOverAllocation">100</profile>
+ <profile namespace="globus" key="lowOverAllocation">100</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ </pool>
+
+</config>
Added: SwiftTutorials/UofC_2013-07-29-uc3/part07/tc
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part07/tc (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part07/tc 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,2 @@
+westmere random4 random4.sh
+localhost average avg.sh
Added: SwiftTutorials/UofC_2013-07-29-uc3/part08/apps
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part08/apps (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part08/apps 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1 @@
+uc3 sh /bin/sh
Added: SwiftTutorials/UofC_2013-07-29-uc3/part08/p00.swift
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part08/p00.swift (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part08/p00.swift 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,28 @@
+type file;
+
+# Application to be called by this script
+
+file simulation_script <"simulate.sh">;
+
+# app() functions for application programs to be called:
+
+app (file out) simulation (file script, int timesteps, int sim_range)
+{
+ sh @filename(script) timesteps sim_range stdout=@filename(out);
+}
+
+# Command line params to this script:
+
+int nsim = @toInt(@arg("nsim", "10")); # number of simulation programs to run
+int range = @toInt(@arg("range", "100")); # range of the generated random numbers
+
+# Main script and data
+
+int steps=3;
+
+tracef("\n*** Script parameters: nsim=%i steps=%i range=%i \n\n", nsim, steps, range);
+
+foreach i in [0:nsim-1] {
+ file simout <single_file_mapper; file=@strcat("data/out_p00/sim_",i,".out")>;
+ simout = simulation(simulation_script, steps, range);
+}
Added: SwiftTutorials/UofC_2013-07-29-uc3/part08/sites.xml
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part08/sites.xml (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part08/sites.xml 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,24 @@
+<config>
+ <pool handle="uc3">
+ <execution provider="coaster" jobmanager="local:condor"/>
+ <profile namespace="karajan" key="jobThrottle">10.00</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ <profile namespace="globus" key="jobsPerNode">1</profile>
+ <profile namespace="globus" key="maxtime">300</profile>
+ <profile namespace="globus" key="maxWalltime">00:01:00</profile>
+ <profile namespace="globus" key="highOverAllocation">100</profile>
+ <profile namespace="globus" key="lowOverAllocation">100</profile>
+ <profile namespace="globus" key="slots">20</profile>
+ <profile namespace="globus" key="maxNodes">1</profile>
+ <profile namespace="globus" key="nodeGranularity">1</profile>
+ <profile namespace="globus" key="condor.+AccountingGroup">"group_friends.{env.USER}"</profile>
+ <profile namespace="globus" key="jobType">nonshared</profile>
+
+ <!-- Requirements = regexp("uc3-c*", Machine) -->
+ <profile namespace="globus" key="condor.Requirements">regexp("uc3-c*", Machine)</profile>
+ <!-- <profile namespace="globus" key="condor.Requirements">UidDomain == "osg-gk.mwt2.org"</profile> -->
+
+ <workdirectory>.</workdirectory>
+ </pool>
+</config>
+
Added: SwiftTutorials/UofC_2013-07-29-uc3/part09/apps
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part09/apps (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part09/apps 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1 @@
+uc3 sh /bin/sh
Added: SwiftTutorials/UofC_2013-07-29-uc3/part09/p01.swift
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part09/p01.swift (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part09/p01.swift 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,41 @@
+type file;
+
+# Applications to be called by this script
+
+file simulation_script <"simulate.sh">;
+file analysis_script <"stats.sh">;
+
+# app() functions for application programs to be called:
+
+app (file out) simulation (file script, int timesteps, int sim_range, file bias_file, int scale, int sim_count)
+{
+ sh @filename(script) timesteps sim_range @filename(bias_file) scale sim_count stdout=@filename(out);
+}
+
+app (file out) analyze (file script, file s[])
+{
+ sh @script @filenames(s) stdout=@filename(out);
+}
+
+# Command line params to this script:
+
+int nsim = @toInt(@arg("nsim", "10")); # number of simulation programs to run
+int steps = @toInt(@arg("steps", "1")); # number of "steps" each simulation (==seconds of runtime)
+int range = @toInt(@arg("range", "100")); # range of the generated random numbers
+int count = @toInt(@arg("count", "10")); # number of random numbers generated per simulation
+
+# Main script and data
+
+tracef("\n*** Script parameters: nsim=%i steps=%i range=%i count=%i\n\n", nsim, steps, range, count);
+
+file sims[]; # Array of files to hold each simulation output
+file bias<"data/bias.dat">; # Input data file to "bias" the numbers:
+ # 1 line: scale offset ( N = n*scale + offset)
+foreach i in [0:nsim-1] {
+ file simout <single_file_mapper; file=@strcat("data/out_p01/sim_",i,".out")>;
+ simout = simulation(simulation_script, steps, range, bias, 100000, count);
+ sims[i] = simout;
+}
+
+file stats<"data/p01_stats.out">; # Final output file: average of all "simulations"
+stats = analyze(analysis_script,sims);
Added: SwiftTutorials/UofC_2013-07-29-uc3/part09/sites.xml
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part09/sites.xml (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part09/sites.xml 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,24 @@
+<config>
+ <pool handle="uc3">
+ <execution provider="coaster" jobmanager="local:condor"/>
+ <profile namespace="karajan" key="jobThrottle">10.00</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ <profile namespace="globus" key="jobsPerNode">1</profile>
+ <profile namespace="globus" key="maxtime">300</profile>
+ <profile namespace="globus" key="maxWalltime">00:01:00</profile>
+ <profile namespace="globus" key="highOverAllocation">100</profile>
+ <profile namespace="globus" key="lowOverAllocation">100</profile>
+ <profile namespace="globus" key="slots">20</profile>
+ <profile namespace="globus" key="maxNodes">1</profile>
+ <profile namespace="globus" key="nodeGranularity">1</profile>
+ <profile namespace="globus" key="condor.+AccountingGroup">"group_friends.{env.USER}"</profile>
+ <profile namespace="globus" key="jobType">nonshared</profile>
+
+ <!-- Requirements = regexp("uc3-c*", Machine) -->
+ <profile namespace="globus" key="condor.Requirements">regexp("uc3-c*", Machine)</profile>
+ <!-- <profile namespace="globus" key="condor.Requirements">UidDomain == "osg-gk.mwt2.org"</profile> -->
+
+ <workdirectory>.</workdirectory>
+ </pool>
+</config>
+
Added: SwiftTutorials/UofC_2013-07-29-uc3/part10/apps
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part10/apps (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part10/apps 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1 @@
+uc3 sh /bin/sh
Added: SwiftTutorials/UofC_2013-07-29-uc3/part10/p02.swift
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part10/p02.swift (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part10/p02.swift 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,52 @@
+type file;
+
+# Applications to be called by this script
+
+file simulation_script <"simulate.sh">;
+file analysis_script <"stats.sh">;
+
+# app() functions for application programs to be called:
+
+app (file out) genrand (file script, int timesteps, int sim_range)
+{
+ sh @filename(script) timesteps sim_range stdout=@filename(out);
+}
+
+app (file out) simulation (file script, int timesteps, int sim_range, file bias_file, int scale, int sim_count)
+{
+ sh @filename(script) timesteps sim_range @filename(bias_file) scale sim_count stdout=@filename(out);
+}
+
+app (file out) analyze (file script, file s[])
+{
+ sh @script @filenames(s) stdout=@filename(out);
+}
+
+# Command line params to this script:
+
+int nsim = @toInt(@arg("nsim", "10")); # number of simulation programs to run
+int range = @toInt(@arg("range", "100")); # range of the generated random numbers
+int count = @toInt(@arg("count", "10")); # number of random numbers generated per simulation
+
+# Main script and data
+
+tracef("\n*** Script parameters: nsim=%i range=%i count=%i\n\n", nsim, range, count);
+
+file bias<"data/dynamic_bias.dat">; # Dynamically generated bias for simulation ensemble
+
+bias = genrand(simulation_script, 1, 1000);
+
+file sims[]; # Array of files to hold each simulation output
+
+foreach i in [0:nsim-1] {
+
+ int steps = readData(genrand(simulation_script, 1, 5));
+ tracef(" for simulation[%i] steps=%i\n", i, steps+1);
+
+ file simout <single_file_mapper; file=@strcat("data/out_p02/sim_",i,".out")>;
+ simout = simulation(simulation_script, steps+1, range, bias, 100000, count);
+ sims[i] = simout;
+}
+
+file stats<"data/p02_stats.out">; # Final output file: average of all "simulations"
+stats = analyze(analysis_script,sims);
Added: SwiftTutorials/UofC_2013-07-29-uc3/part10/sites.xml
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/part10/sites.xml (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/part10/sites.xml 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,24 @@
+<config>
+ <pool handle="uc3">
+ <execution provider="coaster" jobmanager="local:condor"/>
+ <profile namespace="karajan" key="jobThrottle">10.00</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ <profile namespace="globus" key="jobsPerNode">1</profile>
+ <profile namespace="globus" key="maxtime">300</profile>
+ <profile namespace="globus" key="maxWalltime">00:01:00</profile>
+ <profile namespace="globus" key="highOverAllocation">100</profile>
+ <profile namespace="globus" key="lowOverAllocation">100</profile>
+ <profile namespace="globus" key="slots">20</profile>
+ <profile namespace="globus" key="maxNodes">1</profile>
+ <profile namespace="globus" key="nodeGranularity">1</profile>
+ <profile namespace="globus" key="condor.+AccountingGroup">"group_friends.{env.USER}"</profile>
+ <profile namespace="globus" key="jobType">nonshared</profile>
+
+ <!-- Requirements = regexp("uc3-c*", Machine) -->
+ <profile namespace="globus" key="condor.Requirements">regexp("uc3-c*", Machine)</profile>
+ <!-- <profile namespace="globus" key="condor.Requirements">UidDomain == "osg-gk.mwt2.org"</profile> -->
+
+ <workdirectory>.</workdirectory>
+ </pool>
+</config>
+
Added: SwiftTutorials/UofC_2013-07-29-uc3/setup.sh
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/setup.sh (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/setup.sh 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,55 @@
+#export SBATCH_RESERVATION=osg # for Midway
+
+export SWIFT_HEAP_MAX=4G
+
+# SWIFTREL=/home/wilde/swift/src/0.94/cog/modules/swift/dist/rc5.1/bin
+
+if which swift >& /dev/null; then
+ :
+else
+ echo "Trying to load swift module:"
+ module load swift
+ if which swift >& /dev/null; then
+ Swift loaded from module
+ else
+ echo "Cant find a swift to run"
+ return
+ fi
+fi
+
+echo -e "\\nUsing swift release from: \\c"
+which swift
+swift -version
+
+# export PATHPREFIX="$( cd "$( dirname "${BASH_SOURCE[0]}" )/../bin" && pwd )"
+# echo
+# echo Setting PATHPREFIX=$PATHPREFIX
+
+if [ -e $HOME/.swift ]; then
+ savedotswift=$(mktemp -d $HOME/.swift.save.XXXX)
+ echo Saving $HOME/.swift in $savedotswift
+ mv $HOME/.swift/* $savedotswift
+else
+ mkdir -p $HOME/.swift
+fi
+
+cat >>$HOME/.swift/swift.properties <<END
+
+# Properties for Swift Tutorial
+
+sites.file=$HOME/.swift/sites.xml
+tc.file=$HOME/.swift/apps
+
+wrapperlog.always.transfer=true
+sitedir.keep=true
+file.gc.enabled=false
+status.mode=provider
+
+execution.retries=0
+lazy.errors=false
+
+use.wrapper.staging=false
+use.provider.staging=true
+provider.staging.pin.swiftfiles=true
+
+END
Added: SwiftTutorials/UofC_2013-07-29-uc3/simulate.sh
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/simulate.sh (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/simulate.sh 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,23 @@
+#! /bin/bash
+
+runtime=${1:-0}
+range=${2:-100}
+biasfile=${3:-nobias}
+scale=${4:-1}
+n=${5:-1}
+
+if [ $biasfile = nobias ]; then
+ offset=0
+else
+ read offset <$biasfile
+fi
+
+# run for some number of "timesteps"
+
+sleep $runtime
+
+# emit n "simulation results", scaled and biased by the specified argument values
+
+for ((i=0;i<n;i++)); do
+ echo $(( ($RANDOM%range)*scale+offset))
+done
Property changes on: SwiftTutorials/UofC_2013-07-29-uc3/simulate.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: SwiftTutorials/UofC_2013-07-29-uc3/stats.sh
===================================================================
--- SwiftTutorials/UofC_2013-07-29-uc3/stats.sh (rev 0)
+++ SwiftTutorials/UofC_2013-07-29-uc3/stats.sh 2013-07-29 20:22:35 UTC (rev 6681)
@@ -0,0 +1,9 @@
+#! /bin/sh
+
+awk '
+
+{ sum += $1}
+
+END { print sum/NR }
+' $*
+
Property changes on: SwiftTutorials/UofC_2013-07-29-uc3/stats.sh
___________________________________________________________________
Added: svn:executable
+ *
More information about the Swift-commit
mailing list