[Swift-commit] r6570 - in SwiftTutorials/CMTS_2013-06-17/basic_swift: . part08
davidk at ci.uchicago.edu
davidk at ci.uchicago.edu
Mon Jun 17 13:42:31 CDT 2013
Author: davidk
Date: 2013-06-17 13:42:31 -0500 (Mon, 17 Jun 2013)
New Revision: 6570
Added:
SwiftTutorials/CMTS_2013-06-17/basic_swift/part08/
SwiftTutorials/CMTS_2013-06-17/basic_swift/part08/README
SwiftTutorials/CMTS_2013-06-17/basic_swift/part08/apps
SwiftTutorials/CMTS_2013-06-17/basic_swift/part08/p8.swift
SwiftTutorials/CMTS_2013-06-17/basic_swift/part08/sites.xml
Log:
Part08
Added: SwiftTutorials/CMTS_2013-06-17/basic_swift/part08/README
===================================================================
--- SwiftTutorials/CMTS_2013-06-17/basic_swift/part08/README (rev 0)
+++ SwiftTutorials/CMTS_2013-06-17/basic_swift/part08/README 2013-06-17 18:42:31 UTC (rev 6570)
@@ -0,0 +1,80 @@
+
+TOPIC: Example pipelines
+
+
+So far we've been running our "simulations" on the local login host.
+Now we'll run them on the parallel cluster.
+
+The script will be the same as previous, but we change the tool catalog to
+indicate that our "simulation app" (random4.sh) should be run on the
+"westmere" partition of the Midway cluster using the "slurm" scheduler:
+
+We changed file "apps" from:
+
+ localhost random4 random4.sh
+ localhost average avg.sh
+
+to:
+
+ cmtsworkshop random4 random4.sh
+ localhost average avg.sh
+
+And we added a pool entry to "sites.xml" file which specifies two "pools" of resources,
+the "local host" pool (which we've been using till now) and the "cmtsworkshop" pool.
+
+To run:
+
+ $ swift p7.swift -steps=10 -range=1000 -count=6 -nsim=100
+
+Note that we're now asking for 100 simulations to be run, each of which
+is to run for 10 steps (ie 10 seconds).
+
+The sites file looks like this:
+
+<config>
+
+ <pool handle="localhost">
+ <execution provider="local"/>
+ <filesystem provider="local"/>
+ <workdirectory>/scratch/midway/{env.USER}/swiftwork</workdirectory>
+ </pool>
+
+ <pool handle="cmtsworkshop">
+ <execution provider="coaster" jobmanager="local:slurm"/>
+
+ <!-- Set partition and account here: -->
+ <profile namespace="globus" key="queue">cmtsworkshop</profile>
+ <profile namespace="globus" key="ppn">8</profile>
+ <profile namespace="globus" key="project">cmtsworkshop</profile> -->
+
+ <!-- Set number of jobs and nodes per job here: -->
+ <profile namespace="globus" key="slots">1</profile>
+ <profile namespace="globus" key="maxnodes">1</profile>
+ <profile namespace="globus" key="nodegranularity">1</profile>
+ <profile namespace="globus" key="jobsPerNode">8</profile> <!-- apps per node -->
+ <profile namespace="karajan" key="jobThrottle">.07</profile> <!-- eg .07 -> 8 -->
+
+ <!-- Set estimated app time (maxwalltime) and requested job time (maxtime) here: -->
+ <profile namespace="globus" key="maxWalltime">00:15:00</profile>
+ <profile namespace="globus" key="maxtime">1800</profile> <!-- in seconds! -->
+
+ <!-- Set data staging model and work dir here: -->
+ <filesystem provider="local"/>
+ <workdirectory>/scratch/midway/{env.USER}/swiftwork</workdirectory>
+
+ <!-- Typically leave these constant: -->
+ <profile namespace="globus" key="slurm.exclusive">false</profile>
+ <profile namespace="globus" key="highOverAllocation">100</profile>
+ <profile namespace="globus" key="lowOverAllocation">100</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ </pool>
+
+</config>
+
+Try:
+
+ o running more app() calls at once: change jobsPerNode and jobThrottle
+
+ o requesting more than 1 slurm job at once: change slots
+
+ o requesting nodes in units greater than 1: change maxnodes and nodeGranularity
Added: SwiftTutorials/CMTS_2013-06-17/basic_swift/part08/apps
===================================================================
--- SwiftTutorials/CMTS_2013-06-17/basic_swift/part08/apps (rev 0)
+++ SwiftTutorials/CMTS_2013-06-17/basic_swift/part08/apps 2013-06-17 18:42:31 UTC (rev 6570)
@@ -0,0 +1,2 @@
+cmtsworkshop random4 random4.sh
+localhost average avg.sh
Added: SwiftTutorials/CMTS_2013-06-17/basic_swift/part08/p8.swift
===================================================================
--- SwiftTutorials/CMTS_2013-06-17/basic_swift/part08/p8.swift (rev 0)
+++ SwiftTutorials/CMTS_2013-06-17/basic_swift/part08/p8.swift 2013-06-17 18:42:31 UTC (rev 6570)
@@ -0,0 +1,36 @@
+type file;
+
+app (file o) mysim4 (int timesteps, int range_value, int count_value)
+{
+ random4 timesteps range_value count_value stdout=@filename(o);
+}
+
+app (file o) mysim4_filename (int timesteps, file range_value, int count_value)
+{
+ random4 timesteps @filename(range_value) count_value stdout=@filename(o);
+}
+
+app (file o) analyze (file s[])
+{
+ average @filenames(s) stdout=@filename(o);
+}
+
+file sims[];
+int nsim = @toInt(@arg("nsim", "10"));
+int steps = @toInt(@arg("steps", "1"));
+int range = @toInt(@arg("range", "100"));
+int count = @toInt(@arg("count", "10"));
+
+foreach i in [0:nsim-1] {
+
+ file simout <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
+ simout = mysim4(steps, range, 1);
+
+ file simout2 <single_file_mapper; file=@strcat("output/sim_", i, ".tmp.out")>;
+ simout2 = mysim4_filename(steps, simout, nsim);
+
+ sims[i] = simout2;
+}
+
+file stats<"output/average.out">;
+stats = analyze(sims);
Added: SwiftTutorials/CMTS_2013-06-17/basic_swift/part08/sites.xml
===================================================================
--- SwiftTutorials/CMTS_2013-06-17/basic_swift/part08/sites.xml (rev 0)
+++ SwiftTutorials/CMTS_2013-06-17/basic_swift/part08/sites.xml 2013-06-17 18:42:31 UTC (rev 6570)
@@ -0,0 +1,39 @@
+<config>
+
+ <pool handle="localhost">
+ <execution provider="local"/>
+ <filesystem provider="local"/>
+ <workdirectory>/scratch/midway/{env.USER}/swiftwork</workdirectory>
+ </pool>
+
+ <pool handle="cmtsworkshop">
+ <execution provider="coaster" jobmanager="local:slurm"/>
+
+ <!-- Set partition and account here: -->
+ <profile namespace="globus" key="queue">cmtsworkshop</profile>
+ <profile namespace="globus" key="ppn">8</profile>
+ <profile namespace="globus" key="project">cmtsworkshop</profile>
+
+ <!-- Set number of jobs and nodes per job here: -->
+ <profile namespace="globus" key="slots">4</profile>
+ <profile namespace="globus" key="maxnodes">1</profile>
+ <profile namespace="globus" key="nodegranularity">1</profile>
+ <profile namespace="globus" key="jobsPerNode">8</profile> <!-- apps per node! -->
+ <profile namespace="karajan" key="jobThrottle">.31</profile> <!-- eg .31 -> 32 -->
+
+ <!-- Set estimated app time (maxwalltime) and requested job time (maxtime) here: -->
+ <profile namespace="globus" key="maxWalltime">00:15:00</profile>
+ <profile namespace="globus" key="maxtime">1800</profile> <!-- in seconds! -->
+
+ <!-- Set data staging model and work dir here: -->
+ <filesystem provider="local"/>
+ <workdirectory>/scratch/midway/{env.USER}/swiftwork</workdirectory>
+
+ <!-- Typically leave these constant: -->
+ <profile namespace="globus" key="slurm.exclusive">false</profile>
+ <profile namespace="globus" key="highOverAllocation">100</profile>
+ <profile namespace="globus" key="lowOverAllocation">100</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ </pool>
+
+</config>
More information about the Swift-commit
mailing list