[Swift-commit] r7003 - in SwiftTutorials/swift-cray-tutorial: . bin doc tbd-future
wilde at ci.uchicago.edu
wilde at ci.uchicago.edu
Mon Aug 26 11:07:45 CDT 2013
Author: wilde
Date: 2013-08-26 11:07:45 -0500 (Mon, 26 Aug 2013)
New Revision: 7003
Added:
SwiftTutorials/swift-cray-tutorial/tbd-future/
SwiftTutorials/swift-cray-tutorial/tbd-future/cloud/
SwiftTutorials/swift-cray-tutorial/tbd-future/modis/
Removed:
SwiftTutorials/swift-cray-tutorial/cloud/
SwiftTutorials/swift-cray-tutorial/doc/p1.png
SwiftTutorials/swift-cray-tutorial/doc/p2.png
SwiftTutorials/swift-cray-tutorial/doc/p3.png
SwiftTutorials/swift-cray-tutorial/doc/p4.png
SwiftTutorials/swift-cray-tutorial/doc/p5.png
SwiftTutorials/swift-cray-tutorial/doc/p6.png
SwiftTutorials/swift-cray-tutorial/doc/p7.png
SwiftTutorials/swift-cray-tutorial/doc/p8.png
SwiftTutorials/swift-cray-tutorial/doc/p9.png
SwiftTutorials/swift-cray-tutorial/modis/
Modified:
SwiftTutorials/swift-cray-tutorial/bin/plot.sh
SwiftTutorials/swift-cray-tutorial/doc/README
SwiftTutorials/swift-cray-tutorial/setup.sh
Log:
Doc edits; moved tbd stuff under separate dir; added package builder; setup for multiple pool tests/.
Modified: SwiftTutorials/swift-cray-tutorial/bin/plot.sh
===================================================================
--- SwiftTutorials/swift-cray-tutorial/bin/plot.sh 2013-08-26 08:25:18 UTC (rev 7002)
+++ SwiftTutorials/swift-cray-tutorial/bin/plot.sh 2013-08-26 16:07:45 UTC (rev 7003)
@@ -53,13 +53,14 @@
set terminal png enhanced
set nokey
set output "cumulativeplot.png"
-set ylabel "number of completed jobs"
-set title "Cumulative jobs"
+set xlabel "Time in sec"
+set ylabel "number of completed apps"
+set title "Cumulative Apps Completed"
plot "plot_cumulative.txt" using 1:2 with lines
set output "activeplot.png"
set xlabel "Time in sec"
-set ylabel "number of active jobs"
-set title "Active jobs"
+set ylabel "number of active apps"
+set title "Active Apps"
plot "plot_active.txt" using 1:2 with line
EOF1
Modified: SwiftTutorials/swift-cray-tutorial/doc/README
===================================================================
--- SwiftTutorials/swift-cray-tutorial/doc/README 2013-08-26 08:25:18 UTC (rev 7002)
+++ SwiftTutorials/swift-cray-tutorial/doc/README 2013-08-26 16:07:45 UTC (rev 7003)
@@ -28,24 +28,22 @@
ordinary application programs on distributed parallel resources.
Swift scripts run many copies of ordinary programs concurrently, using
statements like this:
-
-----
foreach protein in proteinList {
runBLAST(protein);
}
-----
-
Swift acts like a structured "shell" language. It runs programs
concurrently as soon as their inputs are available, reducing the need
-for complex parallel programming. Swift expresses your workflow
-in a portable fashion: The same script runs on grids like OSG, as well
-as on multicore computers, clusters, clouds, and supercomputers.
+for complex parallel programming. Swift expresses your workflow in a
+portable fashion: The same script runs on multicore computers,
+clusters, clouds, grids, and supercomputers.
In this tutorial, you'll be able to first try a few Swift examples
-(parts 1-3) on the OSG Connect login host, to get a sense of the
-language. Then in parts 4-6 you'll run similar workflows on
-distributed OSG Connect resources, and see how more complex workflows
-can be expressed with Swift scripts.
+(parts 1-3) on the Cray login host, to get a sense of the
+language. Then in parts 4-6 you'll run similar workflows on Cray
+compute nodes, and see how more complex workflows can be expressed
+with Swift scripts.
Swift tutorial setup
@@ -374,26 +372,27 @@
Output files will be named output/sim_N.out.
-In order to run on Cray compute nodes, sites.xml was modified. Here is
-the new sites.xml we are using for this example. Note the changes
-between the sites.xml file in this example which specifies
+In order for our .swift scripts to run apps on compute nodes, we
+specify a more complex sites.xml file. The one in `part04` is shown
+below.
-"execution
-provider=condor", and the sites.xml file in part 1, which runs locally
-by specifying "execution provider=local".
-
.sites.xml
-----
-sys::[cat -n ../part06/sites.xml]
+sys::[cat -n ../part04/sites.xml]
-----
-Below is the updated apps file. Since Swift is staging shell scripts
-remotely to nodes on the cluster, the only application you need to
-define here is the shell.
+Below is the apps file used in this part.
+Since we are running on a single local cluster, and have one or more
+filesystems that are identically mounted on the login node where we
+are running the `swift` command and on the compute nodes where the
+apps will execute, we can specify the location of each app by a a line
+in the `apps` file, with a pathname (absolute, or a name to be located
+in PATH):
+
.apps
-----
-sys::[cat -n ../part06/apps]
+sys::[cat -n ../part04/apps]
-----
@@ -417,33 +416,6 @@
$ swift p5.swift
----
-
-Part 6: Specifying more complex workflow patterns
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-p6.swift build on p5.swift, but adds new apps for generating a random
-seed and a random bias value.
-
-image::part06.png[align="center"]
-
-.p6.swift
-----
-sys::[cat -n ../part06/p6.swift]
-----
-
-
-Use the command below to specify the time for each simulation.
-----
-$ cd ../part06
-$ swift p6.swift -steps=3 # each simulation takes 3 seconds
-----
-
-
-
-
-
-
-
Larger runs
~~~~~~~~~~~
To test with larger runs, there are two changes that are required. The first is a
@@ -473,8 +445,54 @@
the number of active jobs over time. When the worker agents are trasnferring data
to and from nodes, this time does not get marked as active.
-After each swift run, a log file will be created called partNN-<YYYYmmdd>-<hhmm>-<random>.log.
-Once you have identified the log file name, run the following command to generate the plots:
+After each swift run, a log file will be created called
+partNN-<YYYYmmdd>-<hhmm>-<random>.log. Once you have identified the
+log file name, run the command `./plot.sh <logfile>` (where logfile is
+the most recent Swift run log) to generate the plots. For example:
+-----
$ ./plot.sh <logfile>
+-----
+image::activeplot.png[width=700,align=center]
+image::cumulativeplot.png[width=700,align=center]
+////
+image::activeplot.png[align="center",scaledwidth="60%"]
+image::cumulativeplot.png[align="center",scaledwidth="60%"]
+////
+
+
+You can then scp the two resulting plot images to another host to view them. Eg, from a Mac, do:
+
+-----
+$ scp ...
+$ open *.png
+----
+
+
+Part 6: Specifying more complex workflow patterns
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+p6.swift build on p5.swift, but adds new apps for generating a random
+seed and a random bias value.
+
+image::part06.png[align="center"]
+
+.p6.swift
+----
+sys::[cat -n ../part06/p6.swift]
+----
+
+
+Use the command below to specify the time for each simulation.
+----
+$ cd ../part06
+$ swift p6.swift -steps=3 # each simulation takes 3 seconds
+----
+
+
+
+
+
+
+
Deleted: SwiftTutorials/swift-cray-tutorial/doc/p1.png
===================================================================
(Binary files differ)
Deleted: SwiftTutorials/swift-cray-tutorial/doc/p2.png
===================================================================
(Binary files differ)
Deleted: SwiftTutorials/swift-cray-tutorial/doc/p3.png
===================================================================
(Binary files differ)
Deleted: SwiftTutorials/swift-cray-tutorial/doc/p4.png
===================================================================
(Binary files differ)
Deleted: SwiftTutorials/swift-cray-tutorial/doc/p5.png
===================================================================
(Binary files differ)
Deleted: SwiftTutorials/swift-cray-tutorial/doc/p6.png
===================================================================
(Binary files differ)
Deleted: SwiftTutorials/swift-cray-tutorial/doc/p7.png
===================================================================
(Binary files differ)
Deleted: SwiftTutorials/swift-cray-tutorial/doc/p8.png
===================================================================
(Binary files differ)
Deleted: SwiftTutorials/swift-cray-tutorial/doc/p9.png
===================================================================
(Binary files differ)
Modified: SwiftTutorials/swift-cray-tutorial/setup.sh
===================================================================
--- SwiftTutorials/swift-cray-tutorial/setup.sh 2013-08-26 08:25:18 UTC (rev 7002)
+++ SwiftTutorials/swift-cray-tutorial/setup.sh 2013-08-26 16:07:45 UTC (rev 7003)
@@ -72,20 +72,66 @@
END
cat >sites.raven<<END
-<config>
+<?xml version="1.0" encoding="UTF-8"?>
+<config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites">
+
+ <pool handle="localhost">
+ <execution provider="local" />
+ <profile namespace="karajan" key="jobThrottle">0</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ <filesystem provider="local"/>
+ <workdirectory>.swift/tmp</workdirectory>
+ <profile namespace="swift" key="stagingMethod">local</profile>
+ </pool>
+
<pool handle="raven">
- <execution provider="coaster" jobmanager="local:pbs"/>
+ <execution provider="coaster" jobmanager="local:pbs" URL="local:01"/>
<profile namespace="env" key="SWIFT_GEN_SCRIPTS">KEEP</profile>
+ <profile namespace="globus" key="jobsPerNode">32</profile>
+ <profile namespace="globus" key="providerAttributes">pbs.aprun;pbs.mpp;depth=32</profile>
+ <profile namespace="globus" key="maxWallTime">00:01:00</profile>
+ <profile namespace="globus" key="slots">10</profile>
+ <profile namespace="globus" key="maxNodes">1</profile>
+ <profile namespace="karajan" key="jobThrottle">3.20</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ <filesystem provider="local"/>
+ <workdirectory>{env.HOME}/swiftwork</workdirectory>
+ </pool>
+
+ <pool handle="ravenMED">
+ <execution provider="coaster" jobmanager="local:pbs" URL="local:02"/>
+ <profile namespace="env" key="SWIFT_GEN_SCRIPTS">KEEP</profile>
+ <profile namespace="globus" key="queue">medium</profile>
+ <profile namespace="globus" key="jobsPerNode">32</profile>
+ <profile namespace="globus" key="providerAttributes">pbs.aprun;pbs.mpp;depth=32</profile>
+ <profile namespace="globus" key="maxWallTime">00:01:00</profile>
+ <profile namespace="globus" key="slots">1</profile>
+ <profile namespace="globus" key="maxNodes">6</profile>
+ <profile namespace="globus" key="nodeGranularity">6</profile>
+ <profile namespace="karajan" key="jobThrottle">5.00</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ <!-- <filesystem provider="local"/> -->
+ <!-- <workdirectory>{env.HOME}/swiftwork</workdirectory> -->
+ <workdirectory>/dev/shm/</workdirectory> -->
+ </pool>
+
+ <pool handle="ravenGPU">
+ <execution provider="coaster" jobmanager="local:pbs" URL="local:03"/>
+ <profile namespace="env" key="SWIFT_GEN_SCRIPTS">KEEP</profile>
+ <profile namespace="globus" key="queue">gpu_nodes</profile>
<profile namespace="globus" key="jobsPerNode">16</profile>
- <profile namespace="globus" key="providerAttributes">pbs.aprun;pbs.mpp;depth=24</profile>
+ <profile namespace="globus" key="providerAttributes">pbs.aprun;pbs.mpp;depth=16</profile>
<profile namespace="globus" key="maxWallTime">00:01:00</profile>
<profile namespace="globus" key="slots">1</profile>
- <profile namespace="globus" key="maxNodes">2</profile>
+ <profile namespace="globus" key="maxNodes">6</profile>
+ <profile namespace="globus" key="nodeGranularity">6</profile>
<profile namespace="karajan" key="jobThrottle">5.00</profile>
<profile namespace="karajan" key="initialScore">10000</profile>
- <filesystem provider="local"/>
- <workdirectory>{env.HOME}/swiftwork</workdirectory>
+ <!-- <filesystem provider="local"/> -->
+ <!-- <workdirectory>{env.HOME}/swiftwork</workdirectory> -->
+ <workdirectory>/dev/shm/</workdirectory> -->
</pool>
+
</config>
END
More information about the Swift-commit
mailing list