[Swift-commit] r6969 - in SwiftTutorials/CIC_2013-08-09: . doc
wilde at ci.uchicago.edu
wilde at ci.uchicago.edu
Fri Aug 23 00:52:01 CDT 2013
Author: wilde
Date: 2013-08-23 00:52:01 -0500 (Fri, 23 Aug 2013)
New Revision: 6969
Modified:
SwiftTutorials/CIC_2013-08-09/doc/README
SwiftTutorials/CIC_2013-08-09/setup.sh
Log:
Adjust sites file format; center figures.
Modified: SwiftTutorials/CIC_2013-08-09/doc/README
===================================================================
--- SwiftTutorials/CIC_2013-08-09/doc/README 2013-08-23 05:31:45 UTC (rev 6968)
+++ SwiftTutorials/CIC_2013-08-09/doc/README 2013-08-23 05:52:01 UTC (rev 6969)
@@ -1,5 +1,5 @@
-Swift CIC Tutorial - 2013.0827
-==============================
+OSGconnect Swift Tutorial - 2013.0827
+=====================================
////
@@ -22,8 +22,6 @@
////
-
-
Workflow tutorial setup
-----------------------
@@ -145,17 +143,17 @@
The stats.sh script reads a file containing n numbers and prints the average
of those numbers to stdout.
-Introductory exercises
-----------------------
-Parts 1-3 (p1.swift - p3.swift) run locally and serve as examples of the Swift language.
-Parts 4-6 (p4.swift - p6.swift) submit jobs to the site specified the setup stage
+OSG Connect exercises
+---------------------
+Parts 1-3 (p1.swift - p3.swift) run on your login host and serve as examples of the Swift language.
+Parts 4-6 (p4.swift - p6.swift) submit jobs to OSG Connect resources.
-p1 - Run an application under Swift
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Part 1: Run a single application under Swift
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The first swift script, p1.swift, runs simulate.sh to generate a single random
number. It writes the number to a file.
-image:p1.png[]
+image::part01.png["p1 workflow",align="center"]
.p1.swift
-----
@@ -197,13 +195,13 @@
$ ./clean.sh
------
-p2 - Parallel loops with foreach
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Part 2: Running an ensemble of many apps in parallel with "foreach" loops
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The p2.swift script introduces the foreach loop. This script runs many
simulations. The script also shows an example of naming the files. The output files
are now called sim_N.out.
-image:p2.png[]
+image::part02.png[align="center"]
.p2.swift
-----
@@ -216,12 +214,12 @@
$ swift p2.swift
-----
-p3 - Merging/reducing the results of a parallel foreach loop
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Part 3: Analyzing results of a parallel ensemble
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
p3.swift introduces a postprocessing step. After all the parallel simulations have completed, the files
created by simulation.sh will be averaged by stats.sh.
-image:p3.png[]
+image::part03.png[align="center"]
.p3.swift
----
@@ -234,15 +232,18 @@
$ swift p3.swift
----
-p4 - Running on the remote site nodes
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-p4.swift is the first script that will submit jobs to remote site nodes for analysis.
-It is similar to earlier scripts, with a few minor exceptions. To generalize the script
-for other types of remote execution (e.g., when no shared filesystem is available to the compute nodes), the application simulate.sh
-will get transferred to the worker node by Swift, in the same manner as any other input data file.
+Part 4: Running a parallel ensemble on OSG Connect resources
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-image:p4.png[]
+p4.swift is the first script that will submit jobs to remote site
+nodes for analysis. It is similar to earlier scripts, with a few
+minor exceptions. To generalize the script for other types of remote
+execution (e.g., when no shared filesystem is available to the compute
+nodes), the application simulate.sh will get transferred to the worker
+node by Swift, in the same manner as any other input data file.
+image::part04.png[align="center"]
+
.p4.swift
----
sys::[cat -n ../part04/p4.swift]
@@ -255,14 +256,16 @@
Output files will be named output/sim_N.out.
-p5 - Running the stats summary step on the remote site
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-p5.swift introduces a postprocessing step. After all the parallel simulations have completed, the files
-created by simulation.sh will be averaged by stats.sh. This is similar to p3, but all app invocations
-are done on remote nodes with Swift managing file transfers.
+Part 5: Controlling where applications run
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-image:p5.png[]
+p5.swift introduces a postprocessing step. After all the parallel
+simulations have completed, the files created by simulation.sh will be
+averaged by stats.sh. This is similar to p3, but all app invocations
+are done on remote nodes with Swift managing file transfers.
+image::part05.png[align="center"]
+
.p5.swift
----
sys::[cat -n ../part05/p5.swift]
@@ -273,29 +276,32 @@
$ swift p5.swift
----
-p6 - Add additional apps and randomness
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Part 6: Specifying more complex workflow patterns
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
p6.swift build on p5.swift, but adds new apps for generating a random
-seed and a random bias value.
+seed and a random bias value.
-image:p6.png[]
+image::part06.png[align="center"]
.p6.swift
----
sys::[cat -n ../part06/p6.swift]
----
-In order to run on the cluster, sites.xml needed to be modified. Here is
-the new sites.xml we are using for this example. Note the changes between the sites.xml file
-in this example which uses condor, and the sites.xml file in part 1, which runs locally.
+In order to run on the cluster, sites.xml needed to be modified. Here
+is the new sites.xml we are using for this example. Note the changes
+between the sites.xml file in this example which uses condor, and the
+sites.xml file in part 1, which runs locally.
.sites.xml
-----
sys::[cat -n ../part06/sites.xml]
-----
-Below is the updated apps file. Since Swift is staging shell scripts remotely to nodes on the cluster,
-the only application it needs defined here is the shell.
+Below is the updated apps file. Since Swift is staging shell scripts
+remotely to nodes on the cluster, the only application you need to
+define here is the shell.
.apps
-----
@@ -308,6 +314,8 @@
$ swift p6.swift -steps=3 # each simulation takes 3 seconds
----
+////
+
Running Swift scripts on Cloud resources
----------------------------------------
@@ -342,14 +350,12 @@
swift -sites.file sites.xml -config cf -tc.file tc p9.swift
-----
-////
* Add cloud resources to existing examples:
-----
./addcloud.sh <dir> #where dir is a tutorial script directory
e.g.
./addcloud.sh ../part01 #will create a new site pool "both.xml" in ../part01
-----
-////
* Finally, to clean up the log files, kill agent and shutdown the coaster service:
-----
@@ -366,7 +372,6 @@
To learn more about the configuration files, see Swift user-guide:
http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html
-////
Running Swift/T on Vesta with Python and R integration
------------------------------------------------------
Modified: SwiftTutorials/CIC_2013-08-09/setup.sh
===================================================================
--- SwiftTutorials/CIC_2013-08-09/setup.sh 2013-08-23 05:31:45 UTC (rev 6968)
+++ SwiftTutorials/CIC_2013-08-09/setup.sh 2013-08-23 05:52:01 UTC (rev 6969)
@@ -77,7 +77,7 @@
return
fi
-cat >>sites.condor<<END
+cat >sites.condor <<END
<config>
<pool handle="osg">
@@ -89,40 +89,19 @@
<profile namespace="globus" key="maxWalltime">00:01:00</profile>
<profile namespace="globus" key="highOverAllocation">10000</profile>
<profile namespace="globus" key="lowOverAllocation">10000</profile>
- <profile namespace="globus" key="internalHostname">$CONTACTHOST</profile> <!-- FIXME Make Dynamic from setup -->
+ <profile namespace="globus" key="internalHostname">$CONTACTHOST</profile>
<profile namespace="globus" key="slots">20</profile>
<profile namespace="globus" key="maxNodes">1</profile>
<profile namespace="globus" key="nodeGranularity">1</profile>
+ <workdirectory>.</workdirectory> <!-- Alt: /tmp/swift/OSG/{env.USER} -->
<!-- For UC3: -->
<profile namespace="globus" key="condor.+AccountingGroup">"group_friends.{env.USER}"</profile>
<!-- For OSGConnect -->
<profile namespace="globus" key="condor.+ProjectName">"swift"</profile>
<profile namespace="globus" key="jobType">nonshared</profile>
- <!-- UC3 --> <!-- <profile namespace="globus" key="condor.Requirements">regexp("uc3-c*", Machine)</profile> -->
- <!-- UCIT --> <!-- <profile namespace="globus" key="condor.Requirements">regexp("appcloud[0-1][0-9].*", Machine)</profile> -->
- <!-- MWUC --> <!-- <profile namespace="globus" key="condor.Requirements">regexp("uct2-c*", Machine)</profile> -->
- <!-- MWIU --> <!-- <profile namespace="globus" key="condor.Requirements">regexp("iut2-c*", Machine)</profile> -->
- <!-- MWUI --> <!-- <profile namespace="globus" key="condor.Requirements">regexp("taub*", Machine)</profile> -->
- <!-- MWT2 --> <!-- <profile namespace="globus" key="condor.Requirements">UidDomain == "osg-gk.mwt2.org"</profile> -->
- <!-- OSG --> <!-- <profile namespace="globus" key="condor.Requirements">isUndefined(GLIDECLIENT_Name) == FALSE</profile> -->
- <!-- MULT --> <!-- <profile namespace="globus" key="condor.Requirements">UidDomain == "osg-gk.mwt2.org" && (regexp("iut2-c*", Machine) || regexp("uct2-c*", Machine)) </profile> -->
-
- <workdirectory>.</workdirectory> <!-- VERIFY OK FOR ALL OSG! -->
</pool>
</config>
-
-<!--
-
-MWT2 @UC Requirements = regexp("uct2-c*", Machine)
-MWT2 @IU Requirements = regexp("iut2-c*", Machine)
-MWT2 @UIUC Requirements = regexp("taub*", Machine)
-UC ATLAS Tier 3 Requirements = regexp("uct3*", Machine)
-ITS Virtualization Lab Requirements = regexp("appcloud[0-1][0-9].*", Machine)
-Open Science Grid
-Requirements = isUndefined(GLIDECLIENT_Name) == FALSE
-
- -->
END
for p in 04 05 06; do
@@ -130,3 +109,19 @@
done
return
+
+# Integrate somewhere:
+
+cat <<END
+ <!-- OSG Connect Resource selector expressions: -->
+ <!-- UC3S regexp("uc3-c*", Machine) -->
+ <!-- UCIT regexp("appcloud[0-1][0-9].*", Machine)-->
+ <!-- MWUC regexp("uct2-c*", Machine) -->
+ <!-- MWIU regexp("iut2-c*", Machine) -->
+ <!-- MWUI regexp("taub*", Machine) -->
+ <!-- MWT2 UidDomain == "osg-gk.mwt2.org" -->
+ <!-- OSG isUndefined(GLIDECLIENT_Name) == FALSE -->
+ <!-- MULT UidDomain == "osg-gk.mwt2.org" && (regexp("iut2-c*", Machine) || regexp("uct2-c*", Machine)) -->
+ <!-- E.g. for UC3 cycle seeder: -->
+ <!-- <profile namespace="globus" key="condor.Requirements">regexp("uc3-c*", Machine)</profile> -->
+END
More information about the Swift-commit
mailing list