[Swift-commit] r6232 - trunk/docs/cookbook
ketan at ci.uchicago.edu
ketan at ci.uchicago.edu
Mon Feb 4 08:29:41 CST 2013
Author: ketan
Date: 2013-02-04 08:29:39 -0600 (Mon, 04 Feb 2013)
New Revision: 6232
Added:
trunk/docs/cookbook/debugging
trunk/docs/cookbook/infrastructures
Removed:
trunk/docs/cookbook/debugging_swift
trunk/docs/cookbook/diverse_infrastructures
Modified:
trunk/docs/cookbook/cookbook.txt
Log:
cookbook organize
Modified: trunk/docs/cookbook/cookbook.txt
===================================================================
--- trunk/docs/cookbook/cookbook.txt 2013-02-04 14:27:24 UTC (rev 6231)
+++ trunk/docs/cookbook/cookbook.txt 2013-02-04 14:29:39 UTC (rev 6232)
@@ -13,8 +13,8 @@
include::coasters[]
-include::diverse_infrastructures[]
+include::infrastructures[]
-include::debugging_swift[]
+include::debugging[]
include::log_processing[]
Copied: trunk/docs/cookbook/debugging (from rev 6230, trunk/docs/cookbook/debugging_swift)
===================================================================
--- trunk/docs/cookbook/debugging (rev 0)
+++ trunk/docs/cookbook/debugging 2013-02-04 14:29:39 UTC (rev 6232)
@@ -0,0 +1,17 @@
+Debugging Swift
+---------------
+Swift errors are logged in several places:
+
+. All text from standard output and standard error produced by running the
+swift command
+. The .log file from this run. It will be named swiftscript.uniqueID.log
+where "swiftscript" is the name of your *.swift script source file, and
+uniqueID is a long unique id which starts with the date and time you ran the
+swift command.
+. $HOME/.globus/coasters directory on remote machines on which you are
+running coasters
+. $HOME/.globus/scripts directory on the host on which you run the Swift
+command, when swift is submitting to a local scheduler (Condor, PBS, SGE,
+Cobalt)
+. $HOME/.globus/??? on remote systems that you access via Globus
+
Deleted: trunk/docs/cookbook/debugging_swift
===================================================================
--- trunk/docs/cookbook/debugging_swift 2013-02-04 14:27:24 UTC (rev 6231)
+++ trunk/docs/cookbook/debugging_swift 2013-02-04 14:29:39 UTC (rev 6232)
@@ -1,17 +0,0 @@
-Debugging Swift
----------------
-Swift errors are logged in several places:
-
-. All text from standard output and standard error produced by running the
-swift command
-. The .log file from this run. It will be named swiftscript.uniqueID.log
-where "swiftscript" is the name of your *.swift script source file, and
-uniqueID is a long unique id which starts with the date and time you ran the
-swift command.
-. $HOME/.globus/coasters directory on remote machines on which you are
-running coasters
-. $HOME/.globus/scripts directory on the host on which you run the Swift
-command, when swift is submitting to a local scheduler (Condor, PBS, SGE,
-Cobalt)
-. $HOME/.globus/??? on remote systems that you access via Globus
-
Deleted: trunk/docs/cookbook/diverse_infrastructures
===================================================================
--- trunk/docs/cookbook/diverse_infrastructures 2013-02-04 14:27:24 UTC (rev 6231)
+++ trunk/docs/cookbook/diverse_infrastructures 2013-02-04 14:29:39 UTC (rev 6232)
@@ -1,182 +0,0 @@
-Swift on Diverse Infrastructures
----------------------------------
-
-Beagle
-~~~~~~
-Swift is available on Beagle as a software module.
-In this recipe, we will walk you through a very simple example running Swift
-on Beagle. The prerequisites required for running this example on Beagle is an access to Beagle and a project allocation. The example is a simplest form of a real application that involves reading from a disc file and writing onto another disc file. It runs the +/bin/cat+ Unix command to read an input file called +data.txt+ and write it to an output file in a folder +outdir+, called +f.nnn.out+.
-
-image::figures/catsn.png[Swift Example]
-
-The steps are as follows:
-
-*step 1.* Load the Swift module on Beagle commandline as follows: +$ module load swift+
-
-*step 2.* Create and change to a directory where your Swift related work (source, inputs, outputs) will
-stay. (say, +$ mkdir swift-lab+, followed by, +$ cd swift-lab+)
-
-*step 3.* copy the folder at +/home/ketan/labs/catsn+ to the above directory. (+$cp -r /home/ketan/catsn .+ followed by +$ cd catsn+).
-
-*step 4.* In the sites file: +sites.xml+ make the following two
-changes:
-
- 1. change the path of the +$$<workdirectory>$$+ element to your preferred location (this location is *required* to be on /lustre, say to +/lustre/beagle/<yourname>/swift-lab/swift.workdir+), and
-
- 2. Change the project name to your project (for instance, +CI-CCR000013+) . The workdirectory will contain
- execution data related to each run, e.g. wrapper scripts, system information, inputs and outputs.
-
-*step 5.* Run the example using following commandline (also found in run.sh):
-
-----
-$ swift -config cf -tc.file tc -sites.file sites.xml catsn.swift -n=1
-----
-
-You can further change the value of +-n+ to any arbitrary number to run that
-many number of concurrent +cat+
-
-*step 6.* Check the output in the generated +outdir+ directory (+ls outdir+)
-
-Note: Running from sandbox node or requesting 1 hour walltime for upto 3 nodes
-will get fast prioritized execution. Good for small tests
-Swift supports a coasters based computing environment for Beagle. A detailed version of Swift documentation is maintained
-http://www.ci.uchicago.edu/swift/docs/index.php[here].
-
-//
-//Intrepid-BG/P
-//~~~~~~~~~~~~~
-//Swift on Intrepid-BG/P
-//
-
-PADS
-~~~~
-Swift on PADS
-To execute your Swift script on the PADS cluster use this command:
-
-----
-swift -tc.file tc -sites.file pbs.xml modis.swift
-----
-where the contents of a simple pbs.xml sites file could be:
-[xml]
-source~~~~
-<config>
- <pool handle="pbs">
- <execution provider="pbs" url="none"/>
- <profile namespace="globus" key="queue">fast</profile>
- <profile namespace="globus" key="maxwalltime">00:05:00</profile>
- <profile namespace="karajan" key="initialScore">10000</profile>
- <profile namespace="karajan" key="jobThrottle">.10</profile>
- <filesystem provider="local"/>
- <workdirectory >/home/you/swiftwork</workdirectory>
- </pool>
-</config>
-source~~~~
-
-OSG
-~~~
-This section describes how to get Swift running on the OSG Grid. We will use a
-manual coaster setup to get Swift running on OSG.
-
-.Coaster setup on OSG
-The following figure shows an abstract scheme for the manual coasters setup on
-OSG.
-
-image::figures/coaster_setup.png[Coaster setup]
-
-In the following steps, we will go through the process of manually setting
-//Swift on OSG
-//
-
-Bionimbus
-~~~~~~~~~
-This section explains a step by step procedure on getting Swift running on the
-Bionimbus cloud. We will use the _manual_ _coasters_ configuration on the
-Bionimbus cloud.
-
-**step1.** Connect to the gateway (ssh gatewayx.lac.uic.edu)
-
-**step2.** Start a virtual machine (euca-run-instances -n 1 -t m1.small
-emi-17EB1170)
-
-**step3.** Start the coaster-service on gateway
-+coaster-service -port 1984 -localport 35753 -nosec+
-
-**step4.** Start the Swift-script from the gateway using normal Swift commandline
-
-+swift -config cf -tc.file tc -sites.file sites.xml yourscript.swift -aparam=999+
-
-.cf
-----
-wrapperlog.always.transfer=true
-sitedir.keep=true
-execution.retries=1
-lazy.errors=true
-status.mode=provider
-use.provider.staging=true
-provider.staging.pin.swiftfiles=false
-foreach.max.threads=100
-provenance.log=true
-----
-
-.tc
-----
-localhost modftdock /home/ketan/swift-labs/bionimbus-coaster-modftdock/app/modftdock.sh null null GLOBUS::maxwalltime="1:00:00"
-----
-
-(See below for a sample sites.xml for this run)
-
-**step5.** Connect back to gateway from virtual machines using reverse ssh tunneling as follows:
-
-.From the gateway prompt
-
-+ssh -R *:5000:localhost:5000 root at 10.101.8.50 sleep 999+
-
-WHERE:
-*=network interface, should remain the same on all cases
-
-localhost=the gateway host, should remain the same
-
-5000(LEFT OF localhost)=the port number on localhost to listen to **THIS WILL
-vary depending upon which port you want to listen to
-
-5000(RIGHT OF localhost)=the port on target host that you want to forward
-
-root at 10.101.8.50=the ip of the Virtual Machine on Bionimbus cloud, this will
-vary based on what ip you get for your Virtual Machine instance
-
-#On anywhere as long as provide the correct callback uri: here the
-"http://140.221.9.110:42195" is the callback uri of previous ones
-
-**step6.** Start the worker from the virtual machine
-+worker.pl http://localhost:42195 tmp /tmp # where 42195 is the port where the
-coaster service is listening to the workers+
-
-.sites.xml for the above run
-
-----
-<config>
- <pool handle="localhost">
- <execution provider="coaster-persistent" url="http://localhost:1984" jobmanager="local:local"/>
- <profile namespace="globus" key="workerManager">passive</profile>
-
- <profile namespace="globus" key="workersPerNode">4</profile>
- <profile namespace="globus" key="maxTime">10000</profile>
- <profile namespace="globus" key="lowOverAllocation">100</profile>
- <profile namespace="globus" key="highOverAllocation">100</profile>
- <profile namespace="globus" key="slots">100</profile>
- <profile namespace="globus" key="nodeGranularity">1</profile>
- <profile namespace="globus" key="maxNodes">10</profile>
- <profile namespace="karajan" key="jobThrottle">25.00</profile>
- <profile namespace="karajan" key="initialScore">10000</profile>
- <profile namespace="swift" key="stagingMethod">proxy</profile>
- <filesystem provider="local"/>
- <workdirectory>/home/ketan/swift-labs/bionimbus-coaster-modftdock/swift.workdir</workdirectory>
- </pool>
-</config>
-----
-
-//Magellan
-//~~~~~~~~
-//Swift on Magellan
-//
-
Copied: trunk/docs/cookbook/infrastructures (from rev 6230, trunk/docs/cookbook/diverse_infrastructures)
===================================================================
--- trunk/docs/cookbook/infrastructures (rev 0)
+++ trunk/docs/cookbook/infrastructures 2013-02-04 14:29:39 UTC (rev 6232)
@@ -0,0 +1,182 @@
+Swift on Diverse Infrastructures
+---------------------------------
+
+Beagle
+~~~~~~
+Swift is available on Beagle as a software module.
+In this recipe, we will walk you through a very simple example running Swift
+on Beagle. The prerequisites required for running this example on Beagle is an access to Beagle and a project allocation. The example is a simplest form of a real application that involves reading from a disc file and writing onto another disc file. It runs the +/bin/cat+ Unix command to read an input file called +data.txt+ and write it to an output file in a folder +outdir+, called +f.nnn.out+.
+
+image::figures/catsn.png[Swift Example]
+
+The steps are as follows:
+
+*step 1.* Load the Swift module on Beagle commandline as follows: +$ module load swift+
+
+*step 2.* Create and change to a directory where your Swift related work (source, inputs, outputs) will
+stay. (say, +$ mkdir swift-lab+, followed by, +$ cd swift-lab+)
+
+*step 3.* copy the folder at +/home/ketan/labs/catsn+ to the above directory. (+$cp -r /home/ketan/catsn .+ followed by +$ cd catsn+).
+
+*step 4.* In the sites file: +sites.xml+ make the following two
+changes:
+
+ 1. change the path of the +$$<workdirectory>$$+ element to your preferred location (this location is *required* to be on /lustre, say to +/lustre/beagle/<yourname>/swift-lab/swift.workdir+), and
+
+ 2. Change the project name to your project (for instance, +CI-CCR000013+) . The workdirectory will contain
+ execution data related to each run, e.g. wrapper scripts, system information, inputs and outputs.
+
+*step 5.* Run the example using following commandline (also found in run.sh):
+
+----
+$ swift -config cf -tc.file tc -sites.file sites.xml catsn.swift -n=1
+----
+
+You can further change the value of +-n+ to any arbitrary number to run that
+many number of concurrent +cat+
+
+*step 6.* Check the output in the generated +outdir+ directory (+ls outdir+)
+
+Note: Running from sandbox node or requesting 1 hour walltime for upto 3 nodes
+will get fast prioritized execution. Good for small tests
+Swift supports a coasters based computing environment for Beagle. A detailed version of Swift documentation is maintained
+http://www.ci.uchicago.edu/swift/docs/index.php[here].
+
+//
+//Intrepid-BG/P
+//~~~~~~~~~~~~~
+//Swift on Intrepid-BG/P
+//
+
+PADS
+~~~~
+Swift on PADS
+To execute your Swift script on the PADS cluster use this command:
+
+----
+swift -tc.file tc -sites.file pbs.xml modis.swift
+----
+where the contents of a simple pbs.xml sites file could be:
+[xml]
+source~~~~
+<config>
+ <pool handle="pbs">
+ <execution provider="pbs" url="none"/>
+ <profile namespace="globus" key="queue">fast</profile>
+ <profile namespace="globus" key="maxwalltime">00:05:00</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ <profile namespace="karajan" key="jobThrottle">.10</profile>
+ <filesystem provider="local"/>
+ <workdirectory >/home/you/swiftwork</workdirectory>
+ </pool>
+</config>
+source~~~~
+
+OSG
+~~~
+This section describes how to get Swift running on the OSG Grid. We will use a
+manual coaster setup to get Swift running on OSG.
+
+.Coaster setup on OSG
+The following figure shows an abstract scheme for the manual coasters setup on
+OSG.
+
+image::figures/coaster_setup.png[Coaster setup]
+
+In the following steps, we will go through the process of manually setting
+//Swift on OSG
+//
+
+Bionimbus
+~~~~~~~~~
+This section explains a step by step procedure on getting Swift running on the
+Bionimbus cloud. We will use the _manual_ _coasters_ configuration on the
+Bionimbus cloud.
+
+**step1.** Connect to the gateway (ssh gatewayx.lac.uic.edu)
+
+**step2.** Start a virtual machine (euca-run-instances -n 1 -t m1.small
+emi-17EB1170)
+
+**step3.** Start the coaster-service on gateway
++coaster-service -port 1984 -localport 35753 -nosec+
+
+**step4.** Start the Swift-script from the gateway using normal Swift commandline
+
++swift -config cf -tc.file tc -sites.file sites.xml yourscript.swift -aparam=999+
+
+.cf
+----
+wrapperlog.always.transfer=true
+sitedir.keep=true
+execution.retries=1
+lazy.errors=true
+status.mode=provider
+use.provider.staging=true
+provider.staging.pin.swiftfiles=false
+foreach.max.threads=100
+provenance.log=true
+----
+
+.tc
+----
+localhost modftdock /home/ketan/swift-labs/bionimbus-coaster-modftdock/app/modftdock.sh null null GLOBUS::maxwalltime="1:00:00"
+----
+
+(See below for a sample sites.xml for this run)
+
+**step5.** Connect back to gateway from virtual machines using reverse ssh tunneling as follows:
+
+.From the gateway prompt
+
++ssh -R *:5000:localhost:5000 root at 10.101.8.50 sleep 999+
+
+WHERE:
+*=network interface, should remain the same on all cases
+
+localhost=the gateway host, should remain the same
+
+5000(LEFT OF localhost)=the port number on localhost to listen to **THIS WILL
+vary depending upon which port you want to listen to
+
+5000(RIGHT OF localhost)=the port on target host that you want to forward
+
+root at 10.101.8.50=the ip of the Virtual Machine on Bionimbus cloud, this will
+vary based on what ip you get for your Virtual Machine instance
+
+#On anywhere as long as provide the correct callback uri: here the
+"http://140.221.9.110:42195" is the callback uri of previous ones
+
+**step6.** Start the worker from the virtual machine
++worker.pl http://localhost:42195 tmp /tmp # where 42195 is the port where the
+coaster service is listening to the workers+
+
+.sites.xml for the above run
+
+----
+<config>
+ <pool handle="localhost">
+ <execution provider="coaster-persistent" url="http://localhost:1984" jobmanager="local:local"/>
+ <profile namespace="globus" key="workerManager">passive</profile>
+
+ <profile namespace="globus" key="workersPerNode">4</profile>
+ <profile namespace="globus" key="maxTime">10000</profile>
+ <profile namespace="globus" key="lowOverAllocation">100</profile>
+ <profile namespace="globus" key="highOverAllocation">100</profile>
+ <profile namespace="globus" key="slots">100</profile>
+ <profile namespace="globus" key="nodeGranularity">1</profile>
+ <profile namespace="globus" key="maxNodes">10</profile>
+ <profile namespace="karajan" key="jobThrottle">25.00</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ <profile namespace="swift" key="stagingMethod">proxy</profile>
+ <filesystem provider="local"/>
+ <workdirectory>/home/ketan/swift-labs/bionimbus-coaster-modftdock/swift.workdir</workdirectory>
+ </pool>
+</config>
+----
+
+//Magellan
+//~~~~~~~~
+//Swift on Magellan
+//
+
More information about the Swift-commit
mailing list