[Swift-commit] r5086 - in trunk/docs: . cookbook siteguide stylesheets tutorial userguide utils
davidk at ci.uchicago.edu
davidk at ci.uchicago.edu
Wed Sep 7 19:53:13 CDT 2011
Author: davidk
Date: 2011-09-07 19:53:06 -0500 (Wed, 07 Sep 2011)
New Revision: 5086
Added:
trunk/docs/README
trunk/docs/cookbook/cookbook-asciidoc.txt
trunk/docs/print_link.sh
trunk/docs/siteguide/
trunk/docs/siteguide/beagle
trunk/docs/siteguide/futuregrid
trunk/docs/siteguide/grid
trunk/docs/siteguide/images
trunk/docs/siteguide/pads
trunk/docs/siteguide/siteguide.txt
trunk/docs/stylesheets/
trunk/docs/stylesheets/asciidoc.css
trunk/docs/tutorial/abstract
trunk/docs/tutorial/bits
trunk/docs/tutorial/hello_world
trunk/docs/tutorial/images/
trunk/docs/tutorial/introduction
trunk/docs/tutorial/language_features
trunk/docs/tutorial/runtime_features
trunk/docs/tutorial/tutorial.txt
trunk/docs/utils/
trunk/docs/utils/gensites.txt
Removed:
trunk/docs/README
trunk/docs/cookbook/cookbook.txt
trunk/docs/documentation/
trunk/docs/print_link.sh
trunk/docs/quickstart/
trunk/docs/siteguide/
trunk/docs/siteguide/beagle
trunk/docs/siteguide/futuregrid
trunk/docs/siteguide/grid
trunk/docs/siteguide/images
trunk/docs/siteguide/pads
trunk/docs/siteguide/siteguide.txt
trunk/docs/stylesheets/
trunk/docs/stylesheets/asciidoc.css
trunk/docs/tutorial/abstract
trunk/docs/tutorial/bits
trunk/docs/tutorial/hello_world
trunk/docs/tutorial/images/
trunk/docs/tutorial/introduction
trunk/docs/tutorial/language_features
trunk/docs/tutorial/runtime_features
trunk/docs/tutorial/tutorial.txt
trunk/docs/userguide/log-processing
trunk/docs/utils/
trunk/docs/utils/gensites.txt
Modified:
trunk/docs/
trunk/docs/cookbook/
trunk/docs/tutorial/
trunk/docs/userguide/cdm
trunk/docs/userguide/coasters
trunk/docs/userguide/userguide.txt
Log:
Doc merge fixes
Property changes on: trunk/docs
___________________________________________________________________
Deleted: svn:mergeinfo
- /branches/release-0.93/docs:4761-5078
Deleted: trunk/docs/README
===================================================================
--- trunk/docs/README 2011-09-07 21:09:21 UTC (rev 5085)
+++ trunk/docs/README 2011-09-08 00:53:06 UTC (rev 5086)
@@ -1,11 +0,0 @@
-Prerequisites to building documentation:
-
-* asciidoc and a2x (http://www.methods.co.nz/asciidoc)
-* dblatex (http://dblatex.sourceforge.net)
-* source-highlight (http://www.gnu.org/software/src-highlite)
-
-For CI users, compiled versions of these packages can be found in ~davidk.
-
-To build, run:
-
-build_docs.sh <installation_dir>
Copied: trunk/docs/README (from rev 5078, trunk/docs/README)
===================================================================
--- trunk/docs/README (rev 0)
+++ trunk/docs/README 2011-09-08 00:53:06 UTC (rev 5086)
@@ -0,0 +1,11 @@
+Prerequisites to building documentation:
+
+* asciidoc and a2x (http://www.methods.co.nz/asciidoc)
+* dblatex (http://dblatex.sourceforge.net)
+* source-highlight (http://www.gnu.org/software/src-highlite)
+
+For CI users, compiled versions of these packages can be found in ~davidk.
+
+To build, run:
+
+build_docs.sh <installation_dir>
Property changes on: trunk/docs/cookbook
___________________________________________________________________
Added: svn:ignore
+ cookbook-asciidoc.html
cookbook-asciidoc.pdf
Copied: trunk/docs/cookbook/cookbook-asciidoc.txt (from rev 5078, trunk/docs/cookbook/cookbook-asciidoc.txt)
===================================================================
--- trunk/docs/cookbook/cookbook-asciidoc.txt (rev 0)
+++ trunk/docs/cookbook/cookbook-asciidoc.txt 2011-09-08 00:53:06 UTC (rev 5086)
@@ -0,0 +1,20 @@
+// To compile use: asciidoc -a toc -n cookbook-asciidoc.txt
+
+:numbered:
+
+Swift Cookbook
+==============
+Swift team <swift-user at ci.uchicago.edu>
+v0.92, March 2011
+
+include::overview[]
+
+include::swift_basics[]
+
+include::coasters[]
+
+include::diverse_infrastructures[]
+
+include::debugging_swift[]
+
+include::log_processing[]
Deleted: trunk/docs/cookbook/cookbook.txt
===================================================================
--- trunk/docs/cookbook/cookbook.txt 2011-09-07 21:09:21 UTC (rev 5085)
+++ trunk/docs/cookbook/cookbook.txt 2011-09-08 00:53:06 UTC (rev 5086)
@@ -1,16 +0,0 @@
-Swift Cookbook
-==============
-Swift team <swift-user at ci.uchicago.edu>
-v0.92, March 2011
-
-include::overview[]
-
-include::swift_basics[]
-
-include::coasters[]
-
-include::diverse_infrastructures[]
-
-include::debugging_swift[]
-
-include::log_processing[]
Deleted: trunk/docs/print_link.sh
===================================================================
--- trunk/docs/print_link.sh 2011-09-07 21:09:21 UTC (rev 5085)
+++ trunk/docs/print_link.sh 2011-09-08 00:53:06 UTC (rev 5086)
@@ -1,21 +0,0 @@
-#!/bin/bash
-
-# This script is referenced by asciidoc to point to the correct versions of documentation based on version information
-# print_info.sh <guide>
-# This will print the correct URL based on SVN version info
-
-pushd ../.. > /dev/null 2>&1
-VERSION=`svn info |grep URL|awk -F / '{print $NF}'`
-popd > /dev/null 2>&1
-
-# Parse command line arguments
-case "$1" in
- userguide)
- echo http://www.ci.uchicago.edu/swift/guides/$VERSION/userguide/userguide.html[Swift User Guide]
- ;;
- tutorial)
- echo http://www.ci.uchicago.edu/swift/guides/$VERSION/tutorial/tutorial.html[Swift Tutorial]
- ;;
- *) echo http://www.ci.uchicago.edu/swift/docs/index.php[Swift Documentation]
- ;;
-esac
Copied: trunk/docs/print_link.sh (from rev 5078, trunk/docs/print_link.sh)
===================================================================
--- trunk/docs/print_link.sh (rev 0)
+++ trunk/docs/print_link.sh 2011-09-08 00:53:06 UTC (rev 5086)
@@ -0,0 +1,21 @@
+#!/bin/bash
+
+# This script is referenced by asciidoc to point to the correct versions of documentation based on version information
+# print_info.sh <guide>
+# This will print the correct URL based on SVN version info
+
+pushd ../.. > /dev/null 2>&1
+VERSION=`svn info |grep URL|awk -F / '{print $NF}'`
+popd > /dev/null 2>&1
+
+# Parse command line arguments
+case "$1" in
+ userguide)
+ echo http://www.ci.uchicago.edu/swift/guides/$VERSION/userguide/userguide.html[Swift User Guide]
+ ;;
+ tutorial)
+ echo http://www.ci.uchicago.edu/swift/guides/$VERSION/tutorial/tutorial.html[Swift Tutorial]
+ ;;
+ *) echo http://www.ci.uchicago.edu/swift/docs/index.php[Swift Documentation]
+ ;;
+esac
Property changes on: trunk/docs/siteguide
___________________________________________________________________
Added: svn:ignore
+ siteguide.html
siteguide.pdf
Deleted: trunk/docs/siteguide/beagle
===================================================================
--- trunk/docs/siteguide/beagle 2011-09-06 19:12:14 UTC (rev 5078)
+++ trunk/docs/siteguide/beagle 2011-09-08 00:53:06 UTC (rev 5086)
@@ -1,49 +0,0 @@
-Beagle
-------
-
-Requesting Access
-~~~~~~~~~~~~~~~~~
-If you do not already have a Computation Institute account, you can request
-one at https://www.ci.uchicago.edu/accounts/. This page will give you a list
-of resources you can request access to.
-You already have an existing CI account, but do not have access to Beagle,
-send an email to support at ci.uchicago.edu to request access.
-
-Connecting to a login node
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-Once you have account, you should be able to access a Beagle login
-node with the following command:
-
------
-ssh yourusername at login.beagle.ci.uchicago.edu
------
-
-Follow the steps outlined below to get started with Swift on Beagle:
-
-*step 1.* Load the Swift module on Beagle as follows: +module load swift+
-
-*step 2.* Create and change to a directory where your Swift related work will
-stay. (say, +mkdir swift-lab+, followed by, +cd swift-lab+)
-
-*step 3.* To get started with a simple example running +/bin/cat+ to read an
-input file +data.txt+ and write to an output file +f.nnn.out+, copy the folder
-at +/home/ketan/catsn+ to the above directory. (+cp -r /home/ketan/catsn
-.+ followed by +cd catsn+).
-
-*step 4.* In the sites file: +beagle-coaster.xml+, make the following two
-changes: *1)* change the path of +workdirectory+ to your preferred location
-(say to +/lustre/beagle/$USER/swift-lab/swift.workdir+) and *2)* Change the
-project name to your project (+CI-CCR000013+) . The workdirectory will contain
-execution data related to each run, e.g. wrapper scripts, system information,
-inputs and outputs.
-
-*step 5.* Run the example using following commandline (also found in run.sh):
-+swift -config cf -tc.file tc -sites.file beagle-coaster.xml catsn.swift -n=1+
-. You can further change the value of +-n+ to any arbitrary number to run that
-many number of concurrent +cat+
-
-*step 6.* Check the output in the generated +outdir+ directory (+ls outdir+)
-
-Note: Running from sandbox node or requesting 1 hour walltime for upto 3 nodes
-will get fast prioritized execution. Good for small tests.
-
Copied: trunk/docs/siteguide/beagle (from rev 5078, trunk/docs/siteguide/beagle)
===================================================================
--- trunk/docs/siteguide/beagle (rev 0)
+++ trunk/docs/siteguide/beagle 2011-09-08 00:53:06 UTC (rev 5086)
@@ -0,0 +1,49 @@
+Beagle
+------
+
+Requesting Access
+~~~~~~~~~~~~~~~~~
+If you do not already have a Computation Institute account, you can request
+one at https://www.ci.uchicago.edu/accounts/. This page will give you a list
+of resources you can request access to.
+You already have an existing CI account, but do not have access to Beagle,
+send an email to support at ci.uchicago.edu to request access.
+
+Connecting to a login node
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+Once you have account, you should be able to access a Beagle login
+node with the following command:
+
+-----
+ssh yourusername at login.beagle.ci.uchicago.edu
+-----
+
+Follow the steps outlined below to get started with Swift on Beagle:
+
+*step 1.* Load the Swift module on Beagle as follows: +module load swift+
+
+*step 2.* Create and change to a directory where your Swift related work will
+stay. (say, +mkdir swift-lab+, followed by, +cd swift-lab+)
+
+*step 3.* To get started with a simple example running +/bin/cat+ to read an
+input file +data.txt+ and write to an output file +f.nnn.out+, copy the folder
+at +/home/ketan/catsn+ to the above directory. (+cp -r /home/ketan/catsn
+.+ followed by +cd catsn+).
+
+*step 4.* In the sites file: +beagle-coaster.xml+, make the following two
+changes: *1)* change the path of +workdirectory+ to your preferred location
+(say to +/lustre/beagle/$USER/swift-lab/swift.workdir+) and *2)* Change the
+project name to your project (+CI-CCR000013+) . The workdirectory will contain
+execution data related to each run, e.g. wrapper scripts, system information,
+inputs and outputs.
+
+*step 5.* Run the example using following commandline (also found in run.sh):
++swift -config cf -tc.file tc -sites.file beagle-coaster.xml catsn.swift -n=1+
+. You can further change the value of +-n+ to any arbitrary number to run that
+many number of concurrent +cat+
+
+*step 6.* Check the output in the generated +outdir+ directory (+ls outdir+)
+
+Note: Running from sandbox node or requesting 1 hour walltime for upto 3 nodes
+will get fast prioritized execution. Good for small tests.
+
Deleted: trunk/docs/siteguide/futuregrid
===================================================================
--- trunk/docs/siteguide/futuregrid 2011-09-06 19:12:14 UTC (rev 5078)
+++ trunk/docs/siteguide/futuregrid 2011-09-08 00:53:06 UTC (rev 5086)
@@ -1,134 +0,0 @@
-FutureGrid
-----------
-
-Downloading and Building Swift
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The most recent versions of Swift can be found at
-http://www.ci.uchicago.edu/swift/downloads/index.php. To run on futuregrid,
-you will need to download the development/trunk version from SVN.
-
-Adding Swift to your PATH
-~~~~~~~~~~~~~~~~~~~~~~~~~
-Once you have installed Swift, add the Swift binary to your PATH so you can
-easily run it from any directory.
-
-In your home directory, edit the file ".bashrc".
-
-If you have installed Swift via a source repository, add the following line
-at the bottom of .bashrc.
-
------
-export PATH=$PATH:$HOME/cog/modules/swift/dist/swift-svn/bin
------
-
-Requesting Futuregrid Access
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-If you do not already have a futuregrid account, you can follow the
-instructions at https://portal.futuregrid.org/gettingstarted to get started.
-This page provides information on how to create an account, how to join
-a project, how to set up your SSH keys, and how to create a new project.
-
-Downloading Swift VM Tools
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-A set of scripts based around cloudinitd are used to easily start virtual
-machines. To download, change to your home directory and run the
-following command:
-
------
-$ svn co https://svn.ci.uchicago.edu/svn/vdl2/usertools/swift-vm-boot
------
-
-Download your Credentials
-~~~~~~~~~~~~~~~~~~~~~~~~~
-Run the following commands to retrieve your credentials:
------
-$ scp yourusername at hotel.futuregrid.org:nimbus_creds.tar.gz .
-$ tar xvfz nimbus_creds.tar.gz
------
-When you extract your credential file, look at the file called
-hotel.conf. Near the bottom of this file will be two settings
-called vws.repository.s3id and vws.repository.s3key. Copy these
-values for the next step.
-
-Configuring Swift VM Tools
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-Change directories into the swift-vm-boot directory. Edit the file
-called env.sh. There are two settings here that you will need to
-modify: FUTUREGRID_IAAS_ACCESS_KEY and FUTUREGRID_IAAS_SECRET_KEY.
-Paste your access key (s3id) and secret key (s3key) here and save
-the file. By default, env.sh requests 2 nodes on hotel and 2 nodes
-on sierra. Change these values as needed. Run install.sh from the
-swift-vm-boot directory to complete the installation of Swift VM
-Tools. The Swift VM Tools may require a fairly recent version of
-Python. If you run into problems while running the install.sh
-script, please try a more recent version of Python and associated
-libraries.
-
-Configuring coaster-service.conf
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Within your swift distribution's etc directory is a file called
-coaster-service.conf. There are many options here you can modify
-as needed, but these are the settings that will be required
-to run on futuregrid:
-
------
-export WORKER_WORK=/tmp
-export WORKER_MODE=futuregrid
-export SHARED_FILESYSTEM=no
-export WORKER_USERNAME=root
-export SSH_TUNNELING=yes
-export SWIFTVMBOOT_DIR=$HOME/swift-vm-boot
-export WORK=/tmp
------
-
-These is a brief description of these settings in the coaster-service.conf file.
-Tunneling should be used when you are restricted by a firewall. If your local
-machine has multiple network interfaces, you should also manually set the IP
-address of your machine with export IPADDR=your.ip.address.here.
-
-Starting the Coaster Service Script
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Now that everything is configured, change to a temporary directory
-and run this command to start the coaster service:
-
------
-$ start-coaster-service
------
-
-This command will start the VMs, start the required processes on the worker nodes,
-and generate Swift configuration files for you to use. The configuration files
-will be generated in your current directory. These files are sites.xml, tc.data,
-and cf.
-
-Running Swift
-~~~~~~~~~~~~~
-Now that you have all of your configuration files generated, run the following command:
-
------
-$ swift -sites.file sites.xml -tc.file tc.data -config cf <yourscript.swift>
------
-
-If you would like to create a custom tc file for repeated use, rename it to something other
-than tc.data to prevent it from being overwritten. The sites.xml however will need to be
-regenerated every time you start the coaster service. If you need to repeatedly modify some
-sites.xml options, you may edit the template in Swift's etc/sites/persistent-coasters. You
-may also create your own custom tc files with the hostname of persistent-coasters. More
-information about this can be found in the Swift userguide at
-http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html.
-
-Stopping the Coaster Service Script
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-To stop the coaster service, run the following command:
------
-$ stop-coaster-service
------
-
-This will kill the coaster service, kill the worker scripts on remote systems and terminate
-the virtual machines that were created during start-coaster-service.
-
-More Help
-~~~~~~~~~
-The best place for additional help is the Swift user mailing list. You can subscribe to this list at
-http://mail.ci.uchicago.edu/mailman/listinfo/swift-user. When submitting information, please send
-your sites.xml file, your tc.data, and any error messages you run into.
-
Copied: trunk/docs/siteguide/futuregrid (from rev 5078, trunk/docs/siteguide/futuregrid)
===================================================================
--- trunk/docs/siteguide/futuregrid (rev 0)
+++ trunk/docs/siteguide/futuregrid 2011-09-08 00:53:06 UTC (rev 5086)
@@ -0,0 +1,134 @@
+FutureGrid
+----------
+
+Downloading and Building Swift
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The most recent versions of Swift can be found at
+http://www.ci.uchicago.edu/swift/downloads/index.php. To run on futuregrid,
+you will need to download the development/trunk version from SVN.
+
+Adding Swift to your PATH
+~~~~~~~~~~~~~~~~~~~~~~~~~
+Once you have installed Swift, add the Swift binary to your PATH so you can
+easily run it from any directory.
+
+In your home directory, edit the file ".bashrc".
+
+If you have installed Swift via a source repository, add the following line
+at the bottom of .bashrc.
+
+-----
+export PATH=$PATH:$HOME/cog/modules/swift/dist/swift-svn/bin
+-----
+
+Requesting Futuregrid Access
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+If you do not already have a futuregrid account, you can follow the
+instructions at https://portal.futuregrid.org/gettingstarted to get started.
+This page provides information on how to create an account, how to join
+a project, how to set up your SSH keys, and how to create a new project.
+
+Downloading Swift VM Tools
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+A set of scripts based around cloudinitd are used to easily start virtual
+machines. To download, change to your home directory and run the
+following command:
+
+-----
+$ svn co https://svn.ci.uchicago.edu/svn/vdl2/usertools/swift-vm-boot
+-----
+
+Download your Credentials
+~~~~~~~~~~~~~~~~~~~~~~~~~
+Run the following commands to retrieve your credentials:
+-----
+$ scp yourusername at hotel.futuregrid.org:nimbus_creds.tar.gz .
+$ tar xvfz nimbus_creds.tar.gz
+-----
+When you extract your credential file, look at the file called
+hotel.conf. Near the bottom of this file will be two settings
+called vws.repository.s3id and vws.repository.s3key. Copy these
+values for the next step.
+
+Configuring Swift VM Tools
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+Change directories into the swift-vm-boot directory. Edit the file
+called env.sh. There are two settings here that you will need to
+modify: FUTUREGRID_IAAS_ACCESS_KEY and FUTUREGRID_IAAS_SECRET_KEY.
+Paste your access key (s3id) and secret key (s3key) here and save
+the file. By default, env.sh requests 2 nodes on hotel and 2 nodes
+on sierra. Change these values as needed. Run install.sh from the
+swift-vm-boot directory to complete the installation of Swift VM
+Tools. The Swift VM Tools may require a fairly recent version of
+Python. If you run into problems while running the install.sh
+script, please try a more recent version of Python and associated
+libraries.
+
+Configuring coaster-service.conf
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Within your swift distribution's etc directory is a file called
+coaster-service.conf. There are many options here you can modify
+as needed, but these are the settings that will be required
+to run on futuregrid:
+
+-----
+export WORKER_WORK=/tmp
+export WORKER_MODE=futuregrid
+export SHARED_FILESYSTEM=no
+export WORKER_USERNAME=root
+export SSH_TUNNELING=yes
+export SWIFTVMBOOT_DIR=$HOME/swift-vm-boot
+export WORK=/tmp
+-----
+
+These is a brief description of these settings in the coaster-service.conf file.
+Tunneling should be used when you are restricted by a firewall. If your local
+machine has multiple network interfaces, you should also manually set the IP
+address of your machine with export IPADDR=your.ip.address.here.
+
+Starting the Coaster Service Script
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Now that everything is configured, change to a temporary directory
+and run this command to start the coaster service:
+
+-----
+$ start-coaster-service
+-----
+
+This command will start the VMs, start the required processes on the worker nodes,
+and generate Swift configuration files for you to use. The configuration files
+will be generated in your current directory. These files are sites.xml, tc.data,
+and cf.
+
+Running Swift
+~~~~~~~~~~~~~
+Now that you have all of your configuration files generated, run the following command:
+
+-----
+$ swift -sites.file sites.xml -tc.file tc.data -config cf <yourscript.swift>
+-----
+
+If you would like to create a custom tc file for repeated use, rename it to something other
+than tc.data to prevent it from being overwritten. The sites.xml however will need to be
+regenerated every time you start the coaster service. If you need to repeatedly modify some
+sites.xml options, you may edit the template in Swift's etc/sites/persistent-coasters. You
+may also create your own custom tc files with the hostname of persistent-coasters. More
+information about this can be found in the Swift userguide at
+http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html.
+
+Stopping the Coaster Service Script
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+To stop the coaster service, run the following command:
+-----
+$ stop-coaster-service
+-----
+
+This will kill the coaster service, kill the worker scripts on remote systems and terminate
+the virtual machines that were created during start-coaster-service.
+
+More Help
+~~~~~~~~~
+The best place for additional help is the Swift user mailing list. You can subscribe to this list at
+http://mail.ci.uchicago.edu/mailman/listinfo/swift-user. When submitting information, please send
+your sites.xml file, your tc.data, and any error messages you run into.
+
Deleted: trunk/docs/siteguide/grid
===================================================================
--- trunk/docs/siteguide/grid 2011-09-06 19:12:14 UTC (rev 5078)
+++ trunk/docs/siteguide/grid 2011-09-08 00:53:06 UTC (rev 5086)
@@ -1,245 +0,0 @@
-Grids, including OSG and TeraGrid
----------------------------------
-
-Overview of running on grid sites
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-* Get a DOEGrids cert. Then register it in the *OSG Engage VO*, and/or
-map it using +gx-request+ on TeraGrid sites.
-
-* Run +GridSetup+ to configure Swift to use the grid sites. This tests
-for correct operation and creates a "green list" of good sites.
-
-* Prepare an installation package for the programs you want to run on
-grid sites via Swift, and install that package using +foreachsite+.
-
-* Run +swift-workers+ to start and maintain a pool of Swift workers on
-each site.
-
-* Run Swift scripts that use the grid site resources.
-
-NOTE: This revision only supports a single-entry sites file which uses
-provider staging and assumes that the necessary apps are locatable
-through the same tc entries (ie either absolute or PATH-relative
-paths) on all sites.
-
-NOTE: This revision has been testing using the bin/grid code from
-trunk (which gets installed into trunk's bin/ directory, and the base
-swift code from the 0.93 branch. No other configurations have been
-tested at the moment. I intend to put this code in bin/grid in 0.93,
-as it should have no ill affects on other Swift usage.
-
-Requesting Access
-~~~~~~~~~~~~~~~~~
-
-*For OSG*: Obtain a DOEGrids certificate and register the certificate in
-the OSG "Engage" VO following the procedures at:
-
-https://twiki.grid.iu.edu/bin/view/Engagement/EngageNewUserGuide
-
-[red]*FIXME*: access to OSG wiki pages may request the user to present
-a certificate. Is this a problem from users without one? If so, make a
-copy of the page on the Swift web.
-
-*For TeraGrid*: Obtain a DOEGrids certifcate using the OSG ENgage
-instructions above. Ask a TeraGrid PI to add you to a TeraGrid
-project. Once you obtain a login and project access (via US Mail), use
-gx-request to add your certificate
-
-To run jobs using the procedures documented here you need to be logged
-in to a "submit host" on which you will run Swift and other
-grid-related utilities. This can be any host with the OSG client stack
-or equivalent tools installed. Such hosts include the OSG Engage
-submit host, and the two Swift lab servers
-{bridled,communicado}.ci.uchicago.edu.
-
-Obtain a login on engage-submit.renci.org following instructions on
-the OSG URL above.
-
-Obtain a CI login with access to the Swift lab servers by requesting
-"OSG Gridlab" access at:
-
-http://accounts.ci.uchicago.edu
-
-
-Connecting to a submit host
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
------
-ssh yourusername at bridled.ci.uchicago.edu
-ssh yourusername at communicado.ci.uchicago.edu
-ssh yourusername at engage-submit.renci.org
------
-
-Downloading and install Swift
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The current version of Swift can be downloaded from
-http://www.ci.uchicago.edu/swift/downloads/index.php.
-
-Fetch and untar the latest release.
-Then add the Swift bin/ directory to your PATH. For example:
-
------
-cd $HOME
-wget http://www.ci.uchicago.edu/swift/packages/swift-0.92.1.tar.gz
-tar txf swift-0.92.1.tar.gz
-export PATH=$PATH:$HOME/swift-0.92.1/bin
------
-
-Set up OSG environment
-~~~~~~~~~~~~~~~~~~~~~~
-
-Depending on your shell type, run:
-
------
-source /opt/osg/setup.sh
-or
-source /opt/osg/setup.csh
------
-
-Create a VOMS Grid proxy
-~~~~~~~~~~~~~~~~~~~~~~~~
-
------
-$ voms-proxy-init -voms Engage -valid 12:00
------
-
-Generating Configuration Files
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
------
-cd $HOME
-mkdir swiftgrid
-cd swiftgrid
-gen_gridsites
-# Wait a few minutes to a few hours for Swift to validate grid sites
-get_greensites >greensites
------
-
-You can repeatedly try the get_greensites command, which simply
-concatenates all the site names that sucessfully resturned an output
-file from site tests.
-
-Installing software on OSG sites
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The command "foreachsite" will execute a local shell script passed to
-it as an argument, on each OSG site in the Engage VO ReSS site
-catalog. The user's installscript will execute on either the head node
-(as a GRAM "fork" job) or on a worker node, as controlled by the
--resource option. Its syntax is:
-
------
-$ ./foreachsite -help
-./foreachsite foreachsite [-resource fork|worker ] [-sites alt-sites-file] scriptname
-$
------
-
-To install your software, create a script similar to "myapp.sh",
-below, which (in this example) reads a tar file of a pre-compiled
-application "myapp-2.12" and executes a test script for that
-application. The test script should print some reconizable indication
-of its success or failure:
-
------
-$ cat myinstall.sh
-renice 2 -p $$
-IDIR=OSG_APP/extenci/myname/myapp
-mkdir -p $IDIR
-cd $IDIR
-wget http://my.url.edu/~mydir/myapp-2.12.tar.tgz
-tar zxf myapp-2.12.tar.tgz
-myapp-2.12/bin/test_myapp.sh
-if [ $? = 0 ]; then
- echo INSTALL SUCCEEDED
-else
- echo INSTALL FAILED
-fi
-$
-$ foreachsite -resource fork myinstall.sh
-$
-$ # Wait a file here, then poll for successfully installed apps...
-$
-$ grep SUCCEEDED run.89/*/*.stdout
-$
-
-
-Starting a single coaster service
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-This single coaster service will service all grid sites:
-
-start-grid-service --loglevel INFO --throttle 3.99 --jobspernode 1 \
- >& start-grid-service.out
-
-Starting workers on OSG sites
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Make sure that your "greensites" file is in the current working
-directory.
-
-The swiftDemand file should be set to contain the number of workers
-you want to start across all OSG sites. Eventually this will be set
-dynamically by watching your Swift script execute. (Note that this
-number only includes jobs started by the swift-workers factory
-command, not by any workers added manually from the TeraGrid - see
-below.
-
-The condor directory must be pre-created and will be used by Condor to
-return stdout and stderr files from the Condor jobs, which will
-execute the wrapper script "run-workers.sh".
-
-NOTE: this script is current built manually, and wraps around and
-transports the worker.pl script. This needs to be automated.
-
------
-echo 250 >swiftDemand mkdir -p condor
-
-swift-workers greensites extenci \
- http://communicado.ci.uchicago.edu:$(cat service-0.wport) \
- >& swift-workers.out &
------
-
-Adding workers from TeraGrid sites
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The job below can be used to submit jobs to TareGrid (Ranger only at
-the moment) to add more workers to the execution pool. The same
-requirements hold there as for OSG sites, namely, that the app tools
-listed in tc for the single execution site need to be locatable on the
-TeraGrid site(s).
-
------
-start-ranger-service --nodes 1 --walltime 00:10:00 --project TG-DBS123456N \
- --queue development --user tg12345 --startservice no \
- >& start-ranger-service.out
------
-
-NOTE: Change the project and user names to match your TeraGrid
-parameters.
-
-Running Swift
-~~~~~~~~~~~~~
-Now that everything is in place, run Swift with the following command:
-
------
-export SWIFT_HEAP_MAX=6000m # Add this for very large scripts
-
-swift -config cf.ps -tc.file tc -sites.file sites.grid-ps.xml \
- catsn.swift -n=10000 >& swift.out &
------
-
-You should see several new files being created, called catsn.0001.out,
-catsn.0002.out, etc. Each of these files should contain the contents
-of what you placed into data.txt. If this happens, your job has run
-successfully on the grid sites.
-
-More Help
-~~~~~~~~~
-
-The best place for additional help is the Swift user mailing list. You
-can subscribe to this list at
-http://mail.ci.uchicago.edu/mailman/listinfo/swift-user. When
-submitting information, please send your sites.xml file, your tc.data,
-and any Swift log files that were created during your attempt.
Copied: trunk/docs/siteguide/grid (from rev 5078, trunk/docs/siteguide/grid)
===================================================================
--- trunk/docs/siteguide/grid (rev 0)
+++ trunk/docs/siteguide/grid 2011-09-08 00:53:06 UTC (rev 5086)
@@ -0,0 +1,245 @@
+Grids, including OSG and TeraGrid
+---------------------------------
+
+Overview of running on grid sites
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Get a DOEGrids cert. Then register it in the *OSG Engage VO*, and/or
+map it using +gx-request+ on TeraGrid sites.
+
+* Run +GridSetup+ to configure Swift to use the grid sites. This tests
+for correct operation and creates a "green list" of good sites.
+
+* Prepare an installation package for the programs you want to run on
+grid sites via Swift, and install that package using +foreachsite+.
+
+* Run +swift-workers+ to start and maintain a pool of Swift workers on
+each site.
+
+* Run Swift scripts that use the grid site resources.
+
+NOTE: This revision only supports a single-entry sites file which uses
+provider staging and assumes that the necessary apps are locatable
+through the same tc entries (ie either absolute or PATH-relative
+paths) on all sites.
+
+NOTE: This revision has been testing using the bin/grid code from
+trunk (which gets installed into trunk's bin/ directory, and the base
+swift code from the 0.93 branch. No other configurations have been
+tested at the moment. I intend to put this code in bin/grid in 0.93,
+as it should have no ill affects on other Swift usage.
+
+Requesting Access
+~~~~~~~~~~~~~~~~~
+
+*For OSG*: Obtain a DOEGrids certificate and register the certificate in
+the OSG "Engage" VO following the procedures at:
+
+https://twiki.grid.iu.edu/bin/view/Engagement/EngageNewUserGuide
+
+[red]*FIXME*: access to OSG wiki pages may request the user to present
+a certificate. Is this a problem from users without one? If so, make a
+copy of the page on the Swift web.
+
+*For TeraGrid*: Obtain a DOEGrids certifcate using the OSG ENgage
+instructions above. Ask a TeraGrid PI to add you to a TeraGrid
+project. Once you obtain a login and project access (via US Mail), use
+gx-request to add your certificate
+
+To run jobs using the procedures documented here you need to be logged
+in to a "submit host" on which you will run Swift and other
+grid-related utilities. This can be any host with the OSG client stack
+or equivalent tools installed. Such hosts include the OSG Engage
+submit host, and the two Swift lab servers
+{bridled,communicado}.ci.uchicago.edu.
+
+Obtain a login on engage-submit.renci.org following instructions on
+the OSG URL above.
+
+Obtain a CI login with access to the Swift lab servers by requesting
+"OSG Gridlab" access at:
+
+http://accounts.ci.uchicago.edu
+
+
+Connecting to a submit host
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+-----
+ssh yourusername at bridled.ci.uchicago.edu
+ssh yourusername at communicado.ci.uchicago.edu
+ssh yourusername at engage-submit.renci.org
+-----
+
+Downloading and install Swift
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The current version of Swift can be downloaded from
+http://www.ci.uchicago.edu/swift/downloads/index.php.
+
+Fetch and untar the latest release.
+Then add the Swift bin/ directory to your PATH. For example:
+
+-----
+cd $HOME
+wget http://www.ci.uchicago.edu/swift/packages/swift-0.92.1.tar.gz
+tar txf swift-0.92.1.tar.gz
+export PATH=$PATH:$HOME/swift-0.92.1/bin
+-----
+
+Set up OSG environment
+~~~~~~~~~~~~~~~~~~~~~~
+
+Depending on your shell type, run:
+
+-----
+source /opt/osg/setup.sh
+or
+source /opt/osg/setup.csh
+-----
+
+Create a VOMS Grid proxy
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+-----
+$ voms-proxy-init -voms Engage -valid 12:00
+-----
+
+Generating Configuration Files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+-----
+cd $HOME
+mkdir swiftgrid
+cd swiftgrid
+gen_gridsites
+# Wait a few minutes to a few hours for Swift to validate grid sites
+get_greensites >greensites
+-----
+
+You can repeatedly try the get_greensites command, which simply
+concatenates all the site names that sucessfully resturned an output
+file from site tests.
+
+Installing software on OSG sites
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The command "foreachsite" will execute a local shell script passed to
+it as an argument, on each OSG site in the Engage VO ReSS site
+catalog. The user's installscript will execute on either the head node
+(as a GRAM "fork" job) or on a worker node, as controlled by the
+-resource option. Its syntax is:
+
+-----
+$ ./foreachsite -help
+./foreachsite foreachsite [-resource fork|worker ] [-sites alt-sites-file] scriptname
+$
+-----
+
+To install your software, create a script similar to "myapp.sh",
+below, which (in this example) reads a tar file of a pre-compiled
+application "myapp-2.12" and executes a test script for that
+application. The test script should print some reconizable indication
+of its success or failure:
+
+-----
+$ cat myinstall.sh
+renice 2 -p $$
+IDIR=OSG_APP/extenci/myname/myapp
+mkdir -p $IDIR
+cd $IDIR
+wget http://my.url.edu/~mydir/myapp-2.12.tar.tgz
+tar zxf myapp-2.12.tar.tgz
+myapp-2.12/bin/test_myapp.sh
+if [ $? = 0 ]; then
+ echo INSTALL SUCCEEDED
+else
+ echo INSTALL FAILED
+fi
+$
+$ foreachsite -resource fork myinstall.sh
+$
+$ # Wait a file here, then poll for successfully installed apps...
+$
+$ grep SUCCEEDED run.89/*/*.stdout
+$
+
+
+Starting a single coaster service
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This single coaster service will service all grid sites:
+
+start-grid-service --loglevel INFO --throttle 3.99 --jobspernode 1 \
+ >& start-grid-service.out
+
+Starting workers on OSG sites
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Make sure that your "greensites" file is in the current working
+directory.
+
+The swiftDemand file should be set to contain the number of workers
+you want to start across all OSG sites. Eventually this will be set
+dynamically by watching your Swift script execute. (Note that this
+number only includes jobs started by the swift-workers factory
+command, not by any workers added manually from the TeraGrid - see
+below.
+
+The condor directory must be pre-created and will be used by Condor to
+return stdout and stderr files from the Condor jobs, which will
+execute the wrapper script "run-workers.sh".
+
+NOTE: this script is current built manually, and wraps around and
+transports the worker.pl script. This needs to be automated.
+
+-----
+echo 250 >swiftDemand mkdir -p condor
+
+swift-workers greensites extenci \
+ http://communicado.ci.uchicago.edu:$(cat service-0.wport) \
+ >& swift-workers.out &
+-----
+
+Adding workers from TeraGrid sites
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The job below can be used to submit jobs to TareGrid (Ranger only at
+the moment) to add more workers to the execution pool. The same
+requirements hold there as for OSG sites, namely, that the app tools
+listed in tc for the single execution site need to be locatable on the
+TeraGrid site(s).
+
+-----
+start-ranger-service --nodes 1 --walltime 00:10:00 --project TG-DBS123456N \
+ --queue development --user tg12345 --startservice no \
+ >& start-ranger-service.out
+-----
+
+NOTE: Change the project and user names to match your TeraGrid
+parameters.
+
+Running Swift
+~~~~~~~~~~~~~
+Now that everything is in place, run Swift with the following command:
+
+-----
+export SWIFT_HEAP_MAX=6000m # Add this for very large scripts
+
+swift -config cf.ps -tc.file tc -sites.file sites.grid-ps.xml \
+ catsn.swift -n=10000 >& swift.out &
+-----
+
+You should see several new files being created, called catsn.0001.out,
+catsn.0002.out, etc. Each of these files should contain the contents
+of what you placed into data.txt. If this happens, your job has run
+successfully on the grid sites.
+
+More Help
+~~~~~~~~~
+
+The best place for additional help is the Swift user mailing list. You
+can subscribe to this list at
+http://mail.ci.uchicago.edu/mailman/listinfo/swift-user. When
+submitting information, please send your sites.xml file, your tc.data,
+and any Swift log files that were created during your attempt.
Deleted: trunk/docs/siteguide/images
===================================================================
--- trunk/docs/siteguide/images 2011-09-06 19:12:14 UTC (rev 5078)
+++ trunk/docs/siteguide/images 2011-09-08 00:53:06 UTC (rev 5086)
@@ -1 +0,0 @@
-link ../tutorial/images
\ No newline at end of file
Copied: trunk/docs/siteguide/images (from rev 5078, trunk/docs/siteguide/images)
===================================================================
--- trunk/docs/siteguide/images (rev 0)
+++ trunk/docs/siteguide/images 2011-09-08 00:53:06 UTC (rev 5086)
@@ -0,0 +1 @@
+link ../tutorial/images
\ No newline at end of file
Deleted: trunk/docs/siteguide/pads
===================================================================
--- trunk/docs/siteguide/pads 2011-09-06 19:12:14 UTC (rev 5078)
+++ trunk/docs/siteguide/pads 2011-09-08 00:53:06 UTC (rev 5086)
@@ -1,188 +0,0 @@
-PADS
-----
-
-Requesting Access
-~~~~~~~~~~~~~~~~~
-If you do not already have a Computation Institute account, you can request
-one at https://www.ci.uchicago.edu/accounts/. This page will give you a list
-of resources you can request access to. Be sure that PADS is selected. If
-you already have an existing CI account, but do not have access to PADS,
-send an email to support at ci.uchicago.edu to request access.
-
-SSH Keys
-~~~~~~~~
-Before accessing PADS, be sure to have your SSH keys configured correctly.
-There is some basic information about SSH and how to generate your key at
-http://www.ci.uchicago.edu/wiki/bin/view/Resources/SshKeys. Once you have
-followed those instructions, you can add your key at
-https://www.ci.uchicago.edu/support/sshkeys/.
-
-Connecting to a login node
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-Once your keys are configured, you should be able to access a PADS login
-node with the following command:
-
------
-ssh yourusername at login.pads.ci.uchicago.edu
------
-
-Configuring softenv
-~~~~~~~~~~~~~~~~~~~
-Softenv is a system used for managing applications. In order to run Swift,
-the softenv environment will have to be modified slightly. Softenv is
-configured by a file in your home directory called .soft. Edit this file
-to look like this:
------
-+java-sun
-+maui
-+torque
- at default
------
-
-Downloading and building Swift
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The most recent versions of Swift can be found at
-http://www.ci.uchicago.edu/swift/downloads/index.php. Follow the instructions
-provided on that site to download and build Swift.
-
-Adding Swift to your PATH
-~~~~~~~~~~~~~~~~~~~~~~~~~
-Once you have installed Swift, add the Swift binary to your PATH so you can
-easily run it from any directory.
-
-In your home directory, edit the file ".bashrc".
-
-If you have installed Swift via a source repository, add the following line
-at the bottom of .bashrc.
-
------
-export PATH=$PATH:$HOME/cog/modules/swift/dist/swift-svn/bin
------
-
-If you have installed Swift via a binary package, add this line:
-
------
-export PATH=$PATH:$HOME/swift-<version>/bin
------
-
-Replace <version> with the actual name of the swift directory in the example above.
-
-What You Need To Know Before Running Swift
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Before you can create a Swift configuration file, there are some things
-you will need to know.
-
-Swift Work Directory
-^^^^^^^^^^^^^^^^^^^^
-The Swift work directory is a directory which Swift uses for processing work.
-This directory needs to be writable. Common options for this are:
-
------
-/home/username/swiftwork
-/home/username/work
-/tmp
------
-
-Which project(s) are you a member of?
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-PADS requires that you are a member of a project. You can determine this by
-running the following command:
-
------
-$ projects --available
-
-The following projects are available for your use
-
-Project PI Title
-
-CI-CCR000013 Michael Wilde The Swift Parallel Scripting System
------
-
-If you are not a member of a project, you must first request access
-to a project at http://www.ci.uchicago.edu/hpc/projects.
-
-You should make sure that you have a project set as default. Run
-the projects command with no arguments to determine if you have a default.
-
-------
-$ projects
-You have no default project set.
------
-
-To set your default project, use projects --set
-------
-$ projects --set CI-CCR000013 --all
-Your default project for all CI clusters has been set to CI-CCR000013.
------
-
-Determine your Queue
-^^^^^^^^^^^^^^^^^^^^
-
-PADS has several different queues you can submit jobs to depending on
-the type of work you will be doing. The command "qstat -q" will print
-the most up to date list of this information.
-
-.PADS Queues
-[options="header"]
-|=========================================================
-|Queue |Memory|CPU Time|Walltime|Node|Run|Que|Lm |State
-|route |-- |-- |-- |-- | 0| 0|-- | E R
-|short |-- |-- |04:00:00|-- | 64| 0|-- | E R
-|extended|-- |-- |-- |-- | 0| 0|-- | E R
-|fast |-- |-- |01:00:00|1 | 0|152|-- | E R
-|long |-- |-- |24:00:00|-- |232|130|-- | E R
-|=========================================================
-
-Generating Configuration Files
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Now that you know what queue to use, your project, and your work directory, it is time to
-set up Swift. Swift uses a configuration file called sites.xml to determine how it should run.
-There are two methods you can use for creating this file. You can manually edit
-the configuration file, or generate it with a utility called gensites.
-
-Manually Editing sites.xml
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-Below is the template that is used by Swift's test suite for running on PADS.
-
------
-include::../../tests/providers/local-pbs/pads/sites.template.xml[]
------
-
-The values to note here are the ones that are listed between underscores. In the example above, they are _QUEUE_, and _WORK_. Queue is the PADS queue to use and WORK is the swift work directory. These are placeholder values you will need to modify to fit your needs. Copy and paste this template, replace the values, and call it sites.xml.
-
-Manually Editing tc.data
-~~~~~~~~~~~~~~~~~~~~~~~~
-Below is the tc.data file used by Swift's test suite for running on PADS.
-
------
-include::../../tests/providers/local-pbs/pads/tc.template.data[]
-------
-
-Copy these commands and save it as tc.data.
-
-Catsn.swift
-~~~~~~~~~~~
-The swift script we will run is called catsn.swift. It simply cats a file and saves the result. This is a nice simple test to ensure jobs are running correctly. Create a file called data.txt which contains some simple input - a "hello world" will do the trick.
-
------
-include::../../examples/misc/catsn.swift[]
------
-
-Running Swift
-~~~~~~~~~~~~~
-Now that everything is in place, run Swift with the following command:
-
------
-swift -sites.file sites.xml -tc.file tc.data catsn.swift -n=10
------
-
-You should see several new files being created, called catsn.0001.out, catsn.0002.out, etc. Each of these
-files should contain the contents of what you placed into data.txt. If this happens, your job has run
-successfully on PADS!
-
-More Help
-~~~~~~~~~
-The best place for additional help is the Swift user mailing list. You can subscribe to this list at
-https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user. When submitting information, please send your sites.xml file, your tc.data, and any Swift log files that were created during your attempt.
-
-
Copied: trunk/docs/siteguide/pads (from rev 5078, trunk/docs/siteguide/pads)
===================================================================
--- trunk/docs/siteguide/pads (rev 0)
+++ trunk/docs/siteguide/pads 2011-09-08 00:53:06 UTC (rev 5086)
@@ -0,0 +1,188 @@
+PADS
+----
+
+Requesting Access
+~~~~~~~~~~~~~~~~~
+If you do not already have a Computation Institute account, you can request
+one at https://www.ci.uchicago.edu/accounts/. This page will give you a list
+of resources you can request access to. Be sure that PADS is selected. If
+you already have an existing CI account, but do not have access to PADS,
+send an email to support at ci.uchicago.edu to request access.
+
+SSH Keys
+~~~~~~~~
+Before accessing PADS, be sure to have your SSH keys configured correctly.
+There is some basic information about SSH and how to generate your key at
+http://www.ci.uchicago.edu/wiki/bin/view/Resources/SshKeys. Once you have
+followed those instructions, you can add your key at
+https://www.ci.uchicago.edu/support/sshkeys/.
+
+Connecting to a login node
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+Once your keys are configured, you should be able to access a PADS login
+node with the following command:
+
+-----
+ssh yourusername at login.pads.ci.uchicago.edu
+-----
+
+Configuring softenv
+~~~~~~~~~~~~~~~~~~~
+Softenv is a system used for managing applications. In order to run Swift,
+the softenv environment will have to be modified slightly. Softenv is
+configured by a file in your home directory called .soft. Edit this file
+to look like this:
+-----
++java-sun
++maui
++torque
+ at default
+-----
+
+Downloading and building Swift
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The most recent versions of Swift can be found at
+http://www.ci.uchicago.edu/swift/downloads/index.php. Follow the instructions
+provided on that site to download and build Swift.
+
+Adding Swift to your PATH
+~~~~~~~~~~~~~~~~~~~~~~~~~
+Once you have installed Swift, add the Swift binary to your PATH so you can
+easily run it from any directory.
+
+In your home directory, edit the file ".bashrc".
+
+If you have installed Swift via a source repository, add the following line
+at the bottom of .bashrc.
+
+-----
+export PATH=$PATH:$HOME/cog/modules/swift/dist/swift-svn/bin
+-----
+
+If you have installed Swift via a binary package, add this line:
+
+-----
+export PATH=$PATH:$HOME/swift-<version>/bin
+-----
+
+Replace <version> with the actual name of the swift directory in the example above.
+
+What You Need To Know Before Running Swift
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Before you can create a Swift configuration file, there are some things
+you will need to know.
+
+Swift Work Directory
+^^^^^^^^^^^^^^^^^^^^
+The Swift work directory is a directory which Swift uses for processing work.
+This directory needs to be writable. Common options for this are:
+
+-----
+/home/username/swiftwork
+/home/username/work
+/tmp
+-----
+
+Which project(s) are you a member of?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+PADS requires that you are a member of a project. You can determine this by
+running the following command:
+
+-----
+$ projects --available
+
+The following projects are available for your use
+
+Project PI Title
+
+CI-CCR000013 Michael Wilde The Swift Parallel Scripting System
+-----
+
+If you are not a member of a project, you must first request access
+to a project at http://www.ci.uchicago.edu/hpc/projects.
+
+You should make sure that you have a project set as default. Run
+the projects command with no arguments to determine if you have a default.
+
+------
+$ projects
+You have no default project set.
+-----
+
+To set your default project, use projects --set
+------
+$ projects --set CI-CCR000013 --all
+Your default project for all CI clusters has been set to CI-CCR000013.
+-----
+
+Determine your Queue
+^^^^^^^^^^^^^^^^^^^^
+
+PADS has several different queues you can submit jobs to depending on
+the type of work you will be doing. The command "qstat -q" will print
+the most up to date list of this information.
+
+.PADS Queues
+[options="header"]
+|=========================================================
+|Queue |Memory|CPU Time|Walltime|Node|Run|Que|Lm |State
+|route |-- |-- |-- |-- | 0| 0|-- | E R
+|short |-- |-- |04:00:00|-- | 64| 0|-- | E R
+|extended|-- |-- |-- |-- | 0| 0|-- | E R
+|fast |-- |-- |01:00:00|1 | 0|152|-- | E R
+|long |-- |-- |24:00:00|-- |232|130|-- | E R
+|=========================================================
+
+Generating Configuration Files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Now that you know what queue to use, your project, and your work directory, it is time to
+set up Swift. Swift uses a configuration file called sites.xml to determine how it should run.
+There are two methods you can use for creating this file. You can manually edit
+the configuration file, or generate it with a utility called gensites.
+
+Manually Editing sites.xml
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+Below is the template that is used by Swift's test suite for running on PADS.
+
+-----
+include::../../tests/providers/local-pbs/pads/sites.template.xml[]
+-----
+
+The values to note here are the ones that are listed between underscores. In the example above, they are _QUEUE_, and _WORK_. Queue is the PADS queue to use and WORK is the swift work directory. These are placeholder values you will need to modify to fit your needs. Copy and paste this template, replace the values, and call it sites.xml.
+
+Manually Editing tc.data
+~~~~~~~~~~~~~~~~~~~~~~~~
+Below is the tc.data file used by Swift's test suite for running on PADS.
+
+-----
+include::../../tests/providers/local-pbs/pads/tc.template.data[]
+------
+
+Copy these commands and save it as tc.data.
+
+Catsn.swift
+~~~~~~~~~~~
+The swift script we will run is called catsn.swift. It simply cats a file and saves the result. This is a nice simple test to ensure jobs are running correctly. Create a file called data.txt which contains some simple input - a "hello world" will do the trick.
+
+-----
+include::../../examples/misc/catsn.swift[]
+-----
+
+Running Swift
+~~~~~~~~~~~~~
+Now that everything is in place, run Swift with the following command:
+
+-----
+swift -sites.file sites.xml -tc.file tc.data catsn.swift -n=10
+-----
+
+You should see several new files being created, called catsn.0001.out, catsn.0002.out, etc. Each of these
+files should contain the contents of what you placed into data.txt. If this happens, your job has run
+successfully on PADS!
+
+More Help
+~~~~~~~~~
+The best place for additional help is the Swift user mailing list. You can subscribe to this list at
+https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user. When submitting information, please send your sites.xml file, your tc.data, and any Swift log files that were created during your attempt.
+
+
Deleted: trunk/docs/siteguide/siteguide.txt
===================================================================
--- trunk/docs/siteguide/siteguide.txt 2011-09-06 19:12:14 UTC (rev 5078)
+++ trunk/docs/siteguide/siteguide.txt 2011-09-08 00:53:06 UTC (rev 5086)
@@ -1,15 +0,0 @@
-Site Configuration Guide
-========================
-
-:toc:
-:icons:
-:website: http://www.ci.uchicago.edu/swift/guides/siteguide.php
-:numbered:
-
-include::pads[]
-
-include::beagle[]
-
-include::futuregrid[]
-
-include::grid[]
Copied: trunk/docs/siteguide/siteguide.txt (from rev 5078, trunk/docs/siteguide/siteguide.txt)
===================================================================
--- trunk/docs/siteguide/siteguide.txt (rev 0)
+++ trunk/docs/siteguide/siteguide.txt 2011-09-08 00:53:06 UTC (rev 5086)
@@ -0,0 +1,15 @@
+Site Configuration Guide
+========================
+
+:toc:
+:icons:
+:website: http://www.ci.uchicago.edu/swift/guides/siteguide.php
+:numbered:
+
+include::pads[]
+
+include::beagle[]
+
+include::futuregrid[]
+
+include::grid[]
Deleted: trunk/docs/stylesheets/asciidoc.css
===================================================================
--- trunk/docs/stylesheets/asciidoc.css 2011-09-06 19:12:14 UTC (rev 5078)
+++ trunk/docs/stylesheets/asciidoc.css 2011-09-08 00:53:06 UTC (rev 5086)
@@ -1,2 +0,0 @@
-a:link { color:navy; }
-a:visited { color:navy; }
Copied: trunk/docs/stylesheets/asciidoc.css (from rev 5078, trunk/docs/stylesheets/asciidoc.css)
===================================================================
--- trunk/docs/stylesheets/asciidoc.css (rev 0)
+++ trunk/docs/stylesheets/asciidoc.css 2011-09-08 00:53:06 UTC (rev 5086)
@@ -0,0 +1,2 @@
+a:link { color:navy; }
+a:visited { color:navy; }
Property changes on: trunk/docs/tutorial
___________________________________________________________________
Added: svn:ignore
+ tutorial.html
tutorial.pdf
Deleted: trunk/docs/tutorial/abstract
===================================================================
--- trunk/docs/tutorial/abstract 2011-09-07 21:09:21 UTC (rev 5085)
+++ trunk/docs/tutorial/abstract 2011-09-08 00:53:06 UTC (rev 5086)
@@ -1,7 +0,0 @@
-[abstract]
-Abstract
---------
-This is an introductory tutorial on the use of Swift and its programming
-language SwiftScript.
-
-note: merge with intro?
Copied: trunk/docs/tutorial/abstract (from rev 5078, trunk/docs/tutorial/abstract)
===================================================================
--- trunk/docs/tutorial/abstract (rev 0)
+++ trunk/docs/tutorial/abstract 2011-09-08 00:53:06 UTC (rev 5086)
@@ -0,0 +1,7 @@
+[abstract]
+Abstract
+--------
+This is an introductory tutorial on the use of Swift and its programming
+language SwiftScript.
+
+note: merge with intro?
Deleted: trunk/docs/tutorial/bits
===================================================================
--- trunk/docs/tutorial/bits 2011-09-07 21:09:21 UTC (rev 5085)
+++ trunk/docs/tutorial/bits 2011-09-08 00:53:06 UTC (rev 5086)
@@ -1,15 +0,0 @@
-Bits
-----
-
-Named and optional parameters
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-In addition to specifying parameters positionally, parameters can be
-named, and if desired a default value can be specified:
-
-.default.swift
-**************
-----
-include::../../examples/tutorial/default.swift[]
-----
-**************
Copied: trunk/docs/tutorial/bits (from rev 5078, trunk/docs/tutorial/bits)
===================================================================
--- trunk/docs/tutorial/bits (rev 0)
+++ trunk/docs/tutorial/bits 2011-09-08 00:53:06 UTC (rev 5086)
@@ -0,0 +1,15 @@
+Bits
+----
+
+Named and optional parameters
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In addition to specifying parameters positionally, parameters can be
+named, and if desired a default value can be specified:
+
+.default.swift
+**************
+----
+include::../../examples/tutorial/default.swift[]
+----
+**************
Deleted: trunk/docs/tutorial/hello_world
===================================================================
--- trunk/docs/tutorial/hello_world 2011-09-07 21:09:21 UTC (rev 5085)
+++ trunk/docs/tutorial/hello_world 2011-09-08 00:53:06 UTC (rev 5086)
@@ -1,69 +0,0 @@
-Hello World
------------
-The first example program, hello.swift, outputs a hello world message
-into a file called hello.txt.
-
-.hello.swift
-************
-
-----
-include::../../examples/tutorial/hello.swift[]
-----
-************
-
-To run hello.swift, change directories to the location of the script
-and run the swift command as follows.
-
-TIP: Make sure the bin directory of your swift installation is in your PATH.
-----
-$ cd examples/tutorial
-$ swift hello.swift
-Swift svn swift-r3334 (swift modified locally) cog-r2752
-
-RunID: 20100526-1925-8zjupq1b
-Progress:
-Final status: Finished successfully:1
-$ cat hello.txt
-Hello, world!
-----
-
-The basic structure of this program is a type definition, an
-application procedure definition, a variable definition and then a
-call to the procedure.
-
-First we define a new type, called messagefile. In this example, we will
-use this messagefile type for our output message.
-
-----
-sys::[sed -n '1p' ../../examples/tutorial/hello.swift]
-----
-
-All data in SwiftScript must be typed, whether it is stored in memory or
-on disk. This example defines a very simple type. Later on we will see
-more complex type examples.
-
-----
-sys::[sed -n '3,5p' ../../examples/tutorial/hello.swift]
-----
-
-Next we define a procedure called greeting. This procedure will write
-out the "hello world" message to a file. To achieve this, it executes
-the unix utility 'echo' with a parameter "Hello, world!" and directs
-the standard output into the output file.
-
-The actual file to use is specified by the return parameter, t.
------
-sys::[sed -n '7p' ../../examples/tutorial/hello.swift]
------
-Here we define a variable called outfile. The type of this variable is
-messagefile, and we specify that the contents of this variable will be
-stored on disk in a file called hello.txt
-
------
-sys::[sed -n '3,5p' ../../examples/tutorial/hello.swift]
------
-Now we call the greeting procedure, with its output going to the outfile
-variable and therefore to hello.txt on disk.
-
-Over the following exercises, we'll extend this simple hello world
-program to demonstrate various features of Swift.
Copied: trunk/docs/tutorial/hello_world (from rev 5078, trunk/docs/tutorial/hello_world)
===================================================================
--- trunk/docs/tutorial/hello_world (rev 0)
+++ trunk/docs/tutorial/hello_world 2011-09-08 00:53:06 UTC (rev 5086)
@@ -0,0 +1,69 @@
+Hello World
+-----------
+The first example program, hello.swift, outputs a hello world message
+into a file called hello.txt.
+
+.hello.swift
+************
+
+----
+include::../../examples/tutorial/hello.swift[]
+----
+************
+
+To run hello.swift, change directories to the location of the script
+and run the swift command as follows.
+
+TIP: Make sure the bin directory of your swift installation is in your PATH.
+----
+$ cd examples/tutorial
+$ swift hello.swift
+Swift svn swift-r3334 (swift modified locally) cog-r2752
+
+RunID: 20100526-1925-8zjupq1b
+Progress:
+Final status: Finished successfully:1
+$ cat hello.txt
+Hello, world!
+----
+
+The basic structure of this program is a type definition, an
+application procedure definition, a variable definition and then a
+call to the procedure.
+
+First we define a new type, called messagefile. In this example, we will
+use this messagefile type for our output message.
+
+----
+sys::[sed -n '1p' ../../examples/tutorial/hello.swift]
+----
+
+All data in SwiftScript must be typed, whether it is stored in memory or
+on disk. This example defines a very simple type. Later on we will see
+more complex type examples.
+
+----
+sys::[sed -n '3,5p' ../../examples/tutorial/hello.swift]
+----
+
+Next we define a procedure called greeting. This procedure will write
+out the "hello world" message to a file. To achieve this, it executes
+the unix utility 'echo' with a parameter "Hello, world!" and directs
+the standard output into the output file.
+
+The actual file to use is specified by the return parameter, t.
+-----
+sys::[sed -n '7p' ../../examples/tutorial/hello.swift]
+-----
+Here we define a variable called outfile. The type of this variable is
+messagefile, and we specify that the contents of this variable will be
+stored on disk in a file called hello.txt
+
+-----
+sys::[sed -n '3,5p' ../../examples/tutorial/hello.swift]
+-----
+Now we call the greeting procedure, with its output going to the outfile
+variable and therefore to hello.txt on disk.
+
+Over the following exercises, we'll extend this simple hello world
+program to demonstrate various features of Swift.
Deleted: trunk/docs/tutorial/introduction
===================================================================
--- trunk/docs/tutorial/introduction 2011-09-07 21:09:21 UTC (rev 5085)
+++ trunk/docs/tutorial/introduction 2011-09-08 00:53:06 UTC (rev 5086)
@@ -1,16 +0,0 @@
-Introduction
-------------
-This is an introductory tutorial describing the use of Swift and its programming
-language SwiftScript. It is intended to introduce new users to the basics of Swift.
-It is structured as a series of simple exercises/examples which you can
-try for yourself as you read along.
-
-For information on getting an installation of Swift running, consult the
-http://www.ci.uchicago.edu/swift/guides/quickstartguide.php[Swift Quickstart Guide]. We advise you to install the latest stable release of Swift.
-Return to this document when you have successfully run the test SwiftScript program mentioned there.
-
-There is also a
-sys::[../print_link.sh userguide]
-which contains a more detailed reference material on topics covered in this manual. All
-of the programs included in this tutorial can be found in your Swift
-distribution in the examples/tutorial directory.
Copied: trunk/docs/tutorial/introduction (from rev 5078, trunk/docs/tutorial/introduction)
===================================================================
--- trunk/docs/tutorial/introduction (rev 0)
+++ trunk/docs/tutorial/introduction 2011-09-08 00:53:06 UTC (rev 5086)
@@ -0,0 +1,16 @@
+Introduction
+------------
+This is an introductory tutorial describing the use of Swift and its programming
+language SwiftScript. It is intended to introduce new users to the basics of Swift.
+It is structured as a series of simple exercises/examples which you can
+try for yourself as you read along.
+
+For information on getting an installation of Swift running, consult the
+http://www.ci.uchicago.edu/swift/guides/quickstartguide.php[Swift Quickstart Guide]. We advise you to install the latest stable release of Swift.
+Return to this document when you have successfully run the test SwiftScript program mentioned there.
+
+There is also a
+sys::[../print_link.sh userguide]
+which contains a more detailed reference material on topics covered in this manual. All
+of the programs included in this tutorial can be found in your Swift
+distribution in the examples/tutorial directory.
Deleted: trunk/docs/tutorial/language_features
===================================================================
--- trunk/docs/tutorial/language_features 2011-09-07 21:09:21 UTC (rev 5085)
+++ trunk/docs/tutorial/language_features 2011-09-08 00:53:06 UTC (rev 5086)
@@ -1,390 +0,0 @@
-Language features
------------------
-
-Parameters
-~~~~~~~~~~
-
-Procedures can have parameters. Input parameters specify inputs to the
-procedure and output parameters specify outputs. Our hello world greeting
-procedure already uses an output parameter, t, which indicates where the
-greeting output will go. In this section, we will modify the previous
-script to add an input parameter to the greeting function.
-
-.parameter.swift
-****************
-----
-include::../../examples/tutorial/parameter.swift[]
-----
-****************
-
-We have modified the signature of the greeting procedure to indicate
-that it takes a single parameter, s, of type 'string'.
-
-We have modified the invocation of the 'echo' utility so that it takes
-the value of s as a parameter, instead of the string literal "Hello,
-world!".
-
-We have modified the output file definition to point to a different file
-on disk.
-
-We have modified the invocation of greeting so that a greeting string is
-supplied.
-
-The code for this section can be found in parameter.swift. It can be
-invoked using the swift command, with output appearing in parameter.hello.txt:
-
-
-----
-$ swift parameter.swift
-----
-
-Now that we can choose our greeting text, we can call the same procedure
-with different parameters to generate several output files with
-different greetings. The code is in manyparam.swift and can be run as
-before using the swift command.
-
-.manyparam.swift
-****************
-----
-include::../../examples/tutorial/manyparam.swift[]
-----
-****************
-
-Note that we can intermingle definitions of variables with invocations
-of procedures.
-
-When this program runs, there should be three new files in the
-working directory (manyparam.english.txt, manyparam.francais.txt
-and manyparam.nihongo.txt) each containing a greeting in a different
-language.
-
-In addition to specifying parameters positionally, parameters can be
-named, and if desired a default value can be specified.
-
-Adding another application
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-Now we'll define a new application procedure. The procedure we define
-will capitalise all the words in the input file.
-
-To do this, we'll use the unix tr (translate) utility. Here is an
-example of using tr on the unix command line, not using Swift:
-
-----
-$ echo hello | tr '[a-z]' '[A-Z]'
-HELLO
-----
-
-There are two main steps - updating the transformation catalog, and
-updating the application block.
-
-The transformation catalog lists where application executables are located on
-remote sites. We need to modify the transformation catalog to define a logical
-transformation for the tr utility. The transformation catalog can be found
-in etc/tc.data. There are already several entries specifying
-where executables can be found. Add a new line to the file, specifying
-where tr can be found (usually in /usr/bin/tr but it may differ on
-your system), like this:
-
-----
-localhost tr /usr/bin/tr INSTALLED INTEL32::LINUX null
-----
-
-For now, ignore all of the fields except the second and the third. The
-second field tr specifies a logical application name and the third
-specifies the location of the application executable.
-
-Now that we have defined where to find tr, we can use it in SwiftScript.
-
-We can define a new procedure, capitalise, which calls tr.
-
-----
-sys::[sed -n '7,9p' ../../examples/tutorial/capitalise.swift]
-----
-We can call capitalise like this:
-
-----
-sys::[sed -n '12,14p' ../../examples/tutorial/capitalise.swift]
-----
-
-Here is the full program based on this exercise:
-
-.capitalise.swift
-***********************
-----
-include::../../examples/tutorial/capitalise.swift[]
-----
-***********************
-
-Next, run swift and verify the output is correct.
-
-----
-$ swift capitalise.swift
-...
-$ cat capitalise.2.txt
-HELLO FROM SWIFT
-----
-
-Anonymous files
-~~~~~~~~~~~~~~~
-In the previous section, the file hello.txt is used only to store
-an intermediate result. We don't really care about which name is used
-for the file, and we can let Swift choose the name.
-
-To do that, omit the mapping entirely when declaring hellofile:
-
-----
-sys::[sed -n '11p' ../../examples/tutorial/capitalise_anonymous.swift]
-----
-
-Swift will choose a filename, which in the present version will be in a
-subdirectory called _concurrent.
-
-
-Datatypes
-~~~~~~~~~
-All data in variables and files has a data type. So far, we've seen two
-types:
-
- - string - this is a built-in type for storing strings of text in
- memory, much like in other programming languages
- - messagefile - this is a user-defined type used to mark disc resident files as
- containing messages
-
-SwiftScript has the additional built-in types: boolean, integer and
-float that function much like their counterparts in other programming
-languages.
-
-It is also possible to create user defined types with more structure,
-for example:
-
-
-----
-sys::[sed -n '3,6p' ../../examples/tutorial/types.swift]
-----
-
-Each element of the structured type can be accessed using a . like this:
-
-----
-sys::[sed -n '14p' ../../examples/tutorial/types.swift]
-----
-
-The following complete program, types.swift, outputs a greeting using a
-user-defined structure type to hold parameters for the message:
-
-.types.swift
-************
-----
-include::../../examples/tutorial/types.swift[]
-----
-************
-
-Structured types can be comprised of marker types for files. See the
-later section on mappers for more information about this.
-
-
-Arrays
-~~~~~~
-
-We can define arrays using the [] suffix in a variable declaration:
-
-----
-sys::[sed -n '9p' ../../examples/tutorial/arrays.swift]
-----
-
-This program, arrays.swift, will declare an array of message files.
-
-.arrays.swift
-*************
-----
-include::../../examples/tutorial/arrays.swift[]
-----
-*************
-
-Observe that the type of the parameter to greeting is now an array of
-strings, string s[], instead of a single string, string s, that
-elements of the array can be referenced numerically, for example s[0],
-and that the array is initialised using an array literal,
-["how","are","you"].
-
-
-Mappers
-~~~~~~~
-
-A significant difference between SwiftScript and other languages is that
-data can be referred to on disk through variables in a very similar
-fashion to data in memory. For example, in the above examples we have
-seen a variable definition like this:
-
-----
-sys::[sed -n '7p' ../../examples/tutorial/arrays.swift]
-----
-
-This means that outfile is a dataset variable, which is mapped to a
-file on disk called arrays.txt. This variable can be assigned to
-using = in a similar fashion to an in-memory variable. We can say that
-outfile is mapped onto the disk file arrays.txt by a mapper.
-
-There are various ways of mapping in SwiftScript. Two forms of mapping,
-simple named mapping and anonymous mapping, have already
-been seen in this tutorial. Later exercises will introduce more forms.
-
-In simple named mapping, the name of the file that a variable is mapped to
-is explictly listed.
-
-----
-sys::[sed -n '7p' ../../examples/tutorial/hello.swift]
-----
-
-This is useful when you want to explicitly name input and output files
-for your program. An example of this can be seen with 'outfile' in the
-hello world exercise.
-
-With anonymous mapping no name is specified in the source code. A name is
-automatically generated for the file. This is useful for intermediate
-files that are only referenced through SwiftScript.
-A variable declaration is mapped anonymously by ommitting any mapper
-definition.
-
-----
-sys::[sed -n '11p' ../../examples/tutorial/capitalise_anonymous.swift]
-----
-
-Later exercises will introduce other ways of mapping from disk files to
-SwiftScript variables.
-
-
-The Regexp Mapper
-^^^^^^^^^^^^^^^^^
-In this exercise, we introduce the regexp mapper. This mapper
-transforms a string expression using a regular expression, and uses the
-result of that transformation as the filename to map.
-
-regexp.swift demonstrates the use of this by placing output into a
-file that is based on the name of the input file. Our input file is
-mapped to the inputfile variable using the simple named mapper, then
-we use the regular expression mapper to map the output file. We then use
-the countwords() procedure to count the words in the input file and
-store the result in the output file. In order for the countwords()
-procedure to work correctly, add the wc utility (usually found in
-/usr/bin/wc) to tc.data.
-
-The following program replaces the suffix of the input file (regexp_mapper.words.txt)
-with a new suffix (.count) to create regexp_mapper.words.count.
-
-.regexp_mapper.swift
-*************
-----
-include::../../examples/tutorial/regexp_mapper.swift[]
-----
-**************
-
-fixed_array_mapper
-^^^^^^^^^^^^^^^^^^
-The fixed array mapper maps a list of files into an array. Each
-element of the array is mapped into one file in the specified directory.
-See fixed_array_mapper.swift below.
-
-.fixed_array_mapper.swift
-*************************
-----
-include::../../examples/tutorial/fixed_array_mapper.swift[]
-----
-*************************
-
-foreach
-~~~~~~~
-SwiftScript provides a control structure, foreach, to operate on each
-element of an array in parallel.
-
-In this example, we will run the previous word counting example over
-each file in an array without having to explicitly list the array
-elements. The source code for this example is in foreach.swift. This
-program uses three input files: foreach.1.txt, foreach.2.txt, and
-foreach.3.txt. After you have run the workflow, you should see that there are three
-output files: foreach.1.count, foreach.2.count and foreach.3.count, each
-containing the word count for the corresponding input file. We combine
-the use of the fixed_array_mapper and the regexp_mapper.
-
-.foreach.swift
-**************
-----
-include::../../examples/tutorial/foreach.swift[]
-----
-**************
-
-If
-~~
-Decisions can be made using 'if', like this:
-
-----
-sys::[sed -n '11,15p' ../../examples/tutorial/if.swift]
-----
-
-if.swift contains a simple example of this. Compile and run if.swift
-and see that it outputs "good morning". Changing the morning variable
-from true to false will cause the program to output "good afternoon".
-Here is the contents of the full script:
-
-.if.swift
-*********
------
-include::../../examples/tutorial/if.swift[]
------
-*********
-
-Sequential iteration
-~~~~~~~~~~~~~~~~~~~~
-
-A serial execution of instructions can be carried out using the sequential iteration construct.
-
-The following example demonstrates a simple application. Each step of
-the iteration is a string representation of the byte count of the
-previous step's output, with iteration terminating when the byte count
-reaches zero.
-
-Here's the program:
-
-.sequential_iteration.swift
-***************************
-----
-include::../../examples/tutorial/sequential_iteration.swift[]
-----
-***************************
-
-Echo is the standard unix echo utility.
-
-_wcl_ is our application code. It counts the number of bytes in the one
-file and writes that count out to another, like this:
-
-----
-$ cat ../wcl
-#!/bin/bash
-echo -n $(wc -c < $1) > $2
-
-$ echo -n hello > a
-$ wcl a b
-$ cat b
-5
-----
-
-Install the above wcl script somewhere and add a transformation catalog
-(tc) entry for it (see an example below, note that you will need to change the path in third column to the path where wcl is located on your localhost).
-
-----
-localhost wcl /home/ketan/bin/wcl INSTALLED INTEL32::LINUX null
-----
-
-Then run the example program like this:
-
-
-----
-$ swift iterate.swift
-Swift svn swift-r3334 cog-r2752
-
-RunID: 20100526-2259-gtlz8zf4
-Progress:
-SwiftScript trace: extract int value , 16.0
-SwiftScript trace: extract int value , 2.0
-SwiftScript trace: extract int value , 1.0
-Final status: Finished successfully:4
-----
Copied: trunk/docs/tutorial/language_features (from rev 5078, trunk/docs/tutorial/language_features)
===================================================================
--- trunk/docs/tutorial/language_features (rev 0)
+++ trunk/docs/tutorial/language_features 2011-09-08 00:53:06 UTC (rev 5086)
@@ -0,0 +1,390 @@
+Language features
+-----------------
+
+Parameters
+~~~~~~~~~~
+
+Procedures can have parameters. Input parameters specify inputs to the
+procedure and output parameters specify outputs. Our hello world greeting
+procedure already uses an output parameter, t, which indicates where the
+greeting output will go. In this section, we will modify the previous
+script to add an input parameter to the greeting function.
+
+.parameter.swift
+****************
+----
+include::../../examples/tutorial/parameter.swift[]
+----
+****************
+
+We have modified the signature of the greeting procedure to indicate
+that it takes a single parameter, s, of type 'string'.
+
+We have modified the invocation of the 'echo' utility so that it takes
+the value of s as a parameter, instead of the string literal "Hello,
+world!".
+
+We have modified the output file definition to point to a different file
+on disk.
+
+We have modified the invocation of greeting so that a greeting string is
+supplied.
+
+The code for this section can be found in parameter.swift. It can be
+invoked using the swift command, with output appearing in parameter.hello.txt:
+
+
+----
+$ swift parameter.swift
+----
+
+Now that we can choose our greeting text, we can call the same procedure
+with different parameters to generate several output files with
+different greetings. The code is in manyparam.swift and can be run as
+before using the swift command.
+
+.manyparam.swift
+****************
+----
+include::../../examples/tutorial/manyparam.swift[]
+----
+****************
+
+Note that we can intermingle definitions of variables with invocations
+of procedures.
+
+When this program runs, there should be three new files in the
+working directory (manyparam.english.txt, manyparam.francais.txt
+and manyparam.nihongo.txt) each containing a greeting in a different
+language.
+
+In addition to specifying parameters positionally, parameters can be
+named, and if desired a default value can be specified.
+
+Adding another application
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+Now we'll define a new application procedure. The procedure we define
+will capitalise all the words in the input file.
+
+To do this, we'll use the unix tr (translate) utility. Here is an
+example of using tr on the unix command line, not using Swift:
+
+----
+$ echo hello | tr '[a-z]' '[A-Z]'
+HELLO
+----
+
+There are two main steps - updating the transformation catalog, and
+updating the application block.
+
+The transformation catalog lists where application executables are located on
+remote sites. We need to modify the transformation catalog to define a logical
+transformation for the tr utility. The transformation catalog can be found
+in etc/tc.data. There are already several entries specifying
+where executables can be found. Add a new line to the file, specifying
+where tr can be found (usually in /usr/bin/tr but it may differ on
+your system), like this:
+
+----
+localhost tr /usr/bin/tr INSTALLED INTEL32::LINUX null
+----
+
+For now, ignore all of the fields except the second and the third. The
+second field tr specifies a logical application name and the third
+specifies the location of the application executable.
+
+Now that we have defined where to find tr, we can use it in SwiftScript.
+
+We can define a new procedure, capitalise, which calls tr.
+
+----
+sys::[sed -n '7,9p' ../../examples/tutorial/capitalise.swift]
+----
+We can call capitalise like this:
+
+----
+sys::[sed -n '12,14p' ../../examples/tutorial/capitalise.swift]
+----
+
+Here is the full program based on this exercise:
+
+.capitalise.swift
+***********************
+----
+include::../../examples/tutorial/capitalise.swift[]
+----
+***********************
+
+Next, run swift and verify the output is correct.
+
+----
+$ swift capitalise.swift
+...
+$ cat capitalise.2.txt
+HELLO FROM SWIFT
+----
+
+Anonymous files
+~~~~~~~~~~~~~~~
+In the previous section, the file hello.txt is used only to store
+an intermediate result. We don't really care about which name is used
+for the file, and we can let Swift choose the name.
+
+To do that, omit the mapping entirely when declaring hellofile:
+
+----
+sys::[sed -n '11p' ../../examples/tutorial/capitalise_anonymous.swift]
+----
+
+Swift will choose a filename, which in the present version will be in a
+subdirectory called _concurrent.
+
+
+Datatypes
+~~~~~~~~~
+All data in variables and files has a data type. So far, we've seen two
+types:
+
+ - string - this is a built-in type for storing strings of text in
+ memory, much like in other programming languages
+ - messagefile - this is a user-defined type used to mark disc resident files as
+ containing messages
+
+SwiftScript has the additional built-in types: boolean, integer and
+float that function much like their counterparts in other programming
+languages.
+
+It is also possible to create user defined types with more structure,
+for example:
+
+
+----
+sys::[sed -n '3,6p' ../../examples/tutorial/types.swift]
+----
+
+Each element of the structured type can be accessed using a . like this:
+
+----
+sys::[sed -n '14p' ../../examples/tutorial/types.swift]
+----
+
+The following complete program, types.swift, outputs a greeting using a
+user-defined structure type to hold parameters for the message:
+
+.types.swift
+************
+----
+include::../../examples/tutorial/types.swift[]
+----
+************
+
+Structured types can be comprised of marker types for files. See the
+later section on mappers for more information about this.
+
+
+Arrays
+~~~~~~
+
+We can define arrays using the [] suffix in a variable declaration:
+
+----
+sys::[sed -n '9p' ../../examples/tutorial/arrays.swift]
+----
+
+This program, arrays.swift, will declare an array of message files.
+
+.arrays.swift
+*************
+----
+include::../../examples/tutorial/arrays.swift[]
+----
+*************
+
+Observe that the type of the parameter to greeting is now an array of
+strings, string s[], instead of a single string, string s, that
+elements of the array can be referenced numerically, for example s[0],
+and that the array is initialised using an array literal,
+["how","are","you"].
+
+
+Mappers
+~~~~~~~
+
+A significant difference between SwiftScript and other languages is that
+data can be referred to on disk through variables in a very similar
+fashion to data in memory. For example, in the above examples we have
+seen a variable definition like this:
+
+----
+sys::[sed -n '7p' ../../examples/tutorial/arrays.swift]
+----
+
+This means that outfile is a dataset variable, which is mapped to a
+file on disk called arrays.txt. This variable can be assigned to
+using = in a similar fashion to an in-memory variable. We can say that
+outfile is mapped onto the disk file arrays.txt by a mapper.
+
+There are various ways of mapping in SwiftScript. Two forms of mapping,
+simple named mapping and anonymous mapping, have already
+been seen in this tutorial. Later exercises will introduce more forms.
+
+In simple named mapping, the name of the file that a variable is mapped to
+is explictly listed.
+
+----
+sys::[sed -n '7p' ../../examples/tutorial/hello.swift]
+----
+
+This is useful when you want to explicitly name input and output files
+for your program. An example of this can be seen with 'outfile' in the
+hello world exercise.
+
+With anonymous mapping no name is specified in the source code. A name is
+automatically generated for the file. This is useful for intermediate
+files that are only referenced through SwiftScript.
+A variable declaration is mapped anonymously by ommitting any mapper
+definition.
+
+----
+sys::[sed -n '11p' ../../examples/tutorial/capitalise_anonymous.swift]
+----
+
+Later exercises will introduce other ways of mapping from disk files to
+SwiftScript variables.
+
+
+The Regexp Mapper
+^^^^^^^^^^^^^^^^^
+In this exercise, we introduce the regexp mapper. This mapper
+transforms a string expression using a regular expression, and uses the
+result of that transformation as the filename to map.
+
+regexp.swift demonstrates the use of this by placing output into a
+file that is based on the name of the input file. Our input file is
+mapped to the inputfile variable using the simple named mapper, then
+we use the regular expression mapper to map the output file. We then use
+the countwords() procedure to count the words in the input file and
+store the result in the output file. In order for the countwords()
+procedure to work correctly, add the wc utility (usually found in
+/usr/bin/wc) to tc.data.
+
+The following program replaces the suffix of the input file (regexp_mapper.words.txt)
+with a new suffix (.count) to create regexp_mapper.words.count.
+
+.regexp_mapper.swift
+*************
+----
+include::../../examples/tutorial/regexp_mapper.swift[]
+----
+**************
+
+fixed_array_mapper
+^^^^^^^^^^^^^^^^^^
+The fixed array mapper maps a list of files into an array. Each
+element of the array is mapped into one file in the specified directory.
+See fixed_array_mapper.swift below.
+
+.fixed_array_mapper.swift
+*************************
+----
+include::../../examples/tutorial/fixed_array_mapper.swift[]
+----
+*************************
+
+foreach
+~~~~~~~
+SwiftScript provides a control structure, foreach, to operate on each
+element of an array in parallel.
+
+In this example, we will run the previous word counting example over
+each file in an array without having to explicitly list the array
+elements. The source code for this example is in foreach.swift. This
+program uses three input files: foreach.1.txt, foreach.2.txt, and
+foreach.3.txt. After you have run the workflow, you should see that there are three
+output files: foreach.1.count, foreach.2.count and foreach.3.count, each
+containing the word count for the corresponding input file. We combine
+the use of the fixed_array_mapper and the regexp_mapper.
+
+.foreach.swift
+**************
+----
+include::../../examples/tutorial/foreach.swift[]
+----
+**************
+
+If
+~~
+Decisions can be made using 'if', like this:
+
+----
+sys::[sed -n '11,15p' ../../examples/tutorial/if.swift]
+----
+
+if.swift contains a simple example of this. Compile and run if.swift
+and see that it outputs "good morning". Changing the morning variable
+from true to false will cause the program to output "good afternoon".
+Here is the contents of the full script:
+
+.if.swift
+*********
+-----
+include::../../examples/tutorial/if.swift[]
+-----
+*********
+
+Sequential iteration
+~~~~~~~~~~~~~~~~~~~~
+
+A serial execution of instructions can be carried out using the sequential iteration construct.
+
+The following example demonstrates a simple application. Each step of
+the iteration is a string representation of the byte count of the
+previous step's output, with iteration terminating when the byte count
+reaches zero.
+
+Here's the program:
+
+.sequential_iteration.swift
+***************************
+----
+include::../../examples/tutorial/sequential_iteration.swift[]
+----
+***************************
+
+Echo is the standard unix echo utility.
+
+_wcl_ is our application code. It counts the number of bytes in the one
+file and writes that count out to another, like this:
+
+----
+$ cat ../wcl
+#!/bin/bash
+echo -n $(wc -c < $1) > $2
+
+$ echo -n hello > a
+$ wcl a b
+$ cat b
+5
+----
+
+Install the above wcl script somewhere and add a transformation catalog
+(tc) entry for it (see an example below, note that you will need to change the path in third column to the path where wcl is located on your localhost).
+
+----
+localhost wcl /home/ketan/bin/wcl INSTALLED INTEL32::LINUX null
+----
+
+Then run the example program like this:
+
+
+----
+$ swift iterate.swift
+Swift svn swift-r3334 cog-r2752
+
+RunID: 20100526-2259-gtlz8zf4
+Progress:
+SwiftScript trace: extract int value , 16.0
+SwiftScript trace: extract int value , 2.0
+SwiftScript trace: extract int value , 1.0
+Final status: Finished successfully:4
+----
Deleted: trunk/docs/tutorial/runtime_features
===================================================================
--- trunk/docs/tutorial/runtime_features 2011-09-07 21:09:21 UTC (rev 5085)
+++ trunk/docs/tutorial/runtime_features 2011-09-08 00:53:06 UTC (rev 5086)
@@ -1,160 +0,0 @@
-Runtime features
-----------------
-
-Visualizing the workflow as a graph
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-When running a workflow, its possible to generate a provenance graph at
-the same time:
-
-
------
-$ swift -pgraph graph.dot first.swift
-$ dot -ograph.png -Tpng graph.dot
-----
-
-graph.png can then be viewed using your favourite image viewer. The dot application is part of the graphViz project. More information can be found at http://www.graphviz.org.
-
-Running on a remote site
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-As configured by default, all jobs are run locally. In the previous
-examples, we've invoked echo and tr executables from our SwiftScript
-program. These have been run on the local system (the same computer on
-which you ran swift). We can also make our computations run on a
-remote resource. For more information on running Swift on a remote site
-please see the http://www.ci.uchicago.edu/swift/guides/release-0.93/siteguide/siteguide.html[Site Configuration Guide].
-
-
-Starting and restarting
-~~~~~~~~~~~~~~~~~~~~~~~
-
-Now we're going to try out the restart capabilities of Swift. We will
-make a workflow that will deliberately fail, and then we will fix the
-problem so that Swift can continue with the workflow.
-
-First we have the program in working form, restart.swift.
-
-.restart.swift
-**************
-----
-include::../../examples/tutorial/restart.swift[]
-----
-**************
-
-We must define some transformation catalog entries:
-
-----
-localhost touch /usr/bin/touch INSTALLED INTEL32::LINUX null
-localhost broken /bin/true INSTALLED INTEL32::LINUX null
-----
-
-Now we can run the program:
-
-----
-$ swift restart.swift
-Swift 0.9 swift-r2860 cog-r2388
-
-RunID: 20100526-1119-3kgzzi15
-Progress:
-Final status: Finished successfully:4
-----
-
-Four jobs run - touch, echo, broken and a final echo. (note that broken
-isn't actually broken yet).
-
-Now we will break the broken job and see what happens. Replace the
-definition in tc.data for broken with this:
-
-----
-localhost broken /bin/false INSTALLED INTEL32::LINUX null
-----
-
-Now when we run the workflow, the broken task fails:
-
-----
-$ swift restart.swift
-
-Swift 0.9 swift-r2860 cog-r2388
-
-RunID: 20100526-1121-tssdcljg
-Progress:
-Progress: Stage in:1 Finished successfully:2
-Execution failed:
- Exception in broken:
-Arguments: [process]
-Host: localhost
-Directory: restart-20100526-1121-tssdcljg/jobs/1/broken-1i6ufisj
-stderr.txt:
-stdout.txt:
-----
-
-From the output we can see that touch and the first echo completed, but
-then broken failed and so swift did not attempt to execute the final echo.
-
-There will be a restart log with the same name as the RunID:
-
-
-----
-$ ls *20100526-1121-tssdcljg*rlog
-restart-20100526-1121-tssdcljg.0.rlog
-----
-
-This restart log contains enough information for swift to know which
-parts of the workflow were executed successfully.
-
-We can try to rerun it immediately, like this:
-
-
-----
-$ swift -resume restart-20100526-1121-tssdcljg.0.rlog restart.swift
-
-Swift 0.9 swift-r2860 cog-r2388
-
-RunID: 20100526-1125-7yx0zi6d
-Progress:
-Execution failed:
- Exception in broken:
-Arguments: [process]
-Host: localhost
-Directory: restart-20100526-1125-7yx0zi6d/jobs/m/broken-msn1gisj
-stderr.txt:
-stdout.txt:
-
-Caused by:
- Exit code 1
-----
-
-Swift tried to resume the workflow by executing "broken" again. It did
-not try to run the touch or first echo jobs, because the restart log
-says that they do not need to be executed again.
-
-Broken failed again, leaving the original restart log in place.
-
-Now we will fix the problem with "broken" by restoring the original
-tc.data line that works.
-
-Remove the existing "broken" line and replace it with the successful
-tc.data entry above:
-
-
-----
-localhost broken /bin/true INSTALLED INTEL32::LINUX null
-----
-
-Now run again:
-
-
-----
-$ swift -resume restart-20100526-1121-tssdcljg.0.rlog restart.swift
-
-Swift 0.9 swift-r2860 cog-r2388
-
-RunID: 20100526-1128-a2gfuxhg
-Progress:
-Final status: Initializing:2 Finished successfully:2
-----
-
-Swift tries to run "broken" again. This time it works, and so Swift
-continues on to execute the final piece of the workflow as if nothing
-had ever gone wrong.
Copied: trunk/docs/tutorial/runtime_features (from rev 5078, trunk/docs/tutorial/runtime_features)
===================================================================
--- trunk/docs/tutorial/runtime_features (rev 0)
+++ trunk/docs/tutorial/runtime_features 2011-09-08 00:53:06 UTC (rev 5086)
@@ -0,0 +1,317 @@
+Runtime features
+----------------
+
+Visualizing the workflow as a graph
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When running a workflow, its possible to generate a provenance graph at
+the same time:
+
+
+-----
+$ swift -pgraph graph.dot first.swift
+$ dot -ograph.png -Tpng graph.dot
+----
+
+graph.png can then be viewed using your favourite image viewer. The dot application is part of the graphViz project. More information can be found at http://www.graphviz.org.
+
+Running on a remote site
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+As configured by default, all jobs are run locally. In the previous
+examples, we've invoked echo and tr executables from our SwiftScript
+program. These have been run on the local system (the same computer on
+which you ran swift). We can also make our computations run on a
+remote resource.
+
+WARNING: This example is necessarily more vague than previous ones,
+because it requires access to remote resources. You should ensure that
+you can submit a job using the globus-job-run (or globusrun-ws)
+command(s).
+
+We do not need to modify any SwiftScript code to run on another
+resource. Instead, we must modify another catalog, the site catalog.
+This catalog provides details of the location that applications will be
+run, with the default settings referring to the local machine. We will
+modify it to refer to a remote resource - the UC Teraport cluster. If
+you are not a UC Teraport user, you should use details of a different
+resource that you do have access to.
+
+The site catalog is located in etc/sites.xml and is a relatively
+straightforward XML format file. We must modify each of the following
+three settings: gridftp (which indicates how and where data can be
+transferred to the remote resource), jobmanager (which indicates how
+applications can be run on the remote resource) and workdirectory (which
+indicates where working storage can be found on the remote resource).
+
+
+Writing a mapper
+~~~~~~~~~~~~~~~~
+
+This section will introduce writing a custom mapper so that Swift is
+able to access data files laid out in application-specific ways.
+
+An application-specific mapper must take the form of a Java class that
+implements the http://www.ci.uchicago.edu/swift/javadoc/vdsk/org/griphyn/vdl/mapping/Mapper.html[Mapper
+interface].
+
+Usually you don't need to implement this interface directly, because
+Swift provides a number of more concrete classes with some functionality
+already implemented.
+
+The hierarchy of helper classes is:
+
+http://www.ci.uchicago.edu/swift/javadoc/vdsk/org/griphyn/vdl/mapping/Mapper.html[Mapper]
+
+This is the abstract interface for mappers in Swift. You must
+implement methods to provide access to mapper properties, to map from a
+SwiftScript dataset path (such as foo[1].bar) to a file name, to check
+whether a file exists. None of the default Swift mappers implement this
+interface directly - instead they use one of the following helper classes.
+
+http://www.ci.uchicago.edu/swift/javadoc/vdsk/org/griphyn/vdl/mapping/AbstractMapper.html[Abstract Mapper]
+
+This provides helper methods to manage mapper properties and to handle
+existance checking. Examples of mappers which use this class are:
+http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.array_mapper[Array Mapper],
+http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.csv_mapper[CSV Mapper],
+http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.fixed_array_mapper[Fixed Array Mapper],
+http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.regexp_mapper[Regexp Mapper],
+and http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.single_file_mapper[Single File Mapper].
+
+http://www.ci.uchicago.edu/swift/javadoc/vdsk/org/griphyn/vdl/mapping/file/AbstractFileMapper.html[AbstractFileMapper]
+This provides a helper class for mappers which select files based on
+selecting files from a directory listing. It is necessary to write some
+helper methods that are different from the above mapper methods.
+Examples of mappers which use this class are:
+http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.simple_mapper[simple_mapper],
+http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.filesys_mapper[filesys_mapper],
+and the StructuredRegularExpressionMapper.
+
+In general, to write a mapper, choose either the AbstractMapper or the
+AbstractFileMapper and extend those. If your mapper will generally
+select the files it returns based on a directory listing and will
+convert paths to filenames using some regular conversion (for example,
+in the way that simple_mapper maps files in a directory that match a
+particular pattern), then you should probably use the
+AbstractFileMapper. If your mapper will produce a list of files in some
+other way (for example, in the way that csv_mapper maps based on
+filenames given in a CSV file rather than looking at which files are in
+a directory), then you should probably use the AbstractMapper.
+
+
+Writing a very basic mapper
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In this section, we will write a very basic (almost useless) mapper that
+will map a SwiftScript dataset into a hardcoded file called
+myfile.txt, like this:
+
+
+ Swift variable Filename
+
+ var <-----------------------------> myfile.txt
+
+We should be able to use the mapper we write in a SwiftScript program
+like this:
+
+
+----
+type file;
+file f <my_first_mapper>;
+----
+
+First we must choose a base class - AbstractMapper or
+AbstractFileMapper. We aren't going to use a directory listing to decide
+on our mapping - we are getting the mapping from some other source (in
+fact, it will be hard coded). So we will use AbstractMapper.
+
+So now onto the source code. We must define a subclass of AbstractMapper
+and implement several mapper methods: isStatic, existing, and map. These
+methods are documented in the javadoc for the Mapper interface.
+
+Here is the code implementing this mapper. Put this in your source
+vdsk directory, make a directory src/tutorial/|and put this file in
+src/tutorial/MyFirstMapper.java.
+
+.MyFirstMapper.java
+*******************
+----
+include::../../examples/tutorial/MyFirstMapper.java[]
+----
+*******************
+
+Now we need to inform the Swift engine about the existence of this
+mapper. We do that by editing the MapperFactory class definition, in
+src/org/griphyn/vdl/mapping/MapperFactory.java and adding a
+registerMapper call alongside the existing registerMapper calls, like this:
+
+
+----
+registerMapper("my_first_mapper", tutorial.MyFirstMapper.class);
+----
+
+The first parameter is the name of the mapper that will be used in
+SwiftScript program. The second parameter is the new Mapper class that
+we just wrote.
+
+Now rebuild Swift using the "ant redist" target.
+
+This new Swift build will be aware of your new mapper. We can test it
+out with a hello world program:
+
+.my_first_mapper.swift
+**********************
+-----
+include::../../examples/tutorial/my_first_mapper.swift[]
+-----
+**********************
+
+Run this program, and hopefully you will find the "hello" string has
+been output into the hard coded output file myfile.txt:
+
+
+----
+$ cat myfile.txt
+hello
+----
+
+So that's a first very simple mapper implemented. Compare the source
+code to the single_file_mapper in
+http://www.ci.uchicago.edu/trac/swift/browser/trunk/src/org/griphyn/vdl/mapping/file/SingleFileMapper.java[SingleFileMapper.java].
+There is not much more code to the single_file_mapper - mostly code to
+deal with the file parameter.
+
+
+Starting and restarting
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Now we're going to try out the restart capabilities of Swift. We will
+make a workflow that will deliberately fail, and then we will fix the
+problem so that Swift can continue with the workflow.
+
+First we have the program in working form, restart.swift.
+
+.restart.swift
+**************
+----
+include::../../examples/tutorial/restart.swift[]
+----
+**************
+
+We must define some transformation catalog entries:
+
+----
+localhost touch /usr/bin/touch INSTALLED INTEL32::LINUX null
+localhost broken /bin/true INSTALLED INTEL32::LINUX null
+----
+
+Now we can run the program:
+
+----
+$ swift restart.swift
+Swift 0.9 swift-r2860 cog-r2388
+
+RunID: 20100526-1119-3kgzzi15
+Progress:
+Final status: Finished successfully:4
+----
+
+Four jobs run - touch, echo, broken and a final echo. (note that broken
+isn't actually broken yet).
+
+Now we will break the broken job and see what happens. Replace the
+definition in tc.data for broken with this:
+
+----
+localhost broken /bin/false INSTALLED INTEL32::LINUX null
+----
+
+Now when we run the workflow, the broken task fails:
+
+----
+$ swift restart.swift
+
+Swift 0.9 swift-r2860 cog-r2388
+
+RunID: 20100526-1121-tssdcljg
+Progress:
+Progress: Stage in:1 Finished successfully:2
+Execution failed:
+ Exception in broken:
+Arguments: [process]
+Host: localhost
+Directory: restart-20100526-1121-tssdcljg/jobs/1/broken-1i6ufisj
+stderr.txt:
+stdout.txt:
+----
+
+From the output we can see that touch and the first echo completed, but
+then broken failed and so swift did not attempt to execute the final echo.
+
+There will be a restart log with the same name as the RunID:
+
+
+----
+$ ls *20100526-1121-tssdcljg*rlog
+restart-20100526-1121-tssdcljg.0.rlog
+----
+
+This restart log contains enough information for swift to know which
+parts of the workflow were executed successfully.
+
+We can try to rerun it immediately, like this:
+
+
+----
+$ swift -resume restart-20100526-1121-tssdcljg.0.rlog restart.swift
+
+Swift 0.9 swift-r2860 cog-r2388
+
+RunID: 20100526-1125-7yx0zi6d
+Progress:
+Execution failed:
+ Exception in broken:
+Arguments: [process]
+Host: localhost
+Directory: restart-20100526-1125-7yx0zi6d/jobs/m/broken-msn1gisj
+stderr.txt:
+stdout.txt:
+
+Caused by:
+ Exit code 1
+----
+
+Swift tried to resume the workflow by executing "broken" again. It did
+not try to run the touch or first echo jobs, because the restart log
+says that they do not need to be executed again.
+
+Broken failed again, leaving the original restart log in place.
+
+Now we will fix the problem with "broken" by restoring the original
+tc.data line that works.
+
+Remove the existing "broken" line and replace it with the successful
+tc.data entry above:
+
+
+----
+localhost broken /bin/true INSTALLED INTEL32::LINUX null
+----
+
+Now run again:
+
+
+----
+$ swift -resume restart-20100526-1121-tssdcljg.0.rlog restart.swift
+
+Swift 0.9 swift-r2860 cog-r2388
+
+RunID: 20100526-1128-a2gfuxhg
+Progress:
+Final status: Initializing:2 Finished successfully:2
+----
+
+Swift tries to run "broken" again. This time it works, and so Swift
+continues on to execute the final piece of the workflow as if nothing
+had ever gone wrong.
Deleted: trunk/docs/tutorial/tutorial.txt
===================================================================
--- trunk/docs/tutorial/tutorial.txt 2011-09-07 21:09:21 UTC (rev 5085)
+++ trunk/docs/tutorial/tutorial.txt 2011-09-08 00:53:06 UTC (rev 5086)
@@ -1,17 +0,0 @@
-A Swift Tutorial
-================
-
-:toc:
-:icons:
-:website: http://www.ci.uchicago.edu/swift/guides/tutorial.php
-:numbered:
-
-include::introduction[]
-
-include::hello_world[]
-
-include::language_features[]
-
-include::runtime_features[]
-
-include::bits[]
Copied: trunk/docs/tutorial/tutorial.txt (from rev 5078, trunk/docs/tutorial/tutorial.txt)
===================================================================
--- trunk/docs/tutorial/tutorial.txt (rev 0)
+++ trunk/docs/tutorial/tutorial.txt 2011-09-08 00:53:06 UTC (rev 5086)
@@ -0,0 +1,17 @@
+A Swift Tutorial
+================
+
+:toc:
+:icons:
+:website: http://www.ci.uchicago.edu/swift/guides/tutorial.php
+:numbered:
+
+include::introduction[]
+
+include::hello_world[]
+
+include::language_features[]
+
+include::runtime_features[]
+
+include::bits[]
Modified: trunk/docs/userguide/cdm
===================================================================
--- trunk/docs/userguide/cdm 2011-09-07 21:09:21 UTC (rev 5085)
+++ trunk/docs/userguide/cdm 2011-09-08 00:53:06 UTC (rev 5086)
@@ -2,202 +2,202 @@
--------------------------
Overview
-~~~~~~~~
-. The user specifies a CDM policy in a file, customarily fs.data.
-. fs.data is given to Swift on the command line.
-. The Swift data module (org.globus.swift.data) is informed of the CDM policy.
-. At job launch time, the VDL Karajan code queries the CDM policy,
- .. altering the file staging phase, and
- .. sending fs.data to the compute site.
-. At job run time, the Swift wrapper script
- .. consults a Perl script to obtain policy, and
- .. uses wrapper extensions to modify data movement.
-. Similarly, stage out can be changed.
-
-
+~~~~~~~~
+. The user specifies a CDM policy in a file, customarily fs.data.
+. fs.data is given to Swift on the command line.
+. The Swift data module (org.globus.swift.data) is informed of the CDM policy.
+. At job launch time, the VDL Karajan code queries the CDM policy,
+ .. altering the file staging phase, and
+ .. sending fs.data to the compute site.
+. At job run time, the Swift wrapper script
+ .. consults a Perl script to obtain policy, and
+ .. uses wrapper extensions to modify data movement.
+. Similarly, stage out can be changed.
+
+
.Command line
------
-$ swift -sites.file sites.xml -tc.file tc.data -cdm.file fs.data stream.swift
-----
-
+$ swift -sites.file sites.xml -tc.file tc.data -cdm.file fs.data stream.swift
+-----
+
CDM policy file format
~~~~~~~~~~~~~~~~~~~~~~
.Example
------
-# Describe CDM for my job
-property GATHER_LIMIT 1
-rule .*input.txt DIRECT /gpfs/homes/wozniak/data
-rule .*xfile*.data BROADCAST /dev/shm
-rule .* DEFAULT
-----
-
-The lines contain:
-
-. A directive, either rule or property
-. A rule has:
- .. A regular expression
- .. A policy token
- .. Additional policy-specific arguments
-. A property has
- .. A policy property token
- .. The token value
-. Comments with # .
-. Blank lines are ignored.
-
-
+# Describe CDM for my job
+property GATHER_LIMIT 1
+rule .*input.txt DIRECT /gpfs/homes/wozniak/data
+rule .*xfile*.data BROADCAST /dev/shm
+rule .* DEFAULT
+-----
+
+The lines contain:
+
+. A directive, either rule or property
+. A rule has:
+ .. A regular expression
+ .. A policy token
+ .. Additional policy-specific arguments
+. A property has
+ .. A policy property token
+ .. The token value
+. Comments with # .
+. Blank lines are ignored.
+
+
.Notes
-
-. The policy file is used as a lookup database by Swift and Perl methods.
-. For example, a lookup with the database above given the argument input.txt would result in the Direct policy.
-. If the lookup does not succeed, the result is DEFAULT.
- . Policies are listed as subclasses of org.globus.swift.data.policy.Policy .
-
-
+
+. The policy file is used as a lookup database by Swift and Perl methods.
+. For example, a lookup with the database above given the argument input.txt would result in the Direct policy.
+. If the lookup does not succeed, the result is DEFAULT.
+ . Policies are listed as subclasses of org.globus.swift.data.policy.Policy .
+
+
Policy Descriptions
~~~~~~~~~~~~~~~~~~~
-.Default
-
-* Just use file staging as provided by Swift/CoG. Identical to behavior if no CDM file is given.
-
-
+.Default
+
+* Just use file staging as provided by Swift/CoG. Identical to behavior if no CDM file is given.
+
+
.Broadcast
------
-rule .*xfile*.data BROADCAST /dev/shm
------
-* The input files matching the pattern will be stored in the given directory, an LFS location, with links in the job directory.
-* On the BG/P, this will make use of the f2cn tool.
-* On machines not implementing an efficient broadcast, we will just ensure correctness. For example, on a workstation, the location could be in a /tmp RAM FS.
-
-
+-----
+rule .*xfile*.data BROADCAST /dev/shm
+-----
+* The input files matching the pattern will be stored in the given directory, an LFS location, with links in the job directory.
+* On the BG/P, this will make use of the f2cn tool.
+* On machines not implementing an efficient broadcast, we will just ensure correctness. For example, on a workstation, the location could be in a /tmp RAM FS.
+
+
.Direct
------
-rule .*input.txt DIRECT /gpfs/scratch/wozniak/
------
-* Allows for direct I/O to the parallel FS without staging.
-* The input files matching the pattern must already exist in the given directory, a GFS location. Links will be placed in the job directory.
-* The output files matching the pattern will be stored in the given directory, with links in the job directory.
-* Example: In the rule above, the Swift-generated file name ./data/input.txt would be accessed by the user job in /gpfs/homes/wozniak/data/input.txt .
-
-
-.Local
------
-rule .*input.txt LOCAL dd /gpfs/homes/wozniak/data obs=64K
------
-* Allows for client-directed input copy to the compute node.
-* The user may specify cp or dd as the input transfer program.
-* The input files matching the pattern must already exist in the given directory, a GFS location. Copies will be placed in the job directory.
-* Argument list: [tool] [GFS directory] [tool arguments]*
-
-
+-----
+rule .*input.txt DIRECT /gpfs/scratch/wozniak/
+-----
+* Allows for direct I/O to the parallel FS without staging.
+* The input files matching the pattern must already exist in the given directory, a GFS location. Links will be placed in the job directory.
+* The output files matching the pattern will be stored in the given directory, with links in the job directory.
+* Example: In the rule above, the Swift-generated file name ./data/input.txt would be accessed by the user job in /gpfs/homes/wozniak/data/input.txt .
+
+
+.Local
+-----
+rule .*input.txt LOCAL dd /gpfs/homes/wozniak/data obs=64K
+-----
+* Allows for client-directed input copy to the compute node.
+* The user may specify cp or dd as the input transfer program.
+* The input files matching the pattern must already exist in the given directory, a GFS location. Copies will be placed in the job directory.
+* Argument list: [tool] [GFS directory] [tool arguments]*
+
+
.Gather
------
-property GATHER_LIMIT 500000000 # 500 MB
-property GATHER_DIR /dev/shm/gather
-property GATHER_TARGET /gpfs/wozniak/data/gather_target
-rule .*.output.txt GATHER
-----
-
-* The output files matching the pattern will be present to tasks in the job directory as usual but noted in a _swiftwrap shell array GATHER_OUTPUT.
-* The GATHER_OUTPUT files will be cached in the GATHER_DIR, an LFS location.
-* The cache will be flushed when a job ends if a du on GATHER_DIR exceeds GATHER_LIMIT.
-* As the cache fills or on stage out, the files will be bundled into randomly named tarballs in GATHER_TARGET, a GFS location.
-* If the compute node is an SMP, GATHER_DIR is a shared resource. It is protected by the link file GATHER_DIR/.cdm.lock .
-* Unpacking the tarballs in GATHER_TARGET will produce the user-specified filenames.
-
+property GATHER_LIMIT 500000000 # 500 MB
+property GATHER_DIR /dev/shm/gather
+property GATHER_TARGET /gpfs/wozniak/data/gather_target
+rule .*.output.txt GATHER
+-----
+
+* The output files matching the pattern will be present to tasks in the job directory as usual but noted in a _swiftwrap shell array GATHER_OUTPUT.
+* The GATHER_OUTPUT files will be cached in the GATHER_DIR, an LFS location.
+* The cache will be flushed when a job ends if a du on GATHER_DIR exceeds GATHER_LIMIT.
+* As the cache fills or on stage out, the files will be bundled into randomly named tarballs in GATHER_TARGET, a GFS location.
+* If the compute node is an SMP, GATHER_DIR is a shared resource. It is protected by the link file GATHER_DIR/.cdm.lock .
+* Unpacking the tarballs in GATHER_TARGET will produce the user-specified filenames.
+
.Summary
-
-. Files created by application
-. Acquire lock
-. Move files to cache
-. Check cache size
-. If limit exceeded, move all cache files to outbox
-. Release lock
-. If limit was exceeded, stream outbox as tarball to target
-
+
+. Files created by application
+. Acquire lock
+. Move files to cache
+. Check cache size
+. If limit exceeded, move all cache files to outbox
+. Release lock
+. If limit was exceeded, stream outbox as tarball to target
+
.Notes
-
-* Gather required quite a bit of shell functionality to manage the lock, etc. This is placed in cdm_lib.sh .
-* vdl_int.k needed an additional task submission (cdm_cleanup.sh) to perform the final flush at workflow completion time . This task also uses cdm_lib.sh .
-
-
+
+* Gather required quite a bit of shell functionality to manage the lock, etc. This is placed in cdm_lib.sh .
+* vdl_int.k needed an additional task submission (cdm_cleanup.sh) to perform the final flush at workflow completion time . This task also uses cdm_lib.sh .
+
+
VDL/Karajan processing
~~~~~~~~~~~~~~~~~~~~~~
-. CDM functions are available in Karajan via the cdm namespace.
-. These functions are defined in org.globus.swift.data.Query .
-. If CDM is enabled, VDL skips file staging for files unless the policy is DEFAULT.
-
-
-Swift wrapper CDM routines
+. CDM functions are available in Karajan via the cdm namespace.
+. These functions are defined in org.globus.swift.data.Query .
+. If CDM is enabled, VDL skips file staging for files unless the policy is DEFAULT.
+
+
+Swift wrapper CDM routines
~~~~~~~~~~~~~~~~~~~~~~~~~~
-. The cdm.pl script is shipped to the compute node if CDM is enabled.
-. When linking in inputs, CDM is consulted by _swiftwrap:cdm_lookup().
-. The cdm_action() shell function handles CDM methods, typically just producing a link.
-
-
-Test cases
+. The cdm.pl script is shipped to the compute node if CDM is enabled.
+. When linking in inputs, CDM is consulted by _swiftwrap:cdm_lookup().
+. The cdm_action() shell function handles CDM methods, typically just producing a link.
+
+
+Test cases
~~~~~~~~~~
-
-. Simple test cases are in:
- https://svn.mcs.anl.gov/repos/wozniak/collab/cdm/scripts/cdm-direct and
- https://svn.mcs.anl.gov/repos/wozniak/collab/cdm/scripts/all-pairs
-. Do a:
- mkdir cdm
- cd cdm
- svn co https://svn.mcs.anl.gov/repos/wozniak/collab/cdm/scripts
-. In cdm-direct, run:
- source ./setup.sh local local local
-. Run workflow:
- swift -sites.file sites.xml -tc.file tc.data -cdm.file fs.data stream.swift
-. Note that staging is skipped for input.txt
- policy: file://localhost/input.txt : DIRECT
- FILE_STAGE_IN_START file=input.txt ...
- FILE_STAGE_IN_SKIP file=input.txt policy=DIRECT
- FILE_STAGE_IN_END file=input.txt ...
-. In the wrapper output, the input file is handled by CDM functionality:
- Progress 2010-01-21 13:50:32.466572727-0600 LINK_INPUTS
- CDM_POLICY: DIRECT /homes/wozniak/cdm/scripts/cdm-direct
- CDM: jobs/t/cp_sh-tkul4nmj input.txt DIRECT /homes/wozniak/cdm/scripts/cdm-direct
- CDM[DIRECT]: Linking jobs/t/cp_sh-tkul4nmj/input.txt to /homes/wozniak/cdm/scripts/cdm-direct/input.txt
- Progress 2010-01-21 13:50:32.486016708-0600 EXECUTE
-. all-pairs is quite similar but uses more policies.
-
-
+
+. Simple test cases are in:
+ https://svn.mcs.anl.gov/repos/wozniak/collab/cdm/scripts/cdm-direct and
+ https://svn.mcs.anl.gov/repos/wozniak/collab/cdm/scripts/all-pairs
+. Do a:
+ mkdir cdm
+ cd cdm
+ svn co https://svn.mcs.anl.gov/repos/wozniak/collab/cdm/scripts
+. In cdm-direct, run:
+ source ./setup.sh local local local
+. Run workflow:
+ swift -sites.file sites.xml -tc.file tc.data -cdm.file fs.data stream.swift
+. Note that staging is skipped for input.txt
+ policy: file://localhost/input.txt : DIRECT
+ FILE_STAGE_IN_START file=input.txt ...
+ FILE_STAGE_IN_SKIP file=input.txt policy=DIRECT
+ FILE_STAGE_IN_END file=input.txt ...
+. In the wrapper output, the input file is handled by CDM functionality:
+ Progress 2010-01-21 13:50:32.466572727-0600 LINK_INPUTS
+ CDM_POLICY: DIRECT /homes/wozniak/cdm/scripts/cdm-direct
+ CDM: jobs/t/cp_sh-tkul4nmj input.txt DIRECT /homes/wozniak/cdm/scripts/cdm-direct
+ CDM[DIRECT]: Linking jobs/t/cp_sh-tkul4nmj/input.txt to /homes/wozniak/cdm/scripts/cdm-direct/input.txt
+ Progress 2010-01-21 13:50:32.486016708-0600 EXECUTE
+. all-pairs is quite similar but uses more policies.
+
+
PTMap case
-^^^^^^^^^^
-. Start with vanilla PTMap:
- .. cd cdm
- .. mkdir apps
- .. cd apps
- .. https://svn.mcs.anl.gov/repos/wozniak/collab/cdm/apps/ptmap
-. Source setup.sh
-. Use start.sh, which
- .. applies CDM policy from fs.local.data
-
-
+^^^^^^^^^^
+. Start with vanilla PTMap:
+ .. cd cdm
+ .. mkdir apps
+ .. cd apps
+ .. https://svn.mcs.anl.gov/repos/wozniak/collab/cdm/apps/ptmap
+. Source setup.sh
+. Use start.sh, which
+ .. applies CDM policy from fs.local.data
+
+
CDM site-aware policy file format
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
+
Example
-----
-#Describe CDM for my job
-#Use DIRECT and BROADCAST if on cluster1, else use DEFAULT behavior
-property GATHER_LIMIT 1
-rule cluster1 .*input.txt DIRECT /gpfs/homes/wozniak/data
-rule cluster1 .*xfile*.data BROADCAST /dev/shm
-rule ANYWHERE .* DEFAULT
+#Describe CDM for my job
+#Use DIRECT and BROADCAST if on cluster1, else use DEFAULT behavior
+property GATHER_LIMIT 1
+rule cluster1 .*input.txt DIRECT /gpfs/homes/wozniak/data
+rule cluster1 .*xfile*.data BROADCAST /dev/shm
+rule ANYWHERE .* DEFAULT
-----
-
+
The lines contain:
-
-. A directive, either rule or property
-. A rule has:
- .. A regular expression for site matchin
- .. A regular expression for filename matching
- .. A policy token
- .. Additional policy-specific arguments
-. A property has
- .. A policy property token
- .. The token value
-. Comments with # .
-. Blank lines are ignored.
+
+. A directive, either rule or property
+. A rule has:
+ .. A regular expression for site matchin
+ .. A regular expression for filename matching
+ .. A policy token
+ .. Additional policy-specific arguments
+. A property has
+ .. A policy property token
+ .. The token value
+. Comments with # .
+. Blank lines are ignored.
Modified: trunk/docs/userguide/coasters
===================================================================
--- trunk/docs/userguide/coasters 2011-09-07 21:09:21 UTC (rev 5085)
+++ trunk/docs/userguide/coasters 2011-09-08 00:53:06 UTC (rev 5086)
@@ -1,5 +1,7 @@
Coasters
--------
+Coasters were introduced in Swift v0.6 as an experimental feature.
+
In many applications, Swift performance can be greatly enhanced by the
use of CoG coasters. CoG coasters provide a low-overhead job submission
and file transfer mechanism suited for the execution of short jobs (on
Deleted: trunk/docs/userguide/log-processing
===================================================================
--- trunk/docs/userguide/log-processing 2011-09-07 21:09:21 UTC (rev 5085)
+++ trunk/docs/userguide/log-processing 2011-09-08 00:53:06 UTC (rev 5086)
@@ -1,104 +0,0 @@
-
-Log Processing
---------------
-
-To properly generate log plots, you must enable VDL/Karajan logging.
-TODO:How?
-
-
-You should check the scripts that you intend to use to determine
-what log lines they require and ensure that you are generating
-those lines via log4j.properties
-
-Make sure log4.properties contains:
---------------------------------------
-log4j.logger.swift=DEBUG
-log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Cpu=DEBUG
-log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Block=DEBUG
---------------------------------------
-TODO: Does it work for coasters-based runs only?
-
-Normalize event times in the log to the run start time
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-* Generate the log, assuming the log is titled +swift-run.log+
-
-------------------------------------------
-./normalize-log.pl file.contains.start.time swift-run.log > swift-run.norm
-------------------------------------------
-TODO:In what format does the start time be in 'file.contains.start.time'
-
-
-Make a basic load plot from Coasters Cpu log lines
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-. Normalize the log.
-. Build up a load data file:
-+
-------------------------------------------
-./cpu-job-load.pl < swift-run.norm > load.data
-------------------------------------------
-. Plot with the JFreeChart-based plotter in usertools/plotter:
-+
-------------------------------------------
-swift_plotter.zsh -s load.cfg load.eps load.data
-------------------------------------------
-Note: Th load.cfg is available from swift/libexec/log-processing/
-
-
-Make a basic job completion plot from Coasters Cpu log lines
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-. Normalize the log.
-
-. Build up a completed data file:
-+
-------------------------------------------
-./cpu-job-completed.pl < swift-run.norm > completed.data
-------------------------------------------
-
-. Plot with the JFreeChart-based plotter in usertools/plotter:
-+
-------------------------------------------
-swift_plotter.zsh -s completed.cfg completed.eps completed.data
-------------------------------------------
-
-Make a basic Block allocation plot from Coasters Block log lines
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-. Normalize the log.
-
-. Build up a block allocation data file:
-+
-------------------------------------------
-./block-level.pl < swift-run.norm > blocks.data
-------------------------------------------
-
-. Plot with the JFreeChart-based plotter in usertools/plotter:
-+
-------------------------------------------
-swift_plotter.zsh -s blocks.{cfg,eps,data}
-------------------------------------------
-
-Make a job runtime distribution plot from Coasters Cpu log lines
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-. Normalize the log.
-
-. Build up a job runtime file:
-+
-------------------------------------------
-./extract-times.pl < swift-run.norm > times.data
-------------------------------------------
-
-. Put the job runtimes into 1-second buckets:
-+
-------------------------------------------
-./ buckets.pl 1 times.data > buckets.data
-------------------------------------------
-
-. Plot with the JFreeChart-based plotter in usertools/plotter:
-+
-------------------------------------------
-swift_plotter.zsh -s buckets.cfg buckets.eps buckets.data
-------------------------------------------
Modified: trunk/docs/userguide/userguide.txt
===================================================================
--- trunk/docs/userguide/userguide.txt 2011-09-07 21:09:21 UTC (rev 5085)
+++ trunk/docs/userguide/userguide.txt 2011-09-08 00:53:06 UTC (rev 5086)
@@ -37,5 +37,3 @@
include::howto_tips[]
include::cdm[]
-
-include::log-processing[]
Property changes on: trunk/docs/utils
___________________________________________________________________
Added: svn:ignore
+ gensites.html
gensites.pdf
Deleted: trunk/docs/utils/gensites.txt
===================================================================
--- trunk/docs/utils/gensites.txt 2011-09-06 19:12:14 UTC (rev 5078)
+++ trunk/docs/utils/gensites.txt 2011-09-08 00:53:06 UTC (rev 5086)
@@ -1,235 +0,0 @@
-Gensites
-==========
-:toc:
-:icons:
-:website: http://www.ci.uchicago.edu/swift
-:numbered:
-
-Overview
---------
-To simplify this configuration process, versions of Swift starting with
-0.92 include a utility called gensites. The gensites command is used
-to generate a sites.xml file for running a swift workflow on a given
-site. It accomplishes this by using a series of templates. The templates
-used by gensites are the same templates used for internal testing, so
-they are likely up to date and known to work on a given site.
-
-
-Viewing All Available Site Templates
-------------------------------------
-To view a list of all available templates, run the following command:
-
------
-$ gensites -T
------
-
-You should see output similar to this:
------
-intrepid
-local
-local-pbs-coasters
-pads
-queenbee
-sge-local
-ssh
-ssh-pbs-coasters
-surveyor
------
-
-You will notice that the templates can be specific to a particular set
-of machines like Intrepid and Queenbee, or they may be more general and
-aim to work across a variety of machines, as in the case of
-local-pbs-coasters. Gensites will look in three directories for available
-templates: your current directory, $SWIFT_HOME/etc/sites and $HOME/.swift/sites.
-
-Listing the Template
---------------------
-To view the contents of a template, type:
-
------
-$ gensites -l templatename
------
-
-Running this command will print some information about the template and give
-you an idea of what settings you will need to specify. You should see
-something like this:
-
------
-Description: Template for Intrepid. More information about this system can be found at http://www.alcf.anl.gov/support/gettingstarted/index.php
-Required Tokens: host work project queue
------
-
-The required tokens are required to properly use the templates. These are placeholder
-values you will need to specify in the
-following steps.
-
-Providing Site Specific Values
-------------------------------
-
-The gensites script needs to know how to replace the placeholder values
-in the template. This is done by configuring the swift.properties file.
-Gensites will first look for a swift.properties file in the current
-directory. If it does not exist, it will next look in $HOME/.swift.
-
-To add site specific values to swift.properties, add a line in the
-follow format:
-
------
-#site templatename setting=value
------
-
-Continuing the previous example, here is what you could add to
-swift.properties to replace the values of project, queue and work for
-the surveyor template:
-
------
-#site surveyor project=MyProject
-#site surveyor queue=MyQueue
-#site surveyor work=/path/to/workdir
------
-
-Now, running the command 'gensites surveyor' will produce the following
-valid configuration file:
-
------
-<config>
- <pool handle="surveyor">
- <filesystem provider="local" />
- <execution provider="coaster" jobmanager="local:cobalt"/>
- <profile namespace="globus" key="project">MyProject</profile>
- <profile namespace="globus" key="queue">MyQueue</profile>
- <profile namespace="globus" key="kernelprofile">zeptoos</profile>
- <profile namespace="globus" key="alcfbgpnat">true</profile>
- <profile namespace="karajan" key="jobthrottle">21</profile>
- <profile namespace="karajan" key="initialScore">10000</profile>
- <profile namespace="globus" key="workersPerNode">1</profile>
- <profile namespace="globus" key="workerLoggingLevel">DEBUG</profile>
- <profile namespace="globus" key="slots">1</profile>
- <profile namespace="globus" key="maxTime">900</profile>
- <profile namespace="globus" key="nodeGranularity">64</profile>
- <profile namespace="globus" key="maxNodes">64</profile>
- <workdirectory>/path/to/workdir</workdirectory>
- </pool>
-</config>
------
-
-Generating Application Configurations with Gensites
----------------------------------------------------
-Gensites can also be used to create a valid application catalog, commonly
-called tc.data. Here are some examples of how to specify applications within
-your swift.properties file:
-
------
-#app intrepid echo=/usr/bin/echo
------
-
-This first example shows a site specific application. The #app definition
-tells gensites this is related to an application rather than a #site
-definition. In the second part, echo=/usr/bin/echo, the left hand side
-is the name of the application that will be called from within swift. The
-right hand site is the path name which points to the binary.
-
------
-#app intrepid echo=$HOME/bin/echo
------
-
-Environment variables will be interpreted and converted to full path names
-for Swift.
-
------
-#app intrepid echo=bin/echo
------
-
-Gensites can take relative paths (relative to your current directory) and
-translate them to full path names for Swift.
-
-WARNING: Running gensites with #app definitions will replace any file
-called tc.data in your current directory. If a file called tc.data exists,
-it will be renamed to tc.data.old. If you run gensites twice, the original
-contents of your tc.data will be lost. Please either rename your tc file
-or copy to a different location.
-
-Running Swift With the New Configuration
-----------------------------------------
-
-Now that the gensites is configured and producing a valid configuration
-file, Swift needs to know to use it. The first step is to create a
-unique config file based on the preferences you specified.
-
------
-$ gensites surveyor > myconfig.xml
------
-
-This will send the output of gensites to myconfig.xml. This example will
-use a swift.properties location in the default directories (your current
-directory, ~/.swift/swift.properties). To specify a different location
-to the swift.properties, use:
-
------
-$ gensites surveyor -p myswift.properties > myconfig.xml
------
-
-Next, provide the configuration filename to swift:
------
-$ swift -sites.file myconfig.xml myscript.swift
------
-
-Alternatively, if you have specified applications, be sure to load that into
-Swift
------
-$ swift -sites.file myconfig.xml -tc.file tc.data mycript.swift
------
-
-Providing Default Values for All Templates
-------------------------------------------
-It is also possible to specify a default value for a setting, regardless
-of template you use. If you want to set your queue to default to "fast"
-across all templates, you can do this by omitting the template name.
-Consider the following swift.properties:
-
------
-#site queue=fast
-#site surveyor project=MyProject
-#site surveyor work=/path/to/workdir
------
-By omitting the template name, the default value for queue on surveyor
-(and any other template you use) will be set to "fast". One thing to
-keep in mind when setting default values is that order matters. Be sure
-to set your default values first before setting template specific values.
-
-Just like the #site definitions, when a site name is not specified, an app
-will be created for every site that is defined in your template.
------
-#app echo=/bin/echo
------
-
-More Help
----------
-The gensites script provides additional options not discussed here, such
-as using templates and swift.properties in non-standard directories. For
-more information, run gensites -h. Here is a full list of all options
-available.
-
------
-$ gensites -help
-
- usage: gensites template [-p properties.file] [-L template_directory] [-h] [-T] [-l]
-
- template Name of template to use
- -p properties.file Specify a swift.properties to use
- -L template_directory Specify a non-standard template directory
- -T List all templates available
- -h Help / usage information
- -l List the contents of a specific template
-
- Examples:
-
- Create a site configuration file for sites.xml using default properties.file in current directory
- $ gensites pads > sites.xml
-
- Use a specific properties file for a site
- $ gensites -p sites.properties pads > sites.xml
-
- Specify a non-standard directory where templates are located
- $ gensites -L template.dir pads > sites.xml
------
Copied: trunk/docs/utils/gensites.txt (from rev 5078, trunk/docs/utils/gensites.txt)
===================================================================
--- trunk/docs/utils/gensites.txt (rev 0)
+++ trunk/docs/utils/gensites.txt 2011-09-08 00:53:06 UTC (rev 5086)
@@ -0,0 +1,235 @@
+Gensites
+==========
+:toc:
+:icons:
+:website: http://www.ci.uchicago.edu/swift
+:numbered:
+
+Overview
+--------
+To simplify this configuration process, versions of Swift starting with
+0.92 include a utility called gensites. The gensites command is used
+to generate a sites.xml file for running a swift workflow on a given
+site. It accomplishes this by using a series of templates. The templates
+used by gensites are the same templates used for internal testing, so
+they are likely up to date and known to work on a given site.
+
+
+Viewing All Available Site Templates
+------------------------------------
+To view a list of all available templates, run the following command:
+
+-----
+$ gensites -T
+-----
+
+You should see output similar to this:
+-----
+intrepid
+local
+local-pbs-coasters
+pads
+queenbee
+sge-local
+ssh
+ssh-pbs-coasters
+surveyor
+-----
+
+You will notice that the templates can be specific to a particular set
+of machines like Intrepid and Queenbee, or they may be more general and
+aim to work across a variety of machines, as in the case of
+local-pbs-coasters. Gensites will look in three directories for available
+templates: your current directory, $SWIFT_HOME/etc/sites and $HOME/.swift/sites.
+
+Listing the Template
+--------------------
+To view the contents of a template, type:
+
+-----
+$ gensites -l templatename
+-----
+
+Running this command will print some information about the template and give
+you an idea of what settings you will need to specify. You should see
+something like this:
+
+-----
+Description: Template for Intrepid. More information about this system can be found at http://www.alcf.anl.gov/support/gettingstarted/index.php
+Required Tokens: host work project queue
+-----
+
+The required tokens are required to properly use the templates. These are placeholder
+values you will need to specify in the
+following steps.
+
+Providing Site Specific Values
+------------------------------
+
+The gensites script needs to know how to replace the placeholder values
+in the template. This is done by configuring the swift.properties file.
+Gensites will first look for a swift.properties file in the current
+directory. If it does not exist, it will next look in $HOME/.swift.
+
+To add site specific values to swift.properties, add a line in the
+follow format:
+
+-----
+#site templatename setting=value
+-----
+
+Continuing the previous example, here is what you could add to
+swift.properties to replace the values of project, queue and work for
+the surveyor template:
+
+-----
+#site surveyor project=MyProject
+#site surveyor queue=MyQueue
+#site surveyor work=/path/to/workdir
+-----
+
+Now, running the command 'gensites surveyor' will produce the following
+valid configuration file:
+
+-----
+<config>
+ <pool handle="surveyor">
+ <filesystem provider="local" />
+ <execution provider="coaster" jobmanager="local:cobalt"/>
+ <profile namespace="globus" key="project">MyProject</profile>
+ <profile namespace="globus" key="queue">MyQueue</profile>
+ <profile namespace="globus" key="kernelprofile">zeptoos</profile>
+ <profile namespace="globus" key="alcfbgpnat">true</profile>
+ <profile namespace="karajan" key="jobthrottle">21</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ <profile namespace="globus" key="workersPerNode">1</profile>
+ <profile namespace="globus" key="workerLoggingLevel">DEBUG</profile>
+ <profile namespace="globus" key="slots">1</profile>
+ <profile namespace="globus" key="maxTime">900</profile>
+ <profile namespace="globus" key="nodeGranularity">64</profile>
+ <profile namespace="globus" key="maxNodes">64</profile>
+ <workdirectory>/path/to/workdir</workdirectory>
+ </pool>
+</config>
+-----
+
+Generating Application Configurations with Gensites
+---------------------------------------------------
+Gensites can also be used to create a valid application catalog, commonly
+called tc.data. Here are some examples of how to specify applications within
+your swift.properties file:
+
+-----
+#app intrepid echo=/usr/bin/echo
+-----
+
+This first example shows a site specific application. The #app definition
+tells gensites this is related to an application rather than a #site
+definition. In the second part, echo=/usr/bin/echo, the left hand side
+is the name of the application that will be called from within swift. The
+right hand site is the path name which points to the binary.
+
+-----
+#app intrepid echo=$HOME/bin/echo
+-----
+
+Environment variables will be interpreted and converted to full path names
+for Swift.
+
+-----
+#app intrepid echo=bin/echo
+-----
+
+Gensites can take relative paths (relative to your current directory) and
+translate them to full path names for Swift.
+
+WARNING: Running gensites with #app definitions will replace any file
+called tc.data in your current directory. If a file called tc.data exists,
+it will be renamed to tc.data.old. If you run gensites twice, the original
+contents of your tc.data will be lost. Please either rename your tc file
+or copy to a different location.
+
+Running Swift With the New Configuration
+----------------------------------------
+
+Now that the gensites is configured and producing a valid configuration
+file, Swift needs to know to use it. The first step is to create a
+unique config file based on the preferences you specified.
+
+-----
+$ gensites surveyor > myconfig.xml
+-----
+
+This will send the output of gensites to myconfig.xml. This example will
+use a swift.properties location in the default directories (your current
+directory, ~/.swift/swift.properties). To specify a different location
+to the swift.properties, use:
+
+-----
+$ gensites surveyor -p myswift.properties > myconfig.xml
+-----
+
+Next, provide the configuration filename to swift:
+-----
+$ swift -sites.file myconfig.xml myscript.swift
+-----
+
+Alternatively, if you have specified applications, be sure to load that into
+Swift
+-----
+$ swift -sites.file myconfig.xml -tc.file tc.data mycript.swift
+-----
+
+Providing Default Values for All Templates
+------------------------------------------
+It is also possible to specify a default value for a setting, regardless
+of template you use. If you want to set your queue to default to "fast"
+across all templates, you can do this by omitting the template name.
+Consider the following swift.properties:
+
+-----
+#site queue=fast
+#site surveyor project=MyProject
+#site surveyor work=/path/to/workdir
+-----
+By omitting the template name, the default value for queue on surveyor
+(and any other template you use) will be set to "fast". One thing to
+keep in mind when setting default values is that order matters. Be sure
+to set your default values first before setting template specific values.
+
+Just like the #site definitions, when a site name is not specified, an app
+will be created for every site that is defined in your template.
+-----
+#app echo=/bin/echo
+-----
+
+More Help
+---------
+The gensites script provides additional options not discussed here, such
+as using templates and swift.properties in non-standard directories. For
+more information, run gensites -h. Here is a full list of all options
+available.
+
+-----
+$ gensites -help
+
+ usage: gensites template [-p properties.file] [-L template_directory] [-h] [-T] [-l]
+
+ template Name of template to use
+ -p properties.file Specify a swift.properties to use
+ -L template_directory Specify a non-standard template directory
+ -T List all templates available
+ -h Help / usage information
+ -l List the contents of a specific template
+
+ Examples:
+
+ Create a site configuration file for sites.xml using default properties.file in current directory
+ $ gensites pads > sites.xml
+
+ Use a specific properties file for a site
+ $ gensites -p sites.properties pads > sites.xml
+
+ Specify a non-standard directory where templates are located
+ $ gensites -L template.dir pads > sites.xml
+-----
More information about the Swift-commit
mailing list