[Swift-commit] r5034 - in branches/release-0.93/docs: . siteguide tutorial userguide utils
davidk at ci.uchicago.edu
davidk at ci.uchicago.edu
Wed Aug 31 13:43:39 CDT 2011
Author: davidk
Date: 2011-08-31 13:43:39 -0500 (Wed, 31 Aug 2011)
New Revision: 5034
Added:
branches/release-0.93/docs/siteguide/prereqs
branches/release-0.93/docs/utils/
branches/release-0.93/docs/utils/gensites.txt
Modified:
branches/release-0.93/docs/siteguide/pads
branches/release-0.93/docs/siteguide/siteguide.txt
branches/release-0.93/docs/tutorial/runtime_features
branches/release-0.93/docs/userguide/coasters
Log:
Added gensites documentation to 0.93
Added prereqs page to siteguide
Added a PADS quickstart
Some tutorial and userguide cleanup
Modified: branches/release-0.93/docs/siteguide/pads
===================================================================
--- branches/release-0.93/docs/siteguide/pads 2011-08-31 17:37:41 UTC (rev 5033)
+++ branches/release-0.93/docs/siteguide/pads 2011-08-31 18:43:39 UTC (rev 5034)
@@ -1,10 +1,61 @@
PADS
----
+PADS Quickstart
+~~~~~~~~~~~~~~~
+This section will walk you through running a simple Swift script
+on PADS.
+
+In your home directory, edit a file called .soft and add these
+lines (in this order):
+-----
++java-sun
++maui
++torque
+ at default
+-----
+
+Log out of PADS, and log back in.
+
+Next, paste the example below into a file called sites.xml.
+
+-----
+include::../../tests/providers/PADS/coasters/sites.template.xml[]
+-----
+
+Create a directory called swiftwork. Modify _WORK_ in sites.xml
+to point to this new directory. For example
+-----
+<workdirectory>/home/myhome/swiftwork</workdirectory>
+-----
+
+Next, paste the following example into a file called tc.data
+
+-----
+include::../../tests/providers/PADS/coasters/tc.template.data[]
+-----
+
+Copy the catsn Swift script and input text file into your current directory
+-----
+$ cp ~/swift-0.93/examples/misc/catsn.swift .
+$ cp ~/swift-0.93/examples/misc/data.txt .
+-----
+TIP: The location of your swift directory may vary depending on how you installed it. Change this to the examples/misc directory of your installation as needed.
+
+And finally, run the script
+-----
+$ swift -sites.file sites.xml -tc.file tc.data catsn.swift
+-----
+
+You should see 10 new text files get created, named catsn*.out. If
+you see these files, then you have succesfully run Swift on PADS!
+
+TIP: Make sure your default project is defined. Read on for more information.
+
Requesting Access
~~~~~~~~~~~~~~~~~
If you do not already have a Computation Institute account, you can request
-one at https://www.ci.uchicago.edu/accounts/. This page will give you a list
+one at https://www.ci.uchicago.edu/accounts. This page will give you a list
of resources you can request access to. Be sure that PADS is selected. If
you already have an existing CI account, but do not have access to PADS,
send an email to support at ci.uchicago.edu to request access.
@@ -39,34 +90,6 @@
@default
-----
-Downloading and building Swift
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The most recent versions of Swift can be found at
-http://www.ci.uchicago.edu/swift/downloads/index.php. Follow the instructions
-provided on that site to download and build Swift.
-
-Adding Swift to your PATH
-~~~~~~~~~~~~~~~~~~~~~~~~~
-Once you have installed Swift, add the Swift binary to your PATH so you can
-easily run it from any directory.
-
-In your home directory, edit the file ".bashrc".
-
-If you have installed Swift via a source repository, add the following line
-at the bottom of .bashrc.
-
------
-export PATH=$PATH:$HOME/cog/modules/swift/dist/swift-svn/bin
------
-
-If you have installed Swift via a binary package, add this line:
-
------
-export PATH=$PATH:$HOME/swift-<version>/bin
------
-
-Replace <version> with the actual name of the swift directory in the example above.
-
What You Need To Know Before Running Swift
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Before you can create a Swift configuration file, there are some things
@@ -145,17 +168,18 @@
Below is the template that is used by Swift's test suite for running on PADS.
-----
-include::../../tests/providers/local-pbs/pads/sites.template.xml[]
+include::../../tests/providers/PADS/coasters/sites.template.xml[]
-----
-The values to note here are the ones that are listed between underscores. In the example above, they are _QUEUE_, and _WORK_. Queue is the PADS queue to use and WORK is the swift work directory. These are placeholder values you will need to modify to fit your needs. Copy and paste this template, replace the values, and call it sites.xml.
+Before you can use this configuration file, you will need to modify _WORK_ to a directory writable by you,
+and have your default project defined. Copy this template, replace _WORK_, and then save as sites.xml.
Manually Editing tc.data
~~~~~~~~~~~~~~~~~~~~~~~~
Below is the tc.data file used by Swift's test suite for running on PADS.
-----
-include::../../tests/providers/local-pbs/pads/tc.template.data[]
+include::../../tests/providers/PADS/coasters/tc.template.data[]
------
Copy these commands and save it as tc.data.
Added: branches/release-0.93/docs/siteguide/prereqs
===================================================================
--- branches/release-0.93/docs/siteguide/prereqs (rev 0)
+++ branches/release-0.93/docs/siteguide/prereqs 2011-08-31 18:43:39 UTC (rev 5034)
@@ -0,0 +1,8 @@
+Prerequisites
+-------------
+
+This guide assumes that you have already downloaded and installed Swift.
+It assumes that Swift is in your PATH and that you have a working
+version of Sun Java 1.5+. For more information on downloading and
+installing Swift, please see the
+http://www.ci.uchicago.edu/swift/guides/release-0.93/quickstart/quickstart.html[Swift Quickstart Guide].
Modified: branches/release-0.93/docs/siteguide/siteguide.txt
===================================================================
--- branches/release-0.93/docs/siteguide/siteguide.txt 2011-08-31 17:37:41 UTC (rev 5033)
+++ branches/release-0.93/docs/siteguide/siteguide.txt 2011-08-31 18:43:39 UTC (rev 5034)
@@ -6,6 +6,8 @@
:website: http://www.ci.uchicago.edu/swift/guides/siteguide.php
:numbered:
+include::prereqs[]
+
include::pads[]
include::beagle[]
Modified: branches/release-0.93/docs/tutorial/runtime_features
===================================================================
--- branches/release-0.93/docs/tutorial/runtime_features 2011-08-31 17:37:41 UTC (rev 5033)
+++ branches/release-0.93/docs/tutorial/runtime_features 2011-08-31 18:43:39 UTC (rev 5034)
@@ -22,167 +22,10 @@
examples, we've invoked echo and tr executables from our SwiftScript
program. These have been run on the local system (the same computer on
which you ran swift). We can also make our computations run on a
-remote resource.
+remote resource. For more information on running Swift on a remote site
+please see the http://www.ci.uchicago.edu/swift/guides/release-0.93/siteguide/siteguide.html[Site Configuration Guide].
-WARNING: This example is necessarily more vague than previous ones,
-because it requires access to remote resources. You should ensure that
-you can submit a job using the globus-job-run (or globusrun-ws)
-command(s).
-We do not need to modify any SwiftScript code to run on another
-resource. Instead, we must modify another catalog, the site catalog.
-This catalog provides details of the location that applications will be
-run, with the default settings referring to the local machine. We will
-modify it to refer to a remote resource - the UC Teraport cluster. If
-you are not a UC Teraport user, you should use details of a different
-resource that you do have access to.
-
-The site catalog is located in etc/sites.xml and is a relatively
-straightforward XML format file. We must modify each of the following
-three settings: gridftp (which indicates how and where data can be
-transferred to the remote resource), jobmanager (which indicates how
-applications can be run on the remote resource) and workdirectory (which
-indicates where working storage can be found on the remote resource).
-
-
-Writing a mapper
-~~~~~~~~~~~~~~~~
-
-This section will introduce writing a custom mapper so that Swift is
-able to access data files laid out in application-specific ways.
-
-An application-specific mapper must take the form of a Java class that
-implements the http://www.ci.uchicago.edu/swift/javadoc/vdsk/org/griphyn/vdl/mapping/Mapper.html[Mapper
-interface].
-
-Usually you don't need to implement this interface directly, because
-Swift provides a number of more concrete classes with some functionality
-already implemented.
-
-The hierarchy of helper classes is:
-
-http://www.ci.uchicago.edu/swift/javadoc/vdsk/org/griphyn/vdl/mapping/Mapper.html[Mapper]
-
-This is the abstract interface for mappers in Swift. You must
-implement methods to provide access to mapper properties, to map from a
-SwiftScript dataset path (such as foo[1].bar) to a file name, to check
-whether a file exists. None of the default Swift mappers implement this
-interface directly - instead they use one of the following helper classes.
-
-http://www.ci.uchicago.edu/swift/javadoc/vdsk/org/griphyn/vdl/mapping/AbstractMapper.html[Abstract Mapper]
-
-This provides helper methods to manage mapper properties and to handle
-existance checking. Examples of mappers which use this class are:
-http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.array_mapper[Array Mapper],
-http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.csv_mapper[CSV Mapper],
-http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.fixed_array_mapper[Fixed Array Mapper],
-http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.regexp_mapper[Regexp Mapper],
-and http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.single_file_mapper[Single File Mapper].
-
-http://www.ci.uchicago.edu/swift/javadoc/vdsk/org/griphyn/vdl/mapping/file/AbstractFileMapper.html[AbstractFileMapper]
-This provides a helper class for mappers which select files based on
-selecting files from a directory listing. It is necessary to write some
-helper methods that are different from the above mapper methods.
-Examples of mappers which use this class are:
-http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.simple_mapper[simple_mapper],
-http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.filesys_mapper[filesys_mapper],
-and the StructuredRegularExpressionMapper.
-
-In general, to write a mapper, choose either the AbstractMapper or the
-AbstractFileMapper and extend those. If your mapper will generally
-select the files it returns based on a directory listing and will
-convert paths to filenames using some regular conversion (for example,
-in the way that simple_mapper maps files in a directory that match a
-particular pattern), then you should probably use the
-AbstractFileMapper. If your mapper will produce a list of files in some
-other way (for example, in the way that csv_mapper maps based on
-filenames given in a CSV file rather than looking at which files are in
-a directory), then you should probably use the AbstractMapper.
-
-
-Writing a very basic mapper
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-In this section, we will write a very basic (almost useless) mapper that
-will map a SwiftScript dataset into a hardcoded file called
-myfile.txt, like this:
-
-
- Swift variable Filename
-
- var <-----------------------------> myfile.txt
-
-We should be able to use the mapper we write in a SwiftScript program
-like this:
-
-
-----
-type file;
-file f <my_first_mapper>;
-----
-
-First we must choose a base class - AbstractMapper or
-AbstractFileMapper. We aren't going to use a directory listing to decide
-on our mapping - we are getting the mapping from some other source (in
-fact, it will be hard coded). So we will use AbstractMapper.
-
-So now onto the source code. We must define a subclass of AbstractMapper
-and implement several mapper methods: isStatic, existing, and map. These
-methods are documented in the javadoc for the Mapper interface.
-
-Here is the code implementing this mapper. Put this in your source
-vdsk directory, make a directory src/tutorial/|and put this file in
-src/tutorial/MyFirstMapper.java.
-
-.MyFirstMapper.java
-*******************
-----
-include::../../examples/tutorial/MyFirstMapper.java[]
-----
-*******************
-
-Now we need to inform the Swift engine about the existence of this
-mapper. We do that by editing the MapperFactory class definition, in
-src/org/griphyn/vdl/mapping/MapperFactory.java and adding a
-registerMapper call alongside the existing registerMapper calls, like this:
-
-
-----
-registerMapper("my_first_mapper", tutorial.MyFirstMapper.class);
-----
-
-The first parameter is the name of the mapper that will be used in
-SwiftScript program. The second parameter is the new Mapper class that
-we just wrote.
-
-Now rebuild Swift using the "ant redist" target.
-
-This new Swift build will be aware of your new mapper. We can test it
-out with a hello world program:
-
-.my_first_mapper.swift
-**********************
------
-include::../../examples/tutorial/my_first_mapper.swift[]
------
-**********************
-
-Run this program, and hopefully you will find the "hello" string has
-been output into the hard coded output file myfile.txt:
-
-
-----
-$ cat myfile.txt
-hello
-----
-
-So that's a first very simple mapper implemented. Compare the source
-code to the single_file_mapper in
-http://www.ci.uchicago.edu/trac/swift/browser/trunk/src/org/griphyn/vdl/mapping/file/SingleFileMapper.java[SingleFileMapper.java].
-There is not much more code to the single_file_mapper - mostly code to
-deal with the file parameter.
-
-
Starting and restarting
~~~~~~~~~~~~~~~~~~~~~~~
Modified: branches/release-0.93/docs/userguide/coasters
===================================================================
--- branches/release-0.93/docs/userguide/coasters 2011-08-31 17:37:41 UTC (rev 5033)
+++ branches/release-0.93/docs/userguide/coasters 2011-08-31 18:43:39 UTC (rev 5034)
@@ -1,7 +1,5 @@
Coasters
--------
-Coasters were introduced in Swift v0.6 as an experimental feature.
-
In many applications, Swift performance can be greatly enhanced by the
use of CoG coasters. CoG coasters provide a low-overhead job submission
and file transfer mechanism suited for the execution of short jobs (on
Added: branches/release-0.93/docs/utils/gensites.txt
===================================================================
--- branches/release-0.93/docs/utils/gensites.txt (rev 0)
+++ branches/release-0.93/docs/utils/gensites.txt 2011-08-31 18:43:39 UTC (rev 5034)
@@ -0,0 +1,235 @@
+Gensites
+==========
+:toc:
+:icons:
+:website: http://www.ci.uchicago.edu/swift
+:numbered:
+
+Overview
+--------
+To simplify this configuration process, versions of Swift starting with
+0.92 include a utility called gensites. The gensites command is used
+to generate a sites.xml file for running a swift workflow on a given
+site. It accomplishes this by using a series of templates. The templates
+used by gensites are the same templates used for internal testing, so
+they are likely up to date and known to work on a given site.
+
+
+Viewing All Available Site Templates
+------------------------------------
+To view a list of all available templates, run the following command:
+
+-----
+$ gensites -T
+-----
+
+You should see output similar to this:
+-----
+intrepid
+local
+local-pbs-coasters
+pads
+queenbee
+sge-local
+ssh
+ssh-pbs-coasters
+surveyor
+-----
+
+You will notice that the templates can be specific to a particular set
+of machines like Intrepid and Queenbee, or they may be more general and
+aim to work across a variety of machines, as in the case of
+local-pbs-coasters. Gensites will look in three directories for available
+templates: your current directory, $SWIFT_HOME/etc/sites and $HOME/.swift/sites.
+
+Listing the Template
+--------------------
+To view the contents of a template, type:
+
+-----
+$ gensites -l templatename
+-----
+
+Running this command will print some information about the template and give
+you an idea of what settings you will need to specify. You should see
+something like this:
+
+-----
+Description: Template for Intrepid. More information about this system can be found at http://www.alcf.anl.gov/support/gettingstarted/index.php
+Required Tokens: host work project queue
+-----
+
+The required tokens are required to properly use the templates. These are placeholder
+values you will need to specify in the
+following steps.
+
+Providing Site Specific Values
+------------------------------
+
+The gensites script needs to know how to replace the placeholder values
+in the template. This is done by configuring the swift.properties file.
+Gensites will first look for a swift.properties file in the current
+directory. If it does not exist, it will next look in $HOME/.swift.
+
+To add site specific values to swift.properties, add a line in the
+follow format:
+
+-----
+#site templatename setting=value
+-----
+
+Continuing the previous example, here is what you could add to
+swift.properties to replace the values of project, queue and work for
+the surveyor template:
+
+-----
+#site surveyor project=MyProject
+#site surveyor queue=MyQueue
+#site surveyor work=/path/to/workdir
+-----
+
+Now, running the command 'gensites surveyor' will produce the following
+valid configuration file:
+
+-----
+<config>
+ <pool handle="surveyor">
+ <filesystem provider="local" />
+ <execution provider="coaster" jobmanager="local:cobalt"/>
+ <profile namespace="globus" key="project">MyProject</profile>
+ <profile namespace="globus" key="queue">MyQueue</profile>
+ <profile namespace="globus" key="kernelprofile">zeptoos</profile>
+ <profile namespace="globus" key="alcfbgpnat">true</profile>
+ <profile namespace="karajan" key="jobthrottle">21</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ <profile namespace="globus" key="workersPerNode">1</profile>
+ <profile namespace="globus" key="workerLoggingLevel">DEBUG</profile>
+ <profile namespace="globus" key="slots">1</profile>
+ <profile namespace="globus" key="maxTime">900</profile>
+ <profile namespace="globus" key="nodeGranularity">64</profile>
+ <profile namespace="globus" key="maxNodes">64</profile>
+ <workdirectory>/path/to/workdir</workdirectory>
+ </pool>
+</config>
+-----
+
+Generating Application Configurations with Gensites
+---------------------------------------------------
+Gensites can also be used to create a valid application catalog, commonly
+called tc.data. Here are some examples of how to specify applications within
+your swift.properties file:
+
+-----
+#app intrepid echo=/usr/bin/echo
+-----
+
+This first example shows a site specific application. The #app definition
+tells gensites this is related to an application rather than a #site
+definition. In the second part, echo=/usr/bin/echo, the left hand side
+is the name of the application that will be called from within swift. The
+right hand site is the path name which points to the binary.
+
+-----
+#app intrepid echo=$HOME/bin/echo
+-----
+
+Environment variables will be interpreted and converted to full path names
+for Swift.
+
+-----
+#app intrepid echo=bin/echo
+-----
+
+Gensites can take relative paths (relative to your current directory) and
+translate them to full path names for Swift.
+
+WARNING: Running gensites with #app definitions will replace any file
+called tc.data in your current directory. If a file called tc.data exists,
+it will be renamed to tc.data.old. If you run gensites twice, the original
+contents of your tc.data will be lost. Please either rename your tc file
+or copy to a different location.
+
+Running Swift With the New Configuration
+----------------------------------------
+
+Now that the gensites is configured and producing a valid configuration
+file, Swift needs to know to use it. The first step is to create a
+unique config file based on the preferences you specified.
+
+-----
+$ gensites surveyor > myconfig.xml
+-----
+
+This will send the output of gensites to myconfig.xml. This example will
+use a swift.properties location in the default directories (your current
+directory, ~/.swift/swift.properties). To specify a different location
+to the swift.properties, use:
+
+-----
+$ gensites surveyor -p myswift.properties > myconfig.xml
+-----
+
+Next, provide the configuration filename to swift:
+-----
+$ swift -sites.file myconfig.xml myscript.swift
+-----
+
+Alternatively, if you have specified applications, be sure to load that into
+Swift
+-----
+$ swift -sites.file myconfig.xml -tc.file tc.data mycript.swift
+-----
+
+Providing Default Values for All Templates
+------------------------------------------
+It is also possible to specify a default value for a setting, regardless
+of template you use. If you want to set your queue to default to "fast"
+across all templates, you can do this by omitting the template name.
+Consider the following swift.properties:
+
+-----
+#site queue=fast
+#site surveyor project=MyProject
+#site surveyor work=/path/to/workdir
+-----
+By omitting the template name, the default value for queue on surveyor
+(and any other template you use) will be set to "fast". One thing to
+keep in mind when setting default values is that order matters. Be sure
+to set your default values first before setting template specific values.
+
+Just like the #site definitions, when a site name is not specified, an app
+will be created for every site that is defined in your template.
+-----
+#app echo=/bin/echo
+-----
+
+More Help
+---------
+The gensites script provides additional options not discussed here, such
+as using templates and swift.properties in non-standard directories. For
+more information, run gensites -h. Here is a full list of all options
+available.
+
+-----
+$ gensites -help
+
+ usage: gensites template [-p properties.file] [-L template_directory] [-h] [-T] [-l]
+
+ template Name of template to use
+ -p properties.file Specify a swift.properties to use
+ -L template_directory Specify a non-standard template directory
+ -T List all templates available
+ -h Help / usage information
+ -l List the contents of a specific template
+
+ Examples:
+
+ Create a site configuration file for sites.xml using default properties.file in current directory
+ $ gensites pads > sites.xml
+
+ Use a specific properties file for a site
+ $ gensites -p sites.properties pads > sites.xml
+
+ Specify a non-standard directory where templates are located
+ $ gensites -L template.dir pads > sites.xml
+-----
More information about the Swift-commit
mailing list