[Swift-commit] r4544 - in trunk: docs docs/newuser-guide etc/sites examples tests/providers/local-pbs/pads
davidk at ci.uchicago.edu
davidk at ci.uchicago.edu
Thu Jun 2 16:27:04 CDT 2011
Author: davidk
Date: 2011-06-02 16:27:03 -0500 (Thu, 02 Jun 2011)
New Revision: 4544
Added:
trunk/docs/newuser-guide/
trunk/docs/newuser-guide/pads-quickstart.txt
trunk/examples/catsn.swift
Removed:
trunk/etc/sites/coaster-service.conf
Modified:
trunk/tests/providers/local-pbs/pads/sites.template.xml
Log:
Updated PADS pbs-local template and copied to etc/sites
Added catsn.swift to examples directory
Added start of new user guide
Added: trunk/docs/newuser-guide/pads-quickstart.txt
===================================================================
--- trunk/docs/newuser-guide/pads-quickstart.txt (rev 0)
+++ trunk/docs/newuser-guide/pads-quickstart.txt 2011-06-02 21:27:03 UTC (rev 4544)
@@ -0,0 +1,188 @@
+PADS Quickstart Guide
+=====================
+
+Requesting Access
+-----------------
+If you do not already have a Computation Institute account, you can request
+one at https://www.ci.uchicago.edu/accounts/. This page will give you a list
+of resources you can request access to. Be sure that PADS is selected. If
+you already have an existing CI account, but do not have access to PADS,
+send an email to support at ci.uchicago.edu to request access.
+
+SSH Keys
+--------
+Before accessing PADS, be sure to have your SSH keys configured correctly.
+There is some basic information about SSH and how to generate your key at
+http://www.ci.uchicago.edu/wiki/bin/view/Resources/SshKeys. Once you have
+followed those instructions, you can add your key at
+https://www.ci.uchicago.edu/support/sshkeys/.
+
+Connecting to a login node
+--------------------------
+Once your keys are configured, you should be able to access a PADS login
+node with the following command:
+
+-----
+ssh yourusername at login.pads.ci.uchicago.edu
+-----
+
+Configuring softenv
+-------------------
+Softenv is a system used for managing applications. In order to run Swift,
+the softenv environment will have to be modified slightly. Softenv is
+configured by a file in your home directory called .soft. Edit this file
+to look like this:
+-----
++java-sun
++maui
++torque
+ at default
+-----
+
+Downloading and building swift
+------------------------------
+The most recent versions of Swift can be found at
+http://www.ci.uchicago.edu/swift/downloads/index.php. Follow the instructions
+provided on that site to download and build Swift.
+
+Adding Swift to your PATH
+-------------------------
+Once you have installed Swift, add the Swift binary to your PATH so you can
+easily run it from any directory.
+
+In your home directory, edit the file ".bashrc".
+
+If you have installed Swift via a source repository, add the following line
+at the bottom of .bashrc.
+
+-----
+export PATH=$PATH:$HOME/cog/modules/swift/dist/swift-svn/bin
+-----
+
+If you have installed Swift via a binary package, add this line:
+
+-----
+export PATH=$PATH:$HOME/swift-<version>/bin
+-----
+
+Replace <version> with the actual name of the swift directory in the example above.
+
+What You Need To Know Before Running Swift
+------------------------------------------
+Before you can create a Swift configuration file, there are some things
+you will need to know.
+
+Swift Work Directory
+~~~~~~~~~~~~~~~~~~~~
+The Swift work directory is a directory which Swift uses for processing work.
+This directory needs to be writable. Common options for this are:
+
+-----
+/home/username/swiftwork
+/home/username/work
+/tmp
+-----
+
+Which project(s) are you a member of?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+PADS requires that you are a member of a project. You can determine this by
+running the following command:
+
+-----
+$ projects --available
+
+The following projects are available for your use
+
+Project PI Title
+
+CI-CCR000013 Michael Wilde The Swift Parallel Scripting System
+-----
+
+If you are not a member of a project, you must first request access
+to a project at http://www.ci.uchicago.edu/hpc/projects.
+
+You should make sure that you have a project set as default. Run
+the projects command with no arguments to determine if you have a default.
+
+------
+$ projects
+You have no default project set.
+-----
+
+To set your default project, use projects --set
+------
+$ projects --set CI-CCR000013 --all
+Your default project for all CI clusters has been set to CI-CCR000013.
+-----
+
+Determine your Queue
+~~~~~~~~~~~~~~~~~~~~
+
+PADS has several different queues you can submit jobs to depending on
+the type of work you will be doing. The command "qstat -q" will print
+the most up to date list of this information.
+
+.PADS Queues
+[options="header"]
+|=========================================================
+|Queue |Memory|CPU Time|Walltime|Node|Run|Que|Lm |State
+|route |-- |-- |-- |-- | 0| 0|-- | E R
+|short |-- |-- |04:00:00|-- | 64| 0|-- | E R
+|extended|-- |-- |-- |-- | 0| 0|-- | E R
+|fast |-- |-- |01:00:00|1 | 0|152|-- | E R
+|long |-- |-- |24:00:00|-- |232|130|-- | E R
+|=========================================================
+
+Generating Configuration Files
+------------------------------
+Now that you know what queue to use, your project, and your work directory, it is time to
+set up Swift. Swift uses a configuration file called sites.xml to determine how it should run.
+There are two methods you can use for creating this file. You can manually edit
+the configuration file, or generate it with a utility called gensites.
+
+Manually Editing sites.xml
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+Below is the template that is used by Swift's test suite for running on PADS.
+
+-----
+include::../../tests/providers/local-pbs/pads/sites.template.xml[]
+-----
+
+The values to note here are the ones that are listed between underscores. In the example above, they are _QUEUE_, and _WORK_. Queue is the PADS queue to use and WORK is the swift work directory. These are placeholder values you will need to modify to fit your needs. Copy and paste this template, replace the values, and call it sites.xml.
+
+Manually Editing tc.data
+~~~~~~~~~~~~~~~~~~~~~~~~
+Below is the tc.data file used by Swift's test suite for running on PADS.
+
+-----
+include::../../tests/providers/local-pbs/pads/tc.template.data[]
+------
+
+Copy these commands and save it as tc.data.
+
+Catsn.swift
+-----------
+The swift script we will run is called catsn.swift. It simply cats a file and saves the result. This is a nice simple test to ensure jobs are running correctly. Create a file called data.txt which contains some simple input - a "hello world" will do the trick.
+
+-----
+include::../../examples/catsn.swift[]
+-----
+
+Running Swift
+-------------
+Now that everything is in place, run Swift with the following command:
+
+-----
+swift -sites.file sites.xml -tc.file tc.data catsn.swift -n=10
+-----
+
+You should see several new files being created, called catsn.0001.out, catsn.0002.out, etc. Each of these
+files should contain the contents of what you placed into data.txt. If this happens, your job has run
+successfully on PADS!
+
+More Help
+---------
+The best place for additional help is the Swift user mailing list. You can subscribe to this list at
+http://mail.ci.uchicago.edu/mailman/listinfo/swift-user. When submitting information, please send your sites.xml file, your tc.data, and any Swift log files that were created during your attempt.
+
+
Deleted: trunk/etc/sites/coaster-service.conf
===================================================================
--- trunk/etc/sites/coaster-service.conf 2011-06-01 22:15:27 UTC (rev 4543)
+++ trunk/etc/sites/coaster-service.conf 2011-06-02 21:27:03 UTC (rev 4544)
@@ -1,31 +0,0 @@
-# Keep all interesting settings in one place
-# User should modify this to fit environment
-
-# Location of SWIFT. If empty, PATH is referenced
-export SWIFT=
-
-# Where to place/launch worker.pl on the remote machine for sites.xml
-export WORKER_WORK=/home/${USER}/work
-
-# How to launch workers- local or ssh
-export WORKER_MODE=ssh
-
-# Worker logging setting passed to worker.pl for sites.xml
-export WORKER_LOGGING=INFO
-
-# Worker host names for ssh
-export WORKERS_HOSTS="crush thwomp stomp crush crank steamroller grind churn trounce thrash vanquish octagon octopus"
-
-# Directory to keep log files, relative to working directory when launching start-coaster-service
-export LOG_DIR=logs
-
-# Manually define ports. If not specified, ports will be automatically generated
-export LOCAL_PORT=
-export SERVICE_PORT=
-
-# Below are various settings to give information about how to create sites.xml
-# These must remain as comments in order for gensites to recognize them
-#site work=$HOME/work
-#site queue=prod-devel
-#site maxtime=20
-#site nodes=64
Added: trunk/examples/catsn.swift
===================================================================
--- trunk/examples/catsn.swift (rev 0)
+++ trunk/examples/catsn.swift 2011-06-02 21:27:03 UTC (rev 4544)
@@ -0,0 +1,15 @@
+type file;
+
+app (file o) cat (file i)
+{
+ cat @i stdout=@o;
+}
+
+string t = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
+string char[] = @strsplit(t, "");
+
+file out[]<simple_mapper; location=".", prefix="catsn.",suffix=".out">;
+foreach j in [1:@toint(@arg("n","10"))] {
+ file data<"data.txt">;
+ out[j] = cat(data);
+}
Modified: trunk/tests/providers/local-pbs/pads/sites.template.xml
===================================================================
--- trunk/tests/providers/local-pbs/pads/sites.template.xml 2011-06-01 22:15:27 UTC (rev 4543)
+++ trunk/tests/providers/local-pbs/pads/sites.template.xml 2011-06-02 21:27:03 UTC (rev 4544)
@@ -3,6 +3,7 @@
<execution jobmanager="local:pbs" provider="local" url="none" />
<filesystem provider="local" url="none" />
<profile key="jobThrottle" namespace="karajan">0</profile>
+ <profile namespace="globus" key="project">_QUEUE_</profile>
<workdirectory>_WORK_</workdirectory>
</pool>
</config>
More information about the Swift-commit
mailing list