[Swift-commit] r5087 - in trunk/docs: . documentation siteguide tutorial tutorial/images userguide
davidk at ci.uchicago.edu
davidk at ci.uchicago.edu
Wed Sep 7 20:21:11 CDT 2011
Author: davidk
Date: 2011-09-07 20:21:11 -0500 (Wed, 07 Sep 2011)
New Revision: 5087
Added:
trunk/docs/documentation/
trunk/docs/quickstart/
trunk/docs/siteguide/intrepid
trunk/docs/siteguide/prereqs
trunk/docs/tutorial/images/icons/
trunk/docs/userguide/log-processing
Removed:
trunk/docs/documentation/building.html
Modified:
trunk/docs/siteguide/futuregrid
trunk/docs/siteguide/grid
trunk/docs/siteguide/pads
trunk/docs/siteguide/siteguide.txt
trunk/docs/tutorial/runtime_features
trunk/docs/userguide/coasters
trunk/docs/userguide/userguide.txt
Log:
Doc updates
Deleted: trunk/docs/documentation/building.html
===================================================================
--- branches/release-0.93/docs/documentation/building.html 2011-09-07 21:09:21 UTC (rev 5085)
+++ trunk/docs/documentation/building.html 2011-09-08 01:21:11 UTC (rev 5087)
@@ -1,576 +0,0 @@
-<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
- "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
-<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
-<head>
-<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" />
-<meta name="generator" content="AsciiDoc 8.6.3" />
-<title></title>
-<style type="text/css">
-/* Sans-serif font. */
-h1, h2, h3, h4, h5, h6,
-div.title, caption.title,
-thead, p.table.header,
-div#toctitle,
-span#author, span#revnumber, span#revdate, span#revremark,
-div#footer {
- font-family: Arial,Helvetica,sans-serif;
-}
-
-/* Serif font. */
-div.sectionbody {
- font-family: Georgia,"Times New Roman",Times,serif;
-}
-
-/* Monospace font. */
-tt {
- font-size: inherit;
-}
-
-body {
- margin: 1em 5% 1em 5%;
-}
-
-a {
- color: blue;
- text-decoration: underline;
-}
-a:visited {
- color: fuchsia;
-}
-
-em {
- font-style: italic;
- color: navy;
-}
-
-strong {
- font-weight: bold;
- color: #083194;
-}
-
-tt {
- font-size: inherit;
- color: navy;
-}
-
-h1, h2, h3, h4, h5, h6 {
- color: #527bbd;
- margin-top: 1.2em;
- margin-bottom: 0.5em;
- line-height: 1.3;
-}
-
-h1, h2, h3 {
- border-bottom: 2px solid silver;
-}
-h2 {
- padding-top: 0.5em;
-}
-h3 {
- float: left;
-}
-h3 + * {
- clear: left;
-}
-
-div.sectionbody {
- margin-left: 0;
-}
-
-hr {
- border: 1px solid silver;
-}
-
-p {
- margin-top: 0.5em;
- margin-bottom: 0.5em;
-}
-
-ul, ol, li > p {
- margin-top: 0;
-}
-ul > li { color: #aaa; }
-ul > li > * { color: black; }
-
-pre {
- padding: 0;
- margin: 0;
-}
-
-span#author {
- color: #527bbd;
- font-weight: bold;
- font-size: 1.1em;
-}
-span#email {
-}
-span#revnumber, span#revdate, span#revremark {
-}
-
-div#footer {
- font-size: small;
- border-top: 2px solid silver;
- padding-top: 0.5em;
- margin-top: 4.0em;
-}
-div#footer-text {
- float: left;
- padding-bottom: 0.5em;
-}
-div#footer-badges {
- float: right;
- padding-bottom: 0.5em;
-}
-
-div#preamble {
- margin-top: 1.5em;
- margin-bottom: 1.5em;
-}
-div.tableblock, div.imageblock, div.exampleblock, div.verseblock,
-div.quoteblock, div.literalblock, div.listingblock, div.sidebarblock,
-div.admonitionblock {
- margin-top: 1.0em;
- margin-bottom: 1.5em;
-}
-div.admonitionblock {
- margin-top: 2.0em;
- margin-bottom: 2.0em;
- margin-right: 10%;
- color: #606060;
-}
-
-div.content { /* Block element content. */
- padding: 0;
-}
-
-/* Block element titles. */
-div.title, caption.title {
- color: #527bbd;
- font-weight: bold;
- text-align: left;
- margin-top: 1.0em;
- margin-bottom: 0.5em;
-}
-div.title + * {
- margin-top: 0;
-}
-
-td div.title:first-child {
- margin-top: 0.0em;
-}
-div.content div.title:first-child {
- margin-top: 0.0em;
-}
-div.content + div.title {
- margin-top: 0.0em;
-}
-
-div.sidebarblock > div.content {
- background: #ffffee;
- border: 1px solid #dddddd;
- border-left: 4px solid #f0f0f0;
- padding: 0.5em;
-}
-
-div.listingblock > div.content {
- border: 1px solid #dddddd;
- border-left: 5px solid #f0f0f0;
- background: #f8f8f8;
- padding: 0.5em;
-}
-
-div.quoteblock, div.verseblock {
- padding-left: 1.0em;
- margin-left: 1.0em;
- margin-right: 10%;
- border-left: 5px solid #f0f0f0;
- color: #777777;
-}
-
-div.quoteblock > div.attribution {
- padding-top: 0.5em;
- text-align: right;
-}
-
-div.verseblock > pre.content {
- font-family: inherit;
- font-size: inherit;
-}
-div.verseblock > div.attribution {
- padding-top: 0.75em;
- text-align: left;
-}
-/* DEPRECATED: Pre version 8.2.7 verse style literal block. */
-div.verseblock + div.attribution {
- text-align: left;
-}
-
-div.admonitionblock .icon {
- vertical-align: top;
- font-size: 1.1em;
- font-weight: bold;
- text-decoration: underline;
- color: #527bbd;
- padding-right: 0.5em;
-}
-div.admonitionblock td.content {
- padding-left: 0.5em;
- border-left: 3px solid #dddddd;
-}
-
-div.exampleblock > div.content {
- border-left: 3px solid #dddddd;
- padding-left: 0.5em;
-}
-
-div.imageblock div.content { padding-left: 0; }
-span.image img { border-style: none; }
-a.image:visited { color: white; }
-
-dl {
- margin-top: 0.8em;
- margin-bottom: 0.8em;
-}
-dt {
- margin-top: 0.5em;
- margin-bottom: 0;
- font-style: normal;
- color: navy;
-}
-dd > *:first-child {
- margin-top: 0.1em;
-}
-
-ul, ol {
- list-style-position: outside;
-}
-ol.arabic {
- list-style-type: decimal;
-}
-ol.loweralpha {
- list-style-type: lower-alpha;
-}
-ol.upperalpha {
- list-style-type: upper-alpha;
-}
-ol.lowerroman {
- list-style-type: lower-roman;
-}
-ol.upperroman {
- list-style-type: upper-roman;
-}
-
-div.compact ul, div.compact ol,
-div.compact p, div.compact p,
-div.compact div, div.compact div {
- margin-top: 0.1em;
- margin-bottom: 0.1em;
-}
-
-div.tableblock > table {
- border: 3px solid #527bbd;
-}
-thead, p.table.header {
- font-weight: bold;
- color: #527bbd;
-}
-tfoot {
- font-weight: bold;
-}
-td > div.verse {
- white-space: pre;
-}
-p.table {
- margin-top: 0;
-}
-/* Because the table frame attribute is overriden by CSS in most browsers. */
-div.tableblock > table[frame="void"] {
- border-style: none;
-}
-div.tableblock > table[frame="hsides"] {
- border-left-style: none;
- border-right-style: none;
-}
-div.tableblock > table[frame="vsides"] {
- border-top-style: none;
- border-bottom-style: none;
-}
-
-
-div.hdlist {
- margin-top: 0.8em;
- margin-bottom: 0.8em;
-}
-div.hdlist tr {
- padding-bottom: 15px;
-}
-dt.hdlist1.strong, td.hdlist1.strong {
- font-weight: bold;
-}
-td.hdlist1 {
- vertical-align: top;
- font-style: normal;
- padding-right: 0.8em;
- color: navy;
-}
-td.hdlist2 {
- vertical-align: top;
-}
-div.hdlist.compact tr {
- margin: 0;
- padding-bottom: 0;
-}
-
-.comment {
- background: yellow;
-}
-
-.footnote, .footnoteref {
- font-size: 0.8em;
-}
-
-span.footnote, span.footnoteref {
- vertical-align: super;
-}
-
-#footnotes {
- margin: 20px 0 20px 0;
- padding: 7px 0 0 0;
-}
-
-#footnotes div.footnote {
- margin: 0 0 5px 0;
-}
-
-#footnotes hr {
- border: none;
- border-top: 1px solid silver;
- height: 1px;
- text-align: left;
- margin-left: 0;
- width: 20%;
- min-width: 100px;
-}
-
-div.colist td {
- padding-right: 0.5em;
- padding-bottom: 0.3em;
- vertical-align: top;
-}
-div.colist td img {
- margin-top: 0.3em;
-}
-
- at media print {
- div#footer-badges { display: none; }
-}
-
-div#toc {
- margin-bottom: 2.5em;
-}
-
-div#toctitle {
- color: #527bbd;
- font-size: 1.1em;
- font-weight: bold;
- margin-top: 1.0em;
- margin-bottom: 0.1em;
-}
-
-div.toclevel1, div.toclevel2, div.toclevel3, div.toclevel4 {
- margin-top: 0;
- margin-bottom: 0;
-}
-div.toclevel2 {
- margin-left: 2em;
- font-size: 0.9em;
-}
-div.toclevel3 {
- margin-left: 4em;
- font-size: 0.9em;
-}
-div.toclevel4 {
- margin-left: 6em;
- font-size: 0.9em;
-}
-
-</style>
-<script type="text/javascript">
-/*<+'])');
- // Function that scans the DOM tree for header elements (the DOM2
- // nodeIterator API would be a better technique but not supported by all
- // browsers).
- var iterate = function (el) {
- for (var i = el.firstChild; i != null; i = i.nextSibling) {
- if (i.nodeType == 1 /* Node.ELEMENT_NODE */) {
- var mo = re.exec(i.tagName);
- if (mo && (i.getAttribute("class") || i.getAttribute("className")) != "float") {
- result[result.length] = new TocEntry(i, getText(i), mo[1]-1);
- }
- iterate(i);
- }
- }
- }
- iterate(el);
- return result;
- }
-
- var toc = document.getElementById("toc");
- var entries = tocEntries(document.getElementById("content"), toclevels);
- for (var i = 0; i < entries.length; ++i) {
- var entry = entries[i];
- if (entry.element.id == "")
- entry.element.id = "_toc_" + i;
- var a = document.createElement("a");
- a.href = "#" + entry.element.id;
- a.appendChild(document.createTextNode(entry.text));
- var div = document.createElement("div");
- div.appendChild(a);
- div.className = "toclevel" + entry.toclevel;
- toc.appendChild(div);
- }
- if (entries.length == 0)
- toc.parentNode.removeChild(toc);
-},
-
-
-/////////////////////////////////////////////////////////////////////
-// Footnotes generator
-/////////////////////////////////////////////////////////////////////
-
-/* Based on footnote generation code from:
- * http://www.brandspankingnew.net/archive/2005/07/format_footnote.html
- */
-
-footnotes: function () {
- var cont = document.getElementById("content");
- var noteholder = document.getElementById("footnotes");
- var spans = cont.getElementsByTagName("span");
- var refs = {};
- var n = 0;
- for (i=0; i<spans.length; i++) {
- if (spans[i].className == "footnote") {
- n++;
- // Use [\s\S] in place of . so multi-line matches work.
- // Because JavaScript has no s (dotall) regex flag.
- note = spans[i].innerHTML.match(/\s*\[([\s\S]*)]\s*/)[1];
- noteholder.innerHTML +=
- "<div class='footnote' id='_footnote_" + n + "'>" +
- "<a href='#_footnoteref_" + n + "' title='Return to text'>" +
- n + "</a>. " + note + "</div>";
- spans[i].innerHTML =
- "[<a id='_footnoteref_" + n + "' href='#_footnote_" + n +
- "' title='View footnote' class='footnote'>" + n + "</a>]";
- var id =spans[i].getAttribute("id");
- if (id != null) refs["#"+id] = n;
- }
- }
- if (n == 0)
- noteholder.parentNode.removeChild(noteholder);
- else {
- // Process footnoterefs.
- for (i=0; i<spans.length; i++) {
- if (spans[i].className == "footnoteref") {
- var href = spans[i].getElementsByTagName("a")[0].getAttribute("href");
- href = href.match(/#.*/)[0]; // Because IE return full URL.
- n = refs[href];
- spans[i].innerHTML =
- "[<a href='#_footnote_" + n +
- "' title='View footnote' class='footnote'>" + n + "</a>]";
- }
- }
- }
-}
-
-}
-/*]]>*/
-</script>
-</head>
-<body class="article">
-<div id="header">
-</div>
-<div id="content">
-<div class="sect1">
-<h2 id="_prerequisites">Prerequisites</h2>
-<div class="sectionbody">
-<div class="paragraph"><p>There are several prerequisites for building documentation:</p></div>
-<div class="ulist"><ul>
-<li>
-<p>
-asciidoc and a2x (<a href="http://www.methods.co.nz/asciidoc">http://www.methods.co.nz/asciidoc</a>)
-</p>
-</li>
-<li>
-<p>
-dblatex (<a href="http://dblatex.sourceforge.net">http://dblatex.sourceforge.net</a>)
-</p>
-</li>
-<li>
-<p>
-source-highlight (<a href="http://www.gnu.org/software/src-highlite">http://www.gnu.org/software/src-highlite</a>)
-</p>
-</li>
-</ul></div>
-<div class="paragraph"><p>For CI users, compiled versions of these packages can be found in ~davidk.</p></div>
-<div class="sect2">
-<h3 id="_build_docs_script">Build Docs Script</h3>
-<div class="paragraph"><p>Before committing any changes to documentation, it is a good idea manually
-build the documentation. The script to build the documentation is
-docs/build_docs.sh.</p></div>
-<div class="paragraph"><p>This script will convert the documents into HTML and PDF, and then copy the
-files to a given output directory.</p></div>
-</div>
-</div>
-</div>
-</div>
-<div id="footnotes"><hr /></div>
-<div id="footer">
-<div id="footer-text">
-Last updated 2011-08-31 12:17:55 CDT
-</div>
-</div>
-</body>
-</html>
Modified: trunk/docs/siteguide/futuregrid
===================================================================
--- trunk/docs/siteguide/futuregrid 2011-09-08 00:53:06 UTC (rev 5086)
+++ trunk/docs/siteguide/futuregrid 2011-09-08 01:21:11 UTC (rev 5087)
@@ -1,5 +1,5 @@
-FutureGrid
-----------
+Futuregrid Quickstart Guide
+---------------------------
Downloading and Building Swift
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Modified: trunk/docs/siteguide/grid
===================================================================
--- trunk/docs/siteguide/grid 2011-09-08 00:53:06 UTC (rev 5086)
+++ trunk/docs/siteguide/grid 2011-09-08 01:21:11 UTC (rev 5087)
@@ -163,8 +163,8 @@
$
$ grep SUCCEEDED run.89/*/*.stdout
$
+-----
-
Starting a single coaster service
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Added: trunk/docs/siteguide/intrepid
===================================================================
--- trunk/docs/siteguide/intrepid (rev 0)
+++ trunk/docs/siteguide/intrepid 2011-09-08 01:21:11 UTC (rev 5087)
@@ -0,0 +1,168 @@
+Intrepid
+--------
+
+Requesting Access
+~~~~~~~~~~~~~~~~~
+If you do not already have an account on Intrepid, you can request
+one at https://accounts.alcf.anl.gov/accounts/request.php. More information about
+this process and requesting allocations for your project can be found at
+http://www.alcf.anl.gov/support/gettingstarted/index.php.
+
+SSH Keys
+~~~~~~~~
+Accessing the Intrepid via SSH can be done with any SSH software package.
+Before logging in, you will need to generate an SSH public key and send it to
+support at alcf.anl.gov for verification and installation.
+
+Cryptocard
+~~~~~~~~~~
+This security token uses one-time passwords for controlled access to the BG/P login systems.
+
+Connecting to a login node
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+When you gain access to Intrepid, you should receive a cryptocard and a temporary PIN.
+You must have a working cryptocard, know your PIN, and have your SSH key in place before
+you may login.
+
+You can connect to Intrepid with the following command:
+
+-----
+ssh yourusername at intrepid.alcf.anl.gov
+-----
+
+You will be presented with a password prompt. The first part of your password is your PIN. Enter you PIN,
+press the Cryptocard button, and then enter the password your crypocard generates. If this is the first
+time you are logging in, you will be prompted to change your PIN.
+
+Downloading and building Swift
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The most recent versions of Swift can be found at
+http://www.ci.uchicago.edu/swift/downloads/index.php. Follow the instructions
+provided on that site to download and build Swift.
+
+Adding Swift to your PATH
+~~~~~~~~~~~~~~~~~~~~~~~~~
+Once you have installed Swift, add the Swift binary to your PATH so you can
+easily run it from any directory.
+
+In your home directory, edit the file ".bashrc".
+
+If you have installed Swift via a source repository, add the following line
+at the bottom of .bashrc.
+
+-----
+export PATH=$PATH:$HOME/cog/modules/swift/dist/swift-svn/bin
+-----
+
+If you have installed Swift via a binary package, add this line:
+
+-----
+export PATH=$PATH:$HOME/swift-<version>/bin
+-----
+
+Replace <version> with the actual name of the swift directory in the example above.
+
+What You Need To Know Before Running Swift
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Before you can create a Swift configuration file, there are some things
+you will need to know.
+
+Swift Work Directory
+^^^^^^^^^^^^^^^^^^^^
+The Swift work directory is a directory which Swift uses for processing work.
+This directory needs to be writable. Common options for this are:
+
+-----
+/home/username/swiftwork
+/home/username/work
+/tmp
+-----
+
+Which project(s) are you a member of?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Intrepid requires that you are a member of a project. You can determine this by
+running the following command:
+
+-----
+$ projects
+HTCScienceApps
+-----
+
+If you are not a member of a project, you must first request access
+to a project. More information on this process can be found at
+https://wiki.alcf.anl.gov/index.php/Discretionary_Allocations
+
+Determine your Queue
+^^^^^^^^^^^^^^^^^^^^
+
+Intrepid has several different queues you can submit jobs to depending on
+the type of work you will be doing. The command "qstat -q" will print
+the most up to date list of this information.
+
+.Intrepid Queues
+[options="header"]
+|=========================================================
+|User Queue|Queue|Nodes|Time (hours)|User Maxrun|Project maxrun
+|prod-devel|prod-devel|64-512|0-1|5|20
+|prod|prod-short|512-4096|0-6|5|20
+|prod|prod-long|512-4096|6-12|5|20
+|prod|prod-capability|4097-32768|0-12|2|20
+|prod|prod-24k|16385-24576|0-12|2|20
+|prod|prod-bigrun|32769-40960|0-12|2|20
+|prod|backfill|512-8192|0-6|5|10
+|============================================================
+
+Generating Configuration Files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Now that you know what queue to use, your project, and your work directory, it is time to
+set up Swift. Swift uses a configuration file called sites.xml to determine how it should run.
+There are two methods you can use for creating this file. You can manually edit
+the configuration file, or generate it with a utility called gensites.
+
+Manually Editing sites.xml
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+Below is the template that is used by Swift's test suite for running on Intrepid.
+TODO: Update the rest below here
+
+-----
+include::../../tests/providers/intrepid/sites.template.xml[]
+-----
+
+The values to note here are the ones that are listed between underscores. In the example above, they are \_QUEUE_, and \_WORK_. Queue is the PADS queue to use and WORK is the swift work directory. These are placeholder values you will need to modify to fit your needs. Copy and paste this template, replace the values, and call it sites.xml.
+
+Manually Editing tc.data
+~~~~~~~~~~~~~~~~~~~~~~~~
+Below is the tc.data file used by Swift's test suite for running on PADS.
+
+-----
+include::../../tests/providers/intrepid/tc.template.data[]
+------
+
+Copy these commands and save it as tc.data.
+
+Catsn.swift
+~~~~~~~~~~~
+The swift script we will run is called catsn.swift. It simply cats a file and saves the result. This is a nice simple test to ensure jobs are running correctly. Create a file called data.txt which contains some simple input - a "hello world" will do the trick.
+
+-----
+include::../../examples/misc/catsn.swift[]
+-----
+
+Running Swift
+~~~~~~~~~~~~~
+Now that everything is in place, run Swift with the following command:
+
+-----
+swift -sites.file sites.xml -tc.file tc.data catsn.swift -n=10
+-----
+
+You should see several new files being created, called catsn.0001.out, catsn.0002.out, etc. Each of these
+files should contain the contents of what you placed into data.txt. If this happens, your job has run
+successfully on PADS!
+
+More Help
+~~~~~~~~~
+The best place for additional help is the Swift user mailing list. You can subscribe to this list at
+https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user. When submitting information, please send your sites.xml file, your tc.data, and any Swift log files that were created during your attempt.
+
+
Modified: trunk/docs/siteguide/pads
===================================================================
--- trunk/docs/siteguide/pads 2011-09-08 00:53:06 UTC (rev 5086)
+++ trunk/docs/siteguide/pads 2011-09-08 01:21:11 UTC (rev 5087)
@@ -1,10 +1,93 @@
PADS
----
+PADS Quickstart
+~~~~~~~~~~~~~~~
+This section will walk you through running a simple Swift script
+on PADS. It will explain how to setup software packages, how
+to create the required Swift configuration files, and finally
+how to run Swift.
+
+Adding Software Packages
+^^^^^^^^^^^^^^^^^^^^^^^^
+In your home directory, edit a file called .soft and add these
+lines (in this order):
+-----
++java-sun
++maui
++torque
+ at default
+-----
+
+Log out of PADS, and log back in for these changes to take effect.
+
+Creating sites.xml
+^^^^^^^^^^^^^^^^^^
+Swift relies on various configuration files to determine how to
+run. This section will provide a working configuration file which
+you can copy and paste to get running quickly. The sites.xml file
+tells Swift how to submit jobs, where working directories are
+located, and various other configuration information. More
+information on sites.xml can be found in the Swift User's Guide.
+
+The first step is to paste the text below into a file named sites.xml.
+
+-----
+include::../../tests/providers/pads/coasters/sites.template.xml[]
+-----
+
+This file will require just a few customizations. First, create a
+directory called swiftwork. Modify \_WORK_ in sites.xml
+to point to this new directory. For example
+-----
+<workdirectory>/home/myhome/swiftwork</workdirectory>
+-----
+
+Creating tc.data
+^^^^^^^^^^^^^^^^
+The tc.data configuration file gives information about the applications
+that will be called by Swift. More information about the format
+of tc.data can be found in the Swift User's guide.
+
+Paste the following example into a file named tc.data
+
+-----
+include::../../tests/providers/pads/coasters/tc.template.data[]
+-----
+
+Copy a Swift Script
+^^^^^^^^^^^^^^^^^^^
+
+Within the Swift directory is an examples directory which contains
+several introductory Swift scripts. The example we will use in this
+section is called catsn.swift. Copy this script to the same directory
+that your sites.xml and tc.data files are located.
+
+-----
+$ cp ~/swift-0.93/examples/misc/catsn.swift .
+$ cp ~/swift-0.93/examples/misc/data.txt .
+-----
+TIP: The location of your swift directory may vary depending on how you installed it. Change this to the examples/misc directory of your installation as needed.
+
+Run Swift
+^^^^^^^^^
+
+Finally, run the script
+-----
+$ swift -sites.file sites.xml -tc.file tc.data catsn.swift
+-----
+
+You should see 10 new text files get created, named catsn*.out. If
+you see these files, then you have succesfully run Swift on PADS!
+
+TIP: Make sure your default project is defined. Read on for more information.
+
+Read on for more detailed information about running Swift on PADS.
+
Requesting Access
~~~~~~~~~~~~~~~~~
If you do not already have a Computation Institute account, you can request
-one at https://www.ci.uchicago.edu/accounts/. This page will give you a list
+one at https://www.ci.uchicago.edu/accounts. This page will give you a list
of resources you can request access to. Be sure that PADS is selected. If
you already have an existing CI account, but do not have access to PADS,
send an email to support at ci.uchicago.edu to request access.
@@ -39,48 +122,17 @@
@default
-----
-Downloading and building Swift
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The most recent versions of Swift can be found at
-http://www.ci.uchicago.edu/swift/downloads/index.php. Follow the instructions
-provided on that site to download and build Swift.
-
-Adding Swift to your PATH
-~~~~~~~~~~~~~~~~~~~~~~~~~
-Once you have installed Swift, add the Swift binary to your PATH so you can
-easily run it from any directory.
-
-In your home directory, edit the file ".bashrc".
-
-If you have installed Swift via a source repository, add the following line
-at the bottom of .bashrc.
-
------
-export PATH=$PATH:$HOME/cog/modules/swift/dist/swift-svn/bin
------
-
-If you have installed Swift via a binary package, add this line:
-
------
-export PATH=$PATH:$HOME/swift-<version>/bin
------
-
-Replace <version> with the actual name of the swift directory in the example above.
-
-What You Need To Know Before Running Swift
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Before you can create a Swift configuration file, there are some things
-you will need to know.
-
Swift Work Directory
^^^^^^^^^^^^^^^^^^^^
The Swift work directory is a directory which Swift uses for processing work.
-This directory needs to be writable. Common options for this are:
+This directory needs to be writable and available to all worker nodes on
+a shared filesystem.
+.Examples
-----
/home/username/swiftwork
/home/username/work
-/tmp
+/autonfs/gpfs-pads/projects/CI-CCR000013/myusername
-----
Which project(s) are you a member of?
@@ -145,29 +197,35 @@
Below is the template that is used by Swift's test suite for running on PADS.
-----
-include::../../tests/providers/local-pbs/pads/sites.template.xml[]
+include::../../tests/providers/pads/coasters/sites.template.xml[]
-----
-The values to note here are the ones that are listed between underscores. In the example above, they are _QUEUE_, and _WORK_. Queue is the PADS queue to use and WORK is the swift work directory. These are placeholder values you will need to modify to fit your needs. Copy and paste this template, replace the values, and call it sites.xml.
+Before you can use this configuration file, you will need to modify \_WORK_ to a directory writable by you,
+and have your default project defined. Copy this template, replace \_WORK_, and then save as sites.xml.
+If you wanted to use a different queue, replace "fast" with the desired queue name.
+
Manually Editing tc.data
~~~~~~~~~~~~~~~~~~~~~~~~
Below is the tc.data file used by Swift's test suite for running on PADS.
-----
-include::../../tests/providers/local-pbs/pads/tc.template.data[]
+include::../../tests/providers/pads/coasters/tc.template.data[]
------
Copy these commands and save it as tc.data.
Catsn.swift
~~~~~~~~~~~
-The swift script we will run is called catsn.swift. It simply cats a file and saves the result. This is a nice simple test to ensure jobs are running correctly. Create a file called data.txt which contains some simple input - a "hello world" will do the trick.
+The swift script we will run is called catsn.swift. It simply cats a file and saves the result. This is
+a nice simple test to ensure jobs are running correctly. Create a file called data.txt which contains
+some simple input - a "hello world" will do the trick.
-----
include::../../examples/misc/catsn.swift[]
-----
+
Running Swift
~~~~~~~~~~~~~
Now that everything is in place, run Swift with the following command:
Added: trunk/docs/siteguide/prereqs
===================================================================
--- trunk/docs/siteguide/prereqs (rev 0)
+++ trunk/docs/siteguide/prereqs 2011-09-08 01:21:11 UTC (rev 5087)
@@ -0,0 +1,8 @@
+Prerequisites
+-------------
+
+This guide assumes that you have already downloaded and installed Swift.
+It assumes that Swift is in your PATH and that you have a working
+version of Sun Java 1.5+. For more information on downloading and
+installing Swift, please see the
+http://www.ci.uchicago.edu/swift/guides/release-0.93/quickstart/quickstart.html[Swift Quickstart Guide].
Modified: trunk/docs/siteguide/siteguide.txt
===================================================================
--- trunk/docs/siteguide/siteguide.txt 2011-09-08 00:53:06 UTC (rev 5086)
+++ trunk/docs/siteguide/siteguide.txt 2011-09-08 01:21:11 UTC (rev 5087)
@@ -6,10 +6,14 @@
:website: http://www.ci.uchicago.edu/swift/guides/siteguide.php
:numbered:
+include::prereqs[]
+
include::pads[]
include::beagle[]
+include::intrepid[]
+
include::futuregrid[]
include::grid[]
Modified: trunk/docs/tutorial/runtime_features
===================================================================
--- trunk/docs/tutorial/runtime_features 2011-09-08 00:53:06 UTC (rev 5086)
+++ trunk/docs/tutorial/runtime_features 2011-09-08 01:21:11 UTC (rev 5087)
@@ -22,167 +22,10 @@
examples, we've invoked echo and tr executables from our SwiftScript
program. These have been run on the local system (the same computer on
which you ran swift). We can also make our computations run on a
-remote resource.
+remote resource. For more information on running Swift on a remote site
+please see the http://www.ci.uchicago.edu/swift/guides/release-0.93/siteguide/siteguide.html[Site Configuration Guide].
-WARNING: This example is necessarily more vague than previous ones,
-because it requires access to remote resources. You should ensure that
-you can submit a job using the globus-job-run (or globusrun-ws)
-command(s).
-We do not need to modify any SwiftScript code to run on another
-resource. Instead, we must modify another catalog, the site catalog.
-This catalog provides details of the location that applications will be
-run, with the default settings referring to the local machine. We will
-modify it to refer to a remote resource - the UC Teraport cluster. If
-you are not a UC Teraport user, you should use details of a different
-resource that you do have access to.
-
-The site catalog is located in etc/sites.xml and is a relatively
-straightforward XML format file. We must modify each of the following
-three settings: gridftp (which indicates how and where data can be
-transferred to the remote resource), jobmanager (which indicates how
-applications can be run on the remote resource) and workdirectory (which
-indicates where working storage can be found on the remote resource).
-
-
-Writing a mapper
-~~~~~~~~~~~~~~~~
-
-This section will introduce writing a custom mapper so that Swift is
-able to access data files laid out in application-specific ways.
-
-An application-specific mapper must take the form of a Java class that
-implements the http://www.ci.uchicago.edu/swift/javadoc/vdsk/org/griphyn/vdl/mapping/Mapper.html[Mapper
-interface].
-
-Usually you don't need to implement this interface directly, because
-Swift provides a number of more concrete classes with some functionality
-already implemented.
-
-The hierarchy of helper classes is:
-
-http://www.ci.uchicago.edu/swift/javadoc/vdsk/org/griphyn/vdl/mapping/Mapper.html[Mapper]
-
-This is the abstract interface for mappers in Swift. You must
-implement methods to provide access to mapper properties, to map from a
-SwiftScript dataset path (such as foo[1].bar) to a file name, to check
-whether a file exists. None of the default Swift mappers implement this
-interface directly - instead they use one of the following helper classes.
-
-http://www.ci.uchicago.edu/swift/javadoc/vdsk/org/griphyn/vdl/mapping/AbstractMapper.html[Abstract Mapper]
-
-This provides helper methods to manage mapper properties and to handle
-existance checking. Examples of mappers which use this class are:
-http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.array_mapper[Array Mapper],
-http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.csv_mapper[CSV Mapper],
-http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.fixed_array_mapper[Fixed Array Mapper],
-http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.regexp_mapper[Regexp Mapper],
-and http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.single_file_mapper[Single File Mapper].
-
-http://www.ci.uchicago.edu/swift/javadoc/vdsk/org/griphyn/vdl/mapping/file/AbstractFileMapper.html[AbstractFileMapper]
-This provides a helper class for mappers which select files based on
-selecting files from a directory listing. It is necessary to write some
-helper methods that are different from the above mapper methods.
-Examples of mappers which use this class are:
-http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.simple_mapper[simple_mapper],
-http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.filesys_mapper[filesys_mapper],
-and the StructuredRegularExpressionMapper.
-
-In general, to write a mapper, choose either the AbstractMapper or the
-AbstractFileMapper and extend those. If your mapper will generally
-select the files it returns based on a directory listing and will
-convert paths to filenames using some regular conversion (for example,
-in the way that simple_mapper maps files in a directory that match a
-particular pattern), then you should probably use the
-AbstractFileMapper. If your mapper will produce a list of files in some
-other way (for example, in the way that csv_mapper maps based on
-filenames given in a CSV file rather than looking at which files are in
-a directory), then you should probably use the AbstractMapper.
-
-
-Writing a very basic mapper
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-In this section, we will write a very basic (almost useless) mapper that
-will map a SwiftScript dataset into a hardcoded file called
-myfile.txt, like this:
-
-
- Swift variable Filename
-
- var <-----------------------------> myfile.txt
-
-We should be able to use the mapper we write in a SwiftScript program
-like this:
-
-
-----
-type file;
-file f <my_first_mapper>;
-----
-
-First we must choose a base class - AbstractMapper or
-AbstractFileMapper. We aren't going to use a directory listing to decide
-on our mapping - we are getting the mapping from some other source (in
-fact, it will be hard coded). So we will use AbstractMapper.
-
-So now onto the source code. We must define a subclass of AbstractMapper
-and implement several mapper methods: isStatic, existing, and map. These
-methods are documented in the javadoc for the Mapper interface.
-
-Here is the code implementing this mapper. Put this in your source
-vdsk directory, make a directory src/tutorial/|and put this file in
-src/tutorial/MyFirstMapper.java.
-
-.MyFirstMapper.java
-*******************
-----
-include::../../examples/tutorial/MyFirstMapper.java[]
-----
-*******************
-
-Now we need to inform the Swift engine about the existence of this
-mapper. We do that by editing the MapperFactory class definition, in
-src/org/griphyn/vdl/mapping/MapperFactory.java and adding a
-registerMapper call alongside the existing registerMapper calls, like this:
-
-
-----
-registerMapper("my_first_mapper", tutorial.MyFirstMapper.class);
-----
-
-The first parameter is the name of the mapper that will be used in
-SwiftScript program. The second parameter is the new Mapper class that
-we just wrote.
-
-Now rebuild Swift using the "ant redist" target.
-
-This new Swift build will be aware of your new mapper. We can test it
-out with a hello world program:
-
-.my_first_mapper.swift
-**********************
------
-include::../../examples/tutorial/my_first_mapper.swift[]
------
-**********************
-
-Run this program, and hopefully you will find the "hello" string has
-been output into the hard coded output file myfile.txt:
-
-
-----
-$ cat myfile.txt
-hello
-----
-
-So that's a first very simple mapper implemented. Compare the source
-code to the single_file_mapper in
-http://www.ci.uchicago.edu/trac/swift/browser/trunk/src/org/griphyn/vdl/mapping/file/SingleFileMapper.java[SingleFileMapper.java].
-There is not much more code to the single_file_mapper - mostly code to
-deal with the file parameter.
-
-
Starting and restarting
~~~~~~~~~~~~~~~~~~~~~~~
Modified: trunk/docs/userguide/coasters
===================================================================
--- trunk/docs/userguide/coasters 2011-09-08 00:53:06 UTC (rev 5086)
+++ trunk/docs/userguide/coasters 2011-09-08 01:21:11 UTC (rev 5087)
@@ -1,7 +1,5 @@
Coasters
--------
-Coasters were introduced in Swift v0.6 as an experimental feature.
-
In many applications, Swift performance can be greatly enhanced by the
use of CoG coasters. CoG coasters provide a low-overhead job submission
and file transfer mechanism suited for the execution of short jobs (on
Added: trunk/docs/userguide/log-processing
===================================================================
--- trunk/docs/userguide/log-processing (rev 0)
+++ trunk/docs/userguide/log-processing 2011-09-08 01:21:11 UTC (rev 5087)
@@ -0,0 +1,104 @@
+
+Log Processing
+--------------
+
+To properly generate log plots, you must enable VDL/Karajan logging.
+TODO:How?
+
+
+You should check the scripts that you intend to use to determine
+what log lines they require and ensure that you are generating
+those lines via log4j.properties
+
+Make sure log4.properties contains:
+--------------------------------------
+log4j.logger.swift=DEBUG
+log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Cpu=DEBUG
+log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Block=DEBUG
+--------------------------------------
+TODO: Does it work for coasters-based runs only?
+
+Normalize event times in the log to the run start time
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Generate the log, assuming the log is titled +swift-run.log+
+
+------------------------------------------
+./normalize-log.pl file.contains.start.time swift-run.log > swift-run.norm
+------------------------------------------
+TODO:In what format does the start time be in 'file.contains.start.time'
+
+
+Make a basic load plot from Coasters Cpu log lines
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+. Normalize the log.
+. Build up a load data file:
++
+------------------------------------------
+./cpu-job-load.pl < swift-run.norm > load.data
+------------------------------------------
+. Plot with the JFreeChart-based plotter in usertools/plotter:
++
+------------------------------------------
+swift_plotter.zsh -s load.cfg load.eps load.data
+------------------------------------------
+Note: Th load.cfg is available from swift/libexec/log-processing/
+
+
+Make a basic job completion plot from Coasters Cpu log lines
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+. Normalize the log.
+
+. Build up a completed data file:
++
+------------------------------------------
+./cpu-job-completed.pl < swift-run.norm > completed.data
+------------------------------------------
+
+. Plot with the JFreeChart-based plotter in usertools/plotter:
++
+------------------------------------------
+swift_plotter.zsh -s completed.cfg completed.eps completed.data
+------------------------------------------
+
+Make a basic Block allocation plot from Coasters Block log lines
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+. Normalize the log.
+
+. Build up a block allocation data file:
++
+------------------------------------------
+./block-level.pl < swift-run.norm > blocks.data
+------------------------------------------
+
+. Plot with the JFreeChart-based plotter in usertools/plotter:
++
+------------------------------------------
+swift_plotter.zsh -s blocks.{cfg,eps,data}
+------------------------------------------
+
+Make a job runtime distribution plot from Coasters Cpu log lines
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+. Normalize the log.
+
+. Build up a job runtime file:
++
+------------------------------------------
+./extract-times.pl < swift-run.norm > times.data
+------------------------------------------
+
+. Put the job runtimes into 1-second buckets:
++
+------------------------------------------
+./ buckets.pl 1 times.data > buckets.data
+------------------------------------------
+
+. Plot with the JFreeChart-based plotter in usertools/plotter:
++
+------------------------------------------
+swift_plotter.zsh -s buckets.cfg buckets.eps buckets.data
+------------------------------------------
Modified: trunk/docs/userguide/userguide.txt
===================================================================
--- trunk/docs/userguide/userguide.txt 2011-09-08 00:53:06 UTC (rev 5086)
+++ trunk/docs/userguide/userguide.txt 2011-09-08 01:21:11 UTC (rev 5087)
@@ -37,3 +37,5 @@
include::howto_tips[]
include::cdm[]
+
+include::log-processing[]
More information about the Swift-commit
mailing list