[Swift-commit] r4896 - in wwwdev: . cookbook cookbook/_notes cookbook/figures cookbook/figures/_notes
vytas at ci.uchicago.edu
vytas at ci.uchicago.edu
Mon Aug 1 09:23:37 CDT 2011
Author: vytas
Date: 2011-08-01 09:23:26 -0500 (Mon, 01 Aug 2011)
New Revision: 4896
Added:
wwwdev/cookbook/
wwwdev/cookbook/_notes/
wwwdev/cookbook/_notes/dwsync.xml
wwwdev/cookbook/cookbook-asciidoc.html
wwwdev/cookbook/cookbook-asciidoc.txt
wwwdev/cookbook/figures/
wwwdev/cookbook/figures/_notes/
wwwdev/cookbook/figures/_notes/dwsync.xml
wwwdev/cookbook/figures/coaster_setup.png
Log:
new look & feel
Added: wwwdev/cookbook/_notes/dwsync.xml
===================================================================
--- wwwdev/cookbook/_notes/dwsync.xml (rev 0)
+++ wwwdev/cookbook/_notes/dwsync.xml 2011-08-01 14:23:26 UTC (rev 4896)
@@ -0,0 +1,5 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<dwsync>
+<file name="cookbook-asciidoc.html" server="login.ci.uchicago.edu//home/vytas/public_html/projects/2011_07-Swift/" local="3394121904" remote="11984644031" />
+<file name="cookbook-asciidoc.txt" server="login.ci.uchicago.edu//home/vytas/public_html/projects/2011_07-Swift/" local="3394121904" remote="11984644036" />
+</dwsync>
\ No newline at end of file
Added: wwwdev/cookbook/cookbook-asciidoc.html
===================================================================
--- wwwdev/cookbook/cookbook-asciidoc.html (rev 0)
+++ wwwdev/cookbook/cookbook-asciidoc.html 2011-08-01 14:23:26 UTC (rev 4896)
@@ -0,0 +1,1500 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
+ "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
+<head>
+<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" />
+<meta name="generator" content="AsciiDoc 8.6.4" />
+<title>Swift Cookbook</title>
+<style type="text/css">
+/* Sans-serif font. */
+h1, h2, h3, h4, h5, h6,
+div.title, caption.title,
+thead, p.table.header,
+div#toctitle,
+span#author, span#revnumber, span#revdate, span#revremark,
+div#footer {
+ font-family: Arial,Helvetica,sans-serif;
+}
+
+/* Serif font. */
+div.sectionbody {
+ font-family: Georgia,"Times New Roman",Times,serif;
+}
+
+/* Monospace font. */
+tt {
+ font-size: inherit;
+}
+
+body {
+ margin: 1em 5% 1em 5%;
+}
+
+a {
+ color: blue;
+ text-decoration: underline;
+}
+a:visited {
+ color: fuchsia;
+}
+
+em {
+ font-style: italic;
+ color: navy;
+}
+
+strong {
+ font-weight: bold;
+ color: #083194;
+}
+
+tt {
+ font-size: inherit;
+ color: navy;
+}
+
+h1, h2, h3, h4, h5, h6 {
+ color: #527bbd;
+ margin-top: 1.2em;
+ margin-bottom: 0.5em;
+ line-height: 1.3;
+}
+
+h1, h2, h3 {
+ border-bottom: 2px solid silver;
+}
+h2 {
+ padding-top: 0.5em;
+}
+h3 {
+ float: left;
+}
+h3 + * {
+ clear: left;
+}
+
+div.sectionbody {
+ margin-left: 0;
+}
+
+hr {
+ border: 1px solid silver;
+}
+
+p {
+ margin-top: 0.5em;
+ margin-bottom: 0.5em;
+}
+
+ul, ol, li > p {
+ margin-top: 0;
+}
+ul > li { color: #aaa; }
+ul > li > * { color: black; }
+
+pre {
+ padding: 0;
+ margin: 0;
+}
+
+span#author {
+ color: #527bbd;
+ font-weight: bold;
+ font-size: 1.1em;
+}
+span#email {
+}
+span#revnumber, span#revdate, span#revremark {
+}
+
+div#footer {
+ font-size: small;
+ border-top: 2px solid silver;
+ padding-top: 0.5em;
+ margin-top: 4.0em;
+}
+div#footer-text {
+ float: left;
+ padding-bottom: 0.5em;
+}
+div#footer-badges {
+ float: right;
+ padding-bottom: 0.5em;
+}
+
+div#preamble {
+ margin-top: 1.5em;
+ margin-bottom: 1.5em;
+}
+div.tableblock, div.imageblock, div.exampleblock, div.verseblock,
+div.quoteblock, div.literalblock, div.listingblock, div.sidebarblock,
+div.admonitionblock {
+ margin-top: 1.0em;
+ margin-bottom: 1.5em;
+}
+div.admonitionblock {
+ margin-top: 2.0em;
+ margin-bottom: 2.0em;
+ margin-right: 10%;
+ color: #606060;
+}
+
+div.content { /* Block element content. */
+ padding: 0;
+}
+
+/* Block element titles. */
+div.title, caption.title {
+ color: #527bbd;
+ font-weight: bold;
+ text-align: left;
+ margin-top: 1.0em;
+ margin-bottom: 0.5em;
+}
+div.title + * {
+ margin-top: 0;
+}
+
+td div.title:first-child {
+ margin-top: 0.0em;
+}
+div.content div.title:first-child {
+ margin-top: 0.0em;
+}
+div.content + div.title {
+ margin-top: 0.0em;
+}
+
+div.sidebarblock > div.content {
+ background: #ffffee;
+ border: 1px solid #dddddd;
+ border-left: 4px solid #f0f0f0;
+ padding: 0.5em;
+}
+
+div.listingblock > div.content {
+ border: 1px solid #dddddd;
+ border-left: 5px solid #f0f0f0;
+ background: #f8f8f8;
+ padding: 0.5em;
+}
+
+div.quoteblock, div.verseblock {
+ padding-left: 1.0em;
+ margin-left: 1.0em;
+ margin-right: 10%;
+ border-left: 5px solid #f0f0f0;
+ color: #777777;
+}
+
+div.quoteblock > div.attribution {
+ padding-top: 0.5em;
+ text-align: right;
+}
+
+div.verseblock > pre.content {
+ font-family: inherit;
+ font-size: inherit;
+}
+div.verseblock > div.attribution {
+ padding-top: 0.75em;
+ text-align: left;
+}
+/* DEPRECATED: Pre version 8.2.7 verse style literal block. */
+div.verseblock + div.attribution {
+ text-align: left;
+}
+
+div.admonitionblock .icon {
+ vertical-align: top;
+ font-size: 1.1em;
+ font-weight: bold;
+ text-decoration: underline;
+ color: #527bbd;
+ padding-right: 0.5em;
+}
+div.admonitionblock td.content {
+ padding-left: 0.5em;
+ border-left: 3px solid #dddddd;
+}
+
+div.exampleblock > div.content {
+ border-left: 3px solid #dddddd;
+ padding-left: 0.5em;
+}
+
+div.imageblock div.content { padding-left: 0; }
+span.image img { border-style: none; }
+a.image:visited { color: white; }
+
+dl {
+ margin-top: 0.8em;
+ margin-bottom: 0.8em;
+}
+dt {
+ margin-top: 0.5em;
+ margin-bottom: 0;
+ font-style: normal;
+ color: navy;
+}
+dd > *:first-child {
+ margin-top: 0.1em;
+}
+
+ul, ol {
+ list-style-position: outside;
+}
+ol.arabic {
+ list-style-type: decimal;
+}
+ol.loweralpha {
+ list-style-type: lower-alpha;
+}
+ol.upperalpha {
+ list-style-type: upper-alpha;
+}
+ol.lowerroman {
+ list-style-type: lower-roman;
+}
+ol.upperroman {
+ list-style-type: upper-roman;
+}
+
+div.compact ul, div.compact ol,
+div.compact p, div.compact p,
+div.compact div, div.compact div {
+ margin-top: 0.1em;
+ margin-bottom: 0.1em;
+}
+
+div.tableblock > table {
+ border: 3px solid #527bbd;
+}
+thead, p.table.header {
+ font-weight: bold;
+ color: #527bbd;
+}
+tfoot {
+ font-weight: bold;
+}
+td > div.verse {
+ white-space: pre;
+}
+p.table {
+ margin-top: 0;
+}
+/* Because the table frame attribute is overriden by CSS in most browsers. */
+div.tableblock > table[frame="void"] {
+ border-style: none;
+}
+div.tableblock > table[frame="hsides"] {
+ border-left-style: none;
+ border-right-style: none;
+}
+div.tableblock > table[frame="vsides"] {
+ border-top-style: none;
+ border-bottom-style: none;
+}
+
+
+div.hdlist {
+ margin-top: 0.8em;
+ margin-bottom: 0.8em;
+}
+div.hdlist tr {
+ padding-bottom: 15px;
+}
+dt.hdlist1.strong, td.hdlist1.strong {
+ font-weight: bold;
+}
+td.hdlist1 {
+ vertical-align: top;
+ font-style: normal;
+ padding-right: 0.8em;
+ color: navy;
+}
+td.hdlist2 {
+ vertical-align: top;
+}
+div.hdlist.compact tr {
+ margin: 0;
+ padding-bottom: 0;
+}
+
+.comment {
+ background: yellow;
+}
+
+.footnote, .footnoteref {
+ font-size: 0.8em;
+}
+
+span.footnote, span.footnoteref {
+ vertical-align: super;
+}
+
+#footnotes {
+ margin: 20px 0 20px 0;
+ padding: 7px 0 0 0;
+}
+
+#footnotes div.footnote {
+ margin: 0 0 5px 0;
+}
+
+#footnotes hr {
+ border: none;
+ border-top: 1px solid silver;
+ height: 1px;
+ text-align: left;
+ margin-left: 0;
+ width: 20%;
+ min-width: 100px;
+}
+
+div.colist td {
+ padding-right: 0.5em;
+ padding-bottom: 0.3em;
+ vertical-align: top;
+}
+div.colist td img {
+ margin-top: 0.3em;
+}
+
+ at media print {
+ div#footer-badges { display: none; }
+}
+
+div#toc {
+ margin-bottom: 2.5em;
+}
+
+div#toctitle {
+ color: #527bbd;
+ font-size: 1.1em;
+ font-weight: bold;
+ margin-top: 1.0em;
+ margin-bottom: 0.1em;
+}
+
+div.toclevel1, div.toclevel2, div.toclevel3, div.toclevel4 {
+ margin-top: 0;
+ margin-bottom: 0;
+}
+div.toclevel2 {
+ margin-left: 2em;
+ font-size: 0.9em;
+}
+div.toclevel3 {
+ margin-left: 4em;
+ font-size: 0.9em;
+}
+div.toclevel4 {
+ margin-left: 6em;
+ font-size: 0.9em;
+}
+
+span.aqua { color: aqua; }
+span.black { color: black; }
+span.blue { color: blue; }
+span.fuchsia { color: fuchsia; }
+span.gray { color: gray; }
+span.green { color: green; }
+span.lime { color: lime; }
+span.maroon { color: maroon; }
+span.navy { color: navy; }
+span.olive { color: olive; }
+span.purple { color: purple; }
+span.red { color: red; }
+span.silver { color: silver; }
+span.teal { color: teal; }
+span.white { color: white; }
+span.yellow { color: yellow; }
+
+span.aqua-background { background: aqua; }
+span.black-background { background: black; }
+span.blue-background { background: blue; }
+span.fuchsia-background { background: fuchsia; }
+span.gray-background { background: gray; }
+span.green-background { background: green; }
+span.lime-background { background: lime; }
+span.maroon-background { background: maroon; }
+span.navy-background { background: navy; }
+span.olive-background { background: olive; }
+span.purple-background { background: purple; }
+span.red-background { background: red; }
+span.silver-background { background: silver; }
+span.teal-background { background: teal; }
+span.white-background { background: white; }
+span.yellow-background { background: yellow; }
+
+span.big { font-size: 2em; }
+span.small { font-size: 0.6em; }
+</style>
+<script type="text/javascript">
+/*<+'])');
+ // Function that scans the DOM tree for header elements (the DOM2
+ // nodeIterator API would be a better technique but not supported by all
+ // browsers).
+ var iterate = function (el) {
+ for (var i = el.firstChild; i != null; i = i.nextSibling) {
+ if (i.nodeType == 1 /* Node.ELEMENT_NODE */) {
+ var mo = re.exec(i.tagName);
+ if (mo && (i.getAttribute("class") || i.getAttribute("className")) != "float") {
+ result[result.length] = new TocEntry(i, getText(i), mo[1]-1);
+ }
+ iterate(i);
+ }
+ }
+ }
+ iterate(el);
+ return result;
+ }
+
+ var toc = document.getElementById("toc");
+ var entries = tocEntries(document.getElementById("content"), toclevels);
+ for (var i = 0; i < entries.length; ++i) {
+ var entry = entries[i];
+ if (entry.element.id == "")
+ entry.element.id = "_toc_" + i;
+ var a = document.createElement("a");
+ a.href = "#" + entry.element.id;
+ a.appendChild(document.createTextNode(entry.text));
+ var div = document.createElement("div");
+ div.appendChild(a);
+ div.className = "toclevel" + entry.toclevel;
+ toc.appendChild(div);
+ }
+ if (entries.length == 0)
+ toc.parentNode.removeChild(toc);
+},
+
+
+/////////////////////////////////////////////////////////////////////
+// Footnotes generator
+/////////////////////////////////////////////////////////////////////
+
+/* Based on footnote generation code from:
+ * http://www.brandspankingnew.net/archive/2005/07/format_footnote.html
+ */
+
+footnotes: function () {
+ var cont = document.getElementById("content");
+ var noteholder = document.getElementById("footnotes");
+ var spans = cont.getElementsByTagName("span");
+ var refs = {};
+ var n = 0;
+ for (i=0; i<spans.length; i++) {
+ if (spans[i].className == "footnote") {
+ n++;
+ // Use [\s\S] in place of . so multi-line matches work.
+ // Because JavaScript has no s (dotall) regex flag.
+ note = spans[i].innerHTML.match(/\s*\[([\s\S]*)]\s*/)[1];
+ noteholder.innerHTML +=
+ "<div class='footnote' id='_footnote_" + n + "'>" +
+ "<a href='#_footnoteref_" + n + "' title='Return to text'>" +
+ n + "</a>. " + note + "</div>";
+ spans[i].innerHTML =
+ "[<a id='_footnoteref_" + n + "' href='#_footnote_" + n +
+ "' title='View footnote' class='footnote'>" + n + "</a>]";
+ var id =spans[i].getAttribute("id");
+ if (id != null) refs["#"+id] = n;
+ }
+ }
+ if (n == 0)
+ noteholder.parentNode.removeChild(noteholder);
+ else {
+ // Process footnoterefs.
+ for (i=0; i<spans.length; i++) {
+ if (spans[i].className == "footnoteref") {
+ var href = spans[i].getElementsByTagName("a")[0].getAttribute("href");
+ href = href.match(/#.*/)[0]; // Because IE return full URL.
+ n = refs[href];
+ spans[i].innerHTML =
+ "[<a href='#_footnote_" + n +
+ "' title='View footnote' class='footnote'>" + n + "</a>]";
+ }
+ }
+ }
+}
+
+}
+/*]]>*/
+</script>
+</head>
+<body class="article">
+<div id="header">
+<h1>Swift Cookbook</h1>
+<span id="author">Swift team</span><br />
+<span id="email"><tt><<a href="mailto:swift-user at ci.uchicago.edu">swift-user at ci.uchicago.edu</a>></tt></span><br />
+<span id="revnumber">version 0.92,</span>
+<span id="revdate">March 2011</span>
+<div id="toc">
+ <div id="toctitle">Table of Contents</div>
+ <noscript><p><b>JavaScript must be enabled in your browser to display the table of contents.</b></p></noscript>
+</div>
+</div>
+<div id="content">
+<div class="sect1">
+<h2 id="_overview">1. Overview</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>This cookbook covers various recipes involving setting up and running Swift under diverse
+configurations based on the application requirements and the underlying
+infrastructures. The Swift system comprises of SwiftScript language and the
+Swift runtime system. For
+introductory material, consult the Swift tutorial found
+<a href="http://www.ci.uchicago.edu/swift/guides/tutorial.php">here</a>.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_swift_basics">2. Swift Basics</h2>
+<div class="sectionbody">
+<div class="sect2">
+<h3 id="_installation">2.1. Installation</h3>
+<div class="paragraph"><p>This section takes you through the installation of the Swift system on your
+computer. We will start with the prerequisites as explained in the subsequent
+section.</p></div>
+<div class="sect3">
+<h4 id="_prerequisites">2.1.1. Prerequisites</h4>
+<div class="paragraph"><div class="title">Check your Java</div><p>Swift is a Java application. Make sure you are running Java version 5 or higher. You
+can make sure you have Java in your $PATH (or $HOME/.soft file depending upon
+your environment)</p></div>
+<div class="paragraph"><p>Following are the possible ways to detect and run Java:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>$ grep java $HOME/.soft
+#+java-sun # Gives you Java 5
++java-1.6.0_03-sun-r1
+$ which java
+/soft/java-1.6.0_11-sun-r1/bin/java
+$ java -version
+java version "1.6.0_11"
+Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
+Java HotSpot(TM) 64-Bit Server VM (build 11.0-b16, mixed mode)</tt></pre>
+</div></div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_setting_up_to_run_swift">2.2. Setting up to run Swift</h3>
+<div class="paragraph"><p>This is simple. We will be using a pre-compiled version of Swift that can be
+downloaded from <a href="http://www.ci.uchicago.edu/swift/downloads/index.php">here</a>. Download and untar the latest precompiled version as follows:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>$ tar xf swift-0.92.1.tar.gz</tt></pre>
+</div></div>
+<div class="sect3">
+<h4 id="_environment_setup">2.2.1. Environment Setup</h4>
+<div class="paragraph"><p>The examples were tested with Java version 1.6. Make sure you do not already
+have Swift in your PATH. If you do, remove it, or remove any +swift or @swift
+lines from your $HOME/.soft or $HOME/.bash_profile file. Then do:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>PATH=$PATH:/path/to/swift/bin</tt></pre>
+</div></div>
+<div class="paragraph"><p>Note that the environment will be different when using Swift from prebuilt distribution (as above) and trunk. The PATH setup when using swift from trunk would be as follows:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>PATH=$PATH:/path/to/swift/dist/swift-svn/bin</tt></pre>
+</div></div>
+<div class="admonitionblock">
+<table><tr>
+<td class="icon">
+<div class="title">Warning</div>
+</td>
+<td class="content">Do NOT set SWIFT_HOME or CLASSPATH in your environment unless you
+fully understand how these will affect Swift’s execution.</td>
+</tr></table>
+</div>
+<div class="paragraph"><p>To execute your Swift script on a login host (or "localhost") use
+the following command:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>swift -tc.file tc somescript.swift</tt></pre>
+</div></div>
+</div>
+<div class="sect3">
+<h4 id="_setting_transformation_catalog">2.2.2. Setting transformation catalog</h4>
+<div class="paragraph"><p>The transformation catalog lists where application executables are located on
+remote sites.</p></div>
+<div class="paragraph"><p>By default, the site catalog is stored in etc/tc.data. This path can be
+overridden with the tc.file configuration property, either in the Swift
+configuration file or on the command line.</p></div>
+<div class="paragraph"><p>The format is one line per executable per site, with fields separated by tabs
+or spaces.</p></div>
+<div class="paragraph"><p>Some example entries:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>localhost echo /bin/echo INSTALLED INTEL32::LINUX null
+TGUC touch /usr/bin/touch INSTALLED INTEL32::LINUX GLOBUS::maxwalltime="00:00:10"</tt></pre>
+</div></div>
+<div class="paragraph"><p>The fields are: <em>site</em>, <em>transformation-name</em>, <em>executable-path</em>, <em>installation-status</em>, <em>platform</em>, and <em>profile</em> entries.</p></div>
+<div class="paragraph"><p>The <em>site</em> field should correspond to a site name listed in the sites catalog.</p></div>
+<div class="paragraph"><p>The <em>transformation-name</em> should correspond to the transformation name used in a
+SwiftScript app procedure.</p></div>
+<div class="paragraph"><p>The <em>executable-path</em> should specify where the particular executable is located
+on that site.</p></div>
+<div class="paragraph"><p>The <em>installation-status</em> and <em>platform</em> fields are not used. Set them to
+<strong>INSTALLED</strong> and <strong>INTEL32::LINUX</strong> respectively.</p></div>
+<div class="paragraph"><p>The <em>profiles</em> field should be set to null if no profile entries are to be
+specified, or should contain the profile entries separated by semicolons.</p></div>
+</div>
+<div class="sect3">
+<h4 id="_setting_swift_configuration">2.2.3. Setting swift configuration</h4>
+<div class="paragraph"><p>Many configuration properties could be set using the Swift configuration file.
+We will not cover them all in this section. see
+<a href="http://www.ci.uchicago.edu/swift/guides/userguide.php#engineconfiguration">here</a> for details. In this section we will cover a simple configuration file with the most basic properties.</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt># A comment
+wrapperlog.always.transfer=true
+sitedir.keep=true
+execution.retries=1
+lazy.errors=true
+status.mode=provider
+use.provider.staging=true
+provider.staging.pin.swiftfiles=false
+clustering.enabled=false
+clustering.queue.delay=10
+clustering.min.time=86400
+foreach.max.threads=100
+provenance.log=true</tt></pre>
+</div></div>
+</div>
+<div class="sect3">
+<h4 id="_setting_sites_xml">2.2.4. Setting sites.xml</h4>
+<div class="paragraph"><p>sites.xml specifies details of the sites that Swift can run on. Following is
+an example of a simple sites.xml file entry for running Swift on local
+environment:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.4
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="font-weight: bold"><span style="color: #0000FF"><pool</span></span> <span style="color: #009900">handle</span><span style="color: #990000">=</span><span style="color: #FF0000">"localhost"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><filesystem</span></span> <span style="color: #009900">provider</span><span style="color: #990000">=</span><span style="color: #FF0000">"local"</span> <span style="font-weight: bold"><span style="color: #0000FF">/></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><execution</span></span> <span style="color: #009900">provider</span><span style="color: #990000">=</span><span style="color: #FF0000">"local"</span> <span style="font-weight: bold"><span style="color: #0000FF">/></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><workdirectory</span></span> <span style="font-weight: bold"><span style="color: #0000FF">></span></span>/var/tmp<span style="font-weight: bold"><span style="color: #0000FF"></workdirectory></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"karajan"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"jobThrottle"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>.07<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"karajan"</span>
+<span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"initialScore"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>100000<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"></pool></span></span></tt></pre></div></div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_first_swiftscript">2.3. First SwiftScript</h3>
+<div class="paragraph"><p>Your first SwiftScript
+Hello Swift-World!</p></div>
+<div class="paragraph"><p>A good sanity check that Swift is set up and running OK locally is this:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>$ which swift
+
+/home/wilde/swift/src/stable/cog/modules/swift/dist/swift-svn/bin/swift
+
+$ echo 'trace("Hello, Swift world!");' >hello.swift
+
+$ swift hello.swift
+
+Swift svn swift-r3202 cog-r2682
+
+RunID: 20100115-1240-6xhzxuz3
+
+Progress:
+
+SwiftScript trace: Hello, Swift world!
+
+Final status:
+
+$</tt></pre>
+</div></div>
+<div class="paragraph"><p>A good first tutorial in using Swift is at:
+<a href="http://www.ci.uchicago.edu/swift/guides/tutorial.php">http://www.ci.uchicago.edu/swift/guides/tutorial.php</a>. Follow the steps in that
+tutorial to learn how to run a few simple scripts on the login host.</p></div>
+</div>
+<div class="sect2">
+<h3 id="_second_swiftscript">2.4. second SwiftScript</h3>
+<div class="paragraph"><p>Following is a more involved Swift script.</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.4
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #008080">type</span> file<span style="color: #990000">;</span>
+
+<span style="font-weight: bold"><span style="color: #000000">app</span></span> <span style="color: #990000">(</span><span style="color: #008080">file</span> o<span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #000000">cat</span></span> <span style="color: #990000">(</span><span style="color: #008080">file</span> i<span style="color: #990000">)</span>
+<span style="color: #FF0000">{</span>
+ cat @<span style="color: #008080">i</span> stdout<span style="color: #990000">=</span>@o<span style="color: #990000">;</span>
+<span style="color: #FF0000">}</span>
+
+<span style="color: #008080">file</span> out<span style="color: #990000">[]<</span>simple_mapper<span style="color: #990000">;</span> location<span style="color: #990000">=</span><span style="color: #FF0000">"outdir"</span><span style="color: #990000">,</span> prefix<span style="color: #990000">=</span><span style="color: #FF0000">"f."</span><span style="color: #990000">,</span>suffix<span style="color: #990000">=</span><span style="color: #FF0000">".out"</span><span style="color: #990000">>;</span>
+
+foreach <span style="color: #008080">j</span> in <span style="color: #990000">[</span><span style="color: #993399">1</span><span style="color: #990000">:</span>@<span style="font-weight: bold"><span style="color: #000000">toint</span></span><span style="color: #990000">(</span>@<span style="font-weight: bold"><span style="color: #000000">arg</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"n"</span><span style="color: #990000">,</span><span style="color: #FF0000">"1"</span><span style="color: #990000">))]</span> <span style="color: #FF0000">{</span>
+
+ <span style="color: #008080">file</span> data<span style="color: #990000"><</span><span style="color: #FF0000">"data.txt"</span><span style="color: #990000">>;</span>
+
+ out<span style="color: #990000">[</span>j<span style="color: #990000">]</span> <span style="color: #990000">=</span> <span style="font-weight: bold"><span style="color: #000000">cat</span></span><span style="color: #990000">(</span>data<span style="color: #990000">);</span>
+<span style="color: #FF0000">}</span></tt></pre></div></div>
+</div>
+<div class="sect2">
+<h3 id="_swift_commandline_options">2.5. Swift Commandline Options</h3>
+<div class="paragraph"><p>A description of Swift Commandline Options</p></div>
+<div class="paragraph"><p>Also includes a description of Swift inputs and outputs.</p></div>
+</div>
+<div class="sect2">
+<h3 id="_what_if_swift_hangs">2.6. What if Swift hangs</h3>
+<div class="paragraph"><p>Owing to its highly multithreaded architecture it is often the case that the
+underlying java virtual machine gets into deadlock situations or Swift hangs
+because of other complications in its threaded operations. Under such
+situations, Swift <em>hang-checker</em> chips in and resolves the situation.</p></div>
+<div class="olist arabic"><ol class="arabic">
+<li>
+<p>
+how to use the information to identify and correct the deadlock.
+</p>
+</li>
+<li>
+<p>
+How close to the Swift source code can we make the hang-checker messages, so that the user can relate it to Swift functions, expressions, and ideally source code lines?
+</p>
+</li>
+<li>
+<p>
+The current Hang Checker output is actually <strong>very</strong> nice and useful already:
+</p>
+</li>
+</ol></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>Registered futures:
+Rupture[] rups Closed, 1 elements, 0 listeners
+Variation vars - Closed, no listeners
+SgtDim sub - Open, 1 listeners
+string site Closed, no listeners
+Variation[] vars Closed, 72 elements, 0 listeners</tt></pre>
+</div></div>
+</div>
+<div class="sect2">
+<h3 id="_resuming_a_stopped_or_crashed_swift_run">2.7. Resuming a stopped or crashed Swift Run</h3>
+<div class="paragraph"><p>I had a .rlog file from a Swift run that ran out of time. I kicked it off
+using the -resume flag described in section 16.2 of the Swift User Guide and
+it picked up where it left off. Then I killed it because I wanted to make
+changes to my sites file.</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>. . .
+Progress: Selecting site:1150 Stage in:55 Active:3 Checking status:1
+Stage out:37 Finished in previous run:2462 Finished successfully:96
+Progress: Selecting site:1150 Stage in:55 Active:2 Checking status:1
+Stage out:38 Finished in previous run:2462 Finished successfully:96
+Cleaning up...
+Shutting down service at https://192.5.86.6:54813
+Got channel MetaChannel: 1293358091 -> null
++ Done
+Canceling job 9297.svc.pads.ci.uchicago.edu</tt></pre>
+</div></div>
+<div class="paragraph"><p>No new rlog file was emitted but it did recognize the progress that had been
+made, the 96 tasks that finished sucessfully above and resumed from 2558 tasks
+finished.</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>[nbest at login2 files]$ pwd
+/home/nbest/bigdata/files
+[nbest at login2 files]$
+~wilde/swift/src/stable/cog/modules/swift/dist/swift-svn/bin/swift \
+> -tc.file tc -sites.file pbs.xml ~/scripts/mcd12q1.swift -resume
+> mcd12q1-20100310-1326-ptxe1x1d.0.rlog
+Swift svn swift-r3255 (swift modified locally) cog-r2723 (cog modified
+locally)
+RunID: 20100311-1027-148caf0a
+Progress:
+Progress: uninitialized:4
+Progress: Selecting site:671 Initializing site shared directory:1 Finished
+in previous run:1864
+Progress: uninitialized:1 Selecting site:576 Stage in:96 Finished in
+previous run:1864
+Progress: Selecting site:1150 Stage in:94 Submitting:2 Finished in
+previous run:2558
+Progress: Selecting site:1150 Stage in:94 Submitted:2 Finished in previous
+run:2558
+Progress: Selecting site:1150 Stage in:93 Submitting:1 Submitted:2
+Finished in previous run:2558
+Progress: Selecting site:1150 Stage in:90 Submitting:1 Submitted:5
+Finished in previous run:2558
+Progress: Selecting site:1150 Stage in:90 Submitted:5 Active:1 Finished
+in previous run:2558</tt></pre>
+</div></div>
+<div class="paragraph"><p>From Neil: A comment about that section of the user guide: It says "In order
+to restart from a restart log file, the -resume logfile argument can be used
+after the SwiftScript? program file name." and then puts the -resume logfile
+argument before the script file name. I’m sure the order doesn’t matter but
+the contradiction is confusing.</p></div>
+<div class="paragraph"><p>Notes to add (from Mike):</p></div>
+<div class="ulist"><ul>
+<li>
+<p>
+explain what aspects of a Swift script make it restartable, and which
+ aspects are notrestartable. Eg, if your mappers can return different data at
+different times, what happens? What other non-determinsitc behavior would
+cause unpredictable, unexpected, or undesired behavior on resumption?
+</p>
+</li>
+<li>
+<p>
+explain what changes you can make in the execution environment (eg
+ increasing or reducing CPUs to run on or throttles, etc); fixing tc.data
+entries, env vars, or apps, etc.
+</p>
+</li>
+<li>
+<p>
+note that resume will again retry failed app() calls. Explain if the retry
+ count starts over or not.
+</p>
+</li>
+<li>
+<p>
+explain how to resume after multiple failures and resumes - i.e. if a .rlog
+ is generated on each run, which one should you resume from? Do you have a
+choice of resuming from any of them, and what happens if you go backwards to
+an older resume file?
+</p>
+</li>
+<li>
+<p>
+whap happens when you kill (eg with <sup>C) a running swift script? Is the
+ signal caught, and the resume file written out at that point? Or written out
+all along? (Note case in which script ws running for hours, then hit </sup>C, but
+resume fie was short (54 bbytes) and swift shows no sign of doing a resume?
+(It silently ignored resume file instead of acknowleging that it found one
+with not useful resume state in it???) Swift should clearly state that its
+resuming and what its resume state is.
+</p>
+</li>
+</ul></div>
+<div class="paragraph"><p><tt>swift -resume ftdock-[id].0.rlog \[rest of the exact command line from initial
+run\]</tt></p></div>
+</div>
+<div class="sect2">
+<h3 id="_passing_an_array_to_swift">2.8. Passing an array to swift?</h3>
+<div class="paragraph"><p>Arrays can be passed to Swift in one of the following ways:</p></div>
+<div class="olist arabic"><ol class="arabic">
+<li>
+<p>
+You can write the array to a file and read in in swift using
+readData (or readData2).
+</p>
+</li>
+<li>
+<p>
+Direct the array into a file (possibly with a "here document" which expands the array) and then read the file in Swift with readData() or process it with a Swift app() function?
+</p>
+</li>
+<li>
+<p>
+You can use @strsplit on a comma separated command line arg and that works well for me.
+</p>
+</li>
+</ol></div>
+<div class="sect3">
+<h4 id="_mappers">2.8.1. Mappers</h4>
+<div class="paragraph"><p>SimpleMapper</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>$ cat swiftapply.swift</tt></pre>
+</div></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.4
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #008080">type</span> RFile<span style="color: #990000">;</span>
+<span style="font-weight: bold"><span style="color: #000000">trace</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"hi 1"</span><span style="color: #990000">);</span>
+<span style="font-weight: bold"><span style="color: #000000">app</span></span> <span style="color: #990000">(</span><span style="color: #008080">RFile</span> result<span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #000000">RunR</span></span> <span style="color: #990000">(</span><span style="color: #008080">RFile</span> rcall<span style="color: #990000">)</span>
+<span style="color: #FF0000">{</span>
+ RunR @rcall @result<span style="color: #990000">;</span>
+<span style="color: #FF0000">}</span>
+<span style="font-weight: bold"><span style="color: #000000">trace</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"hi 2"</span><span style="color: #990000">);</span>
+<span style="color: #008080">RFile</span> rcalls<span style="color: #990000">[]</span> <span style="color: #990000">;</span>
+<span style="color: #008080">RFile</span> results<span style="color: #990000">[]</span> <span style="color: #990000">;</span>
+<span style="font-weight: bold"><span style="color: #000000">trace</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"start"</span><span style="color: #990000">);</span>
+<span style="color: #008080">foreach</span> c<span style="color: #990000">,</span> i <span style="color: #008080">in</span> rcalls <span style="color: #FF0000">{</span>
+ <span style="font-weight: bold"><span style="color: #000000">trace</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"c"</span><span style="color: #990000">,</span>i<span style="color: #990000">,</span>@c<span style="color: #990000">);</span>
+ <span style="font-weight: bold"><span style="color: #000000">trace</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"r"</span><span style="color: #990000">,</span>i<span style="color: #990000">,</span>@<span style="font-weight: bold"><span style="color: #000000">filename</span></span><span style="color: #990000">(</span>results<span style="color: #990000">[</span>i<span style="color: #990000">]));</span>
+ results<span style="color: #990000">[</span>i<span style="color: #990000">]</span> <span style="color: #990000">=</span> <span style="font-weight: bold"><span style="color: #000000">RunR</span></span><span style="color: #990000">(</span>c<span style="color: #990000">);</span>
+<span style="color: #FF0000">}</span></tt></pre></div></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>$ ls calldir resdir
+calldir:
+rcall.1.Rdata rcall.2.Rdata rcall.3.Rdata rcall.4.Rdata
+resdir:
+result.1.Rdata result.2.Rdata result.3.Rdata result.4.Rdata
+$</tt></pre>
+</div></div>
+<div class="paragraph"><p>Notes:</p></div>
+<div class="paragraph"><p>how the .'s match
+prefix and suffix dont span dirs
+intervening pattern must be digits
+these digits become the array indices
+explain how padding= arg works & helps (including padding=0)
+figure out and explain differences between simple_mapper and
+filesys_mapper
+FIXME: Use the "filesys_mapper" and its "location=" parameter to map the
+input data from /home/wilde/bigdata/*</p></div>
+<div class="paragraph"><p>Abbreviations for SingleFileMapper
+Notes:</p></div>
+<div class="paragraph"><p>within <> you can only have a literal string as in <"filename">, not an
+expression. Someday we will fix this to make <> accept a general expression.
+you can use @filenames( ) (note: plural) to pull off a list of filenames.</p></div>
+<div class="paragraph"><p>writeData()</p></div>
+<div class="paragraph"><p>example here</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>$ cat writedata.swift</tt></pre>
+</div></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.4
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #008080">type</span> file<span style="color: #990000">;</span>
+
+<span style="color: #008080">file</span> f <span style="color: #990000"><</span><span style="color: #FF0000">"filea"</span><span style="color: #990000">>;</span>
+<span style="color: #008080">file</span> nf <span style="color: #990000"><</span><span style="color: #FF0000">"filenames"</span><span style="color: #990000">>;</span>
+nf <span style="color: #990000">=</span> <span style="font-weight: bold"><span style="color: #000000">writeData</span></span><span style="color: #990000">(</span>@f<span style="color: #990000">);</span></tt></pre></div></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>$ swift writedata.swift
+Swift svn swift-r3264 (swift modified locally) cog-r2730 (cog modified
+locally)
+RunID: 20100319-2002-s9vpo0pe
+Progress:
+Final status:
+$ cat filenames
+filea$</tt></pre>
+</div></div>
+<div class="paragraph"><p>StructuredRegexpMapper
+IN PROGRESS This mapper can be used to base the mapped filenames of an output
+array on the mapped filenames of an existing array. landuse outputfiles[]
+<structured_regexp_mapper; source=inputfiles,
+location="./output",match="(.)*tif", transform="\\1histogram">;</p></div>
+<div class="paragraph"><p>Use the undocumented "structured_regexp_mapper" to name the output
+filenames based on the input filenames:</p></div>
+<div class="paragraph"><p>For example:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>login2$ ls /home/wilde/bigdata/data/sample
+h11v04.histogram h11v05.histogram h12v04.histogram h32v08.histogram
+h11v04.tif h11v05.tif h12v04.tif h32v08.tif
+login2$
+
+login2$ cat regexp2.swift
+type tif;
+type mytype;
+
+tif images[]<filesys_mapper;
+location="/home/wilde/bigdata/data/sample", prefix="h", suffix=".tif">;
+
+mytype of[] <structured_regexp_mapper; source=images, match="(h..v..)",
+transform="output/myfile.\\1.mytype">;
+
+foreach image, i in images {
+ trace(i, at filename(images));
+ trace(i, at filename(of[i]));
+}
+login2$
+
+login1$ swift regexp2.swift
+Swift svn swift-r3255 (swift modified locally) cog-r2723 (cog modified
+locally)
+
+RunID: 20100310-1105-4okarq08
+Progress:
+SwiftScript trace: 1, output/myfile.h11v04.mytype
+SwiftScript trace: 2, home/wilde/bigdata/data/sample/h11v05.tif
+SwiftScript trace: 3, home/wilde/bigdata/data/sample/h12v04.tif
+SwiftScript trace: 0, output/myfile.h32v08.mytype
+SwiftScript trace: 0, home/wilde/bigdata/data/sample/h32v08.tif
+SwiftScript trace: 3, output/myfile.h12v04.mytype
+SwiftScript trace: 1, home/wilde/bigdata/data/sample/h11v04.tif
+SwiftScript trace: 2, output/myfile.h11v05.mytype
+Final status:
+login1$</tt></pre>
+</div></div>
+</div>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_coasters">3. Coasters</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>Coasters were introduced in Swift v0.6 as an experimental feature. In many
+applications, Swift performance can be greatly enhanced by the use of CoG
+coasters. CoG coasters provide a low-overhead job submission and file transfer
+mechanism suited for the execution of short jobs (on the order of a few
+seconds). A detailed information on coasters can be found at <a href="http://www.ci.uchicago.edu/swift/guides/userguide.php#coasters">http://www.ci.uchicago.edu/swift/guides/userguide.php#coasters</a>.</p></div>
+<div class="paragraph"><p>Following is a coasters setup case-study for a PBS underlying provider where sites.xml coaster settings were:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.4
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="font-weight: bold"><span style="color: #0000FF"><execution</span></span> <span style="color: #009900">provider</span><span style="color: #990000">=</span><span style="color: #FF0000">"coaster"</span> <span style="color: #009900">jobmanager</span><span style="color: #990000">=</span><span style="color: #FF0000">"local:pbs"</span><span style="font-weight: bold"><span style="color: #0000FF">/></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"project"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>CI-CCR000013<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+
+<span style="font-style: italic"><span style="color: #9A1900"><!-- Note that the following is going to be defunct in the new version (0.93+) and replaced by</span></span>
+<span style="font-style: italic"><span style="color: #9A1900">"ProviderAttributes" key and may not work in the future Swift versions--></span></span>
+
+<span style="font-style: italic"><span style="color: #9A1900"><!--<profile namespace="globus" key="ppn">24:cray:pack</profile>--></span></span>
+
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"providerAttributes"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>
+pbs.aprun
+pbs.mpp=true
+<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"workersPerNode"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>24<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"maxTime"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>100000<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"lowOverallocation"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>100<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"highOverallocation"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>100<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"slots"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>20<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"nodeGranularity"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>5<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"maxNodes"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>5<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"karajan"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"jobThrottle"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>20.00<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"karajan"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"initialScore"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>10000<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span></tt></pre></div></div>
+<div class="paragraph"><p>The following table briefly describes the elements on the coasters setup:</p></div>
+<div class="tableblock">
+<table rules="all"
+width="70%"
+frame="border"
+cellspacing="0" cellpadding="4">
+<col width="23%" />
+<col width="76%" />
+<thead>
+<tr>
+<th align="center" valign="top">profile key </th>
+<th align="left" valign="top"> brief description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td align="center" valign="top"><p class="table">slots</p></td>
+<td align="left" valign="top"><p class="table">How many maximum LRM jobs/worker blocks are allowed</p></td>
+</tr>
+<tr>
+<td align="center" valign="top"><p class="table">workersPerNode</p></td>
+<td align="left" valign="top"><p class="table">How many coaster workers to run per execution node</p></td>
+</tr>
+<tr>
+<td align="center" valign="top"><p class="table">nodeGranularity</p></td>
+<td align="left" valign="top"><p class="table">Each worker block uses a number of nodes that is a multiple of this number</p></td>
+</tr>
+<tr>
+<td align="center" valign="top"><p class="table">lowOverallocation</p></td>
+<td align="left" valign="top"><p class="table">How many times larger than the job walltime should a block’s walltime be if all jobs are 1s long</p></td>
+</tr>
+<tr>
+<td align="center" valign="top"><p class="table">highOverallocation</p></td>
+<td align="left" valign="top"><p class="table">How many times larger than the job walltime should a block’s walltime be if all jobs are infinitely long</p></td>
+</tr>
+<tr>
+<td align="center" valign="top"><p class="table">workersPerNode</p></td>
+<td align="left" valign="top"><p class="table">How many coaster workers to run per execution node reserve How many seconds to reserve in a block’s walltime for starting/shutdown operations</p></td>
+</tr>
+<tr>
+<td align="center" valign="top"><p class="table">maxnodes</p></td>
+<td align="left" valign="top"><p class="table">The maximum number of nodes allowed in a block</p></td>
+</tr>
+<tr>
+<td align="center" valign="top"><p class="table">maxtime</p></td>
+<td align="left" valign="top"><p class="table">The maximum number of walltime allowed for a block coaster service</p></td>
+</tr>
+<tr>
+<td align="center" valign="top"><p class="table">jobThrottle</p></td>
+<td align="left" valign="top"><p class="table">the number of concurrent jobs allowed on a site</p></td>
+</tr>
+</tbody>
+</table>
+</div>
+<div class="sect2">
+<h3 id="_for_advanced_users">3.1. For Advanced Users</h3>
+<div class="paragraph"><p>One of the main reason that one would initially deviate from coaster
+defaults into more complex pool entries is to force jobs to fit into some
+site-imposed constraint. For instance a typical submission to the experimental
+queue requires a user to request upto 3 nodes for under 1 hour. This setup
+could be achieved with a careful tuning of coaters parameters.</p></div>
+</div>
+<div class="sect2">
+<h3 id="_coaster_providers_local_ssh_pbs">3.2. Coaster providers: local, ssh, pbs</h3>
+<div class="paragraph"><p>Settings and examples for different coaster providers mechanisms.</p></div>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_swift_on_diverse_infrastructures">4. Swift on Diverse Infrastructures</h2>
+<div class="sectionbody">
+<div class="sect2">
+<h3 id="_beagle">4.1. Beagle</h3>
+<div class="paragraph"><p>Swift is now installed on Beagle as a module. Swift supports a Coasters based,
+computing environment for Beagle. A detailed Swift documentation is maintained
+[[<a href="http://www.ci.uchicago.edu/swift/docs/index.php]">here</a>]. To get started
+with Swift on Beagle follow the steps outlined below:</p></div>
+<div class="paragraph"><p><strong>step 1.</strong> Load the Swift module on Beagle as follows: <tt>module load swift</tt></p></div>
+<div class="paragraph"><p><strong>step 2.</strong> Create and change to a directory where your Swift related work will
+stay. (say, <tt>mkdir swift-lab</tt>, followed by, <tt>cd swift-lab</tt>)</p></div>
+<div class="paragraph"><p><strong>step 3.</strong> To get started with a simple example running <tt>/bin/cat</tt> to read an input file <tt>data.txt</tt> and write to an output file <tt>f.nnn.out</tt>, copy the folder at <tt>/home/ketan/labs/catsn</tt> to the above directory. (<tt>cp -r /home/ketan/catsn .</tt> followed by <tt>cd catsn</tt>).</p></div>
+<div class="paragraph"><p><strong>step 4.</strong> In the sites file: <tt>beagle-coaster.xml</tt>, make the following two
+changes: <strong>1)</strong> change the path of <tt>workdirectory</tt> to your preferred location
+(say to <tt>/lustre/beagle/$USER/swift-lab/swift.workdir</tt>) and <strong>2)</strong> Change the
+project name to your project (<tt>CI-CCR000013</tt>) . The workdirectory will contain
+execution data related to each run, e.g. wrapper scripts, system information,
+inputs and outputs.</p></div>
+<div class="paragraph"><p><strong>step 5.</strong> Run the example using following commandline (also found in run.sh):
+<tt>swift -config cf -tc.file tc -sites.file beagle-coaster.xml catsn.swift -n=1</tt>
+. You can further change the value of <tt>-n</tt> to any arbitrary number to run that
+many number of concurrent <tt>cat</tt></p></div>
+<div class="paragraph"><p><strong>step 6.</strong> Check the output in the generated <tt>outdir</tt> directory (<tt>ls outdir</tt>)</p></div>
+<div class="paragraph"><p>Note: Running from sandbox node or requesting 1 hour walltime for upto 3 nodes
+will get fast prioritized execution. Good for small tests</p></div>
+</div>
+<div class="sect2">
+<h3 id="_pads">4.2. PADS</h3>
+<div class="paragraph"><p>Swift on PADS
+To execute your Swift script on the PADS cluster use this command:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>swift -tc.file tc -sites.file pbs.xml modis.swift</tt></pre>
+</div></div>
+<div class="paragraph"><p>where the contents of a simple pbs.xml sites file could be:</p></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.4
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="font-weight: bold"><span style="color: #0000FF"><config></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><pool</span></span> <span style="color: #009900">handle</span><span style="color: #990000">=</span><span style="color: #FF0000">"pbs"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><execution</span></span> <span style="color: #009900">provider</span><span style="color: #990000">=</span><span style="color: #FF0000">"pbs"</span> <span style="color: #009900">url</span><span style="color: #990000">=</span><span style="color: #FF0000">"none"</span><span style="font-weight: bold"><span style="color: #0000FF">/></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"queue"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>fast<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"maxwalltime"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>00:05:00<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"karajan"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"initialScore"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>10000<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"karajan"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"jobThrottle"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>.10<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><filesystem</span></span> <span style="color: #009900">provider</span><span style="color: #990000">=</span><span style="color: #FF0000">"local"</span><span style="font-weight: bold"><span style="color: #0000FF">/></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><workdirectory</span></span> <span style="font-weight: bold"><span style="color: #0000FF">></span></span>/home/you/swiftwork<span style="font-weight: bold"><span style="color: #0000FF"></workdirectory></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"></pool></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"></config></span></span></tt></pre></div></div>
+</div>
+<div class="sect2">
+<h3 id="_osg">4.3. OSG</h3>
+<div class="paragraph"><p>This section describes how to get Swift running on the OSG Grid. We will use a
+manual coaster setup to get Swift running on OSG.</p></div>
+<div class="paragraph"><div class="title">Coaster setup on OSG</div><p>The following figure shows an abstract scheme for the manual coasters setup on
+OSG.</p></div>
+<div class="imageblock">
+<div class="content">
+<img src="figures/coaster_setup.png" alt="Coaster setup" />
+</div>
+</div>
+<div class="paragraph"><p>In the following steps, we will go through the process of manually setting</p></div>
+</div>
+<div class="sect2">
+<h3 id="_bionimbus">4.4. Bionimbus</h3>
+<div class="paragraph"><p>This section explains a step by step procedure on getting Swift running on the
+Bionimbus cloud. We will use the <em>manual</em> <em>coasters</em> configuration on the
+Bionimbus cloud.</p></div>
+<div class="paragraph"><p><strong>step1.</strong> Connect to the gateway (ssh gatewayx.lac.uic.edu)</p></div>
+<div class="paragraph"><p><strong>step2.</strong> Start a virtual machine (euca-run-instances -n 1 -t m1.small
+emi-17EB1170)</p></div>
+<div class="paragraph"><p><strong>step3.</strong> Start the coaster-service on gateway
+<tt>coaster-service -port 1984 -localport 35753 -nosec</tt></p></div>
+<div class="paragraph"><p><strong>step4.</strong> Start the Swift-script from the gateway using normal Swift commandline</p></div>
+<div class="paragraph"><p><tt>swift -config cf -tc.file tc -sites.file sites.xml yourscript.swift -aparam=999</tt></p></div>
+<div class="listingblock">
+<div class="title">cf</div>
+<div class="content">
+<pre><tt>wrapperlog.always.transfer=true
+sitedir.keep=true
+execution.retries=1
+lazy.errors=true
+status.mode=provider
+use.provider.staging=true
+provider.staging.pin.swiftfiles=false
+foreach.max.threads=100
+provenance.log=true</tt></pre>
+</div></div>
+<div class="listingblock">
+<div class="title">tc</div>
+<div class="content">
+<pre><tt>localhost modftdock /home/ketan/swift-labs/bionimbus-coaster-modftdock/app/modftdock.sh null null GLOBUS::maxwalltime="1:00:00"</tt></pre>
+</div></div>
+<div class="paragraph"><p>(See below for a sample sites.xml for this run)</p></div>
+<div class="paragraph"><p><strong>step5.</strong> Connect back to gateway from virtual machines using reverse ssh tunneling as follows:</p></div>
+<div class="paragraph"><div class="title">From the gateway prompt</div><p><tt>ssh -R *:5000:localhost:5000 <a href="mailto:root at 10.101.8.50">root at 10.101.8.50</a> sleep 999</tt></p></div>
+<div class="paragraph"><p>WHERE:
+*=network interface, should remain the same on all cases</p></div>
+<div class="paragraph"><p>localhost=the gateway host, should remain the same</p></div>
+<div class="paragraph"><p>5000(LEFT OF localhost)=the port number on localhost to listen to **THIS WILL
+vary depending upon which port you want to listen to</p></div>
+<div class="paragraph"><p>5000(RIGHT OF localhost)=the port on target host that you want to forward</p></div>
+<div class="paragraph"><p><a href="mailto:root at 10.101.8.50">root at 10.101.8.50</a>=the ip of the Virtual Machine on Bionimbus cloud, this will
+vary based on what ip you get for your Virtual Machine instance</p></div>
+<div class="paragraph"><p>#On anywhere as long as provide the correct callback uri: here the
+"http://140.221.9.110:35753" is the callback uri of previous ones</p></div>
+<div class="paragraph"><p><strong>step6.</strong> Copy the <tt>worker.pl</tt> from gatewayx to the VM: <tt>scp
+swift-0.92/bin/worker.pl <a href="mailto:root at 10.101.8.50">root at 10.101.8.50</a>:/bin/</tt>, you will be prompted for the
+root password of the VM.</p></div>
+<div class="paragraph"><p><strong>step7.</strong> Start the worker from the virtual machine
+<tt>worker.pl <a href="http://localhost:35753">http://localhost:35753</a> tmp /tmp</tt>
+where 35753 is the port where the coaster service is listening to the workers+</p></div>
+<div class="listingblock">
+<div class="title">sites.xml for the above run</div>
+<div class="content"><!-- Generator: GNU source-highlight 3.1.4
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="font-weight: bold"><span style="color: #0000FF"><config></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><pool</span></span> <span style="color: #009900">handle</span><span style="color: #990000">=</span><span style="color: #FF0000">"localhost"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><execution</span></span> <span style="color: #009900">provider</span><span style="color: #990000">=</span><span style="color: #FF0000">"coaster-persistent"</span> <span style="color: #009900">url</span><span style="color: #990000">=</span><span style="color: #FF0000">"http://localhost:1984"</span> <span style="color: #009900">jobmanager</span><span style="color: #990000">=</span><span style="color: #FF0000">"local:local"</span><span style="font-weight: bold"><span style="color: #0000FF">/></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"workerManager"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>passive<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"workersPerNode"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>4<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"maxTime"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>10000<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"lowOverAllocation"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>100<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"highOverAllocation"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>100<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"slots"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>100<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"nodeGranularity"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>1<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"maxNodes"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>10<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"karajan"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"jobThrottle"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>25.00<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"karajan"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"initialScore"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>10000<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"swift"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"stagingMethod"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>proxy<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><filesystem</span></span> <span style="color: #009900">provider</span><span style="color: #990000">=</span><span style="color: #FF0000">"local"</span><span style="font-weight: bold"><span style="color: #0000FF">/></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><workdirectory></span></span>/home/ketan/swift-labs/bionimbus-coaster-modftdock/swift.workdir<span style="font-weight: bold"><span style="color: #0000FF"></workdirectory></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"></pool></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"></config></span></span></tt></pre></div></div>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_debugging_swift">5. Debugging Swift</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>Swift errors are logged in several places:</p></div>
+<div class="olist arabic"><ol class="arabic">
+<li>
+<p>
+All text from standard output and standard error produced by running the
+swift command
+</p>
+</li>
+<li>
+<p>
+The .log file from this run. It will be named swiftscript.uniqueID.log
+where "swiftscript" is the name of your *.swift script source file, and
+uniqueID is a long unique id which starts with the date and time you ran the
+swift command.
+</p>
+</li>
+<li>
+<p>
+$HOME/.globus/coasters directory on remote machines on which you are
+running coasters
+</p>
+</li>
+<li>
+<p>
+$HOME/.globus/scripts directory on the host on which you run the Swift
+command, when swift is submitting to a local scheduler (Condor, PBS, SGE,
+Cobalt)
+</p>
+</li>
+<li>
+<p>
+$HOME/.globus/??? on remote systems that you access via Globus
+</p>
+</li>
+</ol></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_log_processing">6. Log Processing</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>To properly generate log plots, you must enable VDL/Karajan logging. Make sure
+log4.properties contains:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>log4j.logger.swift=DEBUG
+log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Cpu=DEBUG</tt></pre>
+</div></div>
+<div class="sect2">
+<h3 id="_make_a_basic_load_plot_from_coasters_cpu_log_lines">6.1. Make a basic load plot from Coasters Cpu log lines</h3>
+<div class="paragraph"><div class="title">Generate the log (may set log4j.logger.swift=INFO for this one)</div><p>(assuming the log is titled swift-run.log)</p></div>
+<div class="listingblock">
+<div class="title">Convert the log times to Unix time</div>
+<div class="content">
+<pre><tt>./iso-to-secs < swift-run.log > swift-run.time</tt></pre>
+</div></div>
+<div class="listingblock">
+<div class="title">Make the start time file (this contains the earliest timestamp)</div>
+<div class="content">
+<pre><tt>make LOG=swift-run.log start-time.tmp</tt></pre>
+</div></div>
+<div class="paragraph"><p>or</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>extract-start-time swift-run.log > start-time.tmp</tt></pre>
+</div></div>
+<div class="listingblock">
+<div class="title">Normalize the transition times</div>
+<div class="content">
+<pre><tt>./normalise-event-start-time < swift-run.time > swift-run.norm</tt></pre>
+</div></div>
+<div class="listingblock">
+<div class="title">Build up a load data file:</div>
+<div class="content">
+<pre><tt>./cpu-job-load.pl < swift-run.norm > load.data</tt></pre>
+</div></div>
+<div class="listingblock">
+<div class="title">Plot with the JFreeChart-based plotter in usertools/plotter:</div>
+<div class="content">
+<pre><tt>lines.zsh load.cfg load.eps load.data</tt></pre>
+</div></div>
+<div class="sect3">
+<h4 id="_make_a_basic_job_completion_plot_from_coasters_cpu_log_lines">6.1.1. Make a basic job completion plot from Coasters Cpu log lines</h4>
+<div class="listingblock">
+<div class="title">Same as above, but, build up a completed data file:</div>
+<div class="content">
+<pre><tt>./cpu-job-completed.pl < swift-run.norm > completed.data</tt></pre>
+</div></div>
+<div class="listingblock">
+<div class="title">Plot with the JFreeChart-based plotter in usertools/plotter:</div>
+<div class="content">
+<pre><tt>lines.zsh completed.cfg completed.eps completed.data</tt></pre>
+</div></div>
+</div>
+<div class="sect3">
+<h4 id="_make_a_basic_block_allocation_plot_from_coasters_block_log_lines">6.1.2. Make a basic Block allocation plot from Coasters Block log lines</h4>
+<div class="paragraph"><p>Same as above, but:</p></div>
+<div class="listingblock">
+<div class="title">Build up a block allocation data file:</div>
+<div class="content">
+<pre><tt>./block-level.pl < swift-run.norm > blocks.data</tt></pre>
+</div></div>
+<div class="listingblock">
+<div class="title">Plot with the JFreeChart-based plotter in usertools/plotter:</div>
+<div class="content">
+<pre><tt>lines.zsh blocks.{cfg,eps,data}</tt></pre>
+</div></div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_problem_reporting">6.2. Problem Reporting</h3>
+<div class="paragraph"><p>When reporting problems to <a href="mailto:swift-user at ci.uchicago.edu">swift-user at ci.uchicago.edu</a>, please attach the
+following files and information:</p></div>
+<div class="olist arabic"><ol class="arabic">
+<li>
+<p>
+tc.data and sites.xml (or whatever you named these files)
+</p>
+</li>
+<li>
+<p>
+your .swift source file and any .swift files it imports
+</p>
+</li>
+<li>
+<p>
+any external mapper scripts called by your .swift script
+</p>
+</li>
+<li>
+<p>
+all text from standard output and standard error produced by running the
+swift command
+</p>
+</li>
+<li>
+<p>
+The .log file from this run. It will be named swiftscript.uniqueID.log
+</p>
+</li>
+<li>
+<p>
+where "swiftscript" is the name of your *.swift script source file, and
+uniqueID is a long unique id which starts with the date and time you ran the
+swift command.
+</p>
+</li>
+<li>
+<p>
+The swift command line you invoked
+</p>
+</li>
+<li>
+<p>
+Any swift.properties entries you over-rode ($HOME/.swift/swift.properties,
+-config.file argument properties file, any changes to etc/swift.proerties from
+your swift distribution)
+</p>
+</li>
+<li>
+<p>
+Which swift distribution you are running (release; svn revisions; other
+local changes you mave have made or included)
+</p>
+</li>
+</ol></div>
+</div>
+</div>
+</div>
+</div>
+<div id="footnotes"><hr /></div>
+<div id="footer">
+<div id="footer-text">
+Version 0.92<br />
+Last updated 2011-05-17 17:02:39 CDT
+</div>
+</div>
+</body>
+</html>
Added: wwwdev/cookbook/cookbook-asciidoc.txt
===================================================================
--- wwwdev/cookbook/cookbook-asciidoc.txt (rev 0)
+++ wwwdev/cookbook/cookbook-asciidoc.txt 2011-08-01 14:23:26 UTC (rev 4896)
@@ -0,0 +1,895 @@
+// To compile use: asciidoc -a toc -n cookbook-asciidoc.txt
+
+
+Swift Cookbook
+==============
+Swift team <swift-user at ci.uchicago.edu>
+v0.92, March 2011
+
+Overview
+--------
+This cookbook covers various recipes involving setting up and running Swift under diverse
+configurations based on the application requirements and the underlying
+infrastructures. The Swift system comprises of SwiftScript language and the
+Swift runtime system. For
+introductory material, consult the Swift tutorial found
+link:http://www.ci.uchicago.edu/swift/guides/tutorial.php[here].
+
+Swift Basics
+------------
+
+Installation
+~~~~~~~~~~~~
+
+This section takes you through the installation of the Swift system on your
+computer. We will start with the prerequisites as explained in the subsequent
+section.
+
+Prerequisites
+^^^^^^^^^^^^^^
+.Check your Java
+Swift is a Java application. Make sure you are running Java version 5 or higher. You
+can make sure you have Java in your $PATH (or $HOME/.soft file depending upon
+your environment)
+
+Following are the possible ways to detect and run Java:
+
+----
+$ grep java $HOME/.soft
+#+java-sun # Gives you Java 5
++java-1.6.0_03-sun-r1
+$ which java
+/soft/java-1.6.0_11-sun-r1/bin/java
+$ java -version
+java version "1.6.0_11"
+Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
+Java HotSpot(TM) 64-Bit Server VM (build 11.0-b16, mixed mode)
+----
+
+Setting up to run Swift
+~~~~~~~~~~~~~~~~~~~~~~~~
+This is simple. We will be using a pre-compiled version of Swift that can be
+downloaded from link:http://www.ci.uchicago.edu/swift/downloads/index.php[here]. Download and untar the latest precompiled version as follows:
+
+----
+$ tar xf swift-0.92.1.tar.gz
+----
+
+Environment Setup
+^^^^^^^^^^^^^^^^^^
+
+The examples were tested with Java version 1.6. Make sure you do not already
+have Swift in your PATH. If you do, remove it, or remove any +swift or @swift
+lines from your $HOME/.soft or $HOME/.bash_profile file. Then do:
+
+----
+PATH=$PATH:/path/to/swift/bin
+----
+
+Note that the environment will be different when using Swift from prebuilt distribution (as above) and trunk. The PATH setup when using swift from trunk would be as follows:
+
+----
+PATH=$PATH:/path/to/swift/dist/swift-svn/bin
+----
+
+WARNING: Do NOT set SWIFT_HOME or CLASSPATH in your environment unless you
+fully understand how these will affect Swift's execution.
+
+To execute your Swift script on a login host (or "localhost") use
+the following command:
+
+----
+swift -tc.file tc somescript.swift
+----
+
+Setting transformation catalog
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The transformation catalog lists where application executables are located on
+remote sites.
+
+By default, the site catalog is stored in etc/tc.data. This path can be
+overridden with the tc.file configuration property, either in the Swift
+configuration file or on the command line.
+
+The format is one line per executable per site, with fields separated by tabs
+or spaces.
+
+Some example entries:
+
+----
+localhost echo /bin/echo INSTALLED INTEL32::LINUX null
+TGUC touch /usr/bin/touch INSTALLED INTEL32::LINUX GLOBUS::maxwalltime="00:00:10"
+----
+The fields are: _site_, _transformation-name_, _executable-path_, _installation-status_, _platform_, and _profile_ entries.
+
+The _site_ field should correspond to a site name listed in the sites catalog.
+
+The _transformation-name_ should correspond to the transformation name used in a
+SwiftScript app procedure.
+
+The _executable-path_ should specify where the particular executable is located
+on that site.
+
+The _installation-status_ and _platform_ fields are not used. Set them to
+**INSTALLED** and **INTEL32::LINUX** respectively.
+
+The _profiles_ field should be set to null if no profile entries are to be
+specified, or should contain the profile entries separated by semicolons.
+
+Setting swift configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Many configuration properties could be set using the Swift configuration file.
+We will not cover them all in this section. see
+link:http://www.ci.uchicago.edu/swift/guides/userguide.php#engineconfiguration[here] for details. In this section we will cover a simple configuration file with the most basic properties.
+
+----
+# A comment
+wrapperlog.always.transfer=true
+sitedir.keep=true
+execution.retries=1
+lazy.errors=true
+status.mode=provider
+use.provider.staging=true
+provider.staging.pin.swiftfiles=false
+clustering.enabled=false
+clustering.queue.delay=10
+clustering.min.time=86400
+foreach.max.threads=100
+provenance.log=true
+----
+
+Setting sites.xml
+^^^^^^^^^^^^^^^^^^
+sites.xml specifies details of the sites that Swift can run on. Following is
+an example of a simple sites.xml file entry for running Swift on local
+environment:
+
+[xml]
+source~~~~~~
+<pool handle="localhost">
+<filesystem provider="local" />
+<execution provider="local" />
+<workdirectory >/var/tmp</workdirectory>
+<profile namespace="karajan" key="jobThrottle">.07</profile>
+<profile namespace="karajan"
+key="initialScore">100000</profile>
+</pool>
+source~~~~~~
+
+First SwiftScript
+~~~~~~~~~~~~~~~~~
+Your first SwiftScript
+Hello Swift-World!
+
+A good sanity check that Swift is set up and running OK locally is this:
+
+----
+$ which swift
+
+/home/wilde/swift/src/stable/cog/modules/swift/dist/swift-svn/bin/swift
+
+$ echo 'trace("Hello, Swift world!");' >hello.swift
+
+$ swift hello.swift
+
+Swift svn swift-r3202 cog-r2682
+
+RunID: 20100115-1240-6xhzxuz3
+
+Progress:
+
+SwiftScript trace: Hello, Swift world!
+
+Final status:
+
+$
+----
+A good first tutorial in using Swift is at:
+http://www.ci.uchicago.edu/swift/guides/tutorial.php. Follow the steps in that
+tutorial to learn how to run a few simple scripts on the login host.
+
+second SwiftScript
+~~~~~~~~~~~~~~~~~~~
+Following is a more involved Swift script.
+[java]
+source~~~~~~~
+type file;
+
+app (file o) cat (file i)
+{
+ cat @i stdout=@o;
+}
+
+file out[]<simple_mapper; location="outdir", prefix="f.",suffix=".out">;
+
+foreach j in [1:@toint(@arg("n","1"))] {
+
+ file data<"data.txt">;
+
+ out[j] = cat(data);
+}
+source~~~~~~~
+
+Swift Commandline Options
+~~~~~~~~~~~~~~~~~~~~~~~~~
+A description of Swift Commandline Options
+
+Also includes a description of Swift inputs and outputs.
+
+What if Swift hangs
+~~~~~~~~~~~~~~~~~~~
+Owing to its highly multithreaded architecture it is often the case that the
+underlying java virtual machine gets into deadlock situations or Swift hangs
+because of other complications in its threaded operations. Under such
+situations, Swift _hang-checker_ chips in and resolves the situation.
+
+. how to use the information to identify and correct the deadlock.
+
+. How close to the Swift source code can we make the hang-checker messages, so that the user can relate it to Swift functions, expressions, and ideally source code lines?
+
+. The current Hang Checker output is actually *very* nice and useful already:
+
+----
+Registered futures:
+Rupture[] rups Closed, 1 elements, 0 listeners
+Variation vars - Closed, no listeners
+SgtDim sub - Open, 1 listeners
+string site Closed, no listeners
+Variation[] vars Closed, 72 elements, 0 listeners
+----
+
+Resuming a stopped or crashed Swift Run
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+I had a .rlog file from a Swift run that ran out of time. I kicked it off
+using the -resume flag described in section 16.2 of the Swift User Guide and
+it picked up where it left off. Then I killed it because I wanted to make
+changes to my sites file.
+
+----
+. . .
+Progress: Selecting site:1150 Stage in:55 Active:3 Checking status:1
+Stage out:37 Finished in previous run:2462 Finished successfully:96
+Progress: Selecting site:1150 Stage in:55 Active:2 Checking status:1
+Stage out:38 Finished in previous run:2462 Finished successfully:96
+Cleaning up...
+Shutting down service at https://192.5.86.6:54813
+Got channel MetaChannel: 1293358091 -> null
++ Done
+Canceling job 9297.svc.pads.ci.uchicago.edu
+----
+
+No new rlog file was emitted but it did recognize the progress that had been
+made, the 96 tasks that finished sucessfully above and resumed from 2558 tasks
+finished.
+
+----
+[nbest at login2 files]$ pwd
+/home/nbest/bigdata/files
+[nbest at login2 files]$
+~wilde/swift/src/stable/cog/modules/swift/dist/swift-svn/bin/swift \
+> -tc.file tc -sites.file pbs.xml ~/scripts/mcd12q1.swift -resume
+> mcd12q1-20100310-1326-ptxe1x1d.0.rlog
+Swift svn swift-r3255 (swift modified locally) cog-r2723 (cog modified
+locally)
+RunID: 20100311-1027-148caf0a
+Progress:
+Progress: uninitialized:4
+Progress: Selecting site:671 Initializing site shared directory:1 Finished
+in previous run:1864
+Progress: uninitialized:1 Selecting site:576 Stage in:96 Finished in
+previous run:1864
+Progress: Selecting site:1150 Stage in:94 Submitting:2 Finished in
+previous run:2558
+Progress: Selecting site:1150 Stage in:94 Submitted:2 Finished in previous
+run:2558
+Progress: Selecting site:1150 Stage in:93 Submitting:1 Submitted:2
+Finished in previous run:2558
+Progress: Selecting site:1150 Stage in:90 Submitting:1 Submitted:5
+Finished in previous run:2558
+Progress: Selecting site:1150 Stage in:90 Submitted:5 Active:1 Finished
+in previous run:2558
+----
+
+
+From Neil: A comment about that section of the user guide: It says "In order
+to restart from a restart log file, the -resume logfile argument can be used
+after the SwiftScript? program file name." and then puts the -resume logfile
+argument before the script file name. I'm sure the order doesn't matter but
+the contradiction is confusing.
+
+Notes to add (from Mike):
+
+- explain what aspects of a Swift script make it restartable, and which
+ aspects are notrestartable. Eg, if your mappers can return different data at
+different times, what happens? What other non-determinsitc behavior would
+cause unpredictable, unexpected, or undesired behavior on resumption?
+
+- explain what changes you can make in the execution environment (eg
+ increasing or reducing CPUs to run on or throttles, etc); fixing tc.data
+entries, env vars, or apps, etc.
+
+- note that resume will again retry failed app() calls. Explain if the retry
+ count starts over or not.
+
+- explain how to resume after multiple failures and resumes - i.e. if a .rlog
+ is generated on each run, which one should you resume from? Do you have a
+choice of resuming from any of them, and what happens if you go backwards to
+an older resume file?
+
+- whap happens when you kill (eg with ^C) a running swift script? Is the
+ signal caught, and the resume file written out at that point? Or written out
+all along? (Note case in which script ws running for hours, then hit ^C, but
+resume fie was short (54 bbytes) and swift shows no sign of doing a resume?
+(It silently ignored resume file instead of acknowleging that it found one
+with not useful resume state in it???) Swift should clearly state that its
+resuming and what its resume state is.
+
++swift -resume ftdock-[id].0.rlog \[rest of the exact command line from initial
+run\]+
+
+
+Passing an array to swift?
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+Arrays can be passed to Swift in one of the following ways:
+
+. You can write the array to a file and read in in swift using
+readData (or readData2).
+. Direct the array into a file (possibly with a "here document" which expands the array) and then read the file in Swift with readData() or process it with a Swift app() function?
+. You can use @strsplit on a comma separated command line arg and that works well for me.
+
+Mappers
+^^^^^^^^
+SimpleMapper
+
+----
+$ cat swiftapply.swift
+----
+
+[java]
+source~~~~
+type RFile;
+trace("hi 1");
+app (RFile result) RunR (RFile rcall)
+{
+ RunR @rcall @result;
+}
+trace("hi 2");
+RFile rcalls[] ;
+RFile results[] ;
+trace("start");
+foreach c, i in rcalls {
+ trace("c",i, at c);
+ trace("r",i, at filename(results[i]));
+ results[i] = RunR(c);
+}
+source~~~~
+
+----
+$ ls calldir resdir
+calldir:
+rcall.1.Rdata rcall.2.Rdata rcall.3.Rdata rcall.4.Rdata
+resdir:
+result.1.Rdata result.2.Rdata result.3.Rdata result.4.Rdata
+$
+----
+
+Notes:
+
+how the .'s match
+prefix and suffix dont span dirs
+intervening pattern must be digits
+these digits become the array indices
+explain how padding= arg works & helps (including padding=0)
+figure out and explain differences between simple_mapper and
+filesys_mapper
+FIXME: Use the "filesys_mapper" and its "location=" parameter to map the
+input data from /home/wilde/bigdata/*
+
+Abbreviations for SingleFileMapper
+Notes:
+
+within <> you can only have a literal string as in <"filename">, not an
+expression. Someday we will fix this to make <> accept a general expression.
+you can use @filenames( ) (note: plural) to pull off a list of filenames.
+
+writeData()
+
+example here
+
+----
+$ cat writedata.swift
+----
+
+[java]
+source~~~~
+type file;
+
+file f <"filea">;
+file nf <"filenames">;
+nf = writeData(@f);
+source~~~~
+
+----
+$ swift writedata.swift
+Swift svn swift-r3264 (swift modified locally) cog-r2730 (cog modified
+locally)
+RunID: 20100319-2002-s9vpo0pe
+Progress:
+Final status:
+$ cat filenames
+filea$
+----
+
+StructuredRegexpMapper
+IN PROGRESS This mapper can be used to base the mapped filenames of an output
+array on the mapped filenames of an existing array. landuse outputfiles[]
+<structured_regexp_mapper; source=inputfiles,
+location="./output",match="(.)*tif", transform="\\1histogram">;
+
+Use the undocumented "structured_regexp_mapper" to name the output
+filenames based on the input filenames:
+
+For example:
+
+----
+login2$ ls /home/wilde/bigdata/data/sample
+h11v04.histogram h11v05.histogram h12v04.histogram h32v08.histogram
+h11v04.tif h11v05.tif h12v04.tif h32v08.tif
+login2$
+
+login2$ cat regexp2.swift
+type tif;
+type mytype;
+
+tif images[]<filesys_mapper;
+location="/home/wilde/bigdata/data/sample", prefix="h", suffix=".tif">;
+
+mytype of[] <structured_regexp_mapper; source=images, match="(h..v..)",
+transform="output/myfile.\\1.mytype">;
+
+foreach image, i in images {
+ trace(i, at filename(images));
+ trace(i, at filename(of[i]));
+}
+login2$
+
+login1$ swift regexp2.swift
+Swift svn swift-r3255 (swift modified locally) cog-r2723 (cog modified
+locally)
+
+RunID: 20100310-1105-4okarq08
+Progress:
+SwiftScript trace: 1, output/myfile.h11v04.mytype
+SwiftScript trace: 2, home/wilde/bigdata/data/sample/h11v05.tif
+SwiftScript trace: 3, home/wilde/bigdata/data/sample/h12v04.tif
+SwiftScript trace: 0, output/myfile.h32v08.mytype
+SwiftScript trace: 0, home/wilde/bigdata/data/sample/h32v08.tif
+SwiftScript trace: 3, output/myfile.h12v04.mytype
+SwiftScript trace: 1, home/wilde/bigdata/data/sample/h11v04.tif
+SwiftScript trace: 2, output/myfile.h11v05.mytype
+Final status:
+login1$
+----
+
+Coasters
+--------
+Coasters were introduced in Swift v0.6 as an experimental feature. In many
+applications, Swift performance can be greatly enhanced by the use of CoG
+coasters. CoG coasters provide a low-overhead job submission and file transfer
+mechanism suited for the execution of short jobs (on the order of a few
+seconds). A detailed information on coasters can be found at http://www.ci.uchicago.edu/swift/guides/userguide.php#coasters.
+//**Include neat diagrams.**
+
+Following is a coasters setup case-study for a PBS underlying provider where sites.xml coaster settings were:
+
+[xml]
+source~~~~
+<execution provider="coaster" jobmanager="local:pbs"/>
+<profile namespace="globus" key="project">CI-CCR000013</profile>
+
+<!-- Note that the following is going to be defunct in the new version (0.93+) and replaced by
+"ProviderAttributes" key and may not work in the future Swift versions-->
+
+<!--<profile namespace="globus" key="ppn">24:cray:pack</profile>-->
+
+<profile namespace="globus" key="providerAttributes">
+pbs.aprun
+pbs.mpp=true
+</profile>
+
+<profile namespace="globus" key="workersPerNode">24</profile>
+<profile namespace="globus" key="maxTime">100000</profile>
+
+<profile namespace="globus" key="lowOverallocation">100</profile>
+<profile namespace="globus" key="highOverallocation">100</profile>
+
+<profile namespace="globus" key="slots">20</profile>
+<profile namespace="globus" key="nodeGranularity">5</profile>
+<profile namespace="globus" key="maxNodes">5</profile>
+<profile namespace="karajan" key="jobThrottle">20.00</profile>
+<profile namespace="karajan" key="initialScore">10000</profile>
+source~~~~
+
+The following table briefly describes the elements on the coasters setup:
+[width="70%", cols="^3,10", options="header"]
+|=============================================================================================
+|profile key | brief description
+|slots | How many maximum LRM jobs/worker blocks are allowed
+|workersPerNode | How many coaster workers to run per execution node
+|nodeGranularity | Each worker block uses a number of nodes that is a multiple of this number
+|lowOverallocation | How many times larger than the job walltime should a block's walltime be if all jobs are 1s long
+|highOverallocation | How many times larger than the job walltime should a block's walltime be if all jobs are infinitely long
+|workersPerNode | How many coaster workers to run per execution node reserve How many seconds to reserve in a block's walltime for starting/shutdown operations
+|maxnodes | The maximum number of nodes allowed in a block
+|maxtime | The maximum number of walltime allowed for a block coaster service
+|jobThrottle |the number of concurrent jobs allowed on a site
+|==============================================================================================
+
+//
+//For Beginners
+//~~~~~~~~~~~~~~
+//Coasters for beginners. Usage of existing, prebuilt templates.
+//
+//For Intermediate Users
+//~~~~~~~~~~~~~~~~~~~~~~~
+//Coasters for intermediate users.
+//
+//Using gensites
+//^^^^^^^^^^^^^^^
+//Usage of gensites to generate your own sites
+//configurations.
+//
+//
+//
+
+For Advanced Users
+~~~~~~~~~~~~~~~~~~
+
+//Coasters for advanced users. Getting your hands dirty.
+
+One of the main reason that one would initially deviate from coaster
+defaults into more complex pool entries is to force jobs to fit into some
+site-imposed constraint. For instance a typical submission to the experimental
+queue requires a user to request upto 3 nodes for under 1 hour. This setup
+could be achieved with a careful tuning of coaters parameters.
+
+//How to run Swift under different Coasters configurations
+//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+//
+//Manual
+//^^^^^^^
+//Todo
+//
+//Passive
+//^^^^^^^^
+//Todo
+//
+//Persistent
+//^^^^^^^^^^^
+//Todo
+//
+//Summary of Differences Between different Coaster types
+//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+//A tabular representations of highlights of different coaster setups
+//
+//
+//Data Management
+//
+
+Coaster providers: local, ssh, pbs
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Settings and examples for different coaster providers mechanisms.
+
+
+
+Swift on Diverse Infrastructures
+---------------------------------
+
+Beagle
+~~~~~~
+Swift is now installed on Beagle as a module. Swift supports a Coasters based,
+computing environment for Beagle. A detailed Swift documentation is maintained
+[[http://www.ci.uchicago.edu/swift/docs/index.php][here]]. To get started
+with Swift on Beagle follow the steps outlined below:
+
+*step 1.* Load the Swift module on Beagle as follows: +module load swift+
+
+*step 2.* Create and change to a directory where your Swift related work will
+stay. (say, +mkdir swift-lab+, followed by, +cd swift-lab+)
+
+*step 3.* To get started with a simple example running +/bin/cat+ to read an input file +data.txt+ and write to an output file +f.nnn.out+, copy the folder at +/home/ketan/labs/catsn+ to the above directory. (+cp -r /home/ketan/catsn .+ followed by +cd catsn+).
+
+*step 4.* In the sites file: +beagle-coaster.xml+, make the following two
+changes: *1)* change the path of +workdirectory+ to your preferred location
+(say to +/lustre/beagle/$USER/swift-lab/swift.workdir+) and *2)* Change the
+project name to your project (+CI-CCR000013+) . The workdirectory will contain
+execution data related to each run, e.g. wrapper scripts, system information,
+inputs and outputs.
+
+*step 5.* Run the example using following commandline (also found in run.sh):
++swift -config cf -tc.file tc -sites.file beagle-coaster.xml catsn.swift -n=1+
+. You can further change the value of +-n+ to any arbitrary number to run that
+many number of concurrent +cat+
+
+*step 6.* Check the output in the generated +outdir+ directory (+ls outdir+)
+
+Note: Running from sandbox node or requesting 1 hour walltime for upto 3 nodes
+will get fast prioritized execution. Good for small tests
+
+//
+//Intrepid-BG/P
+//~~~~~~~~~~~~~
+//Swift on Intrepid-BG/P
+//
+
+PADS
+~~~~
+Swift on PADS
+To execute your Swift script on the PADS cluster use this command:
+
+----
+swift -tc.file tc -sites.file pbs.xml modis.swift
+----
+where the contents of a simple pbs.xml sites file could be:
+[xml]
+source~~~~
+<config>
+ <pool handle="pbs">
+ <execution provider="pbs" url="none"/>
+ <profile namespace="globus" key="queue">fast</profile>
+ <profile namespace="globus" key="maxwalltime">00:05:00</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ <profile namespace="karajan" key="jobThrottle">.10</profile>
+ <filesystem provider="local"/>
+ <workdirectory >/home/you/swiftwork</workdirectory>
+ </pool>
+</config>
+source~~~~
+
+OSG
+~~~
+This section describes how to get Swift running on the OSG Grid. We will use a
+manual coaster setup to get Swift running on OSG.
+
+.Coaster setup on OSG
+The following figure shows an abstract scheme for the manual coasters setup on
+OSG.
+
+image::figures/coaster_setup.png[Coaster setup]
+
+In the following steps, we will go through the process of manually setting
+//Swift on OSG
+//
+
+Bionimbus
+~~~~~~~~~
+This section explains a step by step procedure on getting Swift running on the
+Bionimbus cloud. We will use the _manual_ _coasters_ configuration on the
+Bionimbus cloud.
+
+**step1.** Connect to the gateway (ssh gatewayx.lac.uic.edu)
+
+**step2.** Start a virtual machine (euca-run-instances -n 1 -t m1.small
+emi-17EB1170)
+
+**step3.** Start the coaster-service on gateway
++coaster-service -port 1984 -localport 35753 -nosec+
+
+**step4.** Start the Swift-script from the gateway using normal Swift commandline
+
++swift -config cf -tc.file tc -sites.file sites.xml yourscript.swift -aparam=999+
+
+.cf
+----
+wrapperlog.always.transfer=true
+sitedir.keep=true
+execution.retries=1
+lazy.errors=true
+status.mode=provider
+use.provider.staging=true
+provider.staging.pin.swiftfiles=false
+foreach.max.threads=100
+provenance.log=true
+----
+
+.tc
+----
+localhost modftdock /home/ketan/swift-labs/bionimbus-coaster-modftdock/app/modftdock.sh null null GLOBUS::maxwalltime="1:00:00"
+----
+
+(See below for a sample sites.xml for this run)
+
+**step5.** Connect back to gateway from virtual machines using reverse ssh tunneling as follows:
+
+.From the gateway prompt
+
++ssh -R *:5000:localhost:5000 root at 10.101.8.50 sleep 999+
+
+WHERE:
+*=network interface, should remain the same on all cases
+
+localhost=the gateway host, should remain the same
+
+5000(LEFT OF localhost)=the port number on localhost to listen to **THIS WILL
+vary depending upon which port you want to listen to
+
+5000(RIGHT OF localhost)=the port on target host that you want to forward
+
+root at 10.101.8.50=the ip of the Virtual Machine on Bionimbus cloud, this will
+vary based on what ip you get for your Virtual Machine instance
+
+#On anywhere as long as provide the correct callback uri: here the
+"http://140.221.9.110:35753" is the callback uri of previous ones
+
+**step6.** Copy the +worker.pl+ from gatewayx to the VM: +scp
+swift-0.92/bin/worker.pl root at 10.101.8.50:/bin/+, you will be prompted for the
+root password of the VM.
+
+**step7.** Start the worker from the virtual machine
++worker.pl http://localhost:35753 tmp /tmp+
+where 35753 is the port where the coaster service is listening to the workers+
+
+.sites.xml for the above run
+
+[xml]
+source~~~~
+<config>
+ <pool handle="localhost">
+ <execution provider="coaster-persistent" url="http://localhost:1984" jobmanager="local:local"/>
+ <profile namespace="globus" key="workerManager">passive</profile>
+
+ <profile namespace="globus" key="workersPerNode">4</profile>
+ <profile namespace="globus" key="maxTime">10000</profile>
+ <profile namespace="globus" key="lowOverAllocation">100</profile>
+ <profile namespace="globus" key="highOverAllocation">100</profile>
+ <profile namespace="globus" key="slots">100</profile>
+ <profile namespace="globus" key="nodeGranularity">1</profile>
+ <profile namespace="globus" key="maxNodes">10</profile>
+ <profile namespace="karajan" key="jobThrottle">25.00</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ <profile namespace="swift" key="stagingMethod">proxy</profile>
+ <filesystem provider="local"/>
+ <workdirectory>/home/ketan/swift-labs/bionimbus-coaster-modftdock/swift.workdir</workdirectory>
+ </pool>
+</config>
+source~~~~
+
+//Magellan
+//~~~~~~~~
+//Swift on Magellan
+//
+
+Debugging Swift
+---------------
+Swift errors are logged in several places:
+
+. All text from standard output and standard error produced by running the
+swift command
+. The .log file from this run. It will be named swiftscript.uniqueID.log
+where "swiftscript" is the name of your *.swift script source file, and
+uniqueID is a long unique id which starts with the date and time you ran the
+swift command.
+. $HOME/.globus/coasters directory on remote machines on which you are
+running coasters
+. $HOME/.globus/scripts directory on the host on which you run the Swift
+command, when swift is submitting to a local scheduler (Condor, PBS, SGE,
+Cobalt)
+. $HOME/.globus/??? on remote systems that you access via Globus
+
+
+Log Processing
+---------------
+
+To properly generate log plots, you must enable VDL/Karajan logging. Make sure
+log4.properties contains:
+----
+log4j.logger.swift=DEBUG
+log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Cpu=DEBUG
+----
+
+Make a basic load plot from Coasters Cpu log lines
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.Generate the log (may set log4j.logger.swift=INFO for this one)
+(assuming the log is titled swift-run.log)
+
+.Convert the log times to Unix time
+
+----
+./iso-to-secs < swift-run.log > swift-run.time
+----
+
+.Make the start time file (this contains the earliest timestamp)
+
+----
+make LOG=swift-run.log start-time.tmp
+----
+
+or
+
+----
+extract-start-time swift-run.log > start-time.tmp
+----
+
+.Normalize the transition times
+
+----
+./normalise-event-start-time < swift-run.time > swift-run.norm
+----
+
+.Build up a load data file:
+
+----
+./cpu-job-load.pl < swift-run.norm > load.data
+----
+
+.Plot with the JFreeChart-based plotter in usertools/plotter:
+
+----
+lines.zsh load.cfg load.eps load.data
+----
+
+Make a basic job completion plot from Coasters Cpu log lines
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.Same as above, but, build up a completed data file:
+
+----
+./cpu-job-completed.pl < swift-run.norm > completed.data
+----
+
+.Plot with the JFreeChart-based plotter in usertools/plotter:
+
+----
+lines.zsh completed.cfg completed.eps completed.data
+----
+
+Make a basic Block allocation plot from Coasters Block log lines
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Same as above, but:
+
+.Build up a block allocation data file:
+
+----
+./block-level.pl < swift-run.norm > blocks.data
+----
+
+.Plot with the JFreeChart-based plotter in usertools/plotter:
+
+----
+lines.zsh blocks.{cfg,eps,data}
+----
+
+Problem Reporting
+~~~~~~~~~~~~~~~~~
+When reporting problems to swift-user at ci.uchicago.edu, please attach the
+following files and information:
+
+. tc.data and sites.xml (or whatever you named these files)
+
+. your .swift source file and any .swift files it imports
+
+. any external mapper scripts called by your .swift script
+
+. all text from standard output and standard error produced by running the
+swift command
+
+. The .log file from this run. It will be named swiftscript.uniqueID.log
+
+. where "swiftscript" is the name of your *.swift script source file, and
+uniqueID is a long unique id which starts with the date and time you ran the
+swift command.
+
+. The swift command line you invoked
+
+. Any swift.properties entries you over-rode ($HOME/.swift/swift.properties,
+-config.file argument properties file, any changes to etc/swift.proerties from
+your swift distribution)
+
+. Which swift distribution you are running (release; svn revisions; other
+local changes you mave have made or included)
+
+
Added: wwwdev/cookbook/figures/_notes/dwsync.xml
===================================================================
--- wwwdev/cookbook/figures/_notes/dwsync.xml (rev 0)
+++ wwwdev/cookbook/figures/_notes/dwsync.xml 2011-08-01 14:23:26 UTC (rev 4896)
@@ -0,0 +1,4 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<dwsync>
+<file name="coaster_setup.png" server="login.ci.uchicago.edu//home/vytas/public_html/projects/2011_07-Swift/" local="3394121904" remote="11984644046" />
+</dwsync>
\ No newline at end of file
Added: wwwdev/cookbook/figures/coaster_setup.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/cookbook/figures/coaster_setup.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
More information about the Swift-commit
mailing list