[Swift-commit] r4920 - trunk/docs/siteguide
wilde at ci.uchicago.edu
wilde at ci.uchicago.edu
Mon Aug 1 13:27:32 CDT 2011
Author: wilde
Date: 2011-08-01 13:27:32 -0500 (Mon, 01 Aug 2011)
New Revision: 4920
Added:
trunk/docs/siteguide/grid
Modified:
trunk/docs/siteguide/futuregrid
trunk/docs/siteguide/siteguide.html
trunk/docs/siteguide/siteguide.txt
Log:
Added chapter on grid execution to siteguide. This is a preliminary doc (and a work in progress) for the evolving bin/grid scripts.
Modified: trunk/docs/siteguide/futuregrid
===================================================================
--- trunk/docs/siteguide/futuregrid 2011-08-01 16:52:35 UTC (rev 4919)
+++ trunk/docs/siteguide/futuregrid 2011-08-01 18:27:32 UTC (rev 4920)
@@ -1,5 +1,5 @@
-Futuregrid Quickstart Guide
----------------------------
+FutureGrid
+----------
Downloading and Building Swift
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Added: trunk/docs/siteguide/grid
===================================================================
--- trunk/docs/siteguide/grid (rev 0)
+++ trunk/docs/siteguide/grid 2011-08-01 18:27:32 UTC (rev 4920)
@@ -0,0 +1,125 @@
+Grids, including OSG and TeraGrid
+---------------------------------
+
+Overview of running on grid sites
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Get a DOEGrids cert. Then register it in the *OSG Engage VO*, and/or map it using +gx-request+ on TeraGrid sites.
+
+* Run +GridSetup+ to configure Swift to use the grid sites. This tests
+for correct operation and creates a "green list" of good sites.
+
+* Prepare an installation package for the programs you want to run on
+grid sites via Swift, and install that package using +foreachsite+.
+
+* Run +RunWorkers+ to start and maintain a pool of Swift workers on
+each site.
+
+* Run Swift scripts that use the grid site resources.
+
+
+Requesting Access
+~~~~~~~~~~~~~~~~~
+
+*For OSG*: Obtain a DOEGrids certificate and register the certificate in
+the OSG "Engage" VO following the procedures at:
+
+https://twiki.grid.iu.edu/bin/view/Engagement/EngageNewUserGuide
+
+[red]*FIXME*: access to OSG wiki pages may request the user to present
+a certificate. Is this a problem from users without one? If so, make a
+copy of the page on the Swift web.
+
+*For TeraGrid*: Obtain a DOEGrids certifcate using the OSG ENgage
+instructions above. Ask a TeraGrid PI to add you to a TeraGrid
+project. Once you obtain a login and project access (via US Mail), use
+gx-request to add your certificate
+
+To run jobs using the procedures documented here you need to be logged
+in to a "submit host" on which you will run Swift and other
+grid-related utilities. This can be any host with the OSG client stack
+or equivalent tools installed. Such hosts include the OSG Engage
+submit host, and the two Swift lab servers
+{bridled,communicado}.ci.uchicago.edu.
+
+Obtain a login on engage-submit.renci.org following instructions on
+the OSG URL above.
+
+Obtain a CI login with access to the Swift lab servers by requesting
+"OSG Gridlab" access at:
+
+http://accounts.ci.uchicago.edu
+
+
+Connecting to a submit host
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+-----
+ssh yourusername at bridled.ci.uchicago.edu
+ssh yourusername at communicado.ci.uchicago.edu
+ssh yourusername at engage-submit.renci.org
+-----
+
+Downloading and install Swift
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The current version of Swift can be downloaded from
+http://www.ci.uchicago.edu/swift/downloads/index.php.
+
+Fetch and untar the latest release.
+Then add the Swift bin/ directory to your PATH. For example:
+
+-----
+cd $HOME
+wget http://www.ci.uchicago.edu/swift/packages/swift-0.92.1.tar.gz
+tar txf swift-0.92.1.tar.gz
+export PATH=$PATH:$HOME/swift-0.92.1/bin
+-----
+
+Set up OSG environment
+~~~~~~~~~~~~~~~~~~~~~~
+
+Depending on your shell type, run:
+
+-----
+source /opt/osg/setup.sh
+or
+source /opt/osg/setup.csh
+-----
+
+Create a VOMS Grid proxy
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+-----
+$ voms-proxy-init -voms Engage -valid 12:00
+-----
+
+Generating Configuration Files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+-----
+cd $HOME
+mkdir swiftgrid
+cd swiftgrid
+gen_gridsites
+# Wait a few minutes to a few hours for Swift to validate grid sites
+get_greensites >greensites
+-----
+
+
+Running Swift
+~~~~~~~~~~~~~
+Now that everything is in place, run Swift with the following command:
+
+-----
+swift -sites.file sites.xml -tc.file tc.data catsn.swift -n=10
+-----
+
+You should see several new files being created, called catsn.0001.out, catsn.0002.out, etc. Each of these
+files should contain the contents of what you placed into data.txt. If this happens, your job has run
+successfully on PADS!
+
+More Help
+~~~~~~~~~
+The best place for additional help is the Swift user mailing list. You can subscribe to this list at
+http://mail.ci.uchicago.edu/mailman/listinfo/swift-user. When submitting information, please send your sites.xml file, your tc.data, and any Swift log files that were created during your attempt.
Modified: trunk/docs/siteguide/siteguide.html
===================================================================
--- trunk/docs/siteguide/siteguide.html 2011-08-01 16:52:35 UTC (rev 4919)
+++ trunk/docs/siteguide/siteguide.html 2011-08-01 18:27:32 UTC (rev 4920)
@@ -2,17 +2,30 @@
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
-<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
-<meta name="generator" content="AsciiDoc 8.5.2" />
+<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" />
+<meta name="generator" content="AsciiDoc 8.6.4" />
<title>Site Configuration Guide</title>
<style type="text/css">
-/* Debug borders */
-p, li, dt, dd, div, pre, h1, h2, h3, h4, h5, h6 {
-/*
- border: 1px solid red;
-*/
+/* Sans-serif font. */
+h1, h2, h3, h4, h5, h6,
+div.title, caption.title,
+thead, p.table.header,
+div#toctitle,
+span#author, span#revnumber, span#revdate, span#revremark,
+div#footer {
+ font-family: Arial,Helvetica,sans-serif;
}
+/* Serif font. */
+div.sectionbody {
+ font-family: Georgia,"Times New Roman",Times,serif;
+}
+
+/* Monospace font. */
+tt {
+ font-size: inherit;
+}
+
body {
margin: 1em 5% 1em 5%;
}
@@ -36,12 +49,12 @@
}
tt {
+ font-size: inherit;
color: navy;
}
h1, h2, h3, h4, h5, h6 {
color: #527bbd;
- font-family: sans-serif;
margin-top: 1.2em;
margin-bottom: 0.5em;
line-height: 1.3;
@@ -61,7 +74,6 @@
}
div.sectionbody {
- font-family: serif;
margin-left: 0;
}
@@ -77,6 +89,8 @@
ul, ol, li > p {
margin-top: 0;
}
+ul > li { color: #aaa; }
+ul > li > * { color: black; }
pre {
padding: 0;
@@ -85,18 +99,15 @@
span#author {
color: #527bbd;
- font-family: sans-serif;
font-weight: bold;
font-size: 1.1em;
}
span#email {
}
span#revnumber, span#revdate, span#revremark {
- font-family: sans-serif;
}
div#footer {
- font-family: sans-serif;
font-size: small;
border-top: 2px solid silver;
padding-top: 0.5em;
@@ -135,7 +146,6 @@
/* Block element titles. */
div.title, caption.title {
color: #527bbd;
- font-family: sans-serif;
font-weight: bold;
text-align: left;
margin-top: 1.0em;
@@ -157,13 +167,15 @@
div.sidebarblock > div.content {
background: #ffffee;
- border: 1px solid silver;
+ border: 1px solid #dddddd;
+ border-left: 4px solid #f0f0f0;
padding: 0.5em;
}
div.listingblock > div.content {
- border: 1px solid silver;
- background: #f4f4f4;
+ border: 1px solid #dddddd;
+ border-left: 5px solid #f0f0f0;
+ background: #f8f8f8;
padding: 0.5em;
}
@@ -171,7 +183,7 @@
padding-left: 1.0em;
margin-left: 1.0em;
margin-right: 10%;
- border-left: 5px solid #dddddd;
+ border-left: 5px solid #f0f0f0;
color: #777777;
}
@@ -180,8 +192,9 @@
text-align: right;
}
-div.verseblock > div.content {
- white-space: pre;
+div.verseblock > pre.content {
+ font-family: inherit;
+ font-size: inherit;
}
div.verseblock > div.attribution {
padding-top: 0.75em;
@@ -258,8 +271,8 @@
border: 3px solid #527bbd;
}
thead, p.table.header {
- font-family: sans-serif;
font-weight: bold;
+ color: #527bbd;
}
tfoot {
font-weight: bold;
@@ -339,6 +352,14 @@
min-width: 100px;
}
+div.colist td {
+ padding-right: 0.5em;
+ padding-bottom: 0.3em;
+ vertical-align: top;
+}
+div.colist td img {
+ margin-top: 0.3em;
+}
@media print {
div#footer-badges { display: none; }
@@ -350,7 +371,6 @@
div#toctitle {
color: #527bbd;
- font-family: sans-serif;
font-size: 1.1em;
font-weight: bold;
margin-top: 1.0em;
@@ -373,47 +393,43 @@
margin-left: 6em;
font-size: 0.9em;
}
-/* Workarounds for IE6's broken and incomplete CSS2. */
-div.sidebar-content {
- background: #ffffee;
- border: 1px solid silver;
- padding: 0.5em;
-}
-div.sidebar-title, div.image-title {
- color: #527bbd;
- font-family: sans-serif;
- font-weight: bold;
- margin-top: 0.0em;
- margin-bottom: 0.5em;
-}
+span.aqua { color: aqua; }
+span.black { color: black; }
+span.blue { color: blue; }
+span.fuchsia { color: fuchsia; }
+span.gray { color: gray; }
+span.green { color: green; }
+span.lime { color: lime; }
+span.maroon { color: maroon; }
+span.navy { color: navy; }
+span.olive { color: olive; }
+span.purple { color: purple; }
+span.red { color: red; }
+span.silver { color: silver; }
+span.teal { color: teal; }
+span.white { color: white; }
+span.yellow { color: yellow; }
-div.listingblock div.content {
- border: 1px solid silver;
- background: #f4f4f4;
- padding: 0.5em;
-}
+span.aqua-background { background: aqua; }
+span.black-background { background: black; }
+span.blue-background { background: blue; }
+span.fuchsia-background { background: fuchsia; }
+span.gray-background { background: gray; }
+span.green-background { background: green; }
+span.lime-background { background: lime; }
+span.maroon-background { background: maroon; }
+span.navy-background { background: navy; }
+span.olive-background { background: olive; }
+span.purple-background { background: purple; }
+span.red-background { background: red; }
+span.silver-background { background: silver; }
+span.teal-background { background: teal; }
+span.white-background { background: white; }
+span.yellow-background { background: yellow; }
-div.quoteblock-attribution {
- padding-top: 0.5em;
- text-align: right;
-}
-
-div.verseblock-content {
- white-space: pre;
-}
-div.verseblock-attribution {
- padding-top: 0.75em;
- text-align: left;
-}
-
-div.exampleblock-content {
- border-left: 3px solid #dddddd;
- padding-left: 0.5em;
-}
-
-/* IE6 sets dynamically generated links as visited. */
-div#toc a:visited { color: blue; }
+span.big { font-size: 2em; }
+span.small { font-size: 0.6em; }
</style>
<script type="text/javascript">
/*<![CDATA[*/
@@ -549,37 +565,45 @@
/*]]>*/
</script>
</head>
-<body style="max-width:750px">
+<body class="article" style="max-width:750px">
<div id="header">
<h1>Site Configuration Guide</h1>
-<div id="toc">
- <div id="toctitle">Table of Contents</div>
- <noscript><p><b>JavaScript must be enabled in your browser to display the table of contents.</b></p></noscript>
+<div id="toc">
+ <div id="toctitle">Table of Contents</div>
+ <noscript><p><b>JavaScript must be enabled in your browser to display the table of contents.</b></p></noscript>
</div>
</div>
<div id="content">
+<div class="sect1">
<h2 id="_pads">1. PADS</h2>
<div class="sectionbody">
-<h3 id="_requesting_access">1.1. Requesting Access</h3><div style="clear:left"></div>
+<div class="sect2">
+<h3 id="_requesting_access">1.1. Requesting Access</h3>
<div class="paragraph"><p>If you do not already have a Computation Institute account, you can request
one at <a href="https://www.ci.uchicago.edu/accounts/">https://www.ci.uchicago.edu/accounts/</a>. This page will give you a list
of resources you can request access to. Be sure that PADS is selected. If
you already have an existing CI account, but do not have access to PADS,
send an email to <a href="mailto:support at ci.uchicago.edu">support at ci.uchicago.edu</a> to request access.</p></div>
-<h3 id="_ssh_keys">1.2. SSH Keys</h3><div style="clear:left"></div>
+</div>
+<div class="sect2">
+<h3 id="_ssh_keys">1.2. SSH Keys</h3>
<div class="paragraph"><p>Before accessing PADS, be sure to have your SSH keys configured correctly.
There is some basic information about SSH and how to generate your key at
<a href="http://www.ci.uchicago.edu/wiki/bin/view/Resources/SshKeys">http://www.ci.uchicago.edu/wiki/bin/view/Resources/SshKeys</a>. Once you have
followed those instructions, you can add your key at
<a href="https://www.ci.uchicago.edu/support/sshkeys/">https://www.ci.uchicago.edu/support/sshkeys/</a>.</p></div>
-<h3 id="_connecting_to_a_login_node">1.3. Connecting to a login node</h3><div style="clear:left"></div>
+</div>
+<div class="sect2">
+<h3 id="_connecting_to_a_login_node">1.3. Connecting to a login node</h3>
<div class="paragraph"><p>Once your keys are configured, you should be able to access a PADS login
node with the following command:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt>ssh yourusername at login.pads.ci.uchicago.edu</tt></pre>
</div></div>
-<h3 id="_configuring_softenv">1.4. Configuring softenv</h3><div style="clear:left"></div>
+</div>
+<div class="sect2">
+<h3 id="_configuring_softenv">1.4. Configuring softenv</h3>
<div class="paragraph"><p>Softenv is a system used for managing applications. In order to run Swift,
the softenv environment will have to be modified slightly. Softenv is
configured by a file in your home directory called .soft. Edit this file
@@ -591,11 +615,15 @@
+torque
@default</tt></pre>
</div></div>
-<h3 id="_downloading_and_building_swift">1.5. Downloading and building Swift</h3><div style="clear:left"></div>
+</div>
+<div class="sect2">
+<h3 id="_downloading_and_building_swift">1.5. Downloading and building Swift</h3>
<div class="paragraph"><p>The most recent versions of Swift can be found at
<a href="http://www.ci.uchicago.edu/swift/downloads/index.php">http://www.ci.uchicago.edu/swift/downloads/index.php</a>. Follow the instructions
provided on that site to download and build Swift.</p></div>
-<h3 id="_adding_swift_to_your_path">1.6. Adding Swift to your PATH</h3><div style="clear:left"></div>
+</div>
+<div class="sect2">
+<h3 id="_adding_swift_to_your_path">1.6. Adding Swift to your PATH</h3>
<div class="paragraph"><p>Once you have installed Swift, add the Swift binary to your PATH so you can
easily run it from any directory.</p></div>
<div class="paragraph"><p>In your home directory, edit the file ".bashrc".</p></div>
@@ -611,9 +639,12 @@
<pre><tt>export PATH=$PATH:$HOME/swift-<version>/bin</tt></pre>
</div></div>
<div class="paragraph"><p>Replace <version> with the actual name of the swift directory in the example above.</p></div>
-<h3 id="_what_you_need_to_know_before_running_swift">1.7. What You Need To Know Before Running Swift</h3><div style="clear:left"></div>
+</div>
+<div class="sect2">
+<h3 id="_what_you_need_to_know_before_running_swift">1.7. What You Need To Know Before Running Swift</h3>
<div class="paragraph"><p>Before you can create a Swift configuration file, there are some things
you will need to know.</p></div>
+<div class="sect3">
<h4 id="_swift_work_directory">1.7.1. Swift Work Directory</h4>
<div class="paragraph"><p>The Swift work directory is a directory which Swift uses for processing work.
This directory needs to be writable. Common options for this are:</p></div>
@@ -623,6 +654,8 @@
/home/username/work
/tmp</tt></pre>
</div></div>
+</div>
+<div class="sect3">
<h4 id="_which_project_s_are_you_a_member_of">1.7.2. Which project(s) are you a member of?</h4>
<div class="paragraph"><p>PADS requires that you are a member of a project. You can determine this by
running the following command:</p></div>
@@ -651,6 +684,8 @@
<pre><tt>$ projects --set CI-CCR000013 --all
Your default project for all CI clusters has been set to CI-CCR000013.</tt></pre>
</div></div>
+</div>
+<div class="sect3">
<h4 id="_determine_your_queue">1.7.3. Determine your Queue</h4>
<div class="paragraph"><p>PADS has several different queues you can submit jobs to depending on
the type of work you will be doing. The command "qstat -q" will print
@@ -742,11 +777,15 @@
</tbody>
</table>
</div>
-<h3 id="_generating_configuration_files">1.8. Generating Configuration Files</h3><div style="clear:left"></div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_generating_configuration_files">1.8. Generating Configuration Files</h3>
<div class="paragraph"><p>Now that you know what queue to use, your project, and your work directory, it is time to
set up Swift. Swift uses a configuration file called sites.xml to determine how it should run.
There are two methods you can use for creating this file. You can manually edit
the configuration file, or generate it with a utility called gensites.</p></div>
+<div class="sect3">
<h4 id="_manually_editing_sites_xml">1.8.1. Manually Editing sites.xml</h4>
<div class="paragraph"><p>Below is the template that is used by Swift’s test suite for running on PADS.</p></div>
<div class="listingblock">
@@ -762,7 +801,10 @@
</config></tt></pre>
</div></div>
<div class="paragraph"><p>The values to note here are the ones that are listed between underscores. In the example above, they are <em>QUEUE</em>, and <em>WORK</em>. Queue is the PADS queue to use and WORK is the swift work directory. These are placeholder values you will need to modify to fit your needs. Copy and paste this template, replace the values, and call it sites.xml.</p></div>
-<h3 id="_manually_editing_tc_data">1.9. Manually Editing tc.data</h3><div style="clear:left"></div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_manually_editing_tc_data">1.9. Manually Editing tc.data</h3>
<div class="paragraph"><p>Below is the tc.data file used by Swift’s test suite for running on PADS.</p></div>
<div class="listingblock">
<div class="content">
@@ -775,27 +817,17 @@
pads-pbs wc /usr/bin/wc INSTALLED INTEL32::LINUX</tt></pre>
</div></div>
<div class="paragraph"><p>Copy these commands and save it as tc.data.</p></div>
-<h3 id="_catsn_swift">1.10. Catsn.swift</h3><div style="clear:left"></div>
+</div>
+<div class="sect2">
+<h3 id="_catsn_swift">1.10. Catsn.swift</h3>
<div class="paragraph"><p>The swift script we will run is called catsn.swift. It simply cats a file and saves the result. This is a nice simple test to ensure jobs are running correctly. Create a file called data.txt which contains some simple input - a "hello world" will do the trick.</p></div>
<div class="listingblock">
<div class="content">
-<pre><tt>type file;
-
-app (file o) cat (file i)
-{
- cat @i stdout=@o;
-}
-
-string t = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
-string char[] = @strsplit(t, "");
-
-file out[]<simple_mapper; location=".", prefix="catsn.",suffix=".out">;
-foreach j in [1:@toint(@arg("n","10"))] {
- file data<"data.txt">;
- out[j] = cat(data);
-}</tt></pre>
+<pre><tt></tt></pre>
</div></div>
-<h3 id="_running_swift">1.11. Running Swift</h3><div style="clear:left"></div>
+</div>
+<div class="sect2">
+<h3 id="_running_swift">1.11. Running Swift</h3>
<div class="paragraph"><p>Now that everything is in place, run Swift with the following command:</p></div>
<div class="listingblock">
<div class="content">
@@ -804,19 +836,27 @@
<div class="paragraph"><p>You should see several new files being created, called catsn.0001.out, catsn.0002.out, etc. Each of these
files should contain the contents of what you placed into data.txt. If this happens, your job has run
successfully on PADS!</p></div>
-<h3 id="_more_help">1.12. More Help</h3><div style="clear:left"></div>
+</div>
+<div class="sect2">
+<h3 id="_more_help">1.12. More Help</h3>
<div class="paragraph"><p>The best place for additional help is the Swift user mailing list. You can subscribe to this list at
<a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a>. When submitting information, please send your sites.xml file, your tc.data, and any Swift log files that were created during your attempt.</p></div>
</div>
+</div>
+</div>
+<div class="sect1">
<h2 id="_beagle">2. Beagle</h2>
<div class="sectionbody">
-<h3 id="_requesting_access_2">2.1. Requesting Access</h3><div style="clear:left"></div>
+<div class="sect2">
+<h3 id="_requesting_access_2">2.1. Requesting Access</h3>
<div class="paragraph"><p>If you do not already have a Computation Institute account, you can request
one at <a href="https://www.ci.uchicago.edu/accounts/">https://www.ci.uchicago.edu/accounts/</a>. This page will give you a list
of resources you can request access to.
You already have an existing CI account, but do not have access to Beagle,
send an email to <a href="mailto:support at ci.uchicago.edu">support at ci.uchicago.edu</a> to request access.</p></div>
-<h3 id="_connecting_to_a_login_node_2">2.2. Connecting to a login node</h3><div style="clear:left"></div>
+</div>
+<div class="sect2">
+<h3 id="_connecting_to_a_login_node_2">2.2. Connecting to a login node</h3>
<div class="paragraph"><p>Once you have account, you should be able to access a Beagle login
node with the following command:</p></div>
<div class="listingblock">
@@ -845,13 +885,19 @@
<div class="paragraph"><p>Note: Running from sandbox node or requesting 1 hour walltime for upto 3 nodes
will get fast prioritized execution. Good for small tests.</p></div>
</div>
-<h2 id="_futuregrid_quickstart_guide">3. Futuregrid Quickstart Guide</h2>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_futuregrid">3. FutureGrid</h2>
<div class="sectionbody">
-<h3 id="_downloading_and_building_swift_2">3.1. Downloading and Building Swift</h3><div style="clear:left"></div>
+<div class="sect2">
+<h3 id="_downloading_and_building_swift_2">3.1. Downloading and Building Swift</h3>
<div class="paragraph"><p>The most recent versions of Swift can be found at
<a href="http://www.ci.uchicago.edu/swift/downloads/index.php">http://www.ci.uchicago.edu/swift/downloads/index.php</a>. To run on futuregrid,
you will need to download the development/trunk version from SVN.</p></div>
-<h3 id="_adding_swift_to_your_path_2">3.2. Adding Swift to your PATH</h3><div style="clear:left"></div>
+</div>
+<div class="sect2">
+<h3 id="_adding_swift_to_your_path_2">3.2. Adding Swift to your PATH</h3>
<div class="paragraph"><p>Once you have installed Swift, add the Swift binary to your PATH so you can
easily run it from any directory.</p></div>
<div class="paragraph"><p>In your home directory, edit the file ".bashrc".</p></div>
@@ -861,12 +907,16 @@
<div class="content">
<pre><tt>export PATH=$PATH:$HOME/cog/modules/swift/dist/swift-svn/bin</tt></pre>
</div></div>
-<h3 id="_requesting_futuregrid_access">3.3. Requesting Futuregrid Access</h3><div style="clear:left"></div>
+</div>
+<div class="sect2">
+<h3 id="_requesting_futuregrid_access">3.3. Requesting Futuregrid Access</h3>
<div class="paragraph"><p>If you do not already have a futuregrid account, you can follow the
instructions at <a href="https://portal.futuregrid.org/gettingstarted">https://portal.futuregrid.org/gettingstarted</a> to get started.
This page provides information on how to create an account, how to join
a project, how to set up your SSH keys, and how to create a new project.</p></div>
-<h3 id="_downloading_swift_vm_tools">3.4. Downloading Swift VM Tools</h3><div style="clear:left"></div>
+</div>
+<div class="sect2">
+<h3 id="_downloading_swift_vm_tools">3.4. Downloading Swift VM Tools</h3>
<div class="paragraph"><p>A set of scripts based around cloudinitd are used to easily start virtual
machines. To download, change to your home directory and run the
following command:</p></div>
@@ -874,7 +924,9 @@
<div class="content">
<pre><tt>$ svn co https://svn.ci.uchicago.edu/svn/vdl2/usertools/swift-vm-boot</tt></pre>
</div></div>
-<h3 id="_download_your_credentials">3.5. Download your Credentials</h3><div style="clear:left"></div>
+</div>
+<div class="sect2">
+<h3 id="_download_your_credentials">3.5. Download your Credentials</h3>
<div class="paragraph"><p>Run the following commands to retrieve your credentials:</p></div>
<div class="listingblock">
<div class="content">
@@ -885,7 +937,9 @@
hotel.conf. Near the bottom of this file will be two settings
called vws.repository.s3id and vws.repository.s3key. Copy these
values for the next step.</p></div>
-<h3 id="_configuring_swift_vm_tools">3.6. Configuring Swift VM Tools</h3><div style="clear:left"></div>
+</div>
+<div class="sect2">
+<h3 id="_configuring_swift_vm_tools">3.6. Configuring Swift VM Tools</h3>
<div class="paragraph"><p>Change directories into the swift-vm-boot directory. Edit the file
called env.sh. There are two settings here that you will need to
modify: FUTUREGRID_IAAS_ACCESS_KEY and FUTUREGRID_IAAS_SECRET_KEY.
@@ -897,7 +951,9 @@
Python. If you run into problems while running the install.sh
script, please try a more recent version of Python and associated
libraries.</p></div>
-<h3 id="_configuring_coaster_service_conf">3.7. Configuring coaster-service.conf</h3><div style="clear:left"></div>
+</div>
+<div class="sect2">
+<h3 id="_configuring_coaster_service_conf">3.7. Configuring coaster-service.conf</h3>
<div class="paragraph"><p>Within your swift distribution’s etc directory is a file called
coaster-service.conf. There are many options here you can modify
as needed, but these are the settings that will be required
@@ -916,7 +972,9 @@
Tunneling should be used when you are restricted by a firewall. If your local
machine has multiple network interfaces, you should also manually set the IP
address of your machine with export IPADDR=your.ip.address.here.</p></div>
-<h3 id="_starting_the_coaster_service_script">3.8. Starting the Coaster Service Script</h3><div style="clear:left"></div>
+</div>
+<div class="sect2">
+<h3 id="_starting_the_coaster_service_script">3.8. Starting the Coaster Service Script</h3>
<div class="paragraph"><p>Now that everything is configured, change to a temporary directory
and run this command to start the coaster service:</p></div>
<div class="listingblock">
@@ -927,7 +985,9 @@
and generate Swift configuration files for you to use. The configuration files
will be generated in your current directory. These files are sites.xml, tc.data,
and cf.</p></div>
-<h3 id="_running_swift_2">3.9. Running Swift</h3><div style="clear:left"></div>
+</div>
+<div class="sect2">
+<h3 id="_running_swift_2">3.9. Running Swift</h3>
<div class="paragraph"><p>Now that you have all of your configuration files generated, run the following command:</p></div>
<div class="listingblock">
<div class="content">
@@ -940,7 +1000,9 @@
may also create your own custom tc files with the hostname of persistent-coasters. More
information about this can be found in the Swift userguide at
<a href="http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html">http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html</a>.</p></div>
-<h3 id="_stopping_the_coaster_service_script">3.10. Stopping the Coaster Service Script</h3><div style="clear:left"></div>
+</div>
+<div class="sect2">
+<h3 id="_stopping_the_coaster_service_script">3.10. Stopping the Coaster Service Script</h3>
<div class="paragraph"><p>To stop the coaster service, run the following command:</p></div>
<div class="listingblock">
<div class="content">
@@ -948,16 +1010,150 @@
</div></div>
<div class="paragraph"><p>This will kill the coaster service, kill the worker scripts on remote systems and terminate
the virtual machines that were created during start-coaster-service.</p></div>
-<h3 id="_more_help_2">3.11. More Help</h3><div style="clear:left"></div>
+</div>
+<div class="sect2">
+<h3 id="_more_help_2">3.11. More Help</h3>
<div class="paragraph"><p>The best place for additional help is the Swift user mailing list. You can subscribe to this list at
<a href="http://mail.ci.uchicago.edu/mailman/listinfo/swift-user">http://mail.ci.uchicago.edu/mailman/listinfo/swift-user</a>. When submitting information, please send
your sites.xml file, your tc.data, and any error messages you run into.</p></div>
</div>
</div>
+</div>
+<div class="sect1">
+<h2 id="_grids_including_osg_and_teragrid">4. Grids, including OSG and TeraGrid</h2>
+<div class="sectionbody">
+<div class="sect2">
+<h3 id="_overview_of_running_on_grid_sites">4.1. Overview of running on grid sites</h3>
+<div class="ulist"><ul>
+<li>
+<p>
+Get a DOEGrids cert. Then register it in the <strong>OSG Engage VO</strong>, and/or map it using <tt>gx-request</tt> on TeraGrid sites.
+</p>
+</li>
+<li>
+<p>
+Run <tt>GridSetup</tt> to configure Swift to use the grid sites. This tests
+for correct operation and creates a "green list" of good sites.
+</p>
+</li>
+<li>
+<p>
+Prepare an installation package for the programs you want to run on
+grid sites via Swift, and install that package using <tt>foreachsite</tt>.
+</p>
+</li>
+<li>
+<p>
+Run <tt>RunWorkers</tt> to start and maintain a pool of Swift workers on
+each site.
+</p>
+</li>
+<li>
+<p>
+Run Swift scripts that use the grid site resources.
+</p>
+</li>
+</ul></div>
+</div>
+<div class="sect2">
+<h3 id="_requesting_access_3">4.2. Requesting Access</h3>
+<div class="paragraph"><p><strong>For OSG</strong>: Obtain a DOEGrids certificate and register the certificate in
+the OSG "Engage" VO following the procedures at:</p></div>
+<div class="paragraph"><p><a href="https://twiki.grid.iu.edu/bin/view/Engagement/EngageNewUserGuide">https://twiki.grid.iu.edu/bin/view/Engagement/EngageNewUserGuide</a></p></div>
+<div class="paragraph"><p><strong><span class="red">FIXME</span></strong>: access to OSG wiki pages may request the user to present
+a certificate. Is this a problem from users without one? If so, make a
+copy of the page on the Swift web.</p></div>
+<div class="paragraph"><p><strong>For TeraGrid</strong>: Obtain a DOEGrids certifcate using the OSG ENgage
+instructions above. Ask a TeraGrid PI to add you to a TeraGrid
+project. Once you obtain a login and project access (via US Mail), use
+gx-request to add your certificate</p></div>
+<div class="paragraph"><p>To run jobs using the procedures documented here you need to be logged
+in to a "submit host" on which you will run Swift and other
+grid-related utilities. This can be any host with the OSG client stack
+or equivalent tools installed. Such hosts include the OSG Engage
+submit host, and the two Swift lab servers
+{bridled,communicado}.ci.uchicago.edu.</p></div>
+<div class="paragraph"><p>Obtain a login on engage-submit.renci.org following instructions on
+the OSG URL above.</p></div>
+<div class="paragraph"><p>Obtain a CI login with access to the Swift lab servers by requesting
+"OSG Gridlab" access at:</p></div>
+<div class="paragraph"><p><a href="http://accounts.ci.uchicago.edu">http://accounts.ci.uchicago.edu</a></p></div>
+</div>
+<div class="sect2">
+<h3 id="_connecting_to_a_submit_host">4.3. Connecting to a submit host</h3>
+<div class="listingblock">
+<div class="content">
+<pre><tt>ssh yourusername at bridled.ci.uchicago.edu
+ssh yourusername at communicado.ci.uchicago.edu
+ssh yourusername at engage-submit.renci.org</tt></pre>
+</div></div>
+</div>
+<div class="sect2">
+<h3 id="_downloading_and_install_swift">4.4. Downloading and install Swift</h3>
+<div class="paragraph"><p>The current version of Swift can be downloaded from
+<a href="http://www.ci.uchicago.edu/swift/downloads/index.php">http://www.ci.uchicago.edu/swift/downloads/index.php</a>.</p></div>
+<div class="paragraph"><p>Fetch and untar the latest release.
+Then add the Swift bin/ directory to your PATH. For example:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>cd $HOME
+wget http://www.ci.uchicago.edu/swift/packages/swift-0.92.1.tar.gz
+tar txf swift-0.92.1.tar.gz
+export PATH=$PATH:$HOME/swift-0.92.1/bin</tt></pre>
+</div></div>
+</div>
+<div class="sect2">
+<h3 id="_set_up_osg_environment">4.5. Set up OSG environment</h3>
+<div class="paragraph"><p>Depending on your shell type, run:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>source /opt/osg/setup.sh
+or
+source /opt/osg/setup.csh</tt></pre>
+</div></div>
+</div>
+<div class="sect2">
+<h3 id="_create_a_voms_grid_proxy">4.6. Create a VOMS Grid proxy</h3>
+<div class="listingblock">
+<div class="content">
+<pre><tt>$ voms-proxy-init -voms Engage -valid 12:00</tt></pre>
+</div></div>
+</div>
+<div class="sect2">
+<h3 id="_generating_configuration_files_2">4.7. Generating Configuration Files</h3>
+<div class="listingblock">
+<div class="content">
+<pre><tt>cd $HOME
+mkdir swiftgrid
+cd swiftgrid
+gen_gridsites
+# Wait a few minutes to a few hours for Swift to validate grid sites
+get_greensites >greensites</tt></pre>
+</div></div>
+</div>
+<div class="sect2">
+<h3 id="_running_swift_3">4.8. Running Swift</h3>
+<div class="paragraph"><p>Now that everything is in place, run Swift with the following command:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>swift -sites.file sites.xml -tc.file tc.data catsn.swift -n=10</tt></pre>
+</div></div>
+<div class="paragraph"><p>You should see several new files being created, called catsn.0001.out, catsn.0002.out, etc. Each of these
+files should contain the contents of what you placed into data.txt. If this happens, your job has run
+successfully on PADS!</p></div>
+</div>
+<div class="sect2">
+<h3 id="_more_help_3">4.9. More Help</h3>
+<div class="paragraph"><p>The best place for additional help is the Swift user mailing list. You can subscribe to this list at
+<a href="http://mail.ci.uchicago.edu/mailman/listinfo/swift-user">http://mail.ci.uchicago.edu/mailman/listinfo/swift-user</a>. When submitting information, please send your sites.xml file, your tc.data, and any Swift log files that were created during your attempt.</p></div>
+</div>
+</div>
+</div>
+</div>
<div id="footnotes"><hr /></div>
<div id="footer">
<div id="footer-text">
-Last updated 2011-07-13 00:42:08 CDT
+Last updated 2011-08-01 12:53:52 CDT
</div>
</div>
</body>
Modified: trunk/docs/siteguide/siteguide.txt
===================================================================
--- trunk/docs/siteguide/siteguide.txt 2011-08-01 16:52:35 UTC (rev 4919)
+++ trunk/docs/siteguide/siteguide.txt 2011-08-01 18:27:32 UTC (rev 4920)
@@ -11,3 +11,5 @@
include::beagle[]
include::futuregrid[]
+
+include::grid[]
More information about the Swift-commit
mailing list