[Swift-commit] r4419 - www/cookbook
ketan at ci.uchicago.edu
ketan at ci.uchicago.edu
Tue Apr 26 11:02:55 CDT 2011
Author: ketan
Date: 2011-04-26 11:02:55 -0500 (Tue, 26 Apr 2011)
New Revision: 4419
Modified:
www/cookbook/cookbook-asciidoc.html
www/cookbook/cookbook-asciidoc.txt
Log:
Modified: www/cookbook/cookbook-asciidoc.html
===================================================================
--- www/cookbook/cookbook-asciidoc.html 2011-04-26 06:55:46 UTC (rev 4418)
+++ www/cookbook/cookbook-asciidoc.html 2011-04-26 16:02:55 UTC (rev 4419)
@@ -639,24 +639,17 @@
<div class="title">Note</div>
</td>
<td class="content">The environment will be different when using Swift from prebuilt
-distribution and trunk.</td>
+distribution and trunk. To execute your Swift script on the login host ("localhost") use this
+command:</td>
</tr></table>
</div>
-<div class="literalblock">
+<div class="listingblock">
<div class="content">
-<pre><tt> To execute your Swift script on the login host ("localhost") use this
-command:</tt></pre>
-</div></div>
-<div class="literalblock">
-<div class="content">
<pre><tt>swift -tc.file tc modis.swift</tt></pre>
</div></div>
-<div class="literalblock">
+<div class="paragraph"><p>To execute your Swift script on the PADS cluster use this command:</p></div>
+<div class="listingblock">
<div class="content">
-<pre><tt>To execute your Swift script on the PADS cluster use this command:</tt></pre>
-</div></div>
-<div class="literalblock">
-<div class="content">
<pre><tt>swift -tc.file tc -sites.file pbs.xml modis.swift</tt></pre>
</div></div>
<div class="sect3">
@@ -672,23 +665,31 @@
<div class="paragraph"><p>SimpleMapper</p></div>
<div class="listingblock">
<div class="content">
-<pre><tt>com$ cat swiftapply.swift
-type RFile;
-trace("hi 1");
-app (RFile result) RunR (RFile rcall)
-{
- RunR @rcall @result;
-}
-trace("hi 2");
-RFile rcalls[] ;
-RFile results[] ;
-trace("start");
-foreach c, i in rcalls {
- trace("c",i, at c);
- trace("r",i, at filename(results[i]));
- results[i] = RunR(c);
-}
-com$ ls calldir resdir
+<pre><tt>com$ cat swiftapply.swift</tt></pre>
+</div></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.4
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #008080">type</span> RFile<span style="color: #990000">;</span>
+<span style="font-weight: bold"><span style="color: #000000">trace</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"hi 1"</span><span style="color: #990000">);</span>
+<span style="font-weight: bold"><span style="color: #000000">app</span></span> <span style="color: #990000">(</span><span style="color: #008080">RFile</span> result<span style="color: #990000">)</span> <span style="font-weight: bold"><span style="color: #000000">RunR</span></span> <span style="color: #990000">(</span><span style="color: #008080">RFile</span> rcall<span style="color: #990000">)</span>
+<span style="color: #FF0000">{</span>
+ RunR @rcall @result<span style="color: #990000">;</span>
+<span style="color: #FF0000">}</span>
+<span style="font-weight: bold"><span style="color: #000000">trace</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"hi 2"</span><span style="color: #990000">);</span>
+<span style="color: #008080">RFile</span> rcalls<span style="color: #990000">[]</span> <span style="color: #990000">;</span>
+<span style="color: #008080">RFile</span> results<span style="color: #990000">[]</span> <span style="color: #990000">;</span>
+<span style="font-weight: bold"><span style="color: #000000">trace</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"start"</span><span style="color: #990000">);</span>
+<span style="color: #008080">foreach</span> c<span style="color: #990000">,</span> i <span style="color: #008080">in</span> rcalls <span style="color: #FF0000">{</span>
+ <span style="font-weight: bold"><span style="color: #000000">trace</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"c"</span><span style="color: #990000">,</span>i<span style="color: #990000">,</span>@c<span style="color: #990000">);</span>
+ <span style="font-weight: bold"><span style="color: #000000">trace</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"r"</span><span style="color: #990000">,</span>i<span style="color: #990000">,</span>@<span style="font-weight: bold"><span style="color: #000000">filename</span></span><span style="color: #990000">(</span>results<span style="color: #990000">[</span>i<span style="color: #990000">]));</span>
+ results<span style="color: #990000">[</span>i<span style="color: #990000">]</span> <span style="color: #990000">=</span> <span style="font-weight: bold"><span style="color: #000000">RunR</span></span><span style="color: #990000">(</span>c<span style="color: #990000">);</span>
+<span style="color: #FF0000">}</span></tt></pre></div></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>com$ ls calldir resdir
calldir:
rcall.1.Rdata rcall.2.Rdata rcall.3.Rdata rcall.4.Rdata
resdir:
@@ -714,20 +715,28 @@
<div class="paragraph"><p>example here</p></div>
<div class="listingblock">
<div class="content">
-<pre><tt>$ cat writedata.swift
-type file;
-file f <"filea">;
-file nf <"filenames">;
-nf = writeData(@f);
-$ swift writedata.swift
+<pre><tt>$ cat writedata.swift</tt></pre>
+</div></div>
+<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.4
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="color: #008080">type</span> file<span style="color: #990000">;</span>
+
+<span style="color: #008080">file</span> f <span style="color: #990000"><</span><span style="color: #FF0000">"filea"</span><span style="color: #990000">>;</span>
+<span style="color: #008080">file</span> nf <span style="color: #990000"><</span><span style="color: #FF0000">"filenames"</span><span style="color: #990000">>;</span>
+nf <span style="color: #990000">=</span> <span style="font-weight: bold"><span style="color: #000000">writeData</span></span><span style="color: #990000">(</span>@f<span style="color: #990000">);</span></tt></pre></div></div>
+<div class="listingblock">
+<div class="content">
+<pre><tt>$ swift writedata.swift
Swift svn swift-r3264 (swift modified locally) cog-r2730 (cog modified
locally)
RunID: 20100319-2002-s9vpo0pe
Progress:
Final status:
$ cat filenames
-filea$
-$</tt></pre>
+filea$</tt></pre>
</div></div>
<div class="paragraph"><p>StructuredRegexpMapper
IN PROGRESS This mapper can be used to base the mapped filenames of an output
@@ -1005,26 +1014,55 @@
</div>
<div class="sect2">
<h3 id="_bionimbus">3.3. Bionimbus</h3>
-<div class="paragraph"><p>Swift on Bionimbus
-.very important to read this</p></div>
-<div class="paragraph"><p>contains information on how to connect back to gateway from virtual machines
-using reverse ssh tunneling</p></div>
-<div class="paragraph"><p>to open a reverse ssh tunnel do the following:
-.from the gateway prompt</p></div>
-<div class="paragraph"><p><tt>ssh -R *:5000:localhost:5000 <a href="mailto:root at 10.101.8.50">root at 10.101.8.50</a> sleep 999</tt></p></div>
+<div class="paragraph"><p>This section explains a step by step procedure on getting Swift running on the Bionimbus cloud. We will use the <em>manual</em> <em>coasters</em> configuration on the Bionimbus cloud.</p></div>
+<div class="paragraph"><p><strong>step1.</strong> Connect to the gateway (ssh gatewayx.lac.uic.edu)</p></div>
+<div class="paragraph"><p><strong>step2.</strong> Start a virtual machine (euca-run-instances -n 1 -t m1.small
+emi-17EB1170)</p></div>
+<div class="paragraph"><p><strong>step3.</strong> Start the coaster-service on gateway
+<tt>coaster-service -port 1984 -localport 35753 -nosec</tt></p></div>
+<div class="paragraph"><p><strong>step4.</strong> Start the Swift-script from the gateway using normal Swift commandline</p></div>
+<div class="paragraph"><p><tt>swift -config cf -tc.file tc -sites.file sites.xml yourscript.swift -aparam=999</tt></p></div>
+<div class="paragraph"><p>(See below for a sample sites.xml for this run)</p></div>
+<div class="paragraph"><p><strong>step5.</strong> Connect back to gateway from virtual machines using reverse ssh tunneling as follows:</p></div>
+<div class="paragraph"><div class="title">From the gateway prompt</div><p><tt>ssh -R *:5000:localhost:5000 <a href="mailto:root at 10.101.8.50">root at 10.101.8.50</a> sleep 999</tt></p></div>
<div class="paragraph"><p>WHERE:
-*=network interface, should remain the same on all cases
-localhost=the gateway host, should remain the same</p></div>
+*=network interface, should remain the same on all cases</p></div>
+<div class="paragraph"><p>localhost=the gateway host, should remain the same</p></div>
<div class="paragraph"><p>5000(LEFT OF localhost)=the port number on localhost to listen to **THIS WILL
vary depending upon which port you want to listen to</p></div>
<div class="paragraph"><p>5000(RIGHT OF localhost)=the port on target host that you want to forward</p></div>
-<div class="paragraph"><p><a href="mailto:root at 10.101.8.50">root at 10.101.8.50</a>=the ip of the Virtual Machine on bionimbus cloud, this will
+<div class="paragraph"><p><a href="mailto:root at 10.101.8.50">root at 10.101.8.50</a>=the ip of the Virtual Machine on Bionimbus cloud, this will
vary based on what ip you get for your Virtual Machine instance</p></div>
-<div class="paragraph"><p><tt>coaster-service -port 1984 -localport 35753 -nosec</tt></p></div>
<div class="paragraph"><p>#On anywhere as long as provide the correct callback uri: here the
"http://140.221.9.110:42195" is the callback uri of previous ones</p></div>
-<div class="paragraph"><p><tt>worker.pl <a href="http://localhost:42195">http://localhost:42195</a> tmp /tmp # where 42195 is the port where the
+<div class="paragraph"><p><strong>step6.</strong> Start the worker from the virtual machine
+<tt>worker.pl <a href="http://localhost:42195">http://localhost:42195</a> tmp /tmp # where 42195 is the port where the
coaster service is listening to the workers</tt></p></div>
+<div class="listingblock">
+<div class="title">sites.xml for the above run</div>
+<div class="content"><!-- Generator: GNU source-highlight 3.1.4
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="font-weight: bold"><span style="color: #0000FF"><config></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><pool</span></span> <span style="color: #009900">handle</span><span style="color: #990000">=</span><span style="color: #FF0000">"localhost"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><execution</span></span> <span style="color: #009900">provider</span><span style="color: #990000">=</span><span style="color: #FF0000">"coaster-persistent"</span> <span style="color: #009900">url</span><span style="color: #990000">=</span><span style="color: #FF0000">"http://localhost:1984"</span> <span style="color: #009900">jobmanager</span><span style="color: #990000">=</span><span style="color: #FF0000">"local:local"</span><span style="font-weight: bold"><span style="color: #0000FF">/></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"workerManager"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>passive<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"workersPerNode"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>4<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"maxTime"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>10000<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"lowOverAllocation"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>100<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"highOverAllocation"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>100<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"slots"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>100<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"nodeGranularity"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>1<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"maxNodes"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>10<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"karajan"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"jobThrottle"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>25.00<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"karajan"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"initialScore"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>10000<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"swift"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"stagingMethod"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>proxy<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><filesystem</span></span> <span style="color: #009900">provider</span><span style="color: #990000">=</span><span style="color: #FF0000">"local"</span><span style="font-weight: bold"><span style="color: #0000FF">/></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"><workdirectory></span></span>/home/ketan/swift-labs/bionimbus-coaster-modftdock/swift.workdir<span style="font-weight: bold"><span style="color: #0000FF"></workdirectory></span></span>
+ <span style="font-weight: bold"><span style="color: #0000FF"></pool></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"></config></span></span></tt></pre></div></div>
</div>
</div>
</div>
@@ -1034,22 +1072,26 @@
<div class="paragraph"><p>Describe coasters mechanisms.</p></div>
<div class="paragraph"><p>Following is a nice coasters setup case-study where sites.xml coaster settings were:</p></div>
<div class="listingblock">
+<div class="content"><!-- Generator: GNU source-highlight 3.1.4
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre><tt><span style="font-weight: bold"><span style="color: #0000FF"><execution</span></span> <span style="color: #009900">provider</span><span style="color: #990000">=</span><span style="color: #FF0000">"coaster"</span> <span style="color: #009900">jobmanager</span><span style="color: #990000">=</span><span style="color: #FF0000">"local:pbs"</span><span style="font-weight: bold"><span style="color: #0000FF">/></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"project"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>CI-CCR000013<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"ppn"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>24:cray:pack<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"workersPerNode"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>24<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"maxTime"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>100000<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"lowOverallocation"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>100<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"highOverallocation"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>100<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"slots"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>20<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"nodeGranularity"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>5<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"globus"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"maxNodes"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>5<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"karajan"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"jobThrottle"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>20.00<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span>
+<span style="font-weight: bold"><span style="color: #0000FF"><profile</span></span> <span style="color: #009900">namespace</span><span style="color: #990000">=</span><span style="color: #FF0000">"karajan"</span> <span style="color: #009900">key</span><span style="color: #990000">=</span><span style="color: #FF0000">"initialScore"</span><span style="font-weight: bold"><span style="color: #0000FF">></span></span>10000<span style="font-weight: bold"><span style="color: #0000FF"></profile></span></span></tt></pre></div></div>
+<div class="listingblock">
<div class="content">
-<pre><tt><execution provider="coaster" jobmanager="local:pbs"/>
-<profile namespace="globus" key="project">CI-CCR000013</profile>
-<profile namespace="globus" key="ppn">24:cray:pack</profile>
-<profile namespace="globus" key="workersPerNode">24</profile>
-<profile namespace="globus" key="maxTime">100000</profile>
-<profile namespace="globus" key="lowOverallocation">100</profile>
-<profile namespace="globus" key="highOverallocation">100</profile>
-<profile namespace="globus" key="slots">20</profile>
-<profile namespace="globus" key="nodeGranularity">5</profile>
-<profile namespace="globus" key="maxNodes">5</profile>
-<profile namespace="karajan" key="jobThrottle">20.00</profile>
-<profile namespace="karajan" key="initialScore">10000</profile>
+<pre><tt>Your tc entry (shortened here) was:
-Your tc entry (shortened here) was:
-
pbs modftdock /.../modftdock.sh null null GLOBUS::maxwalltime="02:00:00"
And you said you saw in PBS: 13 jobs of 24 hours and 4 jobs of 22 hours. I
@@ -1291,7 +1333,7 @@
<div id="footer">
<div id="footer-text">
Version 0.92<br />
-Last updated 2011-04-25 14:47:18 CDT
+Last updated 2011-04-26 11:02:24 CDT
</div>
</div>
</body>
Modified: www/cookbook/cookbook-asciidoc.txt
===================================================================
--- www/cookbook/cookbook-asciidoc.txt 2011-04-26 06:55:46 UTC (rev 4418)
+++ www/cookbook/cookbook-asciidoc.txt 2011-04-26 16:02:55 UTC (rev 4419)
@@ -72,18 +72,19 @@
Setting up the environment
[NOTE]
The environment will be different when using Swift from prebuilt
-distribution and trunk.
-
- To execute your Swift script on the login host ("localhost") use this
+distribution and trunk. To execute your Swift script on the login host ("localhost") use this
command:
- swift -tc.file tc modis.swift
+----
+swift -tc.file tc modis.swift
+----
- To execute your Swift script on the PADS cluster use this command:
+To execute your Swift script on the PADS cluster use this command:
- swift -tc.file tc -sites.file pbs.xml modis.swift
+----
+swift -tc.file tc -sites.file pbs.xml modis.swift
+----
-
Setting transformation catalog
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -93,12 +94,17 @@
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cf
+
Mappers
^^^^^^^^
SimpleMapper
----
com$ cat swiftapply.swift
+----
+
+[java]
+source~~~~
type RFile;
trace("hi 1");
app (RFile result) RunR (RFile rcall)
@@ -114,6 +120,9 @@
trace("r",i, at filename(results[i]));
results[i] = RunR(c);
}
+source~~~~
+
+----
com$ ls calldir resdir
calldir:
rcall.1.Rdata rcall.2.Rdata rcall.3.Rdata rcall.4.Rdata
@@ -144,12 +153,21 @@
writeData()
example here
+
----
$ cat writedata.swift
+----
+
+[java]
+source~~~~
type file;
+
file f <"filea">;
file nf <"filenames">;
nf = writeData(@f);
+source~~~~
+
+----
$ swift writedata.swift
Swift svn swift-r3264 (swift modified locally) cog-r2730 (cog modified
locally)
@@ -158,7 +176,6 @@
Final status:
$ cat filenames
filea$
-$
----
StructuredRegexpMapper
@@ -416,19 +433,31 @@
//
Bionimbus
~~~~~~~~~
-Swift on Bionimbus
-.very important to read this
+This section explains a step by step procedure on getting Swift running on the Bionimbus cloud. We will use the _manual_ _coasters_ configuration on the Bionimbus cloud.
-contains information on how to connect back to gateway from virtual machines
-using reverse ssh tunneling
+**step1.** Connect to the gateway (ssh gatewayx.lac.uic.edu)
-to open a reverse ssh tunnel do the following:
-.from the gateway prompt
+**step2.** Start a virtual machine (euca-run-instances -n 1 -t m1.small
+emi-17EB1170)
+**step3.** Start the coaster-service on gateway
++coaster-service -port 1984 -localport 35753 -nosec+
+
+**step4.** Start the Swift-script from the gateway using normal Swift commandline
+
++swift -config cf -tc.file tc -sites.file sites.xml yourscript.swift -aparam=999+
+
+(See below for a sample sites.xml for this run)
+
+**step5.** Connect back to gateway from virtual machines using reverse ssh tunneling as follows:
+
+.From the gateway prompt
+
+ssh -R *:5000:localhost:5000 root at 10.101.8.50 sleep 999+
WHERE:
*=network interface, should remain the same on all cases
+
localhost=the gateway host, should remain the same
5000(LEFT OF localhost)=the port number on localhost to listen to **THIS WILL
@@ -436,17 +465,41 @@
5000(RIGHT OF localhost)=the port on target host that you want to forward
-root at 10.101.8.50=the ip of the Virtual Machine on bionimbus cloud, this will
+root at 10.101.8.50=the ip of the Virtual Machine on Bionimbus cloud, this will
vary based on what ip you get for your Virtual Machine instance
-+coaster-service -port 1984 -localport 35753 -nosec+
-
#On anywhere as long as provide the correct callback uri: here the
"http://140.221.9.110:42195" is the callback uri of previous ones
+**step6.** Start the worker from the virtual machine
+worker.pl http://localhost:42195 tmp /tmp # where 42195 is the port where the
coaster service is listening to the workers+
+.sites.xml for the above run
+
+[xml]
+source~~~~
+<config>
+ <pool handle="localhost">
+ <execution provider="coaster-persistent" url="http://localhost:1984" jobmanager="local:local"/>
+ <profile namespace="globus" key="workerManager">passive</profile>
+
+ <profile namespace="globus" key="workersPerNode">4</profile>
+ <profile namespace="globus" key="maxTime">10000</profile>
+ <profile namespace="globus" key="lowOverAllocation">100</profile>
+ <profile namespace="globus" key="highOverAllocation">100</profile>
+ <profile namespace="globus" key="slots">100</profile>
+ <profile namespace="globus" key="nodeGranularity">1</profile>
+ <profile namespace="globus" key="maxNodes">10</profile>
+ <profile namespace="karajan" key="jobThrottle">25.00</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ <profile namespace="swift" key="stagingMethod">proxy</profile>
+ <filesystem provider="local"/>
+ <workdirectory>/home/ketan/swift-labs/bionimbus-coaster-modftdock/swift.workdir</workdirectory>
+ </pool>
+</config>
+source~~~~
+
//Magellan
//~~~~~~~~
//Swift on Magellan
@@ -457,7 +510,9 @@
//**Include neat diagrams.**
Following is a nice coasters setup case-study where sites.xml coaster settings were:
-----
+
+[xml]
+source~~~~
<execution provider="coaster" jobmanager="local:pbs"/>
<profile namespace="globus" key="project">CI-CCR000013</profile>
<profile namespace="globus" key="ppn">24:cray:pack</profile>
@@ -470,7 +525,9 @@
<profile namespace="globus" key="maxNodes">5</profile>
<profile namespace="karajan" key="jobThrottle">20.00</profile>
<profile namespace="karajan" key="initialScore">10000</profile>
+source~~~~
+----
Your tc entry (shortened here) was:
pbs modftdock /.../modftdock.sh null null GLOBUS::maxwalltime="02:00:00"
More information about the Swift-commit
mailing list