[Swift-commit] r8073 - www/Swift-T
wozniak at ci.uchicago.edu
wozniak at ci.uchicago.edu
Tue Jul 29 11:40:32 CDT 2014
Author: wozniak
Date: 2014-07-29 11:40:26 -0500 (Tue, 29 Jul 2014)
New Revision: 8073
Modified:
www/Swift-T/guide.html
Log:
Updates on annotations
Modified: www/Swift-T/guide.html
===================================================================
--- www/Swift-T/guide.html 2014-07-26 22:36:10 UTC (rev 8072)
+++ www/Swift-T/guide.html 2014-07-29 16:40:26 UTC (rev 8073)
@@ -744,7 +744,7 @@
<body class="article" style="max-width:750px">
<div id="header">
<h1>Swift/T Guide</h1>
-<span id="author">v0.6.0, July 2014</span><br />
+<span id="author">v0.6.1, July 2014</span><br />
<div id="toc">
<div id="toctitle">Table of Contents</div>
<noscript><p><b>JavaScript must be enabled in your browser to display the table of contents.</b></p></noscript>
@@ -2632,8 +2632,57 @@
</dd>
</dl></div>
</div>
+<div class="sect2">
+<h3 id="library_location">12.9. Location</h3>
+<div class="paragraph"><p>See the section about <a href="#location">location</a>.</p></div>
+<div class="dlist"><dl>
+<dt class="hdlist1">
+<code>location_from_rank(int) → location</code>
+</dt>
+<dd>
+<p>
+Convert the rank integer to a
+<code>location</code> variable compatible with <code>@location</code>.
+</p>
+</dd>
+<dt class="hdlist1">
+<code>random_worker() → location</code>
+</dt>
+<dd>
+<p>
+Obtain a worker at random.
+</p>
+</dd>
+<dt class="hdlist1">
+<code>hostmap_list() → string[]</code>
+</dt>
+<dd>
+<p>
+Obtain the whole hostmap.
+</p>
+</dd>
+<dt class="hdlist1">
+<code>hostmap_one(string) → location</code>
+</dt>
+<dd>
+<p>
+Lookup the string as a host in the
+hostmap and return one rank running on that host.
+</p>
+</dd>
+<dt class="hdlist1">
+<code>hostmap_one_worker(string) → location</code>
+</dt>
+<dd>
+<p>
+Lookup the string as a host
+in the hostmap and return one worker rank running on that host.
+</p>
+</dd>
+</dl></div>
</div>
</div>
+</div>
<div class="sect1">
<h2 id="_defining_leaf_functions">13. Defining leaf functions</h2>
<div class="sectionbody">
@@ -2660,13 +2709,6 @@
</li>
<li>
<p>
-For complex types such as structures, arrays and files, you may need
- additional logic to marshal inputs and outputs to/from the global
- data store.
-</p>
-</li>
-<li>
-<p>
The requisite files required to build a Tcl package
(e.g <code>pkgIndex.tcl</code>)
</p>
@@ -2766,7 +2808,7 @@
<li>
<p>
<code>int</code>/<code>float</code>/<code>string</code>/<code>bool</code> are converted to the standard
- Tcl representations
+ Tcl representations.
</p>
</li>
<li>
@@ -2774,7 +2816,7 @@
blobs are represented as a Tcl list with first element a pointer
to the data, the second element the length of the data, and if
the blob was loaded from the ADLB data store, a third element
- which is the ADLB ID of the blob
+ which is the ADLB ID of the blob.
</p>
</li>
<li>
@@ -2783,6 +2825,23 @@
file path, and the second element a reference count
</p>
</li>
+<li>
+<p>
+arrays are represented by Tcl dictionaries with keys and values
+ represented according to their type.
+</p>
+</li>
+<li>
+<p>
+bags are represented by Tcl lists with elements in an arbitrary
+ order.
+</p>
+</li>
+<li>
+<p>
+output voids are set automatically.
+</p>
+</li>
</ul></div>
</div>
<div class="sect3">
@@ -2799,8 +2858,7 @@
</div>
<div class="sect3">
<h4 id="_writing_custom_tcl_interfaces">13.1.5. Writing custom Tcl interfaces</h4>
-<div class="paragraph"><p>For Tcl functions that take complex argument types, such as arrays or
-structures, you may need to write a Tcl wrapper function that is
+<div class="paragraph"><p>It is possible to write a Tcl wrapper function that is
directly passed references to data in Swift’s global data store. In
this case your function must manually retrieve/store data from/to the
global distributed data store. In this case, you do not use the STC
@@ -3097,8 +3155,69 @@
</div>
</div>
<div class="sect1">
-<h2 id="Optimizations">14. Optimizations</h2>
+<h2 id="annotations">14. Annotations</h2>
<div class="sectionbody">
+<div class="paragraph"><p>Swift/T supports many annotations to influence the behavior of the
+run.</p></div>
+<div class="sect2">
+<h3 id="_priority">14.1. Priority</h3>
+<div class="paragraph"><p>Leaf tasks resulting from Swift dataflow may be prioritized by using
+the <code>@priority</code> annotation:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>foreach i in [0:n-1] {
+ @priority=i f(i);
+}</code></pre>
+</div></div>
+<div class="paragraph"><p>In this case, <code>f()</code> will operate on higher values of <code>i</code> first.
+Priority is best-effort; it is local to the ADLB server.</p></div>
+<div class="paragraph"><p>This annotation is applied to the leaf task call.</p></div>
+</div>
+<div class="sect2">
+<h3 id="location">14.2. Location</h3>
+<div class="paragraph"><p>Leaf tasks resulting from Swift dataflow may be assigned to a given
+processor by using the <code>@location</code> annotation:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>foreach i in [0:n-1] {
+ location L = location_from_rank(i);
+ @location=L f(i);
+}</code></pre>
+</div></div>
+<div class="paragraph"><p>In this case, each <code>f(i)</code> will execute on a different worker.</p></div>
+<div class="paragraph"><p>This annotation is applied to the leaf task call.</p></div>
+<div class="sect3">
+<h4 id="_hostmap">14.2.1. Hostmap</h4>
+<div class="paragraph"><p>At startup, by default, Turbine obtains all hostnames used in the run
+and builds up a data structure called the hostmap to map hostnames to
+ranks. You may combine the location features with the hostmap
+features to send work to assigned hostnames. See the
+<a href="#library_location">Location library</a> for usage. The hostmap may be
+<a href="#env_hostmap_debug">debugged</a> or <a href="#env_hostmap_disable">disabled</a>.</p></div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_dispatch">14.3. Dispatch</h3>
+<div class="paragraph"><p>This simply dispatches tasks to the given process type: <code>WORKER</code> or
+<code>SERVER</code>. By default, work executes on a <code>SERVER</code>.</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>@dispatch=WORKER
+(int o) process (int i) "pkg" "0.0.1" [
+ "set <<o>> [ pkg::process <<i>> ]"
+];</code></pre>
+</div></div>
+<div class="paragraph"><p>Swift/T builtin functions (<code>strcat()</code>, arithmetic, etc.) are typically
+implemented in this way as tasks that execute on the server. User
+work should be performed on workers to keep servers free to process
+dataflow.</p></div>
+<div class="paragraph"><p>This annotation is applied to the leaf task definition.</p></div>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="Optimizations">15. Optimizations</h2>
+<div class="sectionbody">
<div class="paragraph"><p>STC performs a range of compiler optimizations that can significantly
speed up most Swift programs. The optimization level can be controlled
by the <code>-O</code> command line option. The default optimization
@@ -3132,11 +3251,11 @@
</div>
</div>
<div class="sect1">
-<h2 id="Turbine">15. Running in Turbine</h2>
+<h2 id="Turbine">16. Running in Turbine</h2>
<div class="sectionbody">
<div class="paragraph"><p>The following describes how to run Turbine programs.</p></div>
<div class="sect2">
-<h3 id="_architecture">15.1. Architecture</h3>
+<h3 id="_architecture">16.1. Architecture</h3>
<div class="paragraph"><p>Turbine runs as an MPI program consisting of many processes. Turbine
programs are ADLB programs. Thus, they produce and execute discrete
tasks that are distributed and load balanced at run time.</p></div>
@@ -3166,7 +3285,7 @@
also store Swift data (integers, strings, etc.).</p></div>
</div>
<div class="sect2">
-<h3 id="_concurrency">15.2. Concurrency</h3>
+<h3 id="_concurrency">16.2. Concurrency</h3>
<div class="paragraph"><p>The available concurrency and efficiency in your Swift script is
limited by the following factors:</p></div>
<div class="ulist"><ul>
@@ -3193,7 +3312,7 @@
</ul></div>
</div>
<div class="sect2">
-<h3 id="_invocation">15.3. Invocation</h3>
+<h3 id="_invocation">16.3. Invocation</h3>
<div class="paragraph"><p>The form of a Turbine invocation for STC-generated
<code>program.tcl</code> is:</p></div>
<div class="listingblock">
@@ -3403,31 +3522,38 @@
layout on a given machine is as intended.
</p>
</dd>
+</dl></div>
+<div class="paragraph"><p><a id="env_hostmap_debug"></a>
+<code>ADLB_DEBUG_HOSTMAP=1</code>:: Enable a report showing the hostmap, which
+maps hostnames to ranks for use with the location functionality.</p></div>
+<div class="paragraph"><p><a id="env_hostmap_disable"></a>
+<code>ADLB_DISABLE_HOSTMAP=1</code>:: Prevent the hostmap from being constructed
+(avoiding some communication overhead).</p></div>
+<div class="dlist"><dl>
<dt class="hdlist1">
-<code>ADLB_DEBUG_HOSTMAP=1</code>
+<code>VALGRIND</code>
</dt>
<dd>
<p>
-Enable a report showing the hostmap, which
-maps hostnames to ranks for use with the location functionality.
+Enable valgrind debugging. Cf. <a href="#valgrind">valgrind</a>.
</p>
</dd>
<dt class="hdlist1">
-<code>ADLB_DISABLE_HOSTMAP=1</code>
+<code>GDB_RANK</code>
</dt>
<dd>
<p>
-Prevent the hostmap from being constructed.
+Enable GDB debugging. Cf. <a href="#gdb">GDB</a>.
</p>
</dd>
</dl></div>
</div>
<div class="sect2">
-<h3 id="Build_configuration">15.4. Build configuration</h3>
+<h3 id="Build_configuration">16.4. Build configuration</h3>
<div class="paragraph"><p>The following describes how to turn Swift/T programs in Turbine
on more complex systems.</p></div>
<div class="sect3">
-<h4 id="_build_troubleshooting">15.4.1. Build troubleshooting</h4>
+<h4 id="_build_troubleshooting">16.4.1. Build troubleshooting</h4>
<div class="paragraph"><p>If <code>exm-setup.zsh</code> does not succeed, you may need to change how it
tries to configure and compile Swift/T.</p></div>
<div class="paragraph"><p>Troubleshooting a build problem can require a few steps. The first
@@ -3460,7 +3586,7 @@
</ol></div>
</div>
<div class="sect3">
-<h4 id="Manual_build_configuration">15.4.2. Manual configuration</h4>
+<h4 id="Manual_build_configuration">16.4.2. Manual configuration</h4>
<div class="paragraph"><p><code>exm-setup.zsh</code> and <code>exm-settings.sh</code> provide a convenient way to install
Swift/T. However, this method does not allow full control over
the configuration. Swift/T is built with standard Ant (Java) and
@@ -3496,11 +3622,11 @@
</ul></div>
</div>
<div class="sect3">
-<h4 id="_non_standard_mpi_locations">15.4.3. Non-standard MPI locations</h4>
+<h4 id="_non_standard_mpi_locations">16.4.3. Non-standard MPI locations</h4>
<div class="paragraph"><p>Sometimes simply specifying the MPI directory is not enough to
configure Swift/T.</p></div>
<div class="paragraph"><p>You can modify these settings in <code>exm-settings.sh</code> to more precisely
-define locations of MPI resources</p></div>
+define locations of MPI resources:</p></div>
<div class="listingblock">
<div class="content">
<pre><code>EXM_CUSTOM_MPI=1
@@ -3518,7 +3644,7 @@
</div>
</div>
<div class="sect2">
-<h3 id="_performance_enhancements">15.5. Performance enhancements</h3>
+<h3 id="_performance_enhancements">16.5. Performance enhancements</h3>
<div class="olist arabic"><ol class="arabic">
<li>
<p>
@@ -3577,7 +3703,7 @@
</ol></div>
</div>
<div class="sect2">
-<h3 id="mkstatic">15.6. Building standalone executables with mkstatic.tcl</h3>
+<h3 id="mkstatic">16.6. Building standalone executables with mkstatic.tcl</h3>
<div class="paragraph"><p>It is possible to build a fully self-contained executable, including
all Tcl scripts and compiled code, provided that all dependencies support
static linking. If not, it is also possible to build an executable with
@@ -3759,13 +3885,63 @@
</dd>
</dl></div>
</div>
+<div class="sect2">
+<h3 id="debugging">16.7. Debugging Swift/T runs</h3>
+<div class="paragraph"><p>Applying the debugger allows you to debug native code linked to Swift/T
+from a normal debugger. This will allow you to step through your code
+(and the Swift/T run time libraries).</p></div>
+<div class="ulist"><ul>
+<li>
+<p>
+When using Swift/T dynamically with Tcl packages (the default), you
+ need to attach to the <code>tclsh</code> process. This process loads your
+ native code and calls into it.
+</p>
+</li>
+<li>
+<p>
+When using <code>mkstatic</code>, you generate a complete executable. You can
+ debug this in the normal method for debugging MPI programs.
+</p>
+</li>
+</ul></div>
+<div class="sect3">
+<h4 id="valgrind">16.7.1. Valgrind</h4>
+<div class="paragraph"><p>The Swift/T launcher scripts support <a href="http://valgrind.org">valgrind</a>.
+Simply set the environment variable <code>VALGRIND</code> to the valgrind command
+you wish to use. A suppressions file is distributed with Turbine to
+ignore known issues. (Swift/T is valgrind-clean but there are some
+issues in Tcl.)</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>export VALGRIND="valgrind --suppressions=$HOME/turbine/turbine.supp"
+turbine program.tcl</code></pre>
+</div></div>
</div>
+<div class="sect3">
+<h4 id="gdb">16.7.2. GDB</h4>
+<div class="paragraph"><p>The Turbine library provides a convenient attachment mechanism
+compatible with debuggers like GDB, Eclipse, etc. You attach to a
+Turbine execution by using the <code>GDB_RANK</code> variable:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>$ export GDB_RANK=0
+$ turbine program.tcl
+Waiting for gdb: rank: 0 pid: 23274
+...</code></pre>
+</div></div>
+<div class="paragraph"><p>Rank 0, running in process <code>23274</code>, has blocked (in a loop) and is
+waiting for the debugger to attach. When you attach, set the variable
+<code>t=1</code> to break out of the loop. Then you can debug normally.</p></div>
</div>
</div>
+</div>
+</div>
+</div>
<div id="footnotes"><hr /></div>
<div id="footer">
<div id="footer-text">
-Last updated 2014-07-21 14:19:02 CDT
+Last updated 2014-07-29 11:37:51 CDT
</div>
</div>
</body>
More information about the Swift-commit
mailing list