[Swift-commit] r8350 - www/Swift-T
wozniak at ci.uchicago.edu
wozniak at ci.uchicago.edu
Wed Dec 17 11:49:09 CST 2014
Author: wozniak
Date: 2014-12-17 11:49:08 -0600 (Wed, 17 Dec 2014)
New Revision: 8350
Modified:
www/Swift-T/guide.html
Log:
Swift/T guide updates
Modified: www/Swift-T/guide.html
===================================================================
--- www/Swift-T/guide.html 2014-12-16 16:40:06 UTC (rev 8349)
+++ www/Swift-T/guide.html 2014-12-17 17:49:08 UTC (rev 8350)
@@ -744,7 +744,7 @@
<body class="article" style="max-width:750px">
<div id="header">
<h1>Swift/T Guide</h1>
-<span id="author">v0.6.1, July 2014</span><br />
+<span id="author">v0.7.0, November 2014</span><br />
<div id="toc">
<div id="toctitle">Table of Contents</div>
<noscript><p><b>JavaScript must be enabled in your browser to display the table of contents.</b></p></noscript>
@@ -871,7 +871,7 @@
</p>
<div class="listingblock">
<div class="content">
-<pre><code>wget http://www.mcs.anl.gov/exm/local/downloads/exm-0.5.0.tar.gz</code></pre>
+<pre><code>wget http://www.swift-lang.org/Swift-T/downloads/exm-0.7.0.tar.gz</code></pre>
</div></div>
<div class="paragraph"><p>Cf. <a href="http://www.mcs.anl.gov/exm/local/downloads/downloads.html">Swift/T Downloads</a>
for other packages</p></div>
@@ -946,19 +946,29 @@
<h2 id="_usage">3. Usage</h2>
<div class="sectionbody">
<div class="paragraph"><p>Swift code is conventionally written in <code>*.swift</code> files. Turbine
-code is stored in Tcl files <code>*.tcl</code>. After writing the Swift program
+code is stored in Tcl files with extension <code>*.tic</code> (for Turbine
+Intermediate Code). After writing the Swift program
<code>program.swift</code>, run:</p></div>
<div class="listingblock">
<div class="content">
<pre><code>stc program.swift</code></pre>
</div></div>
-<div class="paragraph"><p>This will compile the program to <code>program.tcl</code>. A second, optional
+<div class="paragraph"><p>This will compile the program to <code>program.tic</code>. A second, optional
argument may be given as an alternate output file name.</p></div>
<div class="paragraph"><p>Then, to run the program, use Turbine:</p></div>
<div class="listingblock">
<div class="content">
<pre><code>turbine -n 4 program.tcl</code></pre>
</div></div>
+<div class="paragraph"><p>You may compile and run in one step with:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>swift-t -n 4 program.swift</code></pre>
+</div></div>
+<div class="paragraph"><p>In this case, <code>program.tic</code> is created in a temporary file and then
+deleted after execution.</p></div>
+<div class="paragraph"><p><code>swift-t</code> accepts arguments for both STC and Turbine.
+Provide <code>-h</code> to any tool (<code>swift-t</code>, <code>stc</code>, <code>turbine</code>) to obtain help.</p></div>
<div class="paragraph"><p>See the <a href="#Turbine">Turbine section</a> for more information about
running the program.</p></div>
<div class="paragraph" id="stc_arguments"><p>STC accepts the following arguments:</p></div>
@@ -2528,16 +2538,6 @@
</p>
</dd>
<dt class="hdlist1">
-<code>file_lines(file) → string[]</code>
-</dt>
-<dd>
-<p>
-Reads the whole file, returning each line as a separate entry in the
-output array. Comments with <code>#</code> are excised, leading and trailing
-whitespace is trimmed, and blank lines are omitted.
-</p>
-</dd>
-<dt class="hdlist1">
<code>glob(string) → file[]</code>
</dt>
<dd>
@@ -2573,6 +2573,44 @@
</li>
</ul></div>
</dd>
+<dt class="hdlist1">
+<code>file_exists(string) → boolean</code>
+</dt>
+<dd>
+<p>
+Attempt to find a file with the given name in the filesystem: return
+<code>true</code> if found, else <code>false</code>.
+</p>
+</dd>
+<dt class="hdlist1">
+<code>file_lines(file) → string[]</code>
+</dt>
+<dd>
+<p>
+Reads the whole file, returning each line as a separate entry in the
+output array. Comments with <code>#</code> are excised, leading and trailing
+whitespace is trimmed, and blank lines are omitted.
+</p>
+</dd>
+<dt class="hdlist1">
+<code>file_mtime(string) → int</code>
+</dt>
+<dd>
+<p>
+Attempt to find a file with the given name in the filesystem: return
+its POSIX modification time in seconds since the Unix epoch.
+</p>
+</dd>
+<dt class="hdlist1">
+<code>file_type(file) → string</code>
+</dt>
+<dd>
+<p>
+Returns a string giving the type of file, which will be one of
+<code>"file"</code>, <code>"directory"</code>, <code>"characterSpecial"</code>, <code>"blockSpecial"</code>,
+<code>"fifo"</code>, <code>"link"</code>, or <code>"socket"</code>.
+</p>
+</dd>
</dl></div>
</div>
<div class="sect2">
@@ -2787,7 +2825,7 @@
];</code></pre>
</div></div>
<div class="paragraph"><p>We can also tell the Swift compiler a little about the function so
-that it can better optimize your programs. For example, <code>double</code> has
+that it can better optimmize your programs. For example, <code>double</code> has
no side-effects and produces the same result each time for the same
arguments (i.e. is deterministic), so we can annotate it as a <code>@pure</code>
function.</p></div>
@@ -2897,7 +2935,72 @@
</div>
</div>
<div class="sect2">
-<h3 id="app_functions">13.2. App functions</h3>
+<h3 id="_dispatch_and_work_types">13.2. Dispatch and work types</h3>
+<div class="paragraph"><p>Each worker in Swift/T is devoted to executing a single type of work.
+There is a default work type that encompasses Swift/T script logic and
+CPU tasks.</p></div>
+<div class="paragraph"><p>There are two subtypes of the default work type that are treated
+differently by the Swift/T optimizer. <code>CONTROL</code>, the default,
+is intended for functions that run for a short duration, for example
+builtin functions such as <code>strcat()</code>, arithmetic.
+<code>WORKER</code>, the second subtype,
+is intended for functions that perform more computation and I/O,
+which may keep a worker busy for a while. Swift/T may bundle
+together multiple <code>CONTROL</code> tasks to reduce overhead, but will
+keep <code>WORKER</code> tasks separate to increase parallelism.</p></div>
+<div class="paragraph"><p>The work type is specified with a <code>@dispatch</code> annotation above
+leaf function definitions.</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>@dispatch=WORKER
+(int o) process (int i) "pkg" "0.0.1" [
+ "set <<o>> [ pkg::process <<i>> ]"
+];</code></pre>
+</div></div>
+<div class="paragraph"><p>Alternative work-types include custom user-defined work types,
+along with tasks for external executors such as Coasters or GeMTC
+for remote command-line and GPU tasks respectively.
+It is also possible to define custom types of CPU leaf functions.</p></div>
+<div class="sect3">
+<h4 id="_custom_work_types">13.2.1. Custom work types</h4>
+<div class="paragraph"><p>For some applications, it is useful to be able to divide up
+CPU workers into multiple categories that execute different
+kinds of work. For these scenarios, Swift/T provides the
+ability to define custom work types.</p></div>
+<div class="paragraph"><p>This sample program illustrates how to define a custom work
+type <code>foo_work</code> and define a function, <code>hello1</code>, which executes
+on a <code>foo_work</code> worker. For comparison, we also include <code>hello2</code>,
+which uses the default work type.</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>pragma worktypedef foo_work;
+
+ at dispatch=foo_work
+hello1 (string name) "turbine" "0.0" [
+ "puts [ format {Hello %s} <<msg>> ]"
+];
+
+hello2 (string name) "turbine" "0.0" [
+ "puts [ format {Hello %s} <<msg>> ]"
+]
+
+main () {
+ // Hello Foo will be printed on a foo_work worker
+ hello1("Foo");
+
+ // Hello Bar will be printed on a regular worker
+ hello2("Bar");
+}</code></pre>
+</div></div>
+<div class="paragraph"><p>In order to run the above script, we need to ensure that
+a worker is allocated to execute <code>foo_work</code> tasks. This is
+achieved by setting the environment variable <code>TURBINE_FOO_WORK_WORKERS</code>
+to the desired number of workers. The Swift/T workers are then divided
+between default workers and <code>foo_work</code> workers.</p></div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="app_functions">13.3. App functions</h3>
<div class="paragraph"><p>App functions are functions that are implemented as command-line
programs. These command-line programs can be brought into a Swift
program as functions with typed inputs and outputs.
@@ -2980,9 +3083,119 @@
</div></div>
</div>
<div class="sect2">
-<h3 id="_external_scripting_support">13.3. External scripting support</h3>
+<h3 id="_remote_job_execution_with_coasters">13.4. Remote job execution with Coasters</h3>
+<div class="paragraph"><p><strong>Note:</strong> Swift/T and Coasters integration is a work in progress
+and is currently best suited for advanced users.
+Planned future changes will make it easier to install and use.</p></div>
+<div class="paragraph"><p>Swift/T supports execution of command-line <code>app</code> functions on
+a wide range of clusters, clouds, and grids with the
+Coaster executor. In order for an app function to be executed
+through coasters, the annotation <code>@dispatch=COASTER</code> must be
+added to the app function definition:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>@dispatch=COASTER
+app (file out) echo (string args[]) {
+ "echo" args @stdout=out
+}</code></pre>
+</div></div>
+<div class="paragraph"><p>Running a Swift/T script with Coasters function requires
+a few steps to set up all components:</p></div>
+<div class="ulist"><ul>
+<li>
+<p>
+Before starting, you must have Swift/T compiled with Coaster
+ support, and the coaster service from Swift/K installed.
+</p>
+</li>
+<li>
+<p>
+Before running your Swift/T script, start a Coaster service
+ from the shell, for example:
+</p>
+</li>
+</ul></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>export WORKER_MODE=local
+export IPADDR=127.0.0.1
+export SERVICE_PORT=53363
+export JOBSPERNODE=4
+export LOGDIR=/home/user/swift-logs
+export WORKER_LOG_DIR=/home/user/swift-logs
+
+coaster-service -nosec -port ${SERVICE_PORT}</code></pre>
+</div></div>
+<div class="ulist"><ul>
+<li>
+<p>
+Coaster configuration must be set in the the TURBINE_COASTER_CONFIG
+ environment variable, for example:
+</p>
+</li>
+</ul></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>export TURBINE_COASTER_CONFIG="jobManager=local,maxParallelTasks=4,coasterServiceURL=${IPADDR}:${SERVICE_PORT}"</code></pre>
+</div></div>
+<div class="ulist"><ul>
+<li>
+<p>
+Once the Coaster service is running and TURBINE_COASTER_CONFIG is set,
+ you can run your Swift/T program in the normal way, and any coaster app
+ tasks will be dispatched to the Coaster service for execution.
+</p>
+</li>
+</ul></div>
+<div class="paragraph"><p>Configuration keys include:</p></div>
+<div class="dlist"><dl>
+<dt class="hdlist1">
+<code>coasterServiceURL</code>
+</dt>
+<dd>
+<p>
+the url of the coaster service to submit tasks through,
+ e.g. <code>localhost:63001</code>. Default is <code>127.0.0.1:53001</code>.
+</p>
+</dd>
+<dt class="hdlist1">
+<code>jobManager</code>
+</dt>
+<dd>
+<p>
+the Coaster job manager the service should use to submit tasks.
+ E.g. to execute jobs locally, the job manager can be set to <code>local</code>
+ and to execute jobs on resources managed through a batch scheduler such
+ as PBS or Slurm, the job manager should be set to <code>pbs</code>, <code>slurm</code>, or
+ the appropriate scheduler.
+</p>
+</dd>
+<dt class="hdlist1">
+<code>maxParallelTasks</code>
+</dt>
+<dd>
+<p>
+the maximum number of concurrent tasks per Coaster
+ worker. Default is 256.
+</p>
+</dd>
+<dt class="hdlist1">
+Other settings
+</dt>
+<dd>
+<p>
+additional configuration keys are passed through to the
+ Coaster service.
+</p>
+</dd>
+</dl></div>
+<div class="paragraph"><p>For more information on configuring and using Coasters, please refer
+to the <a href="http://swift-lang.org/docs/">Swift/K documentation</a>.</p></div>
+</div>
+<div class="sect2">
+<h3 id="_external_scripting_support">13.5. External scripting support</h3>
<div class="sect3">
-<h4 id="_calling_python">13.3.1. Calling Python</h4>
+<h4 id="_calling_python">13.5.1. Calling Python</h4>
<div class="paragraph"><p>You can evaluate arbitrary Python code from within Swift/T. For
example, you can perform processing with a Python library.
Once you have that working, you can use Swift/T to coordinate
@@ -3073,7 +3286,7 @@
</div>
</div>
<div class="sect3">
-<h4 id="_calling_r">13.3.2. Calling R</h4>
+<h4 id="_calling_r">13.5.2. Calling R</h4>
<div class="paragraph"><p>Consider the following Swift script:</p></div>
<div class="listingblock">
<div class="content">
@@ -3125,7 +3338,7 @@
</div>
</div>
<div class="sect3">
-<h4 id="_calling_julia">13.3.3. Calling Julia</h4>
+<h4 id="_calling_julia">13.5.3. Calling Julia</h4>
<div class="paragraph"><p>Consider the following Swift script:</p></div>
<div class="listingblock">
<div class="content">
@@ -3177,10 +3390,10 @@
</div>
</div>
<div class="sect1">
-<h2 id="annotations">14. Annotations</h2>
+<h2 id="annotations">14. Function call annotations</h2>
<div class="sectionbody">
-<div class="paragraph"><p>Swift/T supports many annotations to influence the behavior of the
-run.</p></div>
+<div class="paragraph"><p>Swift/T supports many annotations to influence the behavior of function
+calls.</p></div>
<div class="sect2">
<h3 id="_priority">14.1. Priority</h3>
<div class="paragraph"><p>Leaf tasks resulting from Swift dataflow may be prioritized by using
@@ -3220,25 +3433,8 @@
<a href="#env_hostmap_debug">debugged</a> or <a href="#env_hostmap_disable">disabled</a>.</p></div>
</div>
</div>
-<div class="sect2">
-<h3 id="_dispatch">14.3. Dispatch</h3>
-<div class="paragraph"><p>This simply dispatches tasks to the given process type: <code>WORKER</code> or
-<code>SERVER</code>. By default, work executes on a <code>SERVER</code>.</p></div>
-<div class="listingblock">
-<div class="content">
-<pre><code>@dispatch=WORKER
-(int o) process (int i) "pkg" "0.0.1" [
- "set <<o>> [ pkg::process <<i>> ]"
-];</code></pre>
-</div></div>
-<div class="paragraph"><p>Swift/T builtin functions (<code>strcat()</code>, arithmetic, etc.) are typically
-implemented in this way as tasks that execute on the server. User
-work should be performed on workers to keep servers free to process
-dataflow.</p></div>
-<div class="paragraph"><p>This annotation is applied to the leaf task definition.</p></div>
</div>
</div>
-</div>
<div class="sect1">
<h2 id="Optimizations">15. Optimizations</h2>
<div class="sectionbody">
@@ -3429,6 +3625,36 @@
Swift (<a href="#Turbine_information">Turbine information</a>).</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
+<code>TURBINE_<type>_WORKERS</code>
+</dt>
+<dd>
+<p>
+Number of workers of specified type.
+</p>
+</dd>
+</dl></div>
+<div class="paragraph"><p>Any workers not allocated to specific types are general-purpose workers
+that execute Swift/T control code and CPU-based tasks. There must
+be at least one leftover worker to serve as a general-purpose worker.</p></div>
+<div class="paragraph"><p>Valid work types include:</p></div>
+<div class="ulist"><ul>
+<li>
+<p>
+<code>TURBINE_COASTER_WORKERS</code>:: for Coaster workers.
+</p>
+</li>
+<li>
+<p>
+A work type defined with <code>pragma worktypedef</code> in Swift.
+ E.g. if you define <code>pragma worktypedef a_new_work_type;" in Swift,
+ the environment variable is +TURBINE_A_NEW_WORK_TYPE_WORKERS</code>.
+</p>
+</li>
+</ul></div>
+<div class="paragraph"><p>Generally you will need to allocate workers for any work type used in
+your Swift program.</p></div>
+<div class="dlist"><dl>
+<dt class="hdlist1">
<code>TURBINE_LOG=0</code>
</dt>
<dd>
@@ -4001,7 +4227,7 @@
<div id="footnotes"><hr /></div>
<div id="footer">
<div id="footer-text">
-Last updated 2014-10-09 13:03:11 CDT
+Last updated 2014-12-17 11:09:58 CST
</div>
</div>
</body>
More information about the Swift-commit
mailing list