[Swift-commit] r6613 - in branches/faster: . docs docs/cookbook docs/documentation docs/userguide docs/utils libexec/log-processing
hategan at ci.uchicago.edu
hategan at ci.uchicago.edu
Mon Jul 8 01:01:40 CDT 2013
Author: hategan
Date: 2013-07-08 01:01:39 -0500 (Mon, 08 Jul 2013)
New Revision: 6613
Added:
branches/faster/docs/cookbook/debugging
branches/faster/docs/cookbook/infrastructures
Removed:
branches/faster/docs/cookbook/debugging_swift
branches/faster/docs/cookbook/diverse_infrastructures
Modified:
branches/faster/
branches/faster/docs/build_docs.sh
branches/faster/docs/cookbook/cookbook.txt
branches/faster/docs/cookbook/log_processing
branches/faster/docs/cookbook/overview
branches/faster/docs/cookbook/swift_basics
branches/faster/docs/documentation/building
branches/faster/docs/userguide/app_procedures
branches/faster/docs/userguide/coasters
branches/faster/docs/userguide/commands
branches/faster/docs/userguide/configuration_properties
branches/faster/docs/userguide/language
branches/faster/docs/userguide/log-processing
branches/faster/docs/userguide/mappers
branches/faster/docs/userguide/overview
branches/faster/docs/userguide/reliability_mechanisms
branches/faster/docs/userguide/transformation_catalog
branches/faster/docs/utils/gensites.txt
branches/faster/libexec/log-processing/README.txt
Log:
merged 6231-6241 from trunk
Property changes on: branches/faster
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/release-0.93:4761-5122
/trunk:6172,6177,6182,6189-6190,6202-6203,6206-6208,6215-6223
+ /branches/release-0.93:4761-5122
/trunk:6172,6177,6182,6189-6190,6202-6203,6206-6208,6215-6223,6231-6241
Modified: branches/faster/docs/build_docs.sh
===================================================================
--- branches/faster/docs/build_docs.sh 2013-07-08 05:54:14 UTC (rev 6612)
+++ branches/faster/docs/build_docs.sh 2013-07-08 06:01:39 UTC (rev 6613)
@@ -57,8 +57,14 @@
chmod $CHMOD_DIRECTORY_MODE $INSTALLATION_DIRECTORY > /dev/null 2>&1
fi
+unamestr=`\uname`
+if [[ "$unamestr" == 'Linux' ]]; then
+ pushd $(dirname $(readlink -f $0)) > /dev/null 2>&1
+else
+ pushd $(dirname $(greadlink -f $0)) > /dev/null 2>&1
+fi
+
# Gather version information
-pushd $(dirname $(readlink -f $0)) > /dev/null 2>&1
pushd .. > /dev/null 2>&1
VERSION=`svn info |grep URL|awk -F / '{print $NF}'`
popd > /dev/null 2>&1
Modified: branches/faster/docs/cookbook/cookbook.txt
===================================================================
--- branches/faster/docs/cookbook/cookbook.txt 2013-07-08 05:54:14 UTC (rev 6612)
+++ branches/faster/docs/cookbook/cookbook.txt 2013-07-08 06:01:39 UTC (rev 6613)
@@ -13,8 +13,8 @@
include::coasters[]
-include::diverse_infrastructures[]
+include::infrastructures[]
-include::debugging_swift[]
+include::debugging[]
include::log_processing[]
Copied: branches/faster/docs/cookbook/debugging (from rev 6232, trunk/docs/cookbook/debugging)
===================================================================
--- branches/faster/docs/cookbook/debugging (rev 0)
+++ branches/faster/docs/cookbook/debugging 2013-07-08 06:01:39 UTC (rev 6613)
@@ -0,0 +1,23 @@
+Debugging Swift
+---------------
+
+Swift errors are logged in several places. In case of a failed Swift run, following places
+should be inspected for clues:
+
+. Text from standard output and standard error produced by running
+Swift
+
+. The .log file from this run. It will be named script.uniqueID.log
+where "script" is the name of your Swift script source file, and
+uniqueID is a long unique id which starts with the date and time you ran the
+swift command.
+
+. $HOME/.globus/scripts directory on the host on which you run the Swift
+command, when swift is submitting to a local scheduler (Condor, PBS, SGE,
+Cobalt)
+
+. $HOME/.globus/coasters directory on remote machines on which you are
+running coasters
+
+. $HOME/.globus/* files on remote systems accessed via Coaster providers
+
Deleted: branches/faster/docs/cookbook/debugging_swift
===================================================================
--- branches/faster/docs/cookbook/debugging_swift 2013-07-08 05:54:14 UTC (rev 6612)
+++ branches/faster/docs/cookbook/debugging_swift 2013-07-08 06:01:39 UTC (rev 6613)
@@ -1,17 +0,0 @@
-Debugging Swift
----------------
-Swift errors are logged in several places:
-
-. All text from standard output and standard error produced by running the
-swift command
-. The .log file from this run. It will be named swiftscript.uniqueID.log
-where "swiftscript" is the name of your *.swift script source file, and
-uniqueID is a long unique id which starts with the date and time you ran the
-swift command.
-. $HOME/.globus/coasters directory on remote machines on which you are
-running coasters
-. $HOME/.globus/scripts directory on the host on which you run the Swift
-command, when swift is submitting to a local scheduler (Condor, PBS, SGE,
-Cobalt)
-. $HOME/.globus/??? on remote systems that you access via Globus
-
Deleted: branches/faster/docs/cookbook/diverse_infrastructures
===================================================================
--- branches/faster/docs/cookbook/diverse_infrastructures 2013-07-08 05:54:14 UTC (rev 6612)
+++ branches/faster/docs/cookbook/diverse_infrastructures 2013-07-08 06:01:39 UTC (rev 6613)
@@ -1,182 +0,0 @@
-Swift on Diverse Infrastructures
----------------------------------
-
-Beagle
-~~~~~~
-Swift is available on Beagle as a software module.
-In this recipe, we will walk you through a very simple example running Swift
-on Beagle. The prerequisites required for running this example on Beagle is an access to Beagle and a project allocation. The example is a simplest form of a real application that involves reading from a disc file and writing onto another disc file. It runs the +/bin/cat+ Unix command to read an input file called +data.txt+ and write it to an output file in a folder +outdir+, called +f.nnn.out+.
-
-image::figures/catsn.png[Swift Example]
-
-The steps are as follows:
-
-*step 1.* Load the Swift module on Beagle commandline as follows: +$ module load swift+
-
-*step 2.* Create and change to a directory where your Swift related work (source, inputs, outputs) will
-stay. (say, +$ mkdir swift-lab+, followed by, +$ cd swift-lab+)
-
-*step 3.* copy the folder at +/home/ketan/labs/catsn+ to the above directory. (+$cp -r /home/ketan/catsn .+ followed by +$ cd catsn+).
-
-*step 4.* In the sites file: +sites.xml+ make the following two
-changes:
-
- 1. change the path of the +$$<workdirectory>$$+ element to your preferred location (this location is *required* to be on /lustre, say to +/lustre/beagle/<yourname>/swift-lab/swift.workdir+), and
-
- 2. Change the project name to your project (for instance, +CI-CCR000013+) . The workdirectory will contain
- execution data related to each run, e.g. wrapper scripts, system information, inputs and outputs.
-
-*step 5.* Run the example using following commandline (also found in run.sh):
-
-----
-$ swift -config cf -tc.file tc -sites.file sites.xml catsn.swift -n=1
-----
-
-You can further change the value of +-n+ to any arbitrary number to run that
-many number of concurrent +cat+
-
-*step 6.* Check the output in the generated +outdir+ directory (+ls outdir+)
-
-Note: Running from sandbox node or requesting 1 hour walltime for upto 3 nodes
-will get fast prioritized execution. Good for small tests
-Swift supports a coasters based computing environment for Beagle. A detailed version of Swift documentation is maintained
-http://www.ci.uchicago.edu/swift/docs/index.php[here].
-
-//
-//Intrepid-BG/P
-//~~~~~~~~~~~~~
-//Swift on Intrepid-BG/P
-//
-
-PADS
-~~~~
-Swift on PADS
-To execute your Swift script on the PADS cluster use this command:
-
-----
-swift -tc.file tc -sites.file pbs.xml modis.swift
-----
-where the contents of a simple pbs.xml sites file could be:
-[xml]
-source~~~~
-<config>
- <pool handle="pbs">
- <execution provider="pbs" url="none"/>
- <profile namespace="globus" key="queue">fast</profile>
- <profile namespace="globus" key="maxwalltime">00:05:00</profile>
- <profile namespace="karajan" key="initialScore">10000</profile>
- <profile namespace="karajan" key="jobThrottle">.10</profile>
- <filesystem provider="local"/>
- <workdirectory >/home/you/swiftwork</workdirectory>
- </pool>
-</config>
-source~~~~
-
-OSG
-~~~
-This section describes how to get Swift running on the OSG Grid. We will use a
-manual coaster setup to get Swift running on OSG.
-
-.Coaster setup on OSG
-The following figure shows an abstract scheme for the manual coasters setup on
-OSG.
-
-image::figures/coaster_setup.png[Coaster setup]
-
-In the following steps, we will go through the process of manually setting
-//Swift on OSG
-//
-
-Bionimbus
-~~~~~~~~~
-This section explains a step by step procedure on getting Swift running on the
-Bionimbus cloud. We will use the _manual_ _coasters_ configuration on the
-Bionimbus cloud.
-
-**step1.** Connect to the gateway (ssh gatewayx.lac.uic.edu)
-
-**step2.** Start a virtual machine (euca-run-instances -n 1 -t m1.small
-emi-17EB1170)
-
-**step3.** Start the coaster-service on gateway
-+coaster-service -port 1984 -localport 35753 -nosec+
-
-**step4.** Start the Swift-script from the gateway using normal Swift commandline
-
-+swift -config cf -tc.file tc -sites.file sites.xml yourscript.swift -aparam=999+
-
-.cf
-----
-wrapperlog.always.transfer=true
-sitedir.keep=true
-execution.retries=1
-lazy.errors=true
-status.mode=provider
-use.provider.staging=true
-provider.staging.pin.swiftfiles=false
-foreach.max.threads=100
-provenance.log=true
-----
-
-.tc
-----
-localhost modftdock /home/ketan/swift-labs/bionimbus-coaster-modftdock/app/modftdock.sh null null GLOBUS::maxwalltime="1:00:00"
-----
-
-(See below for a sample sites.xml for this run)
-
-**step5.** Connect back to gateway from virtual machines using reverse ssh tunneling as follows:
-
-.From the gateway prompt
-
-+ssh -R *:5000:localhost:5000 root at 10.101.8.50 sleep 999+
-
-WHERE:
-*=network interface, should remain the same on all cases
-
-localhost=the gateway host, should remain the same
-
-5000(LEFT OF localhost)=the port number on localhost to listen to **THIS WILL
-vary depending upon which port you want to listen to
-
-5000(RIGHT OF localhost)=the port on target host that you want to forward
-
-root at 10.101.8.50=the ip of the Virtual Machine on Bionimbus cloud, this will
-vary based on what ip you get for your Virtual Machine instance
-
-#On anywhere as long as provide the correct callback uri: here the
-"http://140.221.9.110:42195" is the callback uri of previous ones
-
-**step6.** Start the worker from the virtual machine
-+worker.pl http://localhost:42195 tmp /tmp # where 42195 is the port where the
-coaster service is listening to the workers+
-
-.sites.xml for the above run
-
-----
-<config>
- <pool handle="localhost">
- <execution provider="coaster-persistent" url="http://localhost:1984" jobmanager="local:local"/>
- <profile namespace="globus" key="workerManager">passive</profile>
-
- <profile namespace="globus" key="workersPerNode">4</profile>
- <profile namespace="globus" key="maxTime">10000</profile>
- <profile namespace="globus" key="lowOverAllocation">100</profile>
- <profile namespace="globus" key="highOverAllocation">100</profile>
- <profile namespace="globus" key="slots">100</profile>
- <profile namespace="globus" key="nodeGranularity">1</profile>
- <profile namespace="globus" key="maxNodes">10</profile>
- <profile namespace="karajan" key="jobThrottle">25.00</profile>
- <profile namespace="karajan" key="initialScore">10000</profile>
- <profile namespace="swift" key="stagingMethod">proxy</profile>
- <filesystem provider="local"/>
- <workdirectory>/home/ketan/swift-labs/bionimbus-coaster-modftdock/swift.workdir</workdirectory>
- </pool>
-</config>
-----
-
-//Magellan
-//~~~~~~~~
-//Swift on Magellan
-//
-
Copied: branches/faster/docs/cookbook/infrastructures (from rev 6232, trunk/docs/cookbook/infrastructures)
===================================================================
--- branches/faster/docs/cookbook/infrastructures (rev 0)
+++ branches/faster/docs/cookbook/infrastructures 2013-07-08 06:01:39 UTC (rev 6613)
@@ -0,0 +1,246 @@
+Swift on Computational Infrastructures
+--------------------------------------
+
+Introduction
+~~~~~~~~~~~~
+
+We will use the following Swift example, called "catsn" throughout the present section of this cookbook:
+
+----
+type file;
+
+app (file o) cat (file i)
+{
+ cat @i stdout=@o;
+}
+
+file out[]<simple_mapper; location="odir", prefix="f.", suffix=".out">;
+foreach j in [1:@toint(@arg("n","1"))] {
+ file data<"data.txt">;
+ out[j] = cat(data);
+}
+----
+
+Beagle
+~~~~~~
+Beagle is a University of Chicago's Computational Institute super computer
+dedicated to bioinformatics research. It is a cray XE6 machine running PBS job
+scheduler. More on Beagle could be found
+link:https://beagle.ci.uchicago.edu[here].
+
+Swift is available on Beagle as a software module. This recipe will walk you
+through a very simple example running Swift on Beagle. The prerequisites for
+running this example on Beagle is an access to and a project allocation on
+Beagle. The example is a simplest form of a real application which involves
+reading from a disc file and writing onto another disc file. It runs the
++/bin/cat+ Unix command to read an input file called +data.txt+ and write it to
+an output file in a folder +outdir+, called +f.nnn.out+.
+
+image::figures/catsn.png[Swift Example]
+
+The steps are as follows:
+
+*step 1.* Load the Swift module on Beagle commandline as follows: +$ module
+load swift+
+
+*step 2.* Create and change to a directory where your Swift related work
+(source, inputs, outputs) will stay. (say, +$ mkdir swift-lab+, followed by, +$
+cd swift-lab+)
+
+*step 3.* copy the folder at +/home/ketan/labs/catsn+ to the above directory.
+(+$cp -r /home/ketan/catsn .+ followed by +$ cd catsn+).
+
+*step 4.* In the sites file: +sites.xml+ make the following two changes:
+
+ 1. change the path of the +$$<workdirectory>$$+ element to your preferred
+ location (this location is *required* to be on /lustre, say to
+ +/lustre/beagle/<yourname>/swift-lab/swift.workdir+), and
+
+ 2. Change the project name to your project (for instance, +CI-CCR000013+) .
+ The workdirectory will contain execution data related to each run, e.g.
+ wrapper scripts, system information, inputs and outputs.
+
+*step 5.* Run the example using following commandline:
+
+----
+$ swift -config cf -tc.file tc -sites.file sites.xml catsn.swift -n=1
+----
+
+You can further change the value of +-n+ to any arbitrary number to run that
+many number of concurrent +cat+
+
+*step 6.* Check the output in the generated +outdir+ directory (+ls outdir+)
+
+//
+//Intrepid-BG/P
+//~~~~~~~~~~~~~
+//Swift on Intrepid-BG/P
+//
+
+PADS
+~~~~
+Swift on PADS
+To execute your Swift script on the PADS cluster use this command:
+
+----
+swift -tc.file tc -sites.file pbs.xml catsn.swift
+----
+where the contents of a simple pbs.xml sites file could be:
+[xml]
+source~~~~
+<config>
+ <pool handle="pbs">
+ <execution provider="pbs" url="none"/>
+ <profile namespace="globus" key="queue">fast</profile>
+ <profile namespace="globus" key="maxwalltime">00:05:00</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ <profile namespace="karajan" key="jobThrottle">.10</profile>
+ <filesystem provider="local"/>
+ <workdirectory >/home/you/swiftwork</workdirectory>
+ </pool>
+</config>
+source~~~~
+
+Fusion
+~~~~~~
+Fusion is an Argonne-LCRC computational facility. More information on Fusion can be found link:http://www.lcrc.anl.gov/using-fusion[here].
+
+In the current section, we present a recipe to use Swift on Fusion system.
+Fusion employs a PBS schedular. Consequently, we will use the Coasters PBS provider.
+The jobmanager string will be "local:pbs" as follows:
+
+----
+<execution provider="coaster" jobmanager="local:pbs" url="none" />
+----
+
+The complete sites.xml file is as follows:
+
+[xml]
+source~~~~
+<config>
+ <pool handle="pbs">
+ <execution provider="coaster" jobmanager="local:pbs" url="none" />
+ <profile namespace="globus" key="project">ProteinPrediction</profile>
+ <profile namespace="globus" key="queue">batch</profile>
+ <profile namespace="globus" key="maxtime">1000</profile>
+ <profile namespace="globus" key="slots">1</profile>
+ <profile namespace="globus" key="nodeGranularity">2</profile>
+ <profile namespace="globus" key="maxNodes">2</profile>
+
+ <profile namespace="karajan" key="jobThrottle">0.23</profile>
+
+ <filesystem provider="local"/>
+ <workdirectory >/fusion/gpfs/home/${HOME}/SwiftWork</workdirectory>
+ </pool>
+</config>
+source~~~~
+
+
+OSG
+~~~
+This section describes how to get Swift running on the OSG Grid. We will use a
+manual coaster setup to get Swift running on OSG.
+
+.Coaster setup on OSG
+The following figure shows an abstract scheme for the manual coasters setup on
+OSG.
+
+image::figures/coaster_setup.png[Coaster setup]
+
+In the following steps, we will go through the process of manually setting
+//Swift on OSG
+//
+
+Bionimbus
+~~~~~~~~~
+This section explains a step by step procedure on getting Swift running on the
+Bionimbus cloud. We will use the _manual_ _coasters_ configuration on the
+Bionimbus cloud.
+
+**step1.** Connect to the gateway (ssh gatewayx.lac.uic.edu)
+
+**step2.** Start a virtual machine (euca-run-instances -n 1 -t m1.small
+emi-17EB1170)
+
+**step3.** Start the coaster-service on gateway
++coaster-service -port 1984 -localport 35753 -nosec+
+
+**step4.** Start the Swift-script from the gateway using normal Swift commandline
+
++swift -config cf -tc.file tc -sites.file sites.xml yourscript.swift -aparam=999+
+
+.cf
+----
+wrapperlog.always.transfer=true
+sitedir.keep=true
+execution.retries=1
+lazy.errors=true
+status.mode=provider
+use.provider.staging=true
+provider.staging.pin.swiftfiles=false
+foreach.max.threads=100
+provenance.log=true
+----
+
+.tc
+----
+localhost modftdock /home/ketan/swift-labs/bionimbus-coaster-modftdock/app/modftdock.sh null null GLOBUS::maxwalltime="1:00:00"
+----
+
+(See below for a sample sites.xml for this run)
+
+**step5.** Connect back to gateway from virtual machines using reverse ssh tunneling as follows:
+
+.From the gateway prompt
+
++ssh -R *:5000:localhost:5000 root at 10.101.8.50 sleep 999+
+
+WHERE:
+*=network interface, should remain the same on all cases
+
+localhost=the gateway host, should remain the same
+
+5000(LEFT OF localhost)=the port number on localhost to listen to **THIS WILL
+vary depending upon which port you want to listen to
+
+5000(RIGHT OF localhost)=the port on target host that you want to forward
+
+root at 10.101.8.50=the ip of the Virtual Machine on Bionimbus cloud, this will
+vary based on what ip you get for your Virtual Machine instance
+
+#On anywhere as long as provide the correct callback uri: here the
+"http://140.221.9.110:42195" is the callback uri of previous ones
+
+**step6.** Start the worker from the virtual machine
++worker.pl http://localhost:42195 tmp /tmp # where 42195 is the port where the
+coaster service is listening to the workers+
+
+.sites.xml for the above run
+
+----
+<config>
+ <pool handle="localhost">
+ <execution provider="coaster-persistent" url="http://localhost:1984" jobmanager="local:local"/>
+ <profile namespace="globus" key="workerManager">passive</profile>
+
+ <profile namespace="globus" key="workersPerNode">4</profile>
+ <profile namespace="globus" key="maxTime">10000</profile>
+ <profile namespace="globus" key="lowOverAllocation">100</profile>
+ <profile namespace="globus" key="highOverAllocation">100</profile>
+ <profile namespace="globus" key="slots">100</profile>
+ <profile namespace="globus" key="nodeGranularity">1</profile>
+ <profile namespace="globus" key="maxNodes">10</profile>
+ <profile namespace="karajan" key="jobThrottle">25.00</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ <profile namespace="swift" key="stagingMethod">proxy</profile>
+ <filesystem provider="local"/>
+ <workdirectory>/home/ketan/swift-labs/bionimbus-coaster-modftdock/swift.workdir</workdirectory>
+ </pool>
+</config>
+----
+
+//Magellan
+//~~~~~~~~
+//Swift on Magellan
+//
+
Modified: branches/faster/docs/cookbook/log_processing
===================================================================
--- branches/faster/docs/cookbook/log_processing 2013-07-08 05:54:14 UTC (rev 6612)
+++ branches/faster/docs/cookbook/log_processing 2013-07-08 06:01:39 UTC (rev 6613)
@@ -2,7 +2,7 @@
---------------
To properly generate log plots, you must enable VDL/Karajan logging. Make sure
-log4.properties contains:
+log4j.properties contains:
----
log4j.logger.swift=DEBUG
log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Cpu=DEBUG
Modified: branches/faster/docs/cookbook/overview
===================================================================
--- branches/faster/docs/cookbook/overview 2013-07-08 05:54:14 UTC (rev 6612)
+++ branches/faster/docs/cookbook/overview 2013-07-08 06:01:39 UTC (rev 6613)
@@ -1,11 +1,11 @@
Overview
--------
-This cookbook covers various recipes involving setting up and running Swift under diverse
-configurations based on the application requirements and the underlying
-infrastructures. The Swift system comprises of SwiftScript language and the
-Swift runtime system. For
-introductory material, consult the Swift tutorial found
+This cookbook covers recipes involving setting up and running Swift
+under various computational systems based on the application requirements and the
+underlying infrastructures. The Swift framework comprises of a scripting language
+and the Swift runtime system. For introductory material, consult the Swift
+tutorial found
link:http://www.ci.uchicago.edu/swift/guides/trunk/tutorial/tutorial.html[here].
-WARNING: Please be warned that the cookbook is under construction and the recipes are not tested rigorously.
+WARNING: Note that the cookbook is under construction and the recipes presented here are not tested rigorously.
Modified: branches/faster/docs/cookbook/swift_basics
===================================================================
--- branches/faster/docs/cookbook/swift_basics 2013-07-08 05:54:14 UTC (rev 6612)
+++ branches/faster/docs/cookbook/swift_basics 2013-07-08 06:01:39 UTC (rev 6613)
@@ -12,10 +12,10 @@
^^^^^^^^^^^^^^
.Check your Java
Swift is a Java application. Make sure you are running Java version 5 or higher. You
-can make sure you have Java in your $PATH (or $HOME/.soft file depending upon
+can make sure you have Java in your $PATH (or file such as $HOME/.soft depending upon
your environment)
-Following are the possible ways to detect and run Java:
+Following are some possible ways to detect and run Java:
----
$ grep java $HOME/.soft
@@ -32,10 +32,12 @@
Setting up to run Swift
~~~~~~~~~~~~~~~~~~~~~~~~
This is simple. We will be using a pre-compiled version of Swift that can be
-downloaded from link:http://www.ci.uchicago.edu/swift/downloads/index.php[here]. Download and untar the latest precompiled version as follows:
+downloaded from
+link:http://www.ci.uchicago.edu/swift/downloads/index.php[here]. Download and
+untar the latest precompiled version as follows:
----
-$ tar xf swift-0.92.1.tar.gz
+$ tar zxf swift-0.92.1.tar.gz
----
Environment Setup
@@ -88,7 +90,7 @@
The _site_ field should correspond to a site name listed in the sites catalog.
The _transformation-name_ should correspond to the transformation name used in a
-SwiftScript app procedure.
+Swift script app procedure.
The _executable-path_ should specify where the particular executable is located
on that site.
@@ -99,7 +101,7 @@
The _profiles_ field should be set to null if no profile entries are to be
specified, or should contain the profile entries separated by semicolons.
-Setting swift configuration
+Setting Swift configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Many configuration properties could be set using the Swift configuration file.
We will not cover them all in this section. see
@@ -139,9 +141,9 @@
</pool>
source~~~~~~
-First SwiftScript
+First Swift script
~~~~~~~~~~~~~~~~~
-Your first SwiftScript
+Your first Swift script
Hello Swift-World!
A good sanity check that Swift is set up and running OK locally is this:
@@ -161,7 +163,7 @@
Progress:
-SwiftScript trace: Hello, Swift world!
+Swift trace: Hello, Swift world!
Final status:
@@ -171,7 +173,7 @@
http://www.ci.uchicago.edu/swift/guides/tutorial.php. Follow the steps in that
tutorial to learn how to run a few simple scripts on the login host.
-second SwiftScript
+second Swift script
~~~~~~~~~~~~~~~~~~~
Following is a more involved Swift script.
[java]
@@ -201,16 +203,16 @@
What if Swift hangs
~~~~~~~~~~~~~~~~~~~
-Owing to its highly multithreaded architecture it is often the case that the
+Owing to its multi-threaded architecture it is often the case that the
underlying java virtual machine gets into deadlock situations or Swift hangs
-because of other complications in its threaded operations. Under such
-situations, Swift _hang-checker_ chips in and resolves the situation.
+because of other complications in its operations. Under such
+situations, Swift _hang-checker_ chips in and gives indication of the situation.
. how to use the information to identify and correct the deadlock.
. How close to the Swift source code can we make the hang-checker messages, so that the user can relate it to Swift functions, expressions, and ideally source code lines?
-. The current Hang Checker output is actually *very* nice and useful already:
+. The Hang Checker output is nice and useful:
----
Registered futures:
@@ -224,7 +226,7 @@
Resuming a stopped or crashed Swift Run
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-I had a .rlog file from a Swift run that ran out of time. I kicked it off
+I had a .rlog file from a Swift run that ran out of time. I resumed the run
using the -resume flag described in section 16.2 of the Swift User Guide and
it picked up where it left off. Then I killed it because I wanted to make
changes to my sites file.
@@ -250,8 +252,8 @@
[nbest at login2 files]$ pwd
/home/nbest/bigdata/files
[nbest at login2 files]$
-~wilde/swift/src/stable/cog/modules/swift/dist/swift-svn/bin/swift \
-> -tc.file tc -sites.file pbs.xml ~/scripts/mcd12q1.swift -resume
+~wilde/swift/src/stable/cog/modules/swift/dist/swift-svn/bin/swift -resume \
+> -tc.file tc -sites.file pbs.xml ~/scripts/mcd12q1.swift
> mcd12q1-20100310-1326-ptxe1x1d.0.rlog
Swift svn swift-r3255 (swift modified locally) cog-r2723 (cog modified
locally)
@@ -274,13 +276,6 @@
in previous run:2558
----
-
-From Neil: A comment about that section of the user guide: It says "In order
-to restart from a restart log file, the -resume logfile argument can be used
-after the SwiftScript? program file name." and then puts the -resume logfile
-argument before the script file name. I'm sure the order doesn't matter but
-the contradiction is confusing.
-
Notes to add (from Mike):
- explain what aspects of a Swift script make it restartable, and which
@@ -443,14 +438,14 @@
RunID: 20100310-1105-4okarq08
Progress:
-SwiftScript trace: 1, output/myfile.h11v04.mytype
-SwiftScript trace: 2, home/wilde/bigdata/data/sample/h11v05.tif
-SwiftScript trace: 3, home/wilde/bigdata/data/sample/h12v04.tif
-SwiftScript trace: 0, output/myfile.h32v08.mytype
-SwiftScript trace: 0, home/wilde/bigdata/data/sample/h32v08.tif
-SwiftScript trace: 3, output/myfile.h12v04.mytype
-SwiftScript trace: 1, home/wilde/bigdata/data/sample/h11v04.tif
-SwiftScript trace: 2, output/myfile.h11v05.mytype
+Swift script trace: 1, output/myfile.h11v04.mytype
+Swift script trace: 2, home/wilde/bigdata/data/sample/h11v05.tif
+Swift script trace: 3, home/wilde/bigdata/data/sample/h12v04.tif
+Swift script trace: 0, output/myfile.h32v08.mytype
+Swift script trace: 0, home/wilde/bigdata/data/sample/h32v08.tif
+Swift script trace: 3, output/myfile.h12v04.mytype
+Swift script trace: 1, home/wilde/bigdata/data/sample/h11v04.tif
+Swift script trace: 2, output/myfile.h11v05.mytype
Final status:
login1$
----
Modified: branches/faster/docs/documentation/building
===================================================================
--- branches/faster/docs/documentation/building 2013-07-08 05:54:14 UTC (rev 6612)
+++ branches/faster/docs/documentation/building 2013-07-08 06:01:39 UTC (rev 6613)
@@ -10,6 +10,7 @@
* asciidoc and a2x (http://www.methods.co.nz/asciidoc)
* dblatex (http://dblatex.sourceforge.net)
* source-highlight (http://www.gnu.org/software/src-highlite)
+* For Mac OSX users only: coreutils (installation via macports recommended)
For CI users, compiled versions of these packages can be found in ~davidk.
Modified: branches/faster/docs/userguide/app_procedures
===================================================================
--- branches/faster/docs/userguide/app_procedures 2013-07-08 05:54:14 UTC (rev 6612)
+++ branches/faster/docs/userguide/app_procedures 2013-07-08 06:01:39 UTC (rev 6613)
@@ -15,7 +15,7 @@
mechanisms by which Swift performs that translation; that is described
in the next section.
-In this section, this example SwiftScript program is used for reference:
+In this section, this example Swift script is used for reference:
----
type file;
@@ -40,7 +40,7 @@
This application workspace directory will not be shared with any other
application procedure execution attempt; all application procedure
execution attempts will run with distinct application procedure
-workspaces. (for the avoidance of doubt: If a SwiftScript procedure
+workspaces. (for the avoidance of doubt: If a Swift script procedure
invocation is subject to multiple application procedure execution
attempts (due to Swift-level restarts, retries or replication) then each
of those application procedure execution attempts will be made in a
@@ -87,7 +87,7 @@
of the application executable), then the application executable
should exit with unix return code not equal to 0.
- * Each file mapped from an output parameter of the SwiftScript
+ * Each file mapped from an output parameter of the Swift script
procedure call must exist. Files will be mapped in the same way as
for input files.
@@ -212,7 +212,7 @@
information at execution time to aid in debugging and provenance capture.
-SwiftScript language compilation layer
+Swift script language compilation layer
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Step i: text to XML intermediate form parser/processor. parser written
in ANTLR - see resources/VDL.g. The XML Schema Definition (XSD) for the
@@ -220,7 +220,7 @@
Step ii: XML intermediate form to Karajan workflow. Karajan.java - reads
the XML intermediate form. compiles to karajan workflow language - for
-example, expressions are converted from SwiftScript syntax into Karajan
+example, expressions are converted from Swift script syntax into Karajan
syntax, and function invocations become karajan function invocations
with various modifications to parameters to accomodate return parameters
and dataset handling.
@@ -261,7 +261,7 @@
Function reference
------------------
This section details functions that are available for use in the
-SwiftScript language.
+Swift scripting language.
@arg
~~~~
@@ -287,8 +287,8 @@
Swift v0.3-dev r1674 (modified locally)
RunID: 20080220-1548-ylc4pmda
-SwiftScript trace: defaultvalue
-SwiftScript trace: hello
+Swift trace: defaultvalue
+Swift trace: hello
----
@extractint
@@ -408,13 +408,13 @@
@toint
~~~~~~
@toint(input) will parse its input string into an integer. This can be
-used with @arg to pass input parameters to a SwiftScript program as
+used with @arg to pass input parameters to a Swift script as
integers.
@tofloat
~~~~~~
@tofloat(input) will parse its input string into a floating point number. This can be
-used with @arg to pass input parameters to a SwiftScript program as
+used with @arg to pass input parameters to a Swift script as
floating point numbers.
@length
@@ -428,7 +428,7 @@
Built-in procedure reference
----------------------------
This section details built-in procedures that are available for use in
-the SwiftScript language.
+the Swift scripting language.
readData
~~~~~~~~
Modified: branches/faster/docs/userguide/coasters
===================================================================
--- branches/faster/docs/userguide/coasters 2013-07-08 05:54:14 UTC (rev 6612)
+++ branches/faster/docs/userguide/coasters 2013-07-08 06:01:39 UTC (rev 6613)
@@ -3,40 +3,39 @@
Introduction
~~~~~~~~~~~~
-In many applications, Swift performance can be greatly enhanced by the
-use of coasters. Coasters provide a low-overhead job submission
-and file transfer mechanism suited for the execution of short jobs (on
-the order of a few seconds) and the transfer of small files (on the
-order of a few kilobytes) for which other grid protocols such as GRAM
-and GridFTP are poorly suited.
+Coasters are the Swift's implementation of _pilot job abstraction_.
+In many applications, Swift performance can be greatly enhanced by the use of
+coasters. Coasters provide a low-overhead job submission and file transfer
+mechanism suited for the execution of jobs and the transfer of files for which
+other grid protocols such as GRAM and GridFTP are poorly suited.
+
Benefits
~~~~~~~~
+Much of the overhead associated with other grid protocols (such as
+authentication and authorization, and allocation of worker nodes by the site's
+local resource manager) is reduced, because that overhead is associated with
+the allocation of a coaster pilot or coaster worker, rather than with every
+Swift-level procedure invocation; potentially hundreds or thousands of
+Swift-level procedure invocations can be run through a single worker. Coasters
+can be configured for two purposes: job execution and file staging. In
+practice, the Swift script remains the same while working with coasters. A
+detailed description of coaster mechanism is explained in the next section.
-The coaster mechanism submits a pilot job using some other execution
-mechanism such as GRAM, and for each worker node that will be used in a
-remote cluster, it submits a worker job, again using some other
-execution mechanism such as GRAM. Details on the design of the coaster
-mechanism can be found here. <http://wiki.cogkit.org/wiki/Coasters>
-
Mechanism
~~~~~~~~~
+Coasters run at the task management layer logically under the Swift script. The
+jobs and data movement requirements resulting after the interpretation of a
+Swift script are handled by the coasters. The coaster mechanism submits a pilot
+job using some other execution mechanism such as GRAM, SGE or PBS scheduler,
+and for each worker node that will be used in a remote cluster, it submits a
+worker job, again using some other execution mechanism such as GRAM. Details on
+the design of the coaster mechanism can be found here:
+<http://wiki.cogkit.org/wiki/Coasters>. The pilot job manages file transfers
+and the dispatch of execution jobs to workers.
-The pilot job manages file transfers and the dispatch of execution jobs
-to workers. Much of the overhead associated with other grid protocols
-(such as authentication and authorization, and allocation of worker
-nodes by the site's local resource manager) is reduced, because that
-overhead is associated with the allocation of a coaster pilot or coaster
-worker, rather than with every Swift-level procedure invocation;
-potentially hundreds or thousands of Swift-level procedure invocations
-can be run through a single worker.
-
-Coasters can be configured for two purposes: job execution and
-file staging.
-
Coasters How-to
~~~~~~~~~~~~~~~
-
To use for job execution, specify a sites.xml execution element like this:
----
@@ -45,10 +44,10 @@
The jobmanager string contains more detail than with other providers. It
contains either two or three colon separated fields: 1:the provider to
-be use to execute the coaster head job - this provider will submit from
+be use to execute the coaster pilot job - this provider will submit from
the Swift client side environment. Commonly this will be one of the GRAM
providers; 2: the provider to be used to execute coaster worker jobs.
-This provider will be used to submit from the coaster head job
+This provider will be used to submit from the coaster pilot job
environment, so a local scheduler provider can sometimes be used instead
of GRAM. 3: optionally, the jobmanager to be used when submitting worker
job using the provider specified in field 2.
@@ -60,10 +59,10 @@
----
The url parameter should be a pseudo-URI formed with the URI scheme
-being the name of the provider to use to submit the coaster head job,
+being the name of the provider to use to submit the coaster pilot job,
and the hostname portion being the hostname to be used to execute the
-coaster head job. Note that this provider and hostname will be used for
-execution of a coaster head job, not for file transfer; so for example,
+coaster pilot job. Note that this provider and hostname will be used for
+execution of a coaster pilot job, not for file transfer; so for example,
a GRAM endpoint should be specified here rather than a GridFTP endpoint.
Coasters are affected by the following profile settings, which are
Modified: branches/faster/docs/userguide/commands
===================================================================
--- branches/faster/docs/userguide/commands 2013-07-08 05:54:14 UTC (rev 6612)
+++ branches/faster/docs/userguide/commands 2013-07-08 06:01:39 UTC (rev 6613)
@@ -8,15 +8,15 @@
swift
~~~~~
The swift command is the main command line tool for executing
-SwiftScript programs.
+Swift scripts.
Command-line Syntax
^^^^^^^^^^^^^^^^^^^
The swift command is invoked as follows: swift [options]
-SwiftScript-program [SwiftScript-arguments]* with options taken from the
-following list, and SwiftScript-arguments made available to the
-SwiftScript program through the @arg function.
+Swifti script [Swift-arguments]* with options taken from the
+following list, and Swift script arguments made available to the
+Swift script through the @arg function.
Swift command-line options
@@ -26,11 +26,11 @@
-typecheck
- Does a typecheck of a SwiftScript program, instead of executing it.
+ Does a typecheck of a Swift script, instead of executing it.
-dryrun
- Runs the SwiftScript program without submitting any jobs (can be
+ Runs the Swift script without submitting any jobs (can be
used to get a graph)
-monitor
Modified: branches/faster/docs/userguide/configuration_properties
===================================================================
--- branches/faster/docs/userguide/configuration_properties 2013-07-08 05:54:14 UTC (rev 6612)
+++ branches/faster/docs/userguide/configuration_properties 2013-07-08 06:01:39 UTC (rev 6613)
@@ -173,12 +173,12 @@
value of this property. If set to false, Swift will report the
first error encountered and immediately stop execution. If set to
true, Swift will attempt to run as much as possible from a
- SwiftScript program before stopping execution and reporting all
+ Swift script before stopping execution and reporting all
errors encountered.
- When developing SwiftScript programs, using the default value of
+ When developing Swift scripts, using the default value of
false can make the program easier to debug. However in production
- runs, using true will allow more of a SwiftScript program to be
+ runs, using true will allow more of a Swift script to be
run before Swift aborts execution.
pgraph
@@ -188,7 +188,7 @@
Default value: false
Swift can generate a Graphviz <http://www.graphviz.org/> file
- representing the structure of the SwiftScript program it has run. If
+ representing the structure of the Swift script it has run. If
this property is set to true, Swift will save the provenance graph
in a file named by concatenating the program name and the instance
ID (e.g. helloworld-ht0adgi315l61.dot).
Modified: branches/faster/docs/userguide/language
===================================================================
--- branches/faster/docs/userguide/language 2013-07-08 05:54:14 UTC (rev 6612)
+++ branches/faster/docs/userguide/language 2013-07-08 06:01:39 UTC (rev 6613)
@@ -1,5 +1,5 @@
-The SwiftScript Language
-------------------------
+The Swift scripting Language
+----------------------------
Language basics
~~~~~~~~~~~~~~~
@@ -64,15 +64,15 @@
The examples above have used the type image without any definition of
that type. We can declare it as a marker type which has no structure
-exposed to SwiftScript:
+exposed to Swift script:
----
type image;
----
This does not indicate that the data is unstructured; but it indicates
-that the structure of the data is not exposed to SwiftScript. Instead,
-SwiftScript will treat variables of this type as individual opaque files.
+that the structure of the data is not exposed to Swift. Instead,
+Swift will treat variables of this type as individual opaque files.
With mechanisms to declare types, map variables to data files, and
declare and invoke procedures, we can build a complete (albeit simple)
@@ -172,7 +172,7 @@
z=q(y);
----
-Arrays in SwiftScript are more monotonic - a generalisation of being
+Arrays in Swift are more monotonic - a generalisation of being
assignment. Knowledge about the content of an array increases during
execution, but cannot otherwise change. Each element of the array is
itself single assignment or monotonic (depending on its type). During a
@@ -208,9 +208,9 @@
~~~~~~~~~~~~~~~~~~~
As with many other programming languages, procedures consisting of
-SwiftScript code can be defined. These differ from the previously
+Swift script can be defined. These differ from the previously
mentioned procedures declared with the app keyword, as they invoke
-other SwiftScript procedures rather than a component program.
+other Swift procedures rather than a component program.
----
(file output) process (file input) {
@@ -249,13 +249,13 @@
More about types
~~~~~~~~~~~~~~~~
-Each variable and procedure parameter in SwiftScript is strongly typed.
+Each variable and procedure parameter in Swift script is strongly typed.
Types are used to structure data, to aid in debugging and checking
program correctness and to influence how Swift interacts with data.
The image type declared in previous examples is a marker type.
Marker types indicate that data for a variable is stored in a single
-file with no further structure exposed at the SwiftScript level.
+file with no further structure exposed at the Swift script level.
Arrays have been mentioned above, in the arrays section. A code block
may be applied to each element of an array using foreach; or
Modified: branches/faster/docs/userguide/log-processing
===================================================================
--- branches/faster/docs/userguide/log-processing 2013-07-08 05:54:14 UTC (rev 6612)
+++ branches/faster/docs/userguide/log-processing 2013-07-08 06:01:39 UTC (rev 6613)
@@ -3,71 +3,24 @@
--------------
To properly generate log plots, you must enable VDL/Karajan logging.
-TODO:How?
+This can be done by putting the following lines in log4j.properties file found in the /etc directory of Swift installation:
-You should check the scripts that you intend to use to determine
-what log lines they require and ensure that you are generating
-those lines via log4j.properties
-
-Make sure log4.properties contains:
--------------------------------------
log4j.logger.swift=DEBUG
log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Cpu=DEBUG
log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Block=DEBUG
--------------------------------------
-TODO: Does it work for coasters-based runs only?
-Meaning and interpretation of Swift log messages
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+All the executables, zsh and perl scripts mentioned in the following steps are
+available in the libexec/log-processing directory of your Swift installation.
-A Swift log file is typically a text file with the name of the Swift run and
-its timestamp in the filename and an extension ".log". In addition, a ".rlog"
-file is Swift's resume log which is used by Swift when a run is resumed using
-the "-resume" option. The .rlog file is only for Swift's internal purpose and
-not to be interpreted by the user.
-
-Each line in the log file is typically made up of three parts. The first part
-is the timestamp, the second is the type of log message and the third is the
-message itself. The types of log messages follows the java log4j standard types
-of TRACE, DEBUG, INFO, WARN, ERROR and FATAL. This section lists the various
-Swift log messages and explains the meaning and likely interpretation of those
-messages. Please note that the list is not comprehensive at this time. Also
-note that we will ignore the timestamps here.
-
-. _DEBUG Loader arguments: [-sites.file, sites.xml, -config, cf, -tc.file, tc, postproc-gridftp.swift]_
-. _DEBUG Loader Max heap: 5592449024_
-. _DEBUG textfiles BEGIN_
-. _DEBUG Loader kmlversion is_
-. _DEBUG VDL2ExecutionContext Stack dump_
-. _INFO SetFieldValue Set_
-. _INFO get__site STARTCOMPOUND thread=0-8 name=get__site_
-. _INFO vdl:execute START thread=0-8-0 tr=_
-. _INFO GlobalSubmitQueue No global submit throttle set. Using default (1024)_
-. _DEBUG vdl:execute2 THREAD_ASSOCIATION jobid=getsite-ymj72ook thread=0-8-0-1 host=localhost replicationGroup=xmj72ook_
-. _DEBUG vdl:execute2 JOB_START jobid=getsite-ymj72ook tr=getsite arguments=[644] tmpdir=postproc-gridftp-20120319-0942-adf1o1u2/jobs/y/getsite-ymj72ook host=localhost_
-. _INFO GridExec TASK_DEFINITION_
-. _WARN RemoteConfiguration Find: http://140.221.8.62:38260_
-. _INFO AbstractStreamKarajanChannel$Multiplexer Multiplexer 0 started_
-. _INFO AbstractStreamKarajanChannel$Multiplexer (0) Scheduling SC-null for addition_
-. _INFO AbstractStreamKarajanChannel Channel configured_
-. _INFO MetaChannel MetaChannel: 651528505[1478354072: {}] -> null.bind -> SC-null_
-. _INFO ReadBuffer Will ask for 1 buffers for a size of 6070_
-. _INFO ThrottleManager O maxBuffers=512, crtBuffers=0, allowedTransfers=256, active=0, suspended=0_
-. _INFO ThrottleManager mem=113.54 MB, heap=482.88 MB, maxHeap=5.21 GB_
-. _INFO ThrottleManager I maxBuffers=512, crtBuffers=0, allowedTransfers=256, active=0, suspended=0_
-. _INFO PerformanceDiagnosticInputStream [MEM] Heap total: 482.88 MB, Heap used: 118.58 MB_
-. _INFO vdl:execute END_SUCCESS thread=0-8-0 tr=getsite_
-. _INFO WeightedHostScoreScheduler CONTACT_SELECTED host=localhost, score=99.854_
-. _
-
-
Log plotting
~~~~~~~~~~~~
Normalize event times in the log to the run start time
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-* Generate the log, assuming the log is titled +swift-run.log+
+* Generate the normalized log, assuming the log is titled +swift-run.log+
------------------------------------------
./normalize-log.pl file.contains.start.time swift-run.log > swift-run.norm
@@ -148,3 +101,48 @@
------------------------------------------
swift_plotter.zsh -s buckets.cfg buckets.eps buckets.data
------------------------------------------
+
+
+Meaning and interpretation of Swift log messages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A Swift log file is typically a text file with the name of the Swift run and
+its timestamp in the filename and an extension ".log". In addition, a ".rlog"
+file is Swift's resume log which is used by Swift when a run is resumed using
+the "-resume" option. The .rlog file is only for Swift's internal purpose and
+not to be interpreted by the user.
+
+Each line in the log file is typically made up of three parts. The first part
+is the timestamp, the second is the type of log message and the third is the
+message itself. The types of log messages follows the java log4j standard types
+of TRACE, DEBUG, INFO, WARN, ERROR and FATAL. This section lists the various
+Swift log messages and explains the meaning and likely interpretation of those
+messages. Please note that the list is not comprehensive at this time. Also
+note that we will ignore the timestamps here.
+
+. _DEBUG Loader arguments: [-sites.file, sites.xml, -config, cf, -tc.file, tc, postproc-gridftp.swift]_
+. _DEBUG Loader Max heap: 5592449024_
+. _DEBUG textfiles BEGIN_
+. _DEBUG Loader kmlversion is_
+. _DEBUG VDL2ExecutionContext Stack dump_
+. _INFO SetFieldValue Set_
+. _INFO get__site STARTCOMPOUND thread=0-8 name=get__site_
+. _INFO vdl:execute START thread=0-8-0 tr=_
+. _INFO GlobalSubmitQueue No global submit throttle set. Using default (1024)_
+. _DEBUG vdl:execute2 THREAD_ASSOCIATION jobid=getsite-ymj72ook thread=0-8-0-1 host=localhost replicationGroup=xmj72ook_
+. _DEBUG vdl:execute2 JOB_START jobid=getsite-ymj72ook tr=getsite arguments=[644] tmpdir=postproc-gridftp-20120319-0942-adf1o1u2/jobs/y/getsite-ymj72ook host=localhost_
+. _INFO GridExec TASK_DEFINITION_
+. _WARN RemoteConfiguration Find: http://140.221.8.62:38260_
+. _INFO AbstractStreamKarajanChannel$Multiplexer Multiplexer 0 started_
+. _INFO AbstractStreamKarajanChannel$Multiplexer (0) Scheduling SC-null for addition_
+. _INFO AbstractStreamKarajanChannel Channel configured_
+. _INFO MetaChannel MetaChannel: 651528505[1478354072: {}] -> null.bind -> SC-null_
+. _INFO ReadBuffer Will ask for 1 buffers for a size of 6070_
+. _INFO ThrottleManager O maxBuffers=512, crtBuffers=0, allowedTransfers=256, active=0, suspended=0_
+. _INFO ThrottleManager mem=113.54 MB, heap=482.88 MB, maxHeap=5.21 GB_
+. _INFO ThrottleManager I maxBuffers=512, crtBuffers=0, allowedTransfers=256, active=0, suspended=0_
+. _INFO PerformanceDiagnosticInputStream [MEM] Heap total: 482.88 MB, Heap used: 118.58 MB_
+. _INFO vdl:execute END_SUCCESS thread=0-8-0 tr=getsite_
+. _INFO WeightedHostScoreScheduler CONTACT_SELECTED host=localhost, score=99.854_
+. _
+
Modified: branches/faster/docs/userguide/mappers
===================================================================
--- branches/faster/docs/userguide/mappers 2013-07-08 05:54:14 UTC (rev 6612)
+++ branches/faster/docs/userguide/mappers 2013-07-08 06:01:39 UTC (rev 6613)
@@ -24,7 +24,7 @@
The above example declares a dataset called f1, which uses a single file
mapper to map a file from a specific location.
-SwiftScript has a simplified syntax for this case, since
+Swift has a simplified syntax for this case, since
single_file_mapper is frequently used:
----
@@ -35,20 +35,20 @@
patterns. These are documented in the mappers section of this
guide.
-More technical details about SwiftScript
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The syntax of SwiftScript has a superficial resemblance to C and Java.
+More technical details about Swift script
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The syntax of Swift script has a superficial resemblance to C and Java.
For example, { and } characters are used to enclose blocks of statements.
-A SwiftScript program consists of a number of statements. Statements may
+A Swift script consists of a number of statements. Statements may
declare types, procedures and variables, assign values to variables, and
express operations over arrays.
Variables
^^^^^^^^^
-Variables in SwiftScript are declared to be of a specific type.
-Assignments to those variables must be data of that type. SwiftScript
+Variables in Swift scripts are declared to be of a specific type.
+Assignments to those variables must be data of that type. Swift script
variables are single-assignment - a value may be assigned to a variable
at most once. This assignment can happen at declaration time or later on
in execution. When an attempt to read from a variable that has not yet
@@ -98,10 +98,10 @@
~~~~~~~~~~
There are two kinds of procedure: An atomic procedure, which describes
how an external program can be executed; and compound procedures which
-consist of a sequence of SwiftScript statements.
+consist of a sequence of Swift script statements.
A procedure declaration defines the name of a procedure and its input
-and output parameters. SwiftScript procedures can take multiple inputs
+and output parameters. Swift script procedures can take multiple inputs
and produce multiple outputs. Inputs are specified to the right of the
function name, and outputs are specified to the left. For example:
@@ -155,7 +155,7 @@
Compound procedures
^^^^^^^^^^^^^^^^^^^
-A compound procedure contains a set of SwiftScript statements:
+A compound procedure contains a set of Swift script statements:
----
(type2 b) foo_bar (type1 a) {
@@ -167,7 +167,7 @@
Control Constructs
~~~~~~~~~~~~~~~~~~
-SwiftScript provides if, switch, foreach, and iterate
+Swift script provides if, switch, foreach, and iterate
constructs, with syntax and semantics similar to comparable constructs
in other high-level languages.
@@ -265,7 +265,7 @@
Operators
~~~~~~~~~
-The following infix operators are available for use in SwiftScript
+The following infix operators are available for use in Swift script
expressions.
[options="header, autowidth"]
@@ -286,7 +286,7 @@
Global constants
~~~~~~~~~~~~~~~~
-At the top level of a SwiftScript program, the global modified may be
+At the top level of a Swift script program, the global modified may be
added to a declaration so that it is visible throughout the program,
rather than only at the top level of the program. This allows global
constants (of any type) to be defined. (since Swift 0.10)
@@ -295,9 +295,9 @@
Imports
~~~~~~~
The import directive can be used to import definitions from another
-SwiftScript file. (since Swift 0.10)
+Swift file.
-For example, a SwiftScript program might contain this:
+For example, a Swift script might contain this:
----
import "defs";
@@ -337,7 +337,7 @@
is imported multiple times, for example in different files, then Swift
will only process the imports once.
-Imports may contain anything that is valid in a SwiftScript program,
+Imports may contain anything that is valid in a Swift script,
including the code that causes remote execution.
Mappers
@@ -754,9 +754,11 @@
The output (stdout) of the executable should consist of two columns of data,
separated by a space. The first column should be the path of the mapped
-variable, in SwiftScript syntax (for example [2] means the 2nd element
-of an array) or the symbol $ to represent the root of the mapped
-variable. The following table shows the symbols that should appear in the first column corresponding to the mapping of different types of swift constructs such as scalars, arrays and structs.
+variable, in Swift script syntax (for example [2] means the 2nd element of an
+array) or the symbol $ to represent the root of the mapped variable. The
+following table shows the symbols that should appear in the first column
+corresponding to the mapping of different types of swift constructs such as
+scalars, arrays and structs.
[option="header, autowidth"]
|=============
@@ -793,7 +795,12 @@
|stus[2]|qux
|===========
-Advanced Example: The following mapper.sh is an advanced example of an external mapper that maps a two-dimensional array to a directory of files. The files in the said directory are identified by their names appended by a number between 000 and 099. The first index of the array maps to the first part of the filename while the second index of the array maps to the second part of the filename.
+Advanced Example: The following mapper.sh is an advanced example of an external
+mapper that maps a two-dimensional array to a directory of files. The files in
+the said directory are identified by their names appended by a number between
+000 and 099. The first index of the array maps to the first part of the
+filename while the second index of the array maps to the second part of the
+filename.
----
#!/bin/sh
@@ -837,7 +844,9 @@
----
-Assuming there are 4 files with name aaa, bbb, ccc, ddd and a mod_index of 10, we will have 4x10=40 files mapped to a two-dimensional array in the following pattern:
+Assuming there are 4 files with name aaa, bbb, ccc, ddd and a mod_index of 10,
+we will have 4x10=40 files mapped to a two-dimensional array in the following
+pattern:
[options="header, autowidth"]
|============
Modified: branches/faster/docs/userguide/overview
===================================================================
--- branches/faster/docs/userguide/overview 2013-07-08 05:54:14 UTC (rev 6612)
+++ branches/faster/docs/userguide/overview 2013-07-08 06:01:39 UTC (rev 6613)
@@ -1,6 +1,6 @@
Overview
--------
-This manual provides reference material for Swift: the SwiftScript
+This manual provides reference material for Swift: the Swift
language and the Swift runtime system. For introductory material,
consult the Swift tutorial
<http://www.ci.uchicago.edu/swift/guides/tutorial.php>.
@@ -10,9 +10,9 @@
and procedural composition.
Swift programs (or workflows) are written in a language called
-SwiftScript.
+Swift scripts.
-SwiftScript programs are dataflow oriented - they are primarily
+Swift scripts are dataflow oriented - they are primarily
concerned with processing (possibly large) collections of data files, by
invoking programs to do that processing. Swift handles execution of such
programs on remote sites by choosing sites, handling the staging of
Modified: branches/faster/docs/userguide/reliability_mechanisms
===================================================================
--- branches/faster/docs/userguide/reliability_mechanisms 2013-07-08 05:54:14 UTC (rev 6612)
+++ branches/faster/docs/userguide/reliability_mechanisms 2013-07-08 06:01:39 UTC (rev 6613)
@@ -26,7 +26,7 @@
using the unique job ID and a .rlog extension. This restart log can
then be passed to a subsequent Swift invocation using the -resume
parameter. Swift will resume execution, avoiding execution of
-invocations that have previously completed successfully. The SwiftScript
+invocations that have previously completed successfully. The Swift
source file and input data files should not be modified between runs.
Every run creates a restart log file with a named composed of the file
@@ -38,7 +38,7 @@
fails, swift can use the restart log file to continue execution from a
point before the failure occurred. In order to restart from a restart
log file, the -resume logfile argument can be used after the
-SwiftScript program file name. Example:
+Swift script file name. Example:
----
$ swift -resume example-ht0adgi315l61.0.rlog example.swift.
Modified: branches/faster/docs/userguide/transformation_catalog
===================================================================
--- branches/faster/docs/userguide/transformation_catalog 2013-07-08 05:54:14 UTC (rev 6612)
+++ branches/faster/docs/userguide/transformation_catalog 2013-07-08 06:01:39 UTC (rev 6613)
@@ -22,7 +22,7 @@
The site field should correspond to a site name listed in the sites catalog.
The transformation name should correspond to the transformation name
-used in a SwiftScript app procedure.
+used in a Swift script app procedure.
The executable path should specify where the particular executable is
located on that site.
Modified: branches/faster/docs/utils/gensites.txt
===================================================================
--- branches/faster/docs/utils/gensites.txt 2013-07-08 05:54:14 UTC (rev 6612)
+++ branches/faster/docs/utils/gensites.txt 2013-07-08 06:01:39 UTC (rev 6613)
@@ -51,18 +51,11 @@
-----
Running this command will print some information about the template and give
-you an idea of what settings you will need to specify. You should see
-something like this:
+you an idea of what settings you will need to specify.
------
-Description: Template for Intrepid. More information about this system can be found at http://www.alcf.anl.gov/support/gettingstarted/index.php
-Required Tokens: host work project queue
------
+The required tokens are required to properly use the templates. These are
+placeholder values you will need to specify in the following steps.
-The required tokens are required to properly use the templates. These are placeholder
-values you will need to specify in the
-following steps.
-
Providing Site Specific Values
------------------------------
Modified: branches/faster/libexec/log-processing/README.txt
===================================================================
--- branches/faster/libexec/log-processing/README.txt 2013-07-08 05:54:14 UTC (rev 6612)
+++ branches/faster/libexec/log-processing/README.txt 2013-07-08 06:01:39 UTC (rev 6613)
@@ -30,6 +30,7 @@
------------------------------------------
./normalize-log.pl file.contains.start.time swift-run.log > swift-run.norm
------------------------------------------
+Note: This script not available.
Make a basic load plot from Coasters Cpu log lines
--------------------------------------------------
@@ -40,12 +41,15 @@
------------------------------------------
./cpu-job-load.pl < swift-run.norm > load.data
------------------------------------------
+Note: This script not available.
+
. Plot with the JFreeChart-based plotter in usertools/plotter:
+
------------------------------------------
swift_plotter.zsh -s load.cfg load.eps load.data
------------------------------------------
Note: The load.cfg is available from swift/libexec/log-processing/
+Note: This script not available.
Make a basic job completion plot from Coasters Cpu log lines
@@ -64,6 +68,7 @@
------------------------------------------
swift_plotter.zsh -s completed.cfg completed.eps completed.data
------------------------------------------
+Note: This script not available.
Make a basic Block allocation plot from Coasters Block log lines
----------------------------------------------------------------
@@ -75,12 +80,14 @@
------------------------------------------
./block-level.pl < swift-run.norm > blocks.data
------------------------------------------
+Note: This script not available.
. Plot with the JFreeChart-based plotter in usertools/plotter:
+
------------------------------------------
swift_plotter.zsh -s blocks.{cfg,eps,data}
------------------------------------------
+Note: This script not available.
Make a job run time distribution plot from Coasters Cpu log lines
-----------------------------------------------------------------
@@ -98,12 +105,14 @@
------------------------------------------
./buckets.pl 1 times.data > buckets.data
------------------------------------------
+Note: This script not available.
. Plot with the JFreeChart-based plotter in usertools/plotter:
+
------------------------------------------
swift_plotter.zsh -s buckets.cfg buckets.eps buckets.data
------------------------------------------
+Note: This script not available.
== Utilities
More information about the Swift-commit
mailing list