From wilde at ci.uchicago.edu Mon Aug 1 11:52:35 2011
From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu)
Date: Mon, 1 Aug 2011 11:52:35 -0500 (CDT)
Subject: [Swift-commit] r4919 - in trunk: . bin/grid
Message-ID: <20110801165235.50F629CC9E@svn.ci.uchicago.edu>
Author: wilde
Date: 2011-08-01 11:52:35 -0500 (Mon, 01 Aug 2011)
New Revision: 4919
Modified:
trunk/bin/grid/TODO
trunk/build.xml
Log:
Reinstitute the merging of the bin/grid source dir into the bin/ distribution dir. Updated TODO notes as well.
Modified: trunk/bin/grid/TODO
===================================================================
--- trunk/bin/grid/TODO 2011-08-01 14:34:55 UTC (rev 4918)
+++ trunk/bin/grid/TODO 2011-08-01 16:52:35 UTC (rev 4919)
@@ -1,7 +1,6 @@
+EXTENCI APPLICATION WORK
-ExTENCI Exec
-
create modft install & test file; test under fork and work
run sites gen
@@ -26,15 +25,15 @@
FEATURES
-Add site secltion option to foreachsite
+- Add site selection option to foreachsite
-(Swift feature: foreach site in siwift?_)
-
CLEANUP
-Find all interim p-baked tools under swift/lab/osg and place under grid/ for development
+- Find all interim tools under swift/lab/osg and place under grid/ for development
-Find Glen's tgsites command and integrate
+ENHANCEMENTS
-Merge in gstar
+- Find Glen's tgsites command and integrate
+- incorporate gstar (would be a good Globus Online feature)
+
Modified: trunk/build.xml
===================================================================
--- trunk/build.xml 2011-08-01 14:34:55 UTC (rev 4918)
+++ trunk/build.xml 2011-08-01 16:52:35 UTC (rev 4919)
@@ -75,8 +75,8 @@
-
-
+
+
@@ -94,9 +94,8 @@
-
-
+
From wilde at ci.uchicago.edu Mon Aug 1 13:27:32 2011
From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu)
Date: Mon, 1 Aug 2011 13:27:32 -0500 (CDT)
Subject: [Swift-commit] r4920 - trunk/docs/siteguide
Message-ID: <20110801182732.BE07A9CC9E@svn.ci.uchicago.edu>
Author: wilde
Date: 2011-08-01 13:27:32 -0500 (Mon, 01 Aug 2011)
New Revision: 4920
Added:
trunk/docs/siteguide/grid
Modified:
trunk/docs/siteguide/futuregrid
trunk/docs/siteguide/siteguide.html
trunk/docs/siteguide/siteguide.txt
Log:
Added chapter on grid execution to siteguide. This is a preliminary doc (and a work in progress) for the evolving bin/grid scripts.
Modified: trunk/docs/siteguide/futuregrid
===================================================================
--- trunk/docs/siteguide/futuregrid 2011-08-01 16:52:35 UTC (rev 4919)
+++ trunk/docs/siteguide/futuregrid 2011-08-01 18:27:32 UTC (rev 4920)
@@ -1,5 +1,5 @@
-Futuregrid Quickstart Guide
----------------------------
+FutureGrid
+----------
Downloading and Building Swift
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Added: trunk/docs/siteguide/grid
===================================================================
--- trunk/docs/siteguide/grid (rev 0)
+++ trunk/docs/siteguide/grid 2011-08-01 18:27:32 UTC (rev 4920)
@@ -0,0 +1,125 @@
+Grids, including OSG and TeraGrid
+---------------------------------
+
+Overview of running on grid sites
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Get a DOEGrids cert. Then register it in the *OSG Engage VO*, and/or map it using +gx-request+ on TeraGrid sites.
+
+* Run +GridSetup+ to configure Swift to use the grid sites. This tests
+for correct operation and creates a "green list" of good sites.
+
+* Prepare an installation package for the programs you want to run on
+grid sites via Swift, and install that package using +foreachsite+.
+
+* Run +RunWorkers+ to start and maintain a pool of Swift workers on
+each site.
+
+* Run Swift scripts that use the grid site resources.
+
+
+Requesting Access
+~~~~~~~~~~~~~~~~~
+
+*For OSG*: Obtain a DOEGrids certificate and register the certificate in
+the OSG "Engage" VO following the procedures at:
+
+https://twiki.grid.iu.edu/bin/view/Engagement/EngageNewUserGuide
+
+[red]*FIXME*: access to OSG wiki pages may request the user to present
+a certificate. Is this a problem from users without one? If so, make a
+copy of the page on the Swift web.
+
+*For TeraGrid*: Obtain a DOEGrids certifcate using the OSG ENgage
+instructions above. Ask a TeraGrid PI to add you to a TeraGrid
+project. Once you obtain a login and project access (via US Mail), use
+gx-request to add your certificate
+
+To run jobs using the procedures documented here you need to be logged
+in to a "submit host" on which you will run Swift and other
+grid-related utilities. This can be any host with the OSG client stack
+or equivalent tools installed. Such hosts include the OSG Engage
+submit host, and the two Swift lab servers
+{bridled,communicado}.ci.uchicago.edu.
+
+Obtain a login on engage-submit.renci.org following instructions on
+the OSG URL above.
+
+Obtain a CI login with access to the Swift lab servers by requesting
+"OSG Gridlab" access at:
+
+http://accounts.ci.uchicago.edu
+
+
+Connecting to a submit host
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+-----
+ssh yourusername at bridled.ci.uchicago.edu
+ssh yourusername at communicado.ci.uchicago.edu
+ssh yourusername at engage-submit.renci.org
+-----
+
+Downloading and install Swift
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The current version of Swift can be downloaded from
+http://www.ci.uchicago.edu/swift/downloads/index.php.
+
+Fetch and untar the latest release.
+Then add the Swift bin/ directory to your PATH. For example:
+
+-----
+cd $HOME
+wget http://www.ci.uchicago.edu/swift/packages/swift-0.92.1.tar.gz
+tar txf swift-0.92.1.tar.gz
+export PATH=$PATH:$HOME/swift-0.92.1/bin
+-----
+
+Set up OSG environment
+~~~~~~~~~~~~~~~~~~~~~~
+
+Depending on your shell type, run:
+
+-----
+source /opt/osg/setup.sh
+or
+source /opt/osg/setup.csh
+-----
+
+Create a VOMS Grid proxy
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+-----
+$ voms-proxy-init -voms Engage -valid 12:00
+-----
+
+Generating Configuration Files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+-----
+cd $HOME
+mkdir swiftgrid
+cd swiftgrid
+gen_gridsites
+# Wait a few minutes to a few hours for Swift to validate grid sites
+get_greensites >greensites
+-----
+
+
+Running Swift
+~~~~~~~~~~~~~
+Now that everything is in place, run Swift with the following command:
+
+-----
+swift -sites.file sites.xml -tc.file tc.data catsn.swift -n=10
+-----
+
+You should see several new files being created, called catsn.0001.out, catsn.0002.out, etc. Each of these
+files should contain the contents of what you placed into data.txt. If this happens, your job has run
+successfully on PADS!
+
+More Help
+~~~~~~~~~
+The best place for additional help is the Swift user mailing list. You can subscribe to this list at
+http://mail.ci.uchicago.edu/mailman/listinfo/swift-user. When submitting information, please send your sites.xml file, your tc.data, and any Swift log files that were created during your attempt.
Modified: trunk/docs/siteguide/siteguide.html
===================================================================
--- trunk/docs/siteguide/siteguide.html 2011-08-01 16:52:35 UTC (rev 4919)
+++ trunk/docs/siteguide/siteguide.html 2011-08-01 18:27:32 UTC (rev 4920)
@@ -2,17 +2,30 @@
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
-
-
+
+
Site Configuration Guide
-
+
Site Configuration Guide
-
-
Table of Contents
-
+
+
Table of Contents
+
+
1. PADS
-
1.1. Requesting Access
+
+
1.1. Requesting Access
If you do not already have a Computation Institute account, you can request
one at https://www.ci.uchicago.edu/accounts/. This page will give you a list
of resources you can request access to. Be sure that PADS is selected. If
you already have an existing CI account, but do not have access to PADS,
send an email to support at ci.uchicago.edu to request access.
Once your keys are configured, you should be able to access a PADS login
node with the following command:
ssh yourusername at login.pads.ci.uchicago.edu
-
1.4. Configuring softenv
+
+
+
1.4. Configuring softenv
Softenv is a system used for managing applications. In order to run Swift,
the softenv environment will have to be modified slightly. Softenv is
configured by a file in your home directory called .soft. Edit this file
@@ -591,11 +615,15 @@
+torque
@default
Once you have installed Swift, add the Swift binary to your PATH so you can
easily run it from any directory.
In your home directory, edit the file ".bashrc".
@@ -611,9 +639,12 @@
export PATH=$PATH:$HOME/swift-<version>/bin
Replace <version> with the actual name of the swift directory in the example above.
-
1.7. What You Need To Know Before Running Swift
+
+
+
1.7. What You Need To Know Before Running Swift
Before you can create a Swift configuration file, there are some things
you will need to know.
+
1.7.1. Swift Work Directory
The Swift work directory is a directory which Swift uses for processing work.
This directory needs to be writable. Common options for this are:
@@ -623,6 +654,8 @@
/home/username/work
/tmp
+
+
1.7.2. Which project(s) are you a member of?
PADS requires that you are a member of a project. You can determine this by
running the following command:
@@ -651,6 +684,8 @@
$ projects --set CI-CCR000013 --all
Your default project for all CI clusters has been set to CI-CCR000013.
+
+
1.7.3. Determine your Queue
PADS has several different queues you can submit jobs to depending on
the type of work you will be doing. The command "qstat -q" will print
@@ -742,11 +777,15 @@
-
1.8. Generating Configuration Files
+
+
+
+
1.8. Generating Configuration Files
Now that you know what queue to use, your project, and your work directory, it is time to
set up Swift. Swift uses a configuration file called sites.xml to determine how it should run.
There are two methods you can use for creating this file. You can manually edit
the configuration file, or generate it with a utility called gensites.
+
1.8.1. Manually Editing sites.xml
Below is the template that is used by Swift’s test suite for running on PADS.
@@ -762,7 +801,10 @@
</config>
The values to note here are the ones that are listed between underscores. In the example above, they are QUEUE, and WORK. Queue is the PADS queue to use and WORK is the swift work directory. These are placeholder values you will need to modify to fit your needs. Copy and paste this template, replace the values, and call it sites.xml.
-
1.9. Manually Editing tc.data
+
+
+
+
1.9. Manually Editing tc.data
Below is the tc.data file used by Swift’s test suite for running on PADS.
The swift script we will run is called catsn.swift. It simply cats a file and saves the result. This is a nice simple test to ensure jobs are running correctly. Create a file called data.txt which contains some simple input - a "hello world" will do the trick.
Now that everything is in place, run Swift with the following command:
@@ -804,19 +836,27 @@
You should see several new files being created, called catsn.0001.out, catsn.0002.out, etc. Each of these
files should contain the contents of what you placed into data.txt. If this happens, your job has run
successfully on PADS!
-
1.12. More Help
+
+
+
1.12. More Help
The best place for additional help is the Swift user mailing list. You can subscribe to this list at
https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user. When submitting information, please send your sites.xml file, your tc.data, and any Swift log files that were created during your attempt.
+
+
+
2. Beagle
-
2.1. Requesting Access
+
+
2.1. Requesting Access
If you do not already have a Computation Institute account, you can request
one at https://www.ci.uchicago.edu/accounts/. This page will give you a list
of resources you can request access to.
You already have an existing CI account, but do not have access to Beagle,
send an email to support at ci.uchicago.edu to request access.
-
2.2. Connecting to a login node
+
+
+
2.2. Connecting to a login node
Once you have account, you should be able to access a Beagle login
node with the following command:
@@ -845,13 +885,19 @@
Note: Running from sandbox node or requesting 1 hour walltime for upto 3 nodes
will get fast prioritized execution. Good for small tests.
If you do not already have a futuregrid account, you can follow the
instructions at https://portal.futuregrid.org/gettingstarted to get started.
This page provides information on how to create an account, how to join
a project, how to set up your SSH keys, and how to create a new project.
-
3.4. Downloading Swift VM Tools
+
+
+
3.4. Downloading Swift VM Tools
A set of scripts based around cloudinitd are used to easily start virtual
machines. To download, change to your home directory and run the
following command:
@@ -874,7 +924,9 @@
$ svn co https://svn.ci.uchicago.edu/svn/vdl2/usertools/swift-vm-boot
-
3.5. Download your Credentials
+
+
+
3.5. Download your Credentials
Run the following commands to retrieve your credentials:
@@ -885,7 +937,9 @@
hotel.conf. Near the bottom of this file will be two settings
called vws.repository.s3id and vws.repository.s3key. Copy these
values for the next step.
-
3.6. Configuring Swift VM Tools
+
+
+
3.6. Configuring Swift VM Tools
Change directories into the swift-vm-boot directory. Edit the file
called env.sh. There are two settings here that you will need to
modify: FUTUREGRID_IAAS_ACCESS_KEY and FUTUREGRID_IAAS_SECRET_KEY.
@@ -897,7 +951,9 @@
Python. If you run into problems while running the install.sh
script, please try a more recent version of Python and associated
libraries.
-
3.7. Configuring coaster-service.conf
+
+
+
3.7. Configuring coaster-service.conf
Within your swift distribution’s etc directory is a file called
coaster-service.conf. There are many options here you can modify
as needed, but these are the settings that will be required
@@ -916,7 +972,9 @@
Tunneling should be used when you are restricted by a firewall. If your local
machine has multiple network interfaces, you should also manually set the IP
address of your machine with export IPADDR=your.ip.address.here.
-
3.8. Starting the Coaster Service Script
+
+
+
3.8. Starting the Coaster Service Script
Now that everything is configured, change to a temporary directory
and run this command to start the coaster service:
@@ -927,7 +985,9 @@
and generate Swift configuration files for you to use. The configuration files
will be generated in your current directory. These files are sites.xml, tc.data,
and cf.
-
3.9. Running Swift
+
+
+
3.9. Running Swift
Now that you have all of your configuration files generated, run the following command:
To stop the coaster service, run the following command:
@@ -948,16 +1010,150 @@
This will kill the coaster service, kill the worker scripts on remote systems and terminate
the virtual machines that were created during start-coaster-service.
-
3.11. More Help
+
+
+
3.11. More Help
The best place for additional help is the Swift user mailing list. You can subscribe to this list at
http://mail.ci.uchicago.edu/mailman/listinfo/swift-user. When submitting information, please send
your sites.xml file, your tc.data, and any error messages you run into.
+
+
+
4. Grids, including OSG and TeraGrid
+
+
+
4.1. Overview of running on grid sites
+
+
+
+Get a DOEGrids cert. Then register it in the OSG Engage VO, and/or map it using gx-request on TeraGrid sites.
+
+
+
+
+Run GridSetup to configure Swift to use the grid sites. This tests
+for correct operation and creates a "green list" of good sites.
+
+
+
+
+Prepare an installation package for the programs you want to run on
+grid sites via Swift, and install that package using foreachsite.
+
+
+
+
+Run RunWorkers to start and maintain a pool of Swift workers on
+each site.
+
+
+
+
+Run Swift scripts that use the grid site resources.
+
+
+
+
+
+
4.2. Requesting Access
+
For OSG: Obtain a DOEGrids certificate and register the certificate in
+the OSG "Engage" VO following the procedures at:
FIXME: access to OSG wiki pages may request the user to present
+a certificate. Is this a problem from users without one? If so, make a
+copy of the page on the Swift web.
+
For TeraGrid: Obtain a DOEGrids certifcate using the OSG ENgage
+instructions above. Ask a TeraGrid PI to add you to a TeraGrid
+project. Once you obtain a login and project access (via US Mail), use
+gx-request to add your certificate
+
To run jobs using the procedures documented here you need to be logged
+in to a "submit host" on which you will run Swift and other
+grid-related utilities. This can be any host with the OSG client stack
+or equivalent tools installed. Such hosts include the OSG Engage
+submit host, and the two Swift lab servers
+{bridled,communicado}.ci.uchicago.edu.
+
Obtain a login on engage-submit.renci.org following instructions on
+the OSG URL above.
+
Obtain a CI login with access to the Swift lab servers by requesting
+"OSG Gridlab" access at:
cd $HOME
+mkdir swiftgrid
+cd swiftgrid
+gen_gridsites
+# Wait a few minutes to a few hours for Swift to validate grid sites
+get_greensites >greensites
+
+
+
+
4.8. Running Swift
+
Now that everything is in place, run Swift with the following command:
+
+
+
swift -sites.file sites.xml -tc.file tc.data catsn.swift -n=10
+
+
You should see several new files being created, called catsn.0001.out, catsn.0002.out, etc. Each of these
+files should contain the contents of what you placed into data.txt. If this happens, your job has run
+successfully on PADS!
+
+
+
4.9. More Help
+
The best place for additional help is the Swift user mailing list. You can subscribe to this list at
+http://mail.ci.uchicago.edu/mailman/listinfo/swift-user. When submitting information, please send your sites.xml file, your tc.data, and any Swift log files that were created during your attempt.
+
+
+
+
Modified: trunk/docs/siteguide/siteguide.txt
===================================================================
--- trunk/docs/siteguide/siteguide.txt 2011-08-01 16:52:35 UTC (rev 4919)
+++ trunk/docs/siteguide/siteguide.txt 2011-08-01 18:27:32 UTC (rev 4920)
@@ -11,3 +11,5 @@
include::beagle[]
include::futuregrid[]
+
+include::grid[]
From wilde at ci.uchicago.edu Mon Aug 1 14:04:52 2011
From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu)
Date: Mon, 1 Aug 2011 14:04:52 -0500 (CDT)
Subject: [Swift-commit] r4921 - trunk/docs/tutorial
Message-ID: <20110801190452.AE3029CD25@svn.ci.uchicago.edu>
Author: wilde
Date: 2011-08-01 14:04:52 -0500 (Mon, 01 Aug 2011)
New Revision: 4921
Removed:
trunk/docs/tutorial/tutorial.html
Log:
Removed tutorial.html - thats supposed to be a generated file, should not be in svn.
Deleted: trunk/docs/tutorial/tutorial.html
===================================================================
--- trunk/docs/tutorial/tutorial.html 2011-08-01 18:27:32 UTC (rev 4920)
+++ trunk/docs/tutorial/tutorial.html 2011-08-01 19:04:52 UTC (rev 4921)
@@ -1,1507 +0,0 @@
-
-
-
-
-
-A Swift Tutorial
-
-
-
-
-
-
A Swift Tutorial
-
-
Table of Contents
-
-
-
-
-
1. Introduction
-
-
This is an introductory tutorial describing the use of Swift and its programming
-language SwiftScript. It is intended to introduce new users to the basics of Swift.
-It is structured as a series of simple exercises/examples which you can
-try for yourself as you read along.
-
For information on getting an installation of Swift running, consult the
-Swift Quickstart Guide. We advise you to install the latest stable release of Swift.
-Return to this document when you have successfully run the test SwiftScript program mentioned there.
-
There is also a
-Swift User Guide
-which contains a more detailed reference material on topics covered in this manual. All
-of the programs included in this tutorial can be found in your Swift
-distribution in the examples/tutorial directory.
-
-
2. Hello World
-
-
The first example program, hello.swift, outputs a hello world message
-into a file called hello.txt.
The basic structure of this program is a type definition, an
-application procedure definition, a variable definition and then a
-call to the procedure.
-
First we define a new type, called messagefile. In this example, we will
-use this messagefile type for our output message.
-
-
-
type messagefile;
-
-
All data in SwiftScript must be typed, whether it is stored in memory or
-on disk. This example defines a very simple type. Later on we will see
-more complex type examples.
Next we define a procedure called greeting. This procedure will write
-out the "hello world" message to a file. To achieve this, it executes
-the unix utility echo with a parameter "Hello, world!" and directs
-the standard output into the output file.
-
The actual file to use is specified by the return parameter, t.
-
-
-
messagefile outfile <"hello.txt">;
-
-
Here we define a variable called outfile. The type of this variable is
-messagefile, and we specify that the contents of this variable will be
-stored on disk in a file called hello.txt
Now we call the greeting procedure, with its output going to the outfile
-variable and therefore to hello.txt on disk.
-
Over the following exercises, we’ll extend this simple hello world
-program to demonstrate various features of Swift.
-
-
3. Language features
-
-
3.1. Parameters
-
Procedures can have parameters. Input parameters specify inputs to the
-procedure and output parameters specify outputs. Our hello world greeting
-procedure already uses an output parameter, t, which indicates where the
-greeting output will go. In this section, we will modify the previous
-script to add an input parameter to the greeting function.
We have modified the signature of the greeting procedure to indicate
-that it takes a single parameter, s, of type string.
-
We have modified the invocation of the echo utility so that it takes
-the value of s as a parameter, instead of the string literal "Hello,
-world!".
-
We have modified the output file definition to point to a different file
-on disk.
-
We have modified the invocation of greeting so that a greeting string is
-supplied.
-
The code for this section can be found in parameter.swift. It can be
-invoked using the swift command, with output appearing in parameter.hello.txt:
-
-
-
$ swift parameter.swift
-
-
Now that we can choose our greeting text, we can call the same procedure
-with different parameters to generate several output files with
-different greetings. The code is in manyparam.swift and can be run as
-before using the swift command.
-
-
-
manyparam.swift
-
-
-
type messagefile;
-
-app (messagefile t) greeting (string s) {
- echo s stdout=@filename(t);
-}
-
-messagefile english <"manyparam.english.txt">;
-messagefile french <"manyparam.french.txt">;
-messagefile japanese <"manyparam.japanese.txt">;
-
-english = greeting("hello");
-french = greeting("bonjour");
-japanese = greeting("konnichiwa");
-
-
-
Note that we can intermingle definitions of variables with invocations
-of procedures.
-
When this program runs, there should be three new files in the
-working directory (manyparam.english.txt, manyparam.francais.txt
-and manyparam.nihongo.txt) each containing a greeting in a different
-language.
-
In addition to specifying parameters positionally, parameters can be
-named, and if desired a default value can be specified.
-
3.2. Adding another application
-
Now we’ll define a new application procedure. The procedure we define
-will capitalise all the words in the input file.
-
To do this, we’ll use the unix tr (translate) utility. Here is an
-example of using tr on the unix command line, not using Swift:
-
-
-
$ echo hello | tr '[a-z]' '[A-Z]'
-HELLO
-
-
There are two main steps - updating the transformation catalog, and
-updating the application block.
-
The transformation catalog lists where application executables are located on
-remote sites. We need to modify the transformation catalog to define a logical
-transformation for the tr utility. The transformation catalog can be found
-in etc/tc.data. There are already several entries specifying
-where executables can be found. Add a new line to the file, specifying
-where tr can be found (usually in /usr/bin/tr but it may differ on
-your system), like this:
For now, ignore all of the fields except the second and the third. The
-second field tr specifies a logical application name and the third
-specifies the location of the application executable.
-
Now that we have defined where to find tr, we can use it in SwiftScript.
-
We can define a new procedure, capitalise, which calls tr.
$ swift capitalise.swift
-...
-$ cat capitalise.2.txt
-HELLO FROM SWIFT
-
-
3.3. Anonymous files
-
In the previous section, the file hello.txt is used only to store
-an intermediate result. We don’t really care about which name is used
-for the file, and we can let Swift choose the name.
-
To do that, omit the mapping entirely when declaring hellofile:
-
-
-
messagefile hellofile;
-
-
Swift will choose a filename, which in the present version will be in a
-subdirectory called _concurrent.
-
3.4. Datatypes
-
All data in variables and files has a data type. So far, we’ve seen two
-types:
-
-
-
-string - this is a built-in type for storing strings of text in
- memory, much like in other programming languages
-
-
-
-
-messagefile - this is a user-defined type used to mark disc resident files as
- containing messages
-
-
-
-
SwiftScript has the additional built-in types: boolean, integer and
-float that function much like their counterparts in other programming
-languages.
-
It is also possible to create user defined types with more structure,
-for example:
-
-
-
type details {
- string name;
- int pies;
-}
-
-
Each element of the structured type can be accessed using a . like this:
-
-
-
person.name = "John";
-
-
The following complete program, types.swift, outputs a greeting using a
-user-defined structure type to hold parameters for the message:
-
-
-
types.swift
-
-
-
type messagefile;
-
-type details {
- string name;
- int pies;
-}
-
-app (messagefile t) greeting (details d) {
- echo "Hello. Your name is" d.name "and you have eaten" d.pies "pies." stdout=@filename(t);
-}
-
-details person;
-
-person.name = "John";
-person.pies = 3;
-
-messagefile outfile <"types.pies.txt">;
-
-outfile = greeting(person);
-
-
-
Structured types can be comprised of marker types for files. See the
-later section on mappers for more information about this.
-
3.5. Arrays
-
We can define arrays using the [] suffix in a variable declaration:
-
-
-
string words[] = ["how","are","you"];
-
-
This program, arrays.swift, will declare an array of message files.
Observe that the type of the parameter to greeting is now an array of
-strings, string s[], instead of a single string, string s, that
-elements of the array can be referenced numerically, for example s[0],
-and that the array is initialised using an array literal,
-["how","are","you"].
-
3.6. Mappers
-
A significant difference between SwiftScript and other languages is that
-data can be referred to on disk through variables in a very similar
-fashion to data in memory. For example, in the above examples we have
-seen a variable definition like this:
-
-
-
messagefile outfile <"arrays.txt">;
-
-
This means that outfile is a dataset variable, which is mapped to a
-file on disk called arrays.txt. This variable can be assigned to
-using = in a similar fashion to an in-memory variable. We can say that
-outfile is mapped onto the disk file arrays.txt by a mapper.
-
There are various ways of mapping in SwiftScript. Two forms of mapping,
-simple named mapping and anonymous mapping, have already
-been seen in this tutorial. Later exercises will introduce more forms.
-
In simple named mapping, the name of the file that a variable is mapped to
-is explictly listed.
-
-
-
messagefile outfile <"hello.txt">;
-
-
This is useful when you want to explicitly name input and output files
-for your program. An example of this can be seen with outfile in the
-hello world exercise.
-
With anonymous mapping no name is specified in the source code. A name is
-automatically generated for the file. This is useful for intermediate
-files that are only referenced through SwiftScript.
-A variable declaration is mapped anonymously by ommitting any mapper
-definition.
-
-
-
messagefile hellofile;
-
-
Later exercises will introduce other ways of mapping from disk files to
-SwiftScript variables.
-
3.6.1. The Regexp Mapper
-
In this exercise, we introduce the regexp mapper. This mapper
-transforms a string expression using a regular expression, and uses the
-result of that transformation as the filename to map.
-
regexp.swift demonstrates the use of this by placing output into a
-file that is based on the name of the input file. Our input file is
-mapped to the inputfile variable using the simple named mapper, then
-we use the regular expression mapper to map the output file. We then use
-the countwords() procedure to count the words in the input file and
-store the result in the output file. In order for the countwords()
-procedure to work correctly, add the wc utility (usually found in
-/usr/bin/wc) to tc.data.
-
The following program replaces the suffix of the input file (regexp_mapper.words.txt)
-with a new suffix (.count) to create regexp_mapper.words.count.
The fixed array mapper maps a list of files into an array. Each
-element of the array is mapped into one file in the specified directory.
-See fixed_array_mapper.swift below.
SwiftScript provides a control structure, foreach, to operate on each
-element of an array in parallel.
-
In this example, we will run the previous word counting example over
-each file in an array without having to explicitly list the array
-elements. The source code for this example is in foreach.swift. This
-program uses three input files: foreach.1.txt, foreach.2.txt, and
-foreach.3.txt. After you have run the workflow, you should see that there are three
-output files: foreach.1.count, foreach.2.count and foreach.3.count, each
-containing the word count for the corresponding input file. We combine
-the use of the fixed_array_mapper and the regexp_mapper.
if.swift contains a simple example of this. Compile and run if.swift
-and see that it outputs "good morning". Changing the morning variable
-from true to false will cause the program to output "good afternoon".
-Here is the contents of the full script:
A serial execution of instructions can be carried out using the sequential iteration construct.
-
The following example demonstrates a simple application. Each step of
-the iteration is a string representation of the byte count of the
-previous step’s output, with iteration terminating when the byte count
-reaches zero.
-
Here’s the program:
-
-
-
sequential_iteration.swift
-
-
-
type counterfile;
-
-app (counterfile t) echo(string m) {
- echo m stdout=@filename(t);
-}
-
-app (counterfile t) countstep(counterfile i) {
- wcl @filename(i) @filename(t);
-}
-
-counterfile a[] <simple_mapper;prefix="sequential_iteration.foldout">;
-
-a[0] = echo("793578934574893");
-
-iterate v {
- a[v+1] = countstep(a[v]);
- trace("extract int value ", at extractint(a[v+1]));
-} until (@extractint(a[v+1]) <= 1);
-
-
-
Echo is the standard unix echo utility.
-
wcl is our application code. It counts the number of bytes in the one
-file and writes that count out to another, like this:
-
-
-
$ cat ../wcl
-#!/bin/bash
-echo -n $(wc -c < $1) > $2
-
-$ echo -n hello > a
-$ wcl a b
-$ cat b
-5
-
-
Install the above wcl script somewhere and add a transformation catalog
-(tc) entry for it (see an example below, note that you will need to change the path in third column to the path where wcl is located on your localhost).
$ swift iterate.swift
-Swift svn swift-r3334 cog-r2752
-
-RunID: 20100526-2259-gtlz8zf4
-Progress:
-SwiftScript trace: extract int value , 16.0
-SwiftScript trace: extract int value , 2.0
-SwiftScript trace: extract int value , 1.0
-Final status: Finished successfully:4
-
-
-
4. Runtime features
-
-
4.1. Visualizing the workflow as a graph
-
When running a workflow, its possible to generate a provenance graph at
-the same time:
-
-
-
$ swift -pgraph graph.dot first.swift
-$ dot -ograph.png -Tpng graph.dot
-
-
graph.png can then be viewed using your favourite image viewer. The dot application is part of the graphViz project. More information can be found at http://www.graphviz.org.
-
4.2. Running on a remote site
-
As configured by default, all jobs are run locally. In the previous
-examples, we’ve invoked echo and tr executables from our SwiftScript
-program. These have been run on the local system (the same computer on
-which you ran swift). We can also make our computations run on a
-remote resource.
-
-
-
-
-
-
This example is necessarily more vague than previous ones,
-because it requires access to remote resources. You should ensure that
-you can submit a job using the globus-job-run (or globusrun-ws)
-command(s).
-
-
-
We do not need to modify any SwiftScript code to run on another
-resource. Instead, we must modify another catalog, the site catalog.
-This catalog provides details of the location that applications will be
-run, with the default settings referring to the local machine. We will
-modify it to refer to a remote resource - the UC Teraport cluster. If
-you are not a UC Teraport user, you should use details of a different
-resource that you do have access to.
-
The site catalog is located in etc/sites.xml and is a relatively
-straightforward XML format file. We must modify each of the following
-three settings: gridftp (which indicates how and where data can be
-transferred to the remote resource), jobmanager (which indicates how
-applications can be run on the remote resource) and workdirectory (which
-indicates where working storage can be found on the remote resource).
-
4.3. Writing a mapper
-
This section will introduce writing a custom mapper so that Swift is
-able to access data files laid out in application-specific ways.
-
An application-specific mapper must take the form of a Java class that
-implements the Mapper
-interface.
-
Usually you don’t need to implement this interface directly, because
-Swift provides a number of more concrete classes with some functionality
-already implemented.
This is the abstract interface for mappers in Swift. You must
-implement methods to provide access to mapper properties, to map from a
-SwiftScript dataset path (such as foo[1].bar) to a file name, to check
-whether a file exists. None of the default Swift mappers implement this
-interface directly - instead they use one of the following helper classes.
AbstractFileMapper
-This provides a helper class for mappers which select files based on
-selecting files from a directory listing. It is necessary to write some
-helper methods that are different from the above mapper methods.
-Examples of mappers which use this class are:
-simple_mapper,
-filesys_mapper,
-and the StructuredRegularExpressionMapper.
-
In general, to write a mapper, choose either the AbstractMapper or the
-AbstractFileMapper and extend those. If your mapper will generally
-select the files it returns based on a directory listing and will
-convert paths to filenames using some regular conversion (for example,
-in the way that simple_mapper maps files in a directory that match a
-particular pattern), then you should probably use the
-AbstractFileMapper. If your mapper will produce a list of files in some
-other way (for example, in the way that csv_mapper maps based on
-filenames given in a CSV file rather than looking at which files are in
-a directory), then you should probably use the AbstractMapper.
-
4.4. Writing a very basic mapper
-
In this section, we will write a very basic (almost useless) mapper that
-will map a SwiftScript dataset into a hardcoded file called
-myfile.txt, like this:
-
-
-
Swift variable Filename
-
-
-
-
var <-----------------------------> myfile.txt
-
-
We should be able to use the mapper we write in a SwiftScript program
-like this:
-
-
-
type file;
-file f <my_first_mapper>;
-
-
First we must choose a base class - AbstractMapper or
-AbstractFileMapper. We aren’t going to use a directory listing to decide
-on our mapping - we are getting the mapping from some other source (in
-fact, it will be hard coded). So we will use AbstractMapper.
-
So now onto the source code. We must define a subclass of AbstractMapper
-and implement several mapper methods: isStatic, existing, and map. These
-methods are documented in the javadoc for the Mapper interface.
-
Here is the code implementing this mapper. Put this in your source
-vdsk directory, make a directory src/tutorial/|and put this file in
-src/tutorial/MyFirstMapper.java.
Now we need to inform the Swift engine about the existence of this
-mapper. We do that by editing the MapperFactory class definition, in
-src/org/griphyn/vdl/mapping/MapperFactory.java and adding a
-registerMapper call alongside the existing registerMapper calls, like this:
The first parameter is the name of the mapper that will be used in
-SwiftScript program. The second parameter is the new Mapper class that
-we just wrote.
-
Now rebuild Swift using the "ant redist" target.
-
This new Swift build will be aware of your new mapper. We can test it
-out with a hello world program:
Run this program, and hopefully you will find the "hello" string has
-been output into the hard coded output file myfile.txt:
-
-
-
$ cat myfile.txt
-hello
-
-
So that’s a first very simple mapper implemented. Compare the source
-code to the single_file_mapper in
-SingleFileMapper.java.
-There is not much more code to the single_file_mapper - mostly code to
-deal with the file parameter.
-
4.5. Starting and restarting
-
Now we’re going to try out the restart capabilities of Swift. We will
-make a workflow that will deliberately fail, and then we will fix the
-problem so that Swift can continue with the workflow.
-
First we have the program in working form, restart.swift.
Swift tried to resume the workflow by executing "broken" again. It did
-not try to run the touch or first echo jobs, because the restart log
-says that they do not need to be executed again.
-
Broken failed again, leaving the original restart log in place.
-
Now we will fix the problem with "broken" by restoring the original
-tc.data line that works.
-
Remove the existing "broken" line and replace it with the successful
-tc.data entry above:
Swift tries to run "broken" again. This time it works, and so Swift
-continues on to execute the final piece of the workflow as if nothing
-had ever gone wrong.
-
-
5. Bits
-
-
5.1. Named and optional parameters
-
In addition to specifying parameters positionally, parameters can be
-named, and if desired a default value can be specified:
-
-
-
default.swift
-
-
-
type file;
-
-// s has a default value
-app (file t) echo (string s="hello world") {
- echo s stdout=@filename(t);
-}
-
-file hw1<"default.1.txt">;
-file hw2<"default.2.txt">;
-
-// procedure call using the default value
-hw1 = echo();
-
-// using a different value
-hw2 = echo(s="hello again");
-
-
-
-
-
-
-
-
From ketan at ci.uchicago.edu Mon Aug 1 14:29:06 2011
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Mon, 1 Aug 2011 14:29:06 -0500 (CDT)
Subject: [Swift-commit] r4922 - trunk/bin/grid
Message-ID: <20110801192906.5EA009CD25@svn.ci.uchicago.edu>
Author: ketan
Date: 2011-08-01 14:29:06 -0500 (Mon, 01 Aug 2011)
New Revision: 4922
Added:
trunk/bin/grid/start-mcs
Modified:
trunk/bin/grid/start-ranger-service
Log:
Added passive coasters start script for mcs network of machines: crush, stomp, etc.
Added: trunk/bin/grid/start-mcs
===================================================================
--- trunk/bin/grid/start-mcs (rev 0)
+++ trunk/bin/grid/start-mcs 2011-08-01 19:29:06 UTC (rev 4922)
@@ -0,0 +1,217 @@
+#! /bin/bash
+
+Usage()
+{
+ echo $0: 'Usage: start-mcs [-ls|--local-staging|-ps|--provider-staging|-sp|--service-port portnum] site1 ... siteN'
+}
+
+# Command arguments
+
+STAGINGMETHOD=local-staging
+COMPUTEHOSTS=localhost
+
+while [ $# -gt 0 -a $(expr "$1" : -) = 1 ]; do
+ case "$1" in
+ -ls|--local-staging) STAGINGMETHOD=local-staging; shift ;;
+ -ps|--provider-staging) STAGINGMETHOD=provider-staging; shift ;;
+ *) echo $0: Invalid argument "$1"; Usage; exit 1
+ esac
+done
+
+if [ $# -gt 0 ]; then
+ COMPUTEHOSTS=$*
+fi
+
+echo Staging method: $STAGINGMETHOD
+
+for h in $COMPUTEHOSTS; do
+ echo Host: $h
+ ssh $h rm -rf /tmp/$USER
+ if [ $(expr $h : -) != 0 ]; then
+ echo $0: Error: invalid argument or host name: $h
+ Usage
+ exit 1
+ fi
+done
+
+# COMPUTEHOSTS='crush thwomp stomp crank steamroller grind churn trounce thrash vanquish'
+
+SWIFTBIN=$(dirname $(which swift))
+
+function wait-and-start-workers
+{
+ # Look for:
+ # Passive queue processor initialized. Callback URI is http://140.221.8.62:55379
+
+ for try in $(seq 1 20); do
+ uriline=$(grep "Passive queue processor initialized. Callback URI is" $out 2> /dev/null)
+ if [ "_$uriline" = _ ]; then
+ sleep 1
+ else
+ break;
+ fi
+ done
+
+# FIXME: What happened to the error check here: in case server was not set to passive???
+
+ echo Coaster contact: $SERVICE_URL
+
+ LOGDIR=/tmp/$USER/Swift/workers
+# mkdir -p $LOGDIR # moved this to the ssh command
+
+ IDLETIMEOUT=$((60*60*240)) # 10 days: FIXME: make this a command line arg
+
+ rm -rf remotepid.* # FIXME: should not be needed if we start in a new dir each time
+
+ for host in $(echo $COMPUTEHOSTS); do
+ timestamp=$(date "+%Y.%m%d.%H%M%S")
+ random=$(awk "BEGIN {printf \"%0.5d\", $RANDOM}")
+ ID=$timestamp.$random
+ # FIXME: make logging an argument; set false by default
+ # fixme:send worker.pl to remote host via stdin or scp.
+ # ssh $host '/bin/sh -c '\'"mkdir -p $LOGDIR; WORKER_LOGGING_LEVEL=DEBUG $SWIFTBIN/worker.pl $CONTACT $ID $LOGDIR $IDLETIMEOUT 2>&1 & echo PID=\$!"\' >remotepid.$host &1 & echo PID=\$!"\' >remotepid.$host &1 & echo PID=\$!"\' >remotepid.$host $sshpidfile
+}
+
+# make swiftworkers.XXXX temp dir and link swiftworkers/ to it
+rundir=/tmp/$USER/Swift/server
+mkdir -p $(dirname $rundir)
+trundir=$(mktemp -d $rundir.XXXX)
+rm -rf $rundir
+ln -s $trundir $rundir
+
+echo "Logging to $trundir"
+
+cd $rundir
+out=swift.stdouterr
+
+$SWIFTBIN/coaster-service -nosec -passive -portfile service.sport -localportfile service.wport &> service.log &
+coasterservicepid=$!
+
+#wait until the service properly gets started
+sleep 5
+
+SPORT=$(cat service.sport)
+WPORT=$(cat service.wport)
+SERVICE_URL=http://$(hostname -f):$WPORT
+#$SWIFTBIN/coaster-service -nosec -p $SERVICEPORT >& coaster-service.log &
+echo "service host:port is, " $(hostname -f):${SPORT}
+cat >tc <sites.xml <
+
+
+ passive
+ 4
+ .03
+ 10000
+
+ $HOME/swiftwork
+
+
+ENDS
+
+cat >cf <sites.xml <
+
+
+ passive
+ 4
+ .03
+ 10000
+ proxy
+ /tmp/$USER/SwiftR/swiftwork
+
+
+ENDS
+
+cat >cf <passivate.swift <& /dev/null
+
+ fi
+
+ kill 0 # Kill # FIXME: what was this for????
+}
+
+trap onexit $TRAPS
+
+wait-and-start-workers &
+starterpid=$!
+
+#$SWIFTBIN/swift -config cf -tc.file tc -sites.file sites.xml passivate.swift 2>&1 Service started and set to passive mode. Use ^C to terminate all services and workers."
+
+wait
+
Property changes on: trunk/bin/grid/start-mcs
___________________________________________________________________
Added: svn:executable
+ *
Modified: trunk/bin/grid/start-ranger-service
===================================================================
--- trunk/bin/grid/start-ranger-service 2011-08-01 19:04:52 UTC (rev 4921)
+++ trunk/bin/grid/start-ranger-service 2011-08-01 19:29:06 UTC (rev 4922)
@@ -1,6 +1,5 @@
#! /bin/bash
-# FIXME: make these commandline keyword arguments, eg --nodes=
function usage ()
{
echo "Usage:"
From jonmon at ci.uchicago.edu Mon Aug 1 15:21:21 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Mon, 1 Aug 2011 15:21:21 -0500 (CDT)
Subject: [Swift-commit] r4923 - SwiftApps/GOSwift/pykoa/tools
Message-ID: <20110801202121.71CE19CC9E@svn.ci.uchicago.edu>
Author: jonmon
Date: 2011-08-01 15:21:21 -0500 (Mon, 01 Aug 2011)
New Revision: 4923
Modified:
SwiftApps/GOSwift/pykoa/tools/koa_goswift.py
Log:
o changed the GLOBUS_HOSTNAME variable to a new variable since I have switched VMs.
o changed subprocess.Popen parameter from stderr=subprocess.STDOUT to stderr=subprocess.PIPE
Modified: SwiftApps/GOSwift/pykoa/tools/koa_goswift.py
===================================================================
--- SwiftApps/GOSwift/pykoa/tools/koa_goswift.py 2011-08-01 19:29:06 UTC (rev 4922)
+++ SwiftApps/GOSwift/pykoa/tools/koa_goswift.py 2011-08-01 20:21:21 UTC (rev 4923)
@@ -266,7 +266,7 @@
os.environ["X509_USER_PROXY"] = proxyfile
# Need to figure out how to get the values. Not sure how.
- os.environ["GLOBUS_HOSTNAME"] = "50.16.138.64"
+ os.environ["GLOBUS_HOSTNAME"] = "174.129.136.78"
os.environ["GLOBUS_TCP_PORT_RANGE"] = "50000,51000"
os.environ["GLOBUS_SOURCE_PORT_RANGE"] = "50000,51000"
@@ -296,7 +296,7 @@
preexec_fn=enable_sigint,
cwd=os.getcwd(),
stdout=subprocess.PIPE,
- stderr=subprocess.STDOUT)
+ stderr=subprocess.PIPE)
c = SwiftObserver()
status = c.run(p)
From jonmon at ci.uchicago.edu Mon Aug 1 15:49:57 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Mon, 1 Aug 2011 15:49:57 -0500 (CDT)
Subject: [Swift-commit] r4924 - SwiftApps/GOSwift/pykoa/tools
Message-ID: <20110801204957.08A179CC9E@svn.ci.uchicago.edu>
Author: jonmon
Date: 2011-08-01 15:49:56 -0500 (Mon, 01 Aug 2011)
New Revision: 4924
Modified:
SwiftApps/GOSwift/pykoa/tools/koa_goswift.py
Log:
o whitespace changes and added debug line to koa_goswift.py
Modified: SwiftApps/GOSwift/pykoa/tools/koa_goswift.py
===================================================================
--- SwiftApps/GOSwift/pykoa/tools/koa_goswift.py 2011-08-01 20:21:21 UTC (rev 4923)
+++ SwiftApps/GOSwift/pykoa/tools/koa_goswift.py 2011-08-01 20:49:56 UTC (rev 4924)
@@ -176,7 +176,7 @@
url.update_relative_path()
else:
url = url_parse.TransferURL(arg)
-
+
src_url = None
src_cred = None
default_myproxy = None
@@ -234,7 +234,7 @@
if ep:
check_explicit_activate([ep], options.myproxy_host)
- cred_file = myproxy.contact_myproxy_server(options.myproxy_host,
+ cred_file = myproxy.contact_myproxy_server(options.myproxy_host,
options.myproxy_user, options.myproxy_dn,
lifetime=options.myproxy_lifetime)
@@ -272,7 +272,7 @@
# Execute the script
# cmd = os.path.join(os.environ["HOME"], "Swift_lib/swift-0.92/bin", "swift") # stable 0.92.1 release
- cmd = os.path.join(os.environ["HOME"], "Swift_lib/bin", "swift") # trunk release
+ cmd = os.path.join(os.environ["HOME"], "Swift_lib/bin", "swift") # trunk release
os.chdir(work_directory)
status = -1
@@ -360,14 +360,14 @@
runs.append( int( run_num) )
except ValueError:
pykoa.debug( "Directory "+direc+" is not a run directory" )
-
+
runs.sort()
# if the directories list is not empty, create the name of the run directory
if runs:
last_run = runs[ len(runs)-1 ] + 1
run_directory = "run."+str(last_run)
-
+
work_directory = swift_dir+"/"+run_directory
return (work_directory, last_run)
@@ -414,6 +414,7 @@
pykoa.debug( "Created work_directory: " + work_directory )
except os.error:
sys.stderr.write( work_directory + " already exists, clean up " + os.getenv("HOME") + "/Swift" )
+ pykoa.debug( work_directory + " already exists" )
return 1
with open( work_directory+"/tc", "w" ) as tc_file:
From jonmon at ci.uchicago.edu Mon Aug 1 17:15:41 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Mon, 1 Aug 2011 17:15:41 -0500 (CDT)
Subject: [Swift-commit] r4926 - SwiftApps/GOSwift/pykoa/tools
Message-ID: <20110801221541.7E5C39CD25@svn.ci.uchicago.edu>
Author: jonmon
Date: 2011-08-01 17:15:41 -0500 (Mon, 01 Aug 2011)
New Revision: 4926
Modified:
SwiftApps/GOSwift/pykoa/tools/koa_goswift.py
Log:
o koa_goswift.py
-- write stdout/stderr to file after watch mode completes
-- keep track of stdout/stderr in SwiftObserver
-- comments
-- Add exit code to swift.info file in each run directory
Modified: SwiftApps/GOSwift/pykoa/tools/koa_goswift.py
===================================================================
--- SwiftApps/GOSwift/pykoa/tools/koa_goswift.py 2011-08-01 21:11:20 UTC (rev 4925)
+++ SwiftApps/GOSwift/pykoa/tools/koa_goswift.py 2011-08-01 22:15:41 UTC (rev 4926)
@@ -27,10 +27,12 @@
def stdout_event(self, line):
sys.stdout.write(line)
+ self.lines.append(line)
return
def stderr_event(self, line):
sys.stderr.write(line)
+ self.lines.append(line)
return
def setup_opts(argv):
@@ -271,8 +273,8 @@
os.environ["GLOBUS_SOURCE_PORT_RANGE"] = "50000,51000"
# Execute the script
-# cmd = os.path.join(os.environ["HOME"], "Swift_lib/swift-0.92/bin", "swift") # stable 0.92.1 release
- cmd = os.path.join(os.environ["HOME"], "Swift_lib/bin", "swift") # trunk release
+# cmd = os.path.join(os.environ["HOME"], "Swift_lib/0.92", "swift") # stable 0.92.1 release
+ cmd = os.path.join(os.environ["HOME"], "Swift_lib/trunk", "swift") # trunk release
os.chdir(work_directory)
status = -1
@@ -281,13 +283,16 @@
if options.watch_swift_stdoutput:
try:
with BlockInterrupt():
- execute = cmd + " -runid "+str(run_id)+" -sites.file sites.xml -tc.file tc -config cf script.swift | tee -a swift.out"
+ execute = cmd + " -runid "+str(run_id)+" -sites.file sites.xml -tc.file tc -config cf script.swift"
pykoa.debug( "running %s in watch mode" % (execute))
with open(work_directory+"/run.sh", "w") as run_sh:
run_sh.write( "#!/bin/bash\n" )
run_sh.write( "[ -f /opt/osg/setup.sh ] && source /opt/osg/setup.sh\n")
run_sh.write( execute + "\n" )
+ run_sh.write( "\n" )
+ run_sh.write( "echo \"\nExit Code: $?\" >> swift.info\n" )
+ run_sh.write( "exit $?" )
os.chmod(work_directory+"/run.sh", stat.S_IRUSR | stat.S_IXUSR )
@@ -298,13 +303,25 @@
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
+ pid = p.pid
+
+ with open("swift.info", "w") as swift_info:
+ swift_info.write("StartTime: "+str(datetime.datetime.utcnow())+"\n")
+ swift_info.write("Swift pid: "+str(pid)+"\n")
+ swift_info.write("Run id: "+str(run_id)+"\n")
+ swift_info.write("\n")
+ swift_info.write("Command Line:\n "+execute+"\n")
+
c = SwiftObserver()
status = c.run(p)
- pid = p.pid
finally:
if src_cred:
os.remove( proxyfile )
+ with open("swift.out", "w") as swift_out:
+ for line in c.lines:
+ swift_out.write(line)
+
sys.stdout.write( "Swift exit status: " + str(status) + "\n" )
else:
@@ -316,6 +333,8 @@
run_sh.write( "#!/bin/bash\n" )
run_sh.write( "[ -f /opt/osg/setup.sh ] && source /opt/osg/setup.sh\n")
run_sh.write( execute + "\n" )
+ run_sh.write( "echo \"\nExit Code: $? >> swift.info\"\n" )
+ run_sh.write( "exit $?" )
os.chmod(work_directory+"/run.sh", stat.S_IRUSR | stat.S_IXUSR )
@@ -329,13 +348,6 @@
swift_out.close()
- with open("swift.info", "w") as swift_info:
- swift_info.write("StartTime: "+str(datetime.datetime.utcnow())+"\n")
- swift_info.write("Swift pid: "+str(pid)+"\n")
- swift_info.write("Run id: "+str(run_id)+"\n")
- swift_info.write("\n")
- swift_info.write("Command Line:\n "+execute+"\n")
-
return status
def create_work_directory():
From jonmon at ci.uchicago.edu Tue Aug 2 10:42:49 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Tue, 2 Aug 2011 10:42:49 -0500 (CDT)
Subject: [Swift-commit] r4928 - SwiftApps/GOSwift/pykoa/tools
Message-ID: <20110802154249.02F9E9CCED@svn.ci.uchicago.edu>
Author: jonmon
Date: 2011-08-02 10:42:48 -0500 (Tue, 02 Aug 2011)
New Revision: 4928
Modified:
SwiftApps/GOSwift/pykoa/tools/koa_goswift_list.py
Log:
o use the subprocess module instead of the commands module in koa_goswift_list.py
Modified: SwiftApps/GOSwift/pykoa/tools/koa_goswift_list.py
===================================================================
--- SwiftApps/GOSwift/pykoa/tools/koa_goswift_list.py 2011-08-02 15:25:07 UTC (rev 4927)
+++ SwiftApps/GOSwift/pykoa/tools/koa_goswift_list.py 2011-08-02 15:42:48 UTC (rev 4928)
@@ -1,6 +1,6 @@
import os
import sys
-import commands
+import subprocess
import pykoa
import pykoa.tools
@@ -29,15 +29,17 @@
return 1
run_directory = os.environ["HOME"]+"/Swift";
+ cmd = "/bin/ls -lGht "+run_directory+" | awk '{print $2, $3, $4, $5, $6, $7, $8}'"
- (status, output) = commands.getstatusoutput( "/bin/ls " + run_directory )
+ ret = subprocess.call(cmd, close_fds=True,
+ shell=True)
- if status != 0:
+
+ if ret != 0:
sys.stderr.write( "FAILED TO LIST DIRECTORY\n" )
- pykoa.debug("Listing directory failed with error code: "+str(status) + ": /bin/ls " + run_directory)
- return status
+ pykoa.debug("Listing directory failed with error code: "+str(ret) + ": /bin/ls -lGht -w" + run_directory)
+ return ret
- sys.stdout.write( output+"\n" )
return 0
if __name__ == "__main__":
From jonmon at ci.uchicago.edu Tue Aug 2 10:44:34 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Tue, 2 Aug 2011 10:44:34 -0500 (CDT)
Subject: [Swift-commit] r4929 - SwiftApps/GOSwift/pykoa/tools
Message-ID: <20110802154434.1C2E49CCED@svn.ci.uchicago.edu>
Author: jonmon
Date: 2011-08-02 10:44:33 -0500 (Tue, 02 Aug 2011)
New Revision: 4929
Modified:
SwiftApps/GOSwift/pykoa/tools/koa_goswift_stat.py
Log:
o use the subprocess module instead of the commands moudle in koa_goswift_stat.py
o use os.access to check for existence in koa_goswift_stat.py
o removed the '-n' option, could not make it work at the moment in koa_goswift_stat.py
Modified: SwiftApps/GOSwift/pykoa/tools/koa_goswift_stat.py
===================================================================
--- SwiftApps/GOSwift/pykoa/tools/koa_goswift_stat.py 2011-08-02 15:42:48 UTC (rev 4928)
+++ SwiftApps/GOSwift/pykoa/tools/koa_goswift_stat.py 2011-08-02 15:44:33 UTC (rev 4929)
@@ -1,9 +1,7 @@
#!/usr/bin/python
-import commands
-
import os
import sys
-
+import subprocess
import pykoa
import pykoa.tools
from pykoa.koaexception import cli_exception_handler
@@ -14,7 +12,7 @@
Checks the status of a run started in the background using goswift by reading the last n lines of the stdout/stderr file. Only 1 run is allowed to get the status of at a time.
-Default number of lines is 10. This can be controlled by the -n option.
+Default number of lines is 50.
Type 'man goswift_stat' for details. \
"""
@@ -22,12 +20,8 @@
parser = pykoa.tools.get_option_parser(help_screen)
parser.add_option("--debug", dest="debugging",
default=False, action="store_true",
- help="Get status from the last n lines of the Swift log file. See ")
+ help="Get status from the last 50 lines of the Swift log file.")
- parser.add_option("-n", dest="lines",
- default=10, action="store_true",
- help="Controls the number of lines to print from the swift log file or the stdout/stderr file.")
-
(options, args) = pykoa.tools.parse_args(parser, argv)
return (parser, options, args)
@@ -43,8 +37,14 @@
status = 0
home = os.environ["HOME"]
swift_dir = home +"/Swift"
+
+ if not os.access( swift_dir, os.F_OK ):
+ sys.stderr.write( "Swift has not been run on this machine before.: "+swift_dir+" does not exist\n")
+ return 1
+
work_dir = ""
+ # Get the work directory
try:
work_dir = "run."+str(int(args[0]))
except ValueError:
@@ -52,37 +52,29 @@
return 1
run_dir = swift_dir + "/" + work_dir
- if not os.path.isdir( run_dir ):
+ if not os.access( run_dir, os.F_OK ):
sys.stderr.write( run_dir+" does not exist!\n" )
return 1
+ # output from the log file if the log file exists
if options.debugging:
- if not os.path.isfile( run_dir+"/script-"+args[0]+".log"):
+ if not os.access( run_dir+"/script-"+args[0]+".log", os.F_OK):
sys.stderr.write( run_dir+"/script-"+args[0]+".log does not exitst!\n" )
return 1
- try:
- int(options.lines)
- except ValueError:
- sys.stderr.write( "The option passed to -n is not a number: "+option.lines )
- return 1
-
- (status, output) = commands.getstatusoutput( "tail -"+str(options.lines)+" "+run_dir+"/script-"+args[0]+".log" )
- sys.stdout.write(output+"\n")
+ cmd = "tail -"+str(50)+" "+run_dir+"/script-"+args[0]+".log"
+ status = subprocess.call( cmd, close_fds=True,
+ shell=True )
+ # output from the swift.out file if it exists
else:
- if not os.path.isfile( run_dir+"/swift.out"):
+ if not os.access( run_dir+"/swift.out", os.F_OK):
sys.stderr.write( run_dir+"/swift.out does not exitst!\n" )
return 1
- try:
- int(options.lines)
- except ValueError:
- sys.stderr.write( "The option passed to -n is not a number: "+option.lines )
- return 1
-
- (status, output) = commands.getstatusoutput( "tail -"+str(options.lines)+" "+run_dir+"/swift.out" )
- sys.stdout.write(output+"\n")
-
+ cmd = "tail -"+str(50)+" "+run_dir+"/swift.out"
+ status = subprocess.call( cmd, close_fds=True,
+ shell=True )
+
return status
if __name__ == "__main__":
From jonmon at ci.uchicago.edu Tue Aug 2 11:11:11 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Tue, 2 Aug 2011 11:11:11 -0500 (CDT)
Subject: [Swift-commit] r4930 - SwiftApps/GOSwift/pykoa/tools
Message-ID: <20110802161111.A8B0B9CCED@svn.ci.uchicago.edu>
Author: jonmon
Date: 2011-08-02 11:11:11 -0500 (Tue, 02 Aug 2011)
New Revision: 4930
Modified:
SwiftApps/GOSwift/pykoa/tools/koa_goswift_stat.py
Log:
o fixed the '-n' option in koa_goswift_stat.py. Using the -n option now varies how many lines to be printed out from the log file or the swift.out file.
Modified: SwiftApps/GOSwift/pykoa/tools/koa_goswift_stat.py
===================================================================
--- SwiftApps/GOSwift/pykoa/tools/koa_goswift_stat.py 2011-08-02 15:44:33 UTC (rev 4929)
+++ SwiftApps/GOSwift/pykoa/tools/koa_goswift_stat.py 2011-08-02 16:11:11 UTC (rev 4930)
@@ -12,7 +12,7 @@
Checks the status of a run started in the background using goswift by reading the last n lines of the stdout/stderr file. Only 1 run is allowed to get the status of at a time.
-Default number of lines is 50.
+Default number of lines is 10.
Type 'man goswift_stat' for details. \
"""
@@ -20,8 +20,12 @@
parser = pykoa.tools.get_option_parser(help_screen)
parser.add_option("--debug", dest="debugging",
default=False, action="store_true",
- help="Get status from the last 50 lines of the Swift log file.")
+ help="Get status from the last 10 lines of the Swift log file.")
+ parser.add_option("-n", dest="lines",
+ default="10",
+ help="Set the number of lines to be printed out from the swift.out or log file.")
+
(options, args) = pykoa.tools.parse_args(parser, argv)
return (parser, options, args)
@@ -56,13 +60,19 @@
sys.stderr.write( run_dir+" does not exist!\n" )
return 1
+ try:
+ int(options.lines)
+ except ValueError:
+ sys.stderr.write("Invalid value for '-n': '"+options.lines+"'\n")
+ return 1
+
# output from the log file if the log file exists
if options.debugging:
if not os.access( run_dir+"/script-"+args[0]+".log", os.F_OK):
sys.stderr.write( run_dir+"/script-"+args[0]+".log does not exitst!\n" )
return 1
- cmd = "tail -"+str(50)+" "+run_dir+"/script-"+args[0]+".log"
+ cmd = "tail -"+str(options.lines)+" "+run_dir+"/script-"+args[0]+".log"
status = subprocess.call( cmd, close_fds=True,
shell=True )
# output from the swift.out file if it exists
@@ -71,7 +81,7 @@
sys.stderr.write( run_dir+"/swift.out does not exitst!\n" )
return 1
- cmd = "tail -"+str(50)+" "+run_dir+"/swift.out"
+ cmd = "tail -"+str(options.lines)+" "+run_dir+"/swift.out"
status = subprocess.call( cmd, close_fds=True,
shell=True )
From jonmon at ci.uchicago.edu Tue Aug 2 11:28:12 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Tue, 2 Aug 2011 11:28:12 -0500 (CDT)
Subject: [Swift-commit] r4931 - SwiftApps/GOSwift/pykoa/tools
Message-ID: <20110802162812.DB51D9CC9E@svn.ci.uchicago.edu>
Author: jonmon
Date: 2011-08-02 11:28:12 -0500 (Tue, 02 Aug 2011)
New Revision: 4931
Modified:
SwiftApps/GOSwift/pykoa/tools/koa_goswift.py
Log:
o added options that use either the trunk or stable versions of Swift for execution
o added code that dynamically finds the public ip address for the VM
- Not sure how stable this is.
Modified: SwiftApps/GOSwift/pykoa/tools/koa_goswift.py
===================================================================
--- SwiftApps/GOSwift/pykoa/tools/koa_goswift.py 2011-08-02 16:11:11 UTC (rev 4930)
+++ SwiftApps/GOSwift/pykoa/tools/koa_goswift.py 2011-08-02 16:28:12 UTC (rev 4931)
@@ -3,6 +3,7 @@
import re
import sys
import subprocess
+import commands
import datetime
import stat
@@ -56,6 +57,14 @@
default=False, action="store_true",
help="Run Swift as an immediate command. Watch the output that Swift produces")
+ parser.add_option("--trunk", dest="trunk",
+ default=False, action="store_true",
+ help="Run the development trunk version of Swift")
+
+ parser.add_option("--stable", dest="stable",
+ default=True, action="store_true",
+ help="Run the most stable release version")
+
parser.add_option("-d", dest="deadline",
default="30m",
help="Set the time limit for this workflow. " + \
@@ -267,14 +276,31 @@
os.environ["X509_USER_KEY"] = proxyfile
os.environ["X509_USER_PROXY"] = proxyfile
- # Need to figure out how to get the values. Not sure how.
- os.environ["GLOBUS_HOSTNAME"] = "174.129.136.78"
+ # Not sure this will work all the time, but this is what the internet said i could do.
+ # Need to check for errors, maybe should use the commands module for this part so I can get the status and output at the same time
+ cmd = "curl http://169.254.169.254/2009-04-04/meta-data/public-ipv4"
+ (status, host ) = commands.getstatusoutput( cmd )
+ if status != 0:
+ sys.stderr.write( "Could not determine hostname\n" )
+ return status
+
+ # There has to be an easier way to get the last line of a string
+ # Maybe limit the curl output from above
+ lines = []
+ for line in host.split("\n"):
+ lines.append(line)
+
+ host = lines[len(lines)-1]
+ os.environ["GLOBUS_HOSTNAME"] = host
os.environ["GLOBUS_TCP_PORT_RANGE"] = "50000,51000"
os.environ["GLOBUS_SOURCE_PORT_RANGE"] = "50000,51000"
# Execute the script
-# cmd = os.path.join(os.environ["HOME"], "Swift_lib/0.92", "swift") # stable 0.92.1 release
- cmd = os.path.join(os.environ["HOME"], "Swift_lib/trunk", "swift") # trunk release
+ cmd = ""
+ if options.trunk:
+ cmd = os.path.join(os.environ["HOME"], "Swift_lib/trunk", "swift") # trunk release
+ elif options.stable:
+ cmd = os.path.join(os.environ["HOME"], "Swift_lib/0.92", "swift") # stable 0.92.1 release
os.chdir(work_directory)
status = -1
From jonmon at ci.uchicago.edu Tue Aug 2 16:35:31 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Tue, 2 Aug 2011 16:35:31 -0500 (CDT)
Subject: [Swift-commit] r4932 - in SwiftApps/GOSwift: bin pykoa/tools
Message-ID: <20110802213531.046A09CCED@svn.ci.uchicago.edu>
Author: jonmon
Date: 2011-08-02 16:35:30 -0500 (Tue, 02 Aug 2011)
New Revision: 4932
Added:
SwiftApps/GOSwift/bin/koa-goswift-run
SwiftApps/GOSwift/pykoa/tools/koa_goswift_run.py
Removed:
SwiftApps/GOSwift/bin/koa-goswift
SwiftApps/GOSwift/pykoa/tools/koa_goswift.py
Log:
o rename koa_goswift to koa_goswift_run
o cleaned up the getting of the credentials business
o Fixed issues with writing information to the log. Wrong error code was being written to the log
o remove proxy file when execution is finished
Deleted: SwiftApps/GOSwift/bin/koa-goswift
===================================================================
--- SwiftApps/GOSwift/bin/koa-goswift 2011-08-02 16:28:12 UTC (rev 4931)
+++ SwiftApps/GOSwift/bin/koa-goswift 2011-08-02 21:35:30 UTC (rev 4932)
@@ -1,15 +0,0 @@
-#!/bin/sh
-
-if [ -z "$KOA_HOME" ]; then
- echo "Error: KOA_HOME must be set"
- exit 1
-fi
-
-if [ -s "$KOA_USER" ]; then
- echo "Error: KOA_USER must be set"
- exit 1
-fi
-
-export KOA_API_11=TRUE
-
-exec $KOA_HOME/pykoa/tools/koa_goswift.py "$@"
Copied: SwiftApps/GOSwift/bin/koa-goswift-run (from rev 4924, SwiftApps/GOSwift/bin/koa-goswift)
===================================================================
--- SwiftApps/GOSwift/bin/koa-goswift-run (rev 0)
+++ SwiftApps/GOSwift/bin/koa-goswift-run 2011-08-02 21:35:30 UTC (rev 4932)
@@ -0,0 +1,15 @@
+#!/bin/sh
+
+if [ -z "$KOA_HOME" ]; then
+ echo "Error: KOA_HOME must be set"
+ exit 1
+fi
+
+if [ -s "$KOA_USER" ]; then
+ echo "Error: KOA_USER must be set"
+ exit 1
+fi
+
+export KOA_API_11=TRUE
+
+exec $KOA_HOME/pykoa/tools/koa_goswift_run.py "$@"
Deleted: SwiftApps/GOSwift/pykoa/tools/koa_goswift.py
===================================================================
--- SwiftApps/GOSwift/pykoa/tools/koa_goswift.py 2011-08-02 16:28:12 UTC (rev 4931)
+++ SwiftApps/GOSwift/pykoa/tools/koa_goswift.py 2011-08-02 21:35:30 UTC (rev 4932)
@@ -1,478 +0,0 @@
-#!/usr/bin/python
-import os
-import re
-import sys
-import subprocess
-import commands
-import datetime
-import stat
-
-from sqlalchemy.sql import text
-
-import pykoa
-import pykoa.tools
-from pykoa.process_observer import ProcessObserver
-from pykoa.signal_util import BlockInterrupt, enable_sigint
-from pykoa.tools import url_parse, myproxy
-from pykoa.tools import koa_transfer
-from pykoa.tools.koa_ep_activate import check_explicit_activate
-from pykoa.consumers import guc_util
-from pykoa.constants import get_scheme
-from pykoa.data import endpoint_dao
-from pykoa.data import creds_dao
-from pykoa.koaexception import ToolsException, cli_exception_handler
-
-class SwiftObserver(ProcessObserver):
- def __init__(self):
- self.lines = []
-
- def stdout_event(self, line):
- sys.stdout.write(line)
- self.lines.append(line)
- return
-
- def stderr_event(self, line):
- sys.stderr.write(line)
- self.lines.append(line)
- return
-
-def setup_opts(argv):
- # Set up help message
- help_screen = """ \
-goswift [options] < \"goswift.package\"
-
-Executes Swift under Globus Online. The must be logical sites. This will not work for physical sites.
-Add site using endpoint-add and configure site using endpoint-modify.
-
-Type 'man goswift' for details. \
-"""
-
- # parse out options
- parser = pykoa.tools.get_option_parser(help_screen)
- parser.add_option("--dry-run", dest="do_not_execute",
- default=False, action="store_true",
- help="Output the Swift command line that will execute the workflow and the configuration files")
-
- parser.add_option("--watch", dest="watch_swift_stdoutput",
- default=False, action="store_true",
- help="Run Swift as an immediate command. Watch the output that Swift produces")
-
- parser.add_option("--trunk", dest="trunk",
- default=False, action="store_true",
- help="Run the development trunk version of Swift")
-
- parser.add_option("--stable", dest="stable",
- default=True, action="store_true",
- help="Run the most stable release version")
-
- parser.add_option("-d", dest="deadline",
- default="30m",
- help="Set the time limit for this workflow. " + \
- "Default: 30m (30 minutes)")
-
- parser.add_option("-g", dest="gsi_activation",
- default=False, action="store_true",
- help="Use gsi-ssh delegated credential")
-
-
- myproxy.add_myproxy_options(parser)
- (options, args) = pykoa.tools.parse_args(parser, argv)
-
- if options.gsi_activation and not os.getenv("X509_USER_PROXY"):
- raise ToolsException("ENO_GSI_SSH")
-
- return (parser, options, args)
-
-def parse_stdin():
- lines = []
- index = 0
-
- lines = sys.stdin.readlines()
-
- tc = extract_tc( lines )
- sites_file = extract_sites( lines )
- config = extract_config( lines )
- script = extract_script( lines )
-
- return ( tc, sites_file, config, script )
-
-def extract_tc( lines ):
- begin_tc = False
- tc = ""
-
- for line in lines:
-
- line=line.rstrip()
-
- if line == "# begin tc":
- begin_tc = True
-
- elif line == "# end tc":
- begin_tc = False
- break
-
- elif begin_tc:
- tc += line + "\n"
-
- return tc
-
-def extract_sites( lines ):
- begin_sites = False
- sites = ""
-
- for line in lines:
-
- line=line.rstrip()
-
- if line == "# begin sites":
- begin_sites = True
-
- elif line == "# end sites":
- begin_sites = False
- break
-
- elif begin_sites:
- sites += line + "\n"
-
- return sites
-
-def extract_config( lines ):
- begin_config = False
- config = ""
-
- for line in lines:
-
- line=line.rstrip()
-
- if line == "# begin config":
- begin_config = True
-
- elif line == "# end config":
- begin_config = False
- break
-
- elif begin_config:
- config += line + "\n"
-
- return config
-
-def extract_script( lines ):
- begin_script = False
- script = ""
-
- for line in lines:
-
- line=line.rstrip()
-
- if line == "# begin script":
- begin_script = True
-
- elif line == "# end script":
- begin_script = False
- break
-
- elif begin_script:
- script += line + "\n"
-
- return script
-
-def get_creds(args, conn, user_id, options):
- # A logical endpoint (no dots) can be a scp like path, if it has a :
- # This for loop is wrong. If I am understanding this correctly, this should only parsethe last site I pass the script.
- # A current issue is that the site must also be accompanied by a path, which is not necessary for Swift execution.
- # Unless I leave it and use the path given as the work directory on the site in the sites.xml file?
- for arg in args:
- if re.match("^[^.]+:", arg):
- url = url_parse.SCPUrl(arg, allow_empty_path=True)
- url.update_relative_path()
- else:
- url = url_parse.TransferURL(arg)
-
- src_url = None
- src_cred = None
- default_myproxy = None
- ep = None
-
- if not url.scheme:
- # See if it's a logical endpoint
- ep = endpoint_dao.get_logical_by_name(conn, user_id, url.name)
- if not ep:
- ep = endpoint_dao.get_public_by_name(conn, url.name)
-
- if ep:
- default_myproxy = ep.myproxy_server
- # Get first physical ep, if exists
- phys = endpoint_dao.get_physicals_for_logical(conn, ep.id)
- if not phys:
- raise ToolsException("No physical server exists for '%s'" % (
- url.name))
- phys = phys[0]
- scheme = get_scheme(phys.xfer_service_id)
- src_url = "%s://%s:%d%s" % (scheme, phys.hostname, phys.port, url.path)
- src_subject = phys.x509_dn
-
- # See if LTA endpoint is connected
- if phys.is_lta and phys.alive == 0:
- raise ToolsException("The requested Globus Connect endpoint is not currently connected to Globus Online")
-
- # Get creds
- row = creds_dao.get_cred_for_logical(conn, user_id, ep.id)
- if row and row.exp_time > datetime.datetime.utcnow():
- src_cred = row.sec_info
-
- if not src_url:
- # Not a logical endpoint
- if "." not in url.name:
- raise ToolsException("EEXIST_ENDPOINT", url.name)
- if not url.scheme:
- url.scheme = "gsiftp"
- if not url.port:
- url.port = "2811"
- src_url = url.full_name + url.path
- src_subject = ""
- src_cred = None
-
- # Get creds
- if options.gsi_activation:
- if ep:
- check_explicit_activate([ep])
- cred_file = os.environ["X509_USER_PROXY"]
- src_cred = open(cred_file).read()
-
- elif options.myproxy_host or ((not src_cred) and default_myproxy):
- if not options.myproxy_host:
- options.myproxy_host = default_myproxy
- if ep:
- check_explicit_activate([ep], options.myproxy_host)
-
- cred_file = myproxy.contact_myproxy_server(options.myproxy_host,
- options.myproxy_user, options.myproxy_dn,
- lifetime=options.myproxy_lifetime)
-
- if not cred_file:
- return 1
- try:
- src_cred = open(cred_file).read()
- if ep:
- # Save to the database
- ep_cache = {"foo": ep.id}
- koa_transfer.do_activate(conn, cred_file, ep_cache, user_id,
- quiet=True)
- finally:
- os.remove(cred_file)
-
- if not src_cred:
- raise ToolsException("A credential is needed. Try -g or -m.")
-
- return (src_url, src_cred, src_subject)
-
-def execute_swift(user_id, options, src_url, src_cred, src_subject, work_directory, run_id):
-
- if src_subject:
- os.environ["GLOBUS_FTP_CLIENT_TEST_SUBJECT"] = src_subject
- if src_cred:
- proxyfile = guc_util.sec_text_to_proxy_file(user_id, src_cred)
- os.environ["X509_USER_CERT"] = proxyfile
- os.environ["X509_USER_KEY"] = proxyfile
- os.environ["X509_USER_PROXY"] = proxyfile
-
- # Not sure this will work all the time, but this is what the internet said i could do.
- # Need to check for errors, maybe should use the commands module for this part so I can get the status and output at the same time
- cmd = "curl http://169.254.169.254/2009-04-04/meta-data/public-ipv4"
- (status, host ) = commands.getstatusoutput( cmd )
- if status != 0:
- sys.stderr.write( "Could not determine hostname\n" )
- return status
-
- # There has to be an easier way to get the last line of a string
- # Maybe limit the curl output from above
- lines = []
- for line in host.split("\n"):
- lines.append(line)
-
- host = lines[len(lines)-1]
- os.environ["GLOBUS_HOSTNAME"] = host
- os.environ["GLOBUS_TCP_PORT_RANGE"] = "50000,51000"
- os.environ["GLOBUS_SOURCE_PORT_RANGE"] = "50000,51000"
-
- # Execute the script
- cmd = ""
- if options.trunk:
- cmd = os.path.join(os.environ["HOME"], "Swift_lib/trunk", "swift") # trunk release
- elif options.stable:
- cmd = os.path.join(os.environ["HOME"], "Swift_lib/0.92", "swift") # stable 0.92.1 release
-
- os.chdir(work_directory)
- status = -1
- pid = -1
-
- if options.watch_swift_stdoutput:
- try:
- with BlockInterrupt():
- execute = cmd + " -runid "+str(run_id)+" -sites.file sites.xml -tc.file tc -config cf script.swift"
- pykoa.debug( "running %s in watch mode" % (execute))
-
- with open(work_directory+"/run.sh", "w") as run_sh:
- run_sh.write( "#!/bin/bash\n" )
- run_sh.write( "[ -f /opt/osg/setup.sh ] && source /opt/osg/setup.sh\n")
- run_sh.write( execute + "\n" )
- run_sh.write( "\n" )
- run_sh.write( "echo \"\nExit Code: $?\" >> swift.info\n" )
- run_sh.write( "exit $?" )
-
- os.chmod(work_directory+"/run.sh", stat.S_IRUSR | stat.S_IXUSR )
-
- p = subprocess.Popen("exec "+work_directory+"/run.sh",
- close_fds=True, shell=True,
- preexec_fn=enable_sigint,
- cwd=os.getcwd(),
- stdout=subprocess.PIPE,
- stderr=subprocess.PIPE)
-
- pid = p.pid
-
- with open("swift.info", "w") as swift_info:
- swift_info.write("StartTime: "+str(datetime.datetime.utcnow())+"\n")
- swift_info.write("Swift pid: "+str(pid)+"\n")
- swift_info.write("Run id: "+str(run_id)+"\n")
- swift_info.write("\n")
- swift_info.write("Command Line:\n "+execute+"\n")
-
- c = SwiftObserver()
- status = c.run(p)
- finally:
- if src_cred:
- os.remove( proxyfile )
-
- with open("swift.out", "w") as swift_out:
- for line in c.lines:
- swift_out.write(line)
-
- sys.stdout.write( "Swift exit status: " + str(status) + "\n" )
-
- else:
- swift_out = open("swift.out", "w")
- execute = cmd + " -runid "+str(run_id)+" -sites.file sites.xml -tc.file tc -config cf script.swift"
- pykoa.debug( "running %s in background mode" % (execute))
-
- with open(work_directory+"/run.sh", "w") as run_sh:
- run_sh.write( "#!/bin/bash\n" )
- run_sh.write( "[ -f /opt/osg/setup.sh ] && source /opt/osg/setup.sh\n")
- run_sh.write( execute + "\n" )
- run_sh.write( "echo \"\nExit Code: $? >> swift.info\"\n" )
- run_sh.write( "exit $?" )
-
- os.chmod(work_directory+"/run.sh", stat.S_IRUSR | stat.S_IXUSR )
-
- p = subprocess.Popen("exec "+work_directory+"/run.sh",
- close_fds=True, shell=True,
- preexec_fn=enable_sigint,
- cwd=os.getcwd(),
- stderr=subprocess.STDOUT,
- stdout=swift_out)
- pid = p.pid
-
- swift_out.close()
-
- return status
-
-def create_work_directory():
- # check to see is the swift directory has been created
- home = os.getenv( "HOME" )
- swift_dir = home + "/Swift"
- if not os.access( swift_dir, os.F_OK ):
- os.mkdir( swift_dir )
-
- # default is run.0, if there are other directories then adjust the run directory count
- run_directory = "run.0"
- last_run = 0
- directories = os.listdir( swift_dir )
-
- # This might cause a performance problem(to even begin to start Swift)
- # Need to figure out how maybe purge these run directories
- # For now I can manually purge
- runs = []
- for direc in directories:
- ( name, sep, run_num ) = direc.partition( "." )
- try:
- runs.append( int( run_num) )
- except ValueError:
- pykoa.debug( "Directory "+direc+" is not a run directory" )
-
- runs.sort()
-
- # if the directories list is not empty, create the name of the run directory
- if runs:
- last_run = runs[ len(runs)-1 ] + 1
- run_directory = "run."+str(last_run)
-
- work_directory = swift_dir+"/"+run_directory
- return (work_directory, last_run)
-
-# Do not know what this syntax is but pops up in other pykoa_* files, so I kept it
- at cli_exception_handler
-def main( argv=sys.argv[1:] ):
- (parser, options, args) = setup_opts(argv)
-
- # If I did not specify at least one site, print help message
- if len(args) != 1:
- parser.print_help()
- return 1
-
- # Not sure what these lines do yet, just copied from koa-ly.py
- guc_util.init_guc_env()
-
- conn = pykoa.connect()
- user_row = pykoa.tools.cli_get_user( conn )
- user_id = user_row.id
-
- # get the swift input package
- ( tc, sites, config, script ) = parse_stdin()
-
- (work_directory, run_id) = create_work_directory()
-
- # get a space seperated String of the sites to execute on
- execution_sites=" ".join( map ( str, args ) )
- sys.stdout.write( "\nExecuting Swift with " + execution_sites + " credentials\n" )
- sys.stdout.write( "Execution directory: " + work_directory + "\n" )
-
- # If I do not execute the script, just print out all important information
- # useful to verify that the inputs to the Swift command line have been parsed correctly
- if options.do_not_execute:
- sys.stdout.write( "\nTC file: \n" + tc + "\n" )
- sys.stdout.write( "SITES file: \n" + sites + "\n" )
- sys.stdout.write( "CONFIG file: \n" + config + "\n" )
- sys.stdout.write( "SWIFTSCRIPT file: \n" + script + "\n" )
- sys.stdout.write( "swift -sites.file sites.xml -tc.file tc -config cf script.swift\n\n" )
- return 0
-
- # Set up for the acutal execution
- try:
- os.mkdir( work_directory )
- pykoa.debug( "Created work_directory: " + work_directory )
- except os.error:
- sys.stderr.write( work_directory + " already exists, clean up " + os.getenv("HOME") + "/Swift" )
- pykoa.debug( work_directory + " already exists" )
- return 1
-
- with open( work_directory+"/tc", "w" ) as tc_file:
- tc_file.write( tc )
-
- with open( work_directory+"/sites.xml", "w" ) as sites_file:
- sites_file.write( sites )
-
- with open( work_directory+"/cf", "w" ) as config_file:
- config_file.write( config )
-
- with open( work_directory+"/script.swift", "w" ) as script_file:
- script_file.write( script )
-
- (src_url, src_cred, src_subject ) = get_creds(args, conn, user_id, options)
-
- ret = execute_swift(user_id, options, src_url, src_cred, src_subject, work_directory, run_id)
-
- return ret
-
-if __name__ == "__main__":
- rc = main()
- sys.exit(rc)
Copied: SwiftApps/GOSwift/pykoa/tools/koa_goswift_run.py (from rev 4931, SwiftApps/GOSwift/pykoa/tools/koa_goswift.py)
===================================================================
--- SwiftApps/GOSwift/pykoa/tools/koa_goswift_run.py (rev 0)
+++ SwiftApps/GOSwift/pykoa/tools/koa_goswift_run.py 2011-08-02 21:35:30 UTC (rev 4932)
@@ -0,0 +1,432 @@
+#!/usr/bin/python
+import os
+import re
+import sys
+import subprocess
+import commands
+import datetime
+import stat
+
+from sqlalchemy.sql import text
+
+import pykoa
+import pykoa.tools
+from pykoa.process_observer import ProcessObserver
+from pykoa.signal_util import BlockInterrupt, enable_sigint
+from pykoa.tools import url_parse, myproxy
+from pykoa.tools import koa_transfer
+from pykoa.tools.koa_ep_activate import check_explicit_activate
+from pykoa.consumers import guc_util
+from pykoa.constants import get_scheme
+from pykoa.data import endpoint_dao
+from pykoa.data import creds_dao
+from pykoa.koaexception import ToolsException, cli_exception_handler
+
+class SwiftObserver(ProcessObserver):
+ def __init__(self):
+ self.lines = []
+
+ def stdout_event(self, line):
+ sys.stdout.write(line)
+ self.lines.append(line)
+ return
+
+ def stderr_event(self, line):
+ sys.stderr.write(line)
+ self.lines.append(line)
+ return
+
+def setup_opts(argv):
+ # Set up help message
+ help_screen = """ \
+goswift [options] < \"goswift.package\"
+
+Executes Swift under Globus Online. The must be logical sites. This will not work for physical sites.
+Add site using endpoint-add and configure site using endpoint-modify.
+
+Type 'man goswift' for details. \
+"""
+
+ # parse out options
+ parser = pykoa.tools.get_option_parser(help_screen)
+ parser.add_option("--dry-run", dest="do_not_execute",
+ default=False, action="store_true",
+ help="Output the Swift command line that will execute the workflow and the configuration files")
+
+ parser.add_option("--now", dest="watch_swift_stdoutput",
+ default=False, action="store_true",
+ help="Run Swift as an immediate command. Watch the output that Swift produces")
+
+ parser.add_option("--trunk", dest="trunk",
+ default=False, action="store_true",
+ help="Run the development trunk version of Swift")
+
+ parser.add_option("--stable", dest="stable",
+ default=True, action="store_true",
+ help="Run the most stable release version")
+
+ parser.add_option("-d", dest="deadline",
+ default="30m",
+ help="Set the time limit for this workflow. " + \
+ "Default: 30m (30 minutes)")
+
+ parser.add_option("-g", dest="gsi_activation",
+ default=False, action="store_true",
+ help="Use gsi-ssh delegated credential")
+
+
+ myproxy.add_myproxy_options(parser)
+ (options, args) = pykoa.tools.parse_args(parser, argv)
+
+ if options.gsi_activation and not os.getenv("X509_USER_PROXY"):
+ raise ToolsException("ENO_GSI_SSH")
+
+ return (parser, options, args)
+
+def parse_stdin():
+ lines = []
+ index = 0
+
+ lines = sys.stdin.readlines()
+
+ tc = extract_tc( lines )
+ sites_file = extract_sites( lines )
+ config = extract_config( lines )
+ script = extract_script( lines )
+
+ return ( tc, sites_file, config, script )
+
+def extract_tc( lines ):
+ begin_tc = False
+ tc = ""
+
+ for line in lines:
+
+ line=line.rstrip()
+
+ if line == "# begin tc":
+ begin_tc = True
+
+ elif line == "# end tc":
+ begin_tc = False
+ break
+
+ elif begin_tc:
+ tc += line + "\n"
+
+ return tc
+
+def extract_sites( lines ):
+ begin_sites = False
+ sites = ""
+
+ for line in lines:
+
+ line=line.rstrip()
+
+ if line == "# begin sites":
+ begin_sites = True
+
+ elif line == "# end sites":
+ begin_sites = False
+ break
+
+ elif begin_sites:
+ sites += line + "\n"
+
+ return sites
+
+def extract_config( lines ):
+ begin_config = False
+ config = ""
+
+ for line in lines:
+
+ line=line.rstrip()
+
+ if line == "# begin config":
+ begin_config = True
+
+ elif line == "# end config":
+ begin_config = False
+ break
+
+ elif begin_config:
+ config += line + "\n"
+
+ return config
+
+def extract_script( lines ):
+ begin_script = False
+ script = ""
+
+ for line in lines:
+
+ line=line.rstrip()
+
+ if line == "# begin script":
+ begin_script = True
+
+ elif line == "# end script":
+ begin_script = False
+ break
+
+ elif begin_script:
+ script += line + "\n"
+
+ return script
+
+def get_creds(args, conn, user_id, options):
+ # get activate credentials for a logical endpoint
+ # endpoint must be logical
+ # if endpoint is not activated(or credential is expired), raise ToolsException
+ src_cred = None
+
+ ep = endpoint_dao.get_logical_by_name( conn, user_id, args[0] )
+ if not ep:
+ ep = endpoint_dao.get_public_by_name( conn, args[0] )
+
+ if ep:
+ default_myproxy = ep.myproxy_server
+ # Get first physical ep, if exists
+ phys = endpoint_dao.get_physicals_for_logical(conn, ep.id)
+ if not phys:
+ raise ToolsException("No physical server exists for '%s'" % (
+ args[0]))
+ phys = phys[0]
+ scheme = get_scheme(phys.xfer_service_id)
+ src_subject = phys.x509_dn
+
+ # See if LTA endpoint is connected
+ if phys.is_lta and phys.alive == 0:
+ raise ToolsException("The requested Globus Connect endpoint is not currently connected to Globus Online")
+
+ # Get creds
+ row = creds_dao.get_cred_for_logical(conn, user_id, ep.id)
+ if row and row.exp_time > datetime.datetime.utcnow():
+ src_cred = row.sec_info
+
+ if not src_cred:
+ raise ToolsException("A credential is needed. Activate the endpoint with endpoint-activate")
+
+ return ( src_cred, src_subject)
+
+def execute_swift(user_id, options, src_cred, src_subject, work_directory, run_id):
+
+ if src_subject:
+ os.environ["GLOBUS_FTP_CLIENT_TEST_SUBJECT"] = src_subject
+ if src_cred:
+ proxyfile = guc_util.sec_text_to_proxy_file(user_id, src_cred)
+ os.environ["X509_USER_CERT"] = proxyfile
+ os.environ["X509_USER_KEY"] = proxyfile
+ os.environ["X509_USER_PROXY"] = proxyfile
+
+ # Not sure this will work all the time, but this is what the internet said i could do.
+ cmd = "curl http://169.254.169.254/2009-04-04/meta-data/public-ipv4"
+ (status, host ) = commands.getstatusoutput( cmd )
+ if status != 0:
+ sys.stderr.write( "Could not determine hostname\n" )
+ return status
+
+ # There has to be an easier way to get the last line of a string
+ # Maybe limit the curl output from above?
+ lines = []
+ for line in host.split("\n"):
+ lines.append(line)
+
+ host = lines[len(lines)-1]
+ os.environ["GLOBUS_HOSTNAME"] = host
+ os.environ["GLOBUS_TCP_PORT_RANGE"] = "50000,51000"
+ os.environ["GLOBUS_SOURCE_PORT_RANGE"] = "50000,51000"
+
+ # Execute the script
+ cmd = ""
+ if options.trunk:
+ cmd = os.path.join(os.environ["HOME"], "Swift_lib/trunk", "swift") # trunk release
+ elif options.stable:
+ cmd = os.path.join(os.environ["HOME"], "Swift_lib/0.92", "swift") # stable 0.92.1 release
+
+ os.chdir(work_directory)
+ status = -1
+ pid = -1
+
+ if options.watch_swift_stdoutput:
+ try:
+ with BlockInterrupt():
+ execute = cmd + " -runid "+str(run_id)+" -sites.file sites.xml -tc.file tc -config cf script.swift"
+ pykoa.debug( "running %s in watch mode" % (execute))
+
+ with open(work_directory+"/run.sh", "w") as run_sh:
+ run_sh.write( "#!/bin/bash\n" )
+ run_sh.write( "[ -f /opt/osg/setup.sh ] && source /opt/osg/setup.sh\n")
+ run_sh.write( execute + "\n" )
+ run_sh.write( "ret=$?\n" )
+ run_sh.write( "\n" )
+ run_sh.write( "echo \"\nExit Code: $ret\" >> swift.info\n" )
+ run_sh.write( "exit $ret" )
+
+ os.chmod(work_directory+"/run.sh", stat.S_IRUSR | stat.S_IXUSR )
+
+ p = subprocess.Popen("exec "+work_directory+"/run.sh",
+ close_fds=True, shell=True,
+ preexec_fn=enable_sigint,
+ cwd=os.getcwd(),
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE)
+
+ pid = p.pid
+
+ with open("swift.info", "w") as swift_info:
+ swift_info.write("StartTime: "+str(datetime.datetime.utcnow())+"\n")
+ swift_info.write("Swift pid: "+str(pid)+"\n")
+ swift_info.write("Run id: "+str(run_id)+"\n")
+ swift_info.write("\n")
+ swift_info.write("Command Line:\n "+execute+"\n")
+
+ c = SwiftObserver()
+ status = c.run(p)
+ finally:
+ if src_cred:
+ os.remove( proxyfile )
+
+ with open("swift.out", "a") as swift_out:
+ for line in c.lines:
+ swift_out.write(line)
+
+ sys.stdout.write( "Swift exit status: " + str(status) + "\n" )
+
+ else:
+ swift_out = open("swift.out", "w")
+ execute = cmd + " -runid "+str(run_id)+" -sites.file sites.xml -tc.file tc -config cf script.swift"
+ pykoa.debug( "running %s in background mode" % (execute))
+
+ with open(work_directory+"/run.sh", "w") as run_sh:
+ run_sh.write( "#!/bin/bash\n" )
+ run_sh.write( "[ -f /opt/osg/setup.sh ] && source /opt/osg/setup.sh\n")
+ run_sh.write( execute + "\n" )
+ run_sh.write( "ret=$?\n" )
+ run_sh.write( "echo \"\nExit Code: $ret\" >> swift.info\n" )
+ run_sh.write( "rm -f $X509_USER_PROXY\n" )
+ run_sh.write( "exit $ret" )
+
+ os.chmod(work_directory+"/run.sh", stat.S_IRUSR | stat.S_IXUSR )
+
+ p = subprocess.Popen("exec "+work_directory+"/run.sh",
+ close_fds=True, shell=True,
+ preexec_fn=enable_sigint,
+ cwd=os.getcwd(),
+ stderr=subprocess.STDOUT,
+ stdout=swift_out)
+ pid = p.pid
+
+ with open("swift.info", "w") as swift_info:
+ swift_info.write("StartTime: "+str(datetime.datetime.utcnow())+"\n")
+ swift_info.write("Swift pid: "+str(pid)+"\n")
+ swift_info.write("Run id: "+str(run_id)+"\n")
+ swift_info.write("\n")
+ swift_info.write("Command Line:\n "+execute+"\n")
+
+ status = 0
+
+ return status
+
+def create_work_directory():
+ # check to see is the swift directory has been created
+ home = os.getenv( "HOME" )
+ swift_dir = home + "/Swift"
+ if not os.access( swift_dir, os.F_OK ):
+ os.mkdir( swift_dir )
+
+ # default is run.0, if there are other directories then adjust the run directory count
+ run_directory = "run.0"
+ last_run = 0
+ directories = os.listdir( swift_dir )
+
+ # This might cause a performance problem(to even begin to start Swift)
+ # Need to figure out how maybe purge these run directories
+ # For now I can manually purge
+ runs = []
+ for direc in directories:
+ ( name, sep, run_num ) = direc.partition( "." )
+ try:
+ runs.append( int( run_num) )
+ except ValueError:
+ pykoa.debug( "Directory "+direc+" is not a run directory" )
+
+ runs.sort()
+
+ # if the directories list is not empty, create the name of the run directory
+ if runs:
+ last_run = runs[ len(runs)-1 ] + 1
+ run_directory = "run."+str(last_run)
+
+ work_directory = swift_dir+"/"+run_directory
+ return (work_directory, last_run)
+
+# Do not know what this syntax is but pops up in other pykoa_* files, so I kept it
+ at cli_exception_handler
+def main( argv=sys.argv[1:] ):
+ (parser, options, args) = setup_opts(argv)
+
+ # If I did not specify at least one site, print help message
+ if len(args) != 1:
+ parser.print_help()
+ return 1
+
+ # Not sure what these lines do yet, just copied from koa-ly.py
+ guc_util.init_guc_env()
+
+ conn = pykoa.connect()
+ user_row = pykoa.tools.cli_get_user( conn )
+ user_id = user_row.id
+
+ # get the swift input package
+ ( tc, sites, config, script ) = parse_stdin()
+
+ (work_directory, run_id) = create_work_directory()
+
+ # get a space seperated String of the sites to execute on
+ execution_sites=" ".join( map ( str, args ) )
+ sys.stdout.write( "\nExecuting Swift with " + execution_sites + " credentials\n" )
+ sys.stdout.write( "Execution directory: " + work_directory + "\n" )
+
+ # If I do not execute the script, just print out all important information
+ # useful to verify that the inputs to the Swift command line have been parsed correctly
+ if options.do_not_execute:
+ sys.stdout.write( "\nTC file: \n" + tc + "\n" )
+ sys.stdout.write( "SITES file: \n" + sites + "\n" )
+ sys.stdout.write( "CONFIG file: \n" + config + "\n" )
+ sys.stdout.write( "SWIFTSCRIPT file: \n" + script + "\n" )
+ sys.stdout.write( "swift -sites.file sites.xml -tc.file tc -config cf script.swift\n\n" )
+ return 0
+
+ # Set up for the acutal execution
+ try:
+ os.mkdir( work_directory )
+ pykoa.debug( "Created work_directory: " + work_directory )
+ except os.error:
+ sys.stderr.write( work_directory + " already exists, clean up " + os.getenv("HOME") + "/Swift" )
+ pykoa.debug( work_directory + " already exists" )
+ return 1
+
+ with open( work_directory+"/tc", "w" ) as tc_file:
+ tc_file.write( tc )
+
+ with open( work_directory+"/sites.xml", "w" ) as sites_file:
+ sites_file.write( sites )
+
+ with open( work_directory+"/cf", "w" ) as config_file:
+ config_file.write( config )
+
+ with open( work_directory+"/script.swift", "w" ) as script_file:
+ script_file.write( script )
+
+ ( src_cred, src_subject ) = get_creds(args, conn, user_id, options)
+
+ ret = execute_swift(user_id, options, src_cred, src_subject, work_directory, run_id)
+
+ return ret
+
+if __name__ == "__main__":
+ rc = main()
+ sys.exit(rc)
From jonmon at ci.uchicago.edu Tue Aug 2 17:02:58 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Tue, 2 Aug 2011 17:02:58 -0500 (CDT)
Subject: [Swift-commit] r4933 - SwiftApps/GOSwift/pykoa/tools
Message-ID: <20110802220258.08D9C9CCED@svn.ci.uchicago.edu>
Author: jonmon
Date: 2011-08-02 17:02:57 -0500 (Tue, 02 Aug 2011)
New Revision: 4933
Modified:
SwiftApps/GOSwift/pykoa/tools/koa_goswift_run.py
Log:
o added which Swift release was used for execution to the swift.info file.
Modified: SwiftApps/GOSwift/pykoa/tools/koa_goswift_run.py
===================================================================
--- SwiftApps/GOSwift/pykoa/tools/koa_goswift_run.py 2011-08-02 21:35:30 UTC (rev 4932)
+++ SwiftApps/GOSwift/pykoa/tools/koa_goswift_run.py 2011-08-02 22:02:57 UTC (rev 4933)
@@ -241,10 +241,13 @@
# Execute the script
cmd = ""
+ release = ""
if options.trunk:
cmd = os.path.join(os.environ["HOME"], "Swift_lib/trunk", "swift") # trunk release
+ release = "trunk"
elif options.stable:
cmd = os.path.join(os.environ["HOME"], "Swift_lib/0.92", "swift") # stable 0.92.1 release
+ release = "0.92.1"
os.chdir(work_directory)
status = -1
@@ -277,9 +280,10 @@
pid = p.pid
with open("swift.info", "w") as swift_info:
- swift_info.write("StartTime: "+str(datetime.datetime.utcnow())+"\n")
- swift_info.write("Swift pid: "+str(pid)+"\n")
- swift_info.write("Run id: "+str(run_id)+"\n")
+ swift_info.write("StartTime: "+str(datetime.datetime.utcnow())+"\n")
+ swift_info.write("Swift pid: "+str(pid)+"\n")
+ swift_info.write("Run id: "+str(run_id)+"\n")
+ swift_info.write("Swift release: "+release+"\n")
swift_info.write("\n")
swift_info.write("Command Line:\n "+execute+"\n")
@@ -320,9 +324,10 @@
pid = p.pid
with open("swift.info", "w") as swift_info:
- swift_info.write("StartTime: "+str(datetime.datetime.utcnow())+"\n")
- swift_info.write("Swift pid: "+str(pid)+"\n")
- swift_info.write("Run id: "+str(run_id)+"\n")
+ swift_info.write("StartTime: "+str(datetime.datetime.utcnow())+"\n")
+ swift_info.write("Swift pid: "+str(pid)+"\n")
+ swift_info.write("Run id: "+str(run_id)+"\n")
+ swift_info.write("Swift release: "+release+"\n")
swift_info.write("\n")
swift_info.write("Command Line:\n "+execute+"\n")
From jonmon at ci.uchicago.edu Tue Aug 2 19:17:45 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Tue, 2 Aug 2011 19:17:45 -0500 (CDT)
Subject: [Swift-commit] r4934 - SwiftApps/GOSwift/pykoa/tools
Message-ID: <20110803001745.41A4E9CCED@svn.ci.uchicago.edu>
Author: jonmon
Date: 2011-08-02 19:17:44 -0500 (Tue, 02 Aug 2011)
New Revision: 4934
Modified:
SwiftApps/GOSwift/pykoa/tools/koa_goswift_run.py
Log:
o minor change to how the script determines the public ip address for the VM.
Modified: SwiftApps/GOSwift/pykoa/tools/koa_goswift_run.py
===================================================================
--- SwiftApps/GOSwift/pykoa/tools/koa_goswift_run.py 2011-08-02 22:02:57 UTC (rev 4933)
+++ SwiftApps/GOSwift/pykoa/tools/koa_goswift_run.py 2011-08-03 00:17:44 UTC (rev 4934)
@@ -222,7 +222,7 @@
os.environ["X509_USER_PROXY"] = proxyfile
# Not sure this will work all the time, but this is what the internet said i could do.
- cmd = "curl http://169.254.169.254/2009-04-04/meta-data/public-ipv4"
+ cmd = "curl http://169.254.169.254/latest/meta-data/public-ipv4"
(status, host ) = commands.getstatusoutput( cmd )
if status != 0:
sys.stderr.write( "Could not determine hostname\n" )
From hategan at ci.uchicago.edu Wed Aug 3 00:27:09 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Wed, 3 Aug 2011 00:27:09 -0500 (CDT)
Subject: [Swift-commit] r4935 - trunk/src/org/griphyn/vdl/karajan/lib
Message-ID: <20110803052709.079A19CC9E@svn.ci.uchicago.edu>
Author: hategan
Date: 2011-08-03 00:27:08 -0500 (Wed, 03 Aug 2011)
New Revision: 4935
Modified:
trunk/src/org/griphyn/vdl/karajan/lib/InfiniteCountingWhile.java
Log:
ints are Integers now
Modified: trunk/src/org/griphyn/vdl/karajan/lib/InfiniteCountingWhile.java
===================================================================
--- trunk/src/org/griphyn/vdl/karajan/lib/InfiniteCountingWhile.java 2011-08-03 00:17:44 UTC (rev 4934)
+++ trunk/src/org/griphyn/vdl/karajan/lib/InfiniteCountingWhile.java 2011-08-03 05:27:08 UTC (rev 4935)
@@ -33,7 +33,7 @@
stack.setVar("#iteratethread", tc);
stack.setVar("#thread", tc.split(0));
stack.setVar(COUNTER_NAME, Collections.singletonList(0));
- stack.setVar((String) VAR.getStatic(this), new RootDataNode(Types.INT, 0.0));
+ stack.setVar((String) VAR.getStatic(this), new RootDataNode(Types.INT, 0));
super.pre(stack);
}
@@ -48,7 +48,7 @@
return;
}
FlowElement fn = null;
-
+
if (index == elementCount() - 1) {
// the condition is always compiled as the last thing in the loop
// but the increment needs to happen before the condition is
@@ -60,7 +60,7 @@
ThreadingContext tc = (ThreadingContext)stack.getVar("#iteratethread");
stack.setVar("#thread", tc.split(i));
stack.setVar(COUNTER_NAME, Collections.singletonList(i));
- stack.setVar((String) VAR.getStatic(this), new RootDataNode(Types.INT, Double.valueOf(i)));
+ stack.setVar((String) VAR.getStatic(this), new RootDataNode(Types.INT, i));
}
if (index >= elementCount()) {
// starting new iteration
From davidk at ci.uchicago.edu Wed Aug 3 09:22:32 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Wed, 3 Aug 2011 09:22:32 -0500 (CDT)
Subject: [Swift-commit] r4937 - wwwdev/usage
Message-ID: <20110803142232.D27869CC9E@svn.ci.uchicago.edu>
Author: davidk
Date: 2011-08-03 09:22:32 -0500 (Wed, 03 Aug 2011)
New Revision: 4937
Modified:
wwwdev/usage/update_data.php
Log:
Path corrections
Modified: wwwdev/usage/update_data.php
===================================================================
--- wwwdev/usage/update_data.php 2011-08-03 13:48:41 UTC (rev 4936)
+++ wwwdev/usage/update_data.php 2011-08-03 14:22:32 UTC (rev 4937)
@@ -158,7 +158,7 @@
# Unique users per month
-$users_per_month_file = fopen("/home/davidk/public_html/usage/users_per_month.xml", 'w');
+$users_per_month_file = fopen("users_per_month.xml", 'w');
fwrite($users_per_month_file, "\n\n\n\n\n\n\n");
foreach($months as $key => $value) {
fwrite($users_per_month_file, "$key\n");
@@ -171,7 +171,7 @@
fclose($users_per_month_file);
# New users per month
-$new_users_per_month_file = fopen("/home/davidk/public_html/usage/new_users_per_month.xml", 'w');
+$new_users_per_month_file = fopen("new_users_per_month.xml", 'w');
fwrite($new_users_per_month_file, "\n\n\n\n\n\n\n");
foreach($months as $key => $value) {
fwrite($new_users_per_month_file, "$key\n");
From jonmon at ci.uchicago.edu Wed Aug 3 15:52:04 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Wed, 3 Aug 2011 15:52:04 -0500 (CDT)
Subject: [Swift-commit] r4938 - in SwiftApps/GOSwift: bin pykoa/tools
Message-ID: <20110803205204.4327E9CD1A@svn.ci.uchicago.edu>
Author: jonmon
Date: 2011-08-03 15:52:04 -0500 (Wed, 03 Aug 2011)
New Revision: 4938
Added:
SwiftApps/GOSwift/bin/koa-goswift-manage
SwiftApps/GOSwift/pykoa/tools/koa_goswift_manage.py
Removed:
SwiftApps/GOSwift/bin/koa-goswift-kill
Log:
o commit files for the goswift_manage command. Resume is not implemented and stop is broken.
Deleted: SwiftApps/GOSwift/bin/koa-goswift-kill
===================================================================
--- SwiftApps/GOSwift/bin/koa-goswift-kill 2011-08-03 14:22:32 UTC (rev 4937)
+++ SwiftApps/GOSwift/bin/koa-goswift-kill 2011-08-03 20:52:04 UTC (rev 4938)
@@ -1,15 +0,0 @@
-#!/bin/sh
-
-if [ -z "$KOA_HOME" ]; then
- echo "Error: KOA_HOME must be set"
- exit 1
-fi
-
-if [ -s "$KOA_USER" ]; then
- echo "Error: KOA_USER must be set"
- exit 1
-fi
-
-export KOA_API_11=TRUE
-
-exec $KOA_HOME/pykoa/tools/koa_goswift_kill.py "$@"
\ No newline at end of file
Copied: SwiftApps/GOSwift/bin/koa-goswift-manage (from rev 4937, SwiftApps/GOSwift/bin/koa-goswift-kill)
===================================================================
--- SwiftApps/GOSwift/bin/koa-goswift-manage (rev 0)
+++ SwiftApps/GOSwift/bin/koa-goswift-manage 2011-08-03 20:52:04 UTC (rev 4938)
@@ -0,0 +1,15 @@
+#!/bin/sh
+
+if [ -z "$KOA_HOME" ]; then
+ echo "Error: KOA_HOME must be set"
+ exit 1
+fi
+
+if [ -s "$KOA_USER" ]; then
+ echo "Error: KOA_USER must be set"
+ exit 1
+fi
+
+export KOA_API_11=TRUE
+
+exec $KOA_HOME/pykoa/tools/koa_goswift_kill.py "$@"
\ No newline at end of file
Added: SwiftApps/GOSwift/pykoa/tools/koa_goswift_manage.py
===================================================================
--- SwiftApps/GOSwift/pykoa/tools/koa_goswift_manage.py (rev 0)
+++ SwiftApps/GOSwift/pykoa/tools/koa_goswift_manage.py 2011-08-03 20:52:04 UTC (rev 4938)
@@ -0,0 +1,64 @@
+import os
+import sys
+import commands
+
+import pykoa
+import pykoa.tools
+from pykoa.koaexception import cli_exception_handler
+
+def setup_opts(argv):
+ help_screen = """
+goswift_manage [option]
+
+Stop a currently running run or resume a stopped run.
+
+Type 'man goswift_manage' for details. \
+"""
+
+ parser = pykoa.tools.get_option_parser(help_screen)
+ parser.add_option("--stop", dest="stop",
+ default=False, action="store_true",
+ help="Stop/resume a run my the run id")
+
+ (options, args) pykoa.tools.parse_args(parser, argv)
+
+ return (parser, options, args)
+
+def stop(run_id):
+ user = os.environ["USER"]
+ (status, output) = commands.getstatusoutput("ps -u "+user+" -f")
+ if status != 0:
+ sys.stderr.write("Failed to get a list of running processes\n")
+ return 1
+
+ ppid = ??
+ processes = [ppid]
+ lines = []
+ for line in ouput.split("\n"):
+ parsed_line = line.split(" ")
+ if ppid == parsed_line[2]:
+ processes.append(parsed_line[1])
+ ppid = int(parsed_line[1])
+
+ for pid in processes:
+ os.kill(pid, 9)
+
+ return 0
+
+ at cli_exception_handler
+def main( argv=sys.argv[1:] ):
+ (parser, options, args) = setup_opts(argv)
+ ret = -1
+
+ if len(args) != 1:
+ parser.print_help()
+ return 1
+
+ if options.stop:
+ ret = stop(args[0])
+
+ return ret
+
+if __name__ == "__main__":
+ rc = main()
+ sys.exit(rc)
Property changes on: SwiftApps/GOSwift/pykoa/tools/koa_goswift_manage.py
___________________________________________________________________
Added: svn:executable
+ *
From jonmon at ci.uchicago.edu Thu Aug 4 14:15:54 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Thu, 4 Aug 2011 14:15:54 -0500 (CDT)
Subject: [Swift-commit] r4940 - SwiftApps/GOSwift/pykoa/tools
Message-ID: <20110804191554.2217E9CC9E@svn.ci.uchicago.edu>
Author: jonmon
Date: 2011-08-04 14:15:53 -0500 (Thu, 04 Aug 2011)
New Revision: 4940
Modified:
SwiftApps/GOSwift/pykoa/tools/koa_goswift_list.py
Log:
o Modifed the output from goswift_list to be less noisy.
Modified: SwiftApps/GOSwift/pykoa/tools/koa_goswift_list.py
===================================================================
--- SwiftApps/GOSwift/pykoa/tools/koa_goswift_list.py 2011-08-03 22:51:06 UTC (rev 4939)
+++ SwiftApps/GOSwift/pykoa/tools/koa_goswift_list.py 2011-08-04 19:15:53 UTC (rev 4940)
@@ -29,7 +29,7 @@
return 1
run_directory = os.environ["HOME"]+"/Swift";
- cmd = "/bin/ls -lGht "+run_directory+" | awk '{print $2, $3, $4, $5, $6, $7, $8}'"
+ cmd = "/bin/ls -lGht "+run_directory+" | awk '{print $5, $6, $7, $8}'"
ret = subprocess.call(cmd, close_fds=True,
shell=True)
From jonmon at ci.uchicago.edu Thu Aug 4 14:18:21 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Thu, 4 Aug 2011 14:18:21 -0500 (CDT)
Subject: [Swift-commit] r4941 - SwiftApps/SwiftMontage/scripts
Message-ID: <20110804191821.163E79CC9E@svn.ci.uchicago.edu>
Author: jonmon
Date: 2011-08-04 14:18:20 -0500 (Thu, 04 Aug 2011)
New Revision: 4941
Modified:
SwiftApps/SwiftMontage/scripts/SwiftMontage_Apps.swift
Log:
o Have mProject call the mProject_wrap python script
o changed the parameters in mJPEG
Modified: SwiftApps/SwiftMontage/scripts/SwiftMontage_Apps.swift
===================================================================
--- SwiftApps/SwiftMontage/scripts/SwiftMontage_Apps.swift 2011-08-04 19:15:53 UTC (rev 4940)
+++ SwiftApps/SwiftMontage/scripts/SwiftMontage_Apps.swift 2011-08-04 19:18:20 UTC (rev 4941)
@@ -54,7 +54,7 @@
app ( JPEG mos_img_jpg ) mJPEG( Image mos_img )
{
- mJPEG "-gray" @mos_img "20%" "99.98%" "loglog" "-out" @mos_img_jpg;
+ mJPEG "-gray" @mos_img "0s" "max" "gaussian-log" "-out" @mos_img_jpg;
}
@@ -73,7 +73,8 @@
app ( Image proj_img ) mProject( Image raw_img, Header hdr )
{
- mProject "-X" @raw_img @proj_img @hdr;
+ mProject_wrap "-X" @raw_img @proj_img @hdr;
+// mProject "-X" @raw_img @proj_img @hdr;
}
app ( Image shrunk ) mShrink( Image original, int factor )
From jonmon at ci.uchicago.edu Thu Aug 4 14:19:33 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Thu, 4 Aug 2011 14:19:33 -0500 (CDT)
Subject: [Swift-commit] r4942 - SwiftApps/SwiftMontage/etc
Message-ID: <20110804191933.8BE429CC9E@svn.ci.uchicago.edu>
Author: jonmon
Date: 2011-08-04 14:19:33 -0500 (Thu, 04 Aug 2011)
New Revision: 4942
Modified:
SwiftApps/SwiftMontage/etc/swift_montage.properties
Log:
o set retries to from 0 to 3.
o turned off provider staging.
Modified: SwiftApps/SwiftMontage/etc/swift_montage.properties
===================================================================
--- SwiftApps/SwiftMontage/etc/swift_montage.properties 2011-08-04 19:18:20 UTC (rev 4941)
+++ SwiftApps/SwiftMontage/etc/swift_montage.properties 2011-08-04 19:19:33 UTC (rev 4942)
@@ -1,8 +1,8 @@
-execution.retries=0
+execution.retries=3
sitedir.keep=true
status.mode=provider
wrapper.log.always.transfer=true
foreach.maxthreads=1024
wrapper.parameter.mode=files
-use.provider.staging=true
+use.provider.staging=false
provider.staging.pin.swiftfiles=false
From jonmon at ci.uchicago.edu Thu Aug 4 14:30:50 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Thu, 4 Aug 2011 14:30:50 -0500 (CDT)
Subject: [Swift-commit] r4943 - trunk/bin
Message-ID: <20110804193050.D7D239CC9E@svn.ci.uchicago.edu>
Author: jonmon
Date: 2011-08-04 14:30:50 -0500 (Thu, 04 Aug 2011)
New Revision: 4943
Modified:
trunk/bin/swift
Log:
o run the chxml command using the SWIFT_HOME environment variable.
Modified: trunk/bin/swift
===================================================================
--- trunk/bin/swift 2011-08-04 19:19:33 UTC (rev 4942)
+++ trunk/bin/swift 2011-08-04 19:30:50 UTC (rev 4943)
@@ -63,7 +63,7 @@
# make sure sites.xml file is well-formed
-chxml $CMDLINE
+$SWIFT_HOME/bin/chxml $CMDLINE
### SETUP OTHER ENV VARIABLES ####
From jonmon at ci.uchicago.edu Thu Aug 4 15:02:30 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Thu, 4 Aug 2011 15:02:30 -0500 (CDT)
Subject: [Swift-commit] r4944 - SwiftApps/GOSwift/man
Message-ID: <20110804200230.E0F749CC9E@svn.ci.uchicago.edu>
Author: jonmon
Date: 2011-08-04 15:02:30 -0500 (Thu, 04 Aug 2011)
New Revision: 4944
Added:
SwiftApps/GOSwift/man/goswift_run.1
Removed:
SwiftApps/GOSwift/man/goswift.1
Log:
o renamed the man page from goswift to goswift_run
Deleted: SwiftApps/GOSwift/man/goswift.1
===================================================================
--- SwiftApps/GOSwift/man/goswift.1 2011-08-04 19:30:50 UTC (rev 4943)
+++ SwiftApps/GOSwift/man/goswift.1 2011-08-04 20:02:30 UTC (rev 4944)
@@ -1,38 +0,0 @@
-.TH GOSWIFT 1 development "dev" "Development"
-.SH NAME
-goswift \- Execute Swift underneath the Globus Online infastructure
-.SH SYNOPSIS
-.B goswift
-.B [ --dry-run ]
-.BI
-.I < goswift.package
-
-.SH DESCRIPTION
-The
-.B
-goswift
-command executes the Swift parallel scripting language underneath Globus Online on user specified execute points.
-The execute points must be a logical endpoint.
-
-.SH OPTIONS
-.TP
-.B --dry-run
-This will go through the process of parsing the data from given to the
-.BI
-goswift
-command and output all the pertatent information.
-
-.TP
-.B -d
-Set the deadline of the task. If the task has not completed by the deadline it will beautomatically canceled. A suffix of 'm', 'h', or 'd' may be specified to indicate minutes, hours, or days. If a unit suffix is not given it is assumed to be minutes. Example: '30m', '4h', '1d'. The default is '30m' (30 minutes).
-
-
-.SH EXAMPLES
-.TP
-.B goswift localhost ci#pads ci#beagle
-Note: that localhost is automatically defined and it is the VM that Globus Online is executing on.
-
-.SH EXIT STATUS
-.B
-goswift
-can return several different error codes.
Copied: SwiftApps/GOSwift/man/goswift_run.1 (from rev 4924, SwiftApps/GOSwift/man/goswift.1)
===================================================================
--- SwiftApps/GOSwift/man/goswift_run.1 (rev 0)
+++ SwiftApps/GOSwift/man/goswift_run.1 2011-08-04 20:02:30 UTC (rev 4944)
@@ -0,0 +1,38 @@
+.TH GOSWIFT 1 development "dev" "Development"
+.SH NAME
+goswift \- Execute Swift underneath the Globus Online infastructure
+.SH SYNOPSIS
+.B goswift
+.B [ --dry-run ]
+.BI
+.I < goswift.package
+
+.SH DESCRIPTION
+The
+.B
+goswift
+command executes the Swift parallel scripting language underneath Globus Online on user specified execute points.
+The execute points must be a logical endpoint.
+
+.SH OPTIONS
+.TP
+.B --dry-run
+This will go through the process of parsing the data from given to the
+.BI
+goswift
+command and output all the pertatent information.
+
+.TP
+.B -d
+Set the deadline of the task. If the task has not completed by the deadline it will beautomatically canceled. A suffix of 'm', 'h', or 'd' may be specified to indicate minutes, hours, or days. If a unit suffix is not given it is assumed to be minutes. Example: '30m', '4h', '1d'. The default is '30m' (30 minutes).
+
+
+.SH EXAMPLES
+.TP
+.B goswift localhost ci#pads ci#beagle
+Note: that localhost is automatically defined and it is the VM that Globus Online is executing on.
+
+.SH EXIT STATUS
+.B
+goswift
+can return several different error codes.
From jonmon at ci.uchicago.edu Thu Aug 4 16:30:54 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Thu, 4 Aug 2011 16:30:54 -0500 (CDT)
Subject: [Swift-commit] r4945 - SwiftApps/GOSwift/pykoa/tools
Message-ID: <20110804213054.8E0DE9CC9E@svn.ci.uchicago.edu>
Author: jonmon
Date: 2011-08-04 16:30:54 -0500 (Thu, 04 Aug 2011)
New Revision: 4945
Modified:
SwiftApps/GOSwift/pykoa/tools/koa_goswift_manage.py
Log:
o Changed the 'ps -u $USER -f' to 'ps -u $USER -H -f'
o Added get the Swift pid from the swift.info file
o Getting list of processes to kill is broken. Don't parse the lines from the 'ps' output correctly.
Modified: SwiftApps/GOSwift/pykoa/tools/koa_goswift_manage.py
===================================================================
--- SwiftApps/GOSwift/pykoa/tools/koa_goswift_manage.py 2011-08-04 20:02:30 UTC (rev 4944)
+++ SwiftApps/GOSwift/pykoa/tools/koa_goswift_manage.py 2011-08-04 21:30:54 UTC (rev 4945)
@@ -20,22 +20,34 @@
default=False, action="store_true",
help="Stop/resume a run my the run id")
- (options, args) pykoa.tools.parse_args(parser, argv)
+ (options, args) = pykoa.tools.parse_args(parser, argv)
return (parser, options, args)
def stop(run_id):
user = os.environ["USER"]
- (status, output) = commands.getstatusoutput("ps -u "+user+" -f")
+ (status, output) = commands.getstatusoutput("ps -u "+user+" -H -f")
if status != 0:
sys.stderr.write("Failed to get a list of running processes\n")
return 1
- ppid = ??
+ ppid = -1
+
+ with open(os.environ["HOME"]+"/Swift/run."+run_id+"/swift.info") as swift_info:
+ for line in swift_info.readlines():
+ line_parsed = line.split(":")
+ if line_parsed[0] == "Swift pid":
+ ppid = int(line_parsed[1].strip())
+ break
+
+ if ppid == -1:
+ sys.stderr.write("Could not find the Swift pid from the swift.info file\n")
+ return 1
+
processes = [ppid]
- lines = []
- for line in ouput.split("\n"):
- parsed_line = line.split(" ")
+ # BROKEN: Doesn't parse the 'ps' output correctly. Fix the 'parsed_line' split delimiter
+ for line in output.split("\n"):
+ parsed_line = line.split()
if ppid == parsed_line[2]:
processes.append(parsed_line[1])
ppid = int(parsed_line[1])
From hategan at ci.uchicago.edu Thu Aug 4 19:18:28 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Thu, 4 Aug 2011 19:18:28 -0500 (CDT)
Subject: [Swift-commit] r4946 -
trunk/src/org/griphyn/vdl/karajan/lib/swiftscript
Message-ID: <20110805001828.DDB3F9CC9E@svn.ci.uchicago.edu>
Author: hategan
Date: 2011-08-04 19:18:28 -0500 (Thu, 04 Aug 2011)
New Revision: 4946
Modified:
trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/ExtractInt.java
Log:
extractint should return an int not a float
Modified: trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/ExtractInt.java
===================================================================
--- trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/ExtractInt.java 2011-08-04 21:30:54 UTC (rev 4945)
+++ trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/ExtractInt.java 2011-08-05 00:18:28 UTC (rev 4946)
@@ -31,8 +31,7 @@
BufferedReader breader = new BufferedReader(freader);
String str = breader.readLine();
freader.close();
- Double i = new Double(str);
- DSHandle result = new RootDataNode(Types.FLOAT, i);
+ DSHandle result = new RootDataNode(Types.INT, Integer.parseInt(str));
int provid = VDLFunction.nextProvenanceID();
VDLFunction.logProvenanceResult(provid, result, "extractint");
VDLFunction.logProvenanceParameter(provid, handle, "filename");
From hategan at ci.uchicago.edu Fri Aug 5 00:14:57 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Fri, 5 Aug 2011 00:14:57 -0500 (CDT)
Subject: [Swift-commit] r4947 - in branches/release-0.93/tests: . bugs
groups language/working language-behaviour/mappers
providers/local-pbs/fusion providers/local-pbs-coasters
providers/local-pbs-coasters/fusion
providers/local-pbs-coasters/pads
Message-ID: <20110805051457.716CE9CCEC@svn.ci.uchicago.edu>
Author: hategan
Date: 2011-08-05 00:14:57 -0500 (Fri, 05 Aug 2011)
New Revision: 4947
Added:
branches/release-0.93/tests/bugs/
branches/release-0.93/tests/bugs/481-array-hang.swift
branches/release-0.93/tests/language/working/015-mapper-literal.in
branches/release-0.93/tests/providers/local-pbs-coasters/fusion/001-catsn.check.sh
branches/release-0.93/tests/providers/local-pbs-coasters/fusion/001-catsn.setup.sh
branches/release-0.93/tests/providers/local-pbs-coasters/fusion/001-catsn.swift
branches/release-0.93/tests/providers/local-pbs-coasters/fusion/001-catsn.timeout
branches/release-0.93/tests/providers/local-pbs-coasters/fusion/data.txt
branches/release-0.93/tests/providers/local-pbs-coasters/fusion/sites.template.xml
branches/release-0.93/tests/providers/local-pbs-coasters/fusion/tc.template.data
branches/release-0.93/tests/providers/local-pbs-coasters/pads/
branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.check.sh
branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.setup.sh
branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.swift
branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.timeout
branches/release-0.93/tests/providers/local-pbs-coasters/pads/data.txt
branches/release-0.93/tests/providers/local-pbs-coasters/pads/sites.template.xml
branches/release-0.93/tests/providers/local-pbs-coasters/pads/tc.template.data
Removed:
branches/release-0.93/tests/bugs/481-array-hang.swift
branches/release-0.93/tests/providers/local-pbs-coasters/001-catsn-pads-local-pbs-coasters.check.sh
branches/release-0.93/tests/providers/local-pbs-coasters/001-catsn-pads-local-pbs-coasters.setup.sh
branches/release-0.93/tests/providers/local-pbs-coasters/001-catsn-pads-local-pbs-coasters.swift
branches/release-0.93/tests/providers/local-pbs-coasters/001-catsn-pads-local-pbs-coasters.timeout
branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0001.out.expected
branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0002.out.expected
branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0003.out.expected
branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0004.out.expected
branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0005.out.expected
branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0006.out.expected
branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0007.out.expected
branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0008.out.expected
branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0009.out.expected
branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0010.out.expected
branches/release-0.93/tests/providers/local-pbs-coasters/data.txt
branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.check.sh
branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.setup.sh
branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.swift
branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.timeout
branches/release-0.93/tests/providers/local-pbs-coasters/pads/data.txt
branches/release-0.93/tests/providers/local-pbs-coasters/pads/sites.template.xml
branches/release-0.93/tests/providers/local-pbs-coasters/pads/tc.template.data
branches/release-0.93/tests/providers/local-pbs-coasters/sites.template.xml
branches/release-0.93/tests/providers/local-pbs-coasters/tc.template.data
Modified:
branches/release-0.93/tests/
branches/release-0.93/tests/groups/group-all-local.sh
branches/release-0.93/tests/language-behaviour/mappers/07511-fixed-array-mapper-input.check.sh
branches/release-0.93/tests/language/working/015-mapper-literal.swift
branches/release-0.93/tests/providers/local-pbs/fusion/001-catsn.check.sh
branches/release-0.93/tests/providers/local-pbs/fusion/001-catsn.setup.sh
branches/release-0.93/tests/suite.sh
Log:
backported test fixes from trunk
Property changes on: branches/release-0.93/tests
___________________________________________________________________
Added: svn:mergeinfo
+ /trunk/tests:4762-4946
Deleted: branches/release-0.93/tests/bugs/481-array-hang.swift
===================================================================
--- trunk/tests/bugs/481-array-hang.swift 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/bugs/481-array-hang.swift 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1,4 +0,0 @@
-int[] a = [1, 2, 3];
-
-// hangs waiting for a[4]
-trace(a[4]);
\ No newline at end of file
Copied: branches/release-0.93/tests/bugs/481-array-hang.swift (from rev 4946, trunk/tests/bugs/481-array-hang.swift)
===================================================================
--- branches/release-0.93/tests/bugs/481-array-hang.swift (rev 0)
+++ branches/release-0.93/tests/bugs/481-array-hang.swift 2011-08-05 05:14:57 UTC (rev 4947)
@@ -0,0 +1,4 @@
+int[] a = [1, 2, 3];
+
+// hangs waiting for a[4]
+trace(a[4]);
\ No newline at end of file
Modified: branches/release-0.93/tests/groups/group-all-local.sh
===================================================================
--- branches/release-0.93/tests/groups/group-all-local.sh 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/groups/group-all-local.sh 2011-08-05 05:14:57 UTC (rev 4947)
@@ -20,6 +20,7 @@
$TESTDIR/language-behaviour/procedures \
$TESTDIR/language-behaviour/strings \
$TESTDIR/language-behaviour/variables \
+ $TESTDIR/bugs \
$TESTDIR/documentation/tutorial \
# $TESTDIR/cdm/ps/pinned
# $TESTDIR/site/intrepid
Copied: branches/release-0.93/tests/language/working/015-mapper-literal.in (from rev 4946, trunk/tests/language/working/015-mapper-literal.in)
===================================================================
Modified: branches/release-0.93/tests/language/working/015-mapper-literal.swift
===================================================================
--- branches/release-0.93/tests/language/working/015-mapper-literal.swift 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/language/working/015-mapper-literal.swift 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1,2 +1,2 @@
type myfile;
-myfile v<"literalfilename.txt">;
+myfile v<"015-mapper-literal.in">;
Modified: branches/release-0.93/tests/language-behaviour/mappers/07511-fixed-array-mapper-input.check.sh
===================================================================
--- branches/release-0.93/tests/language-behaviour/mappers/07511-fixed-array-mapper-input.check.sh 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/language-behaviour/mappers/07511-fixed-array-mapper-input.check.sh 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1,6 +1,6 @@
#!/bin/bash
set -x
-cat $OUTPUT | grep "file: 07511-fixed-array-mapper-input.first.in" || exit 1
-cat $OUTPUT | grep "file: 07511-fixed-array-mapper-input.second.in" || exit 1
-cat $OUTPUT | grep "file: 07511-fixed-array-mapper-input.third.in" || exit 1
+cat $TEST_LOG | grep "file: 07511-fixed-array-mapper-input.first.in" || exit 1
+cat $TEST_LOG | grep "file: 07511-fixed-array-mapper-input.second.in" || exit 1
+cat $TEST_LOG | grep "file: 07511-fixed-array-mapper-input.third.in" || exit 1
exit 0
Modified: branches/release-0.93/tests/providers/local-pbs/fusion/001-catsn.check.sh
===================================================================
--- branches/release-0.93/tests/providers/local-pbs/fusion/001-catsn.check.sh 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs/fusion/001-catsn.check.sh 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1,9 +1,9 @@
#!/bin/bash
-# Pseudo-code for now
+set -x
-# for f in *.out.expected
-# do
-# assert *.out exists
-# assert *.out.expected == *.out
-# done
+LINES=$( ls catsn*.out | wc -l )
+[[ ${?} == 0 ]] || exit 1
+[[ ${LINES} == 10 ]] || exit 1
+
+exit 0
Modified: branches/release-0.93/tests/providers/local-pbs/fusion/001-catsn.setup.sh
===================================================================
--- branches/release-0.93/tests/providers/local-pbs/fusion/001-catsn.setup.sh 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs/fusion/001-catsn.setup.sh 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1,4 +1,4 @@
#!/bin/bash
cp -v $GROUP/data.txt . || exit 1
-cp -v $GROUP/*expected . || exit 1
+
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/001-catsn-pads-local-pbs-coasters.check.sh
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/001-catsn-pads-local-pbs-coasters.check.sh 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/001-catsn-pads-local-pbs-coasters.check.sh 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1,12 +0,0 @@
-#!/bin/bash
-
-set -x
-
-for count in `seq --format "%04.f" 1 1 10`
-do
- [ -f catsn.$count.out ] || exit 1
- CONTENTS1=$( cat catsn.$count.out.expected )
- CONTENTS2=$( cat catsn.$count.out )
- [[ $CONTENTS1 == $CONTENTS2 ]] || exit 1
-done
-exit 0
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/001-catsn-pads-local-pbs-coasters.setup.sh
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/001-catsn-pads-local-pbs-coasters.setup.sh 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/001-catsn-pads-local-pbs-coasters.setup.sh 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1,4 +0,0 @@
-#!/bin/bash
-
-cp -v $GROUP/data.txt . || exit 1
-cp -v $GROUP/*expected . || exit 1
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/001-catsn-pads-local-pbs-coasters.swift
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/001-catsn-pads-local-pbs-coasters.swift 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/001-catsn-pads-local-pbs-coasters.swift 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1,15 +0,0 @@
-type file;
-
-app (file o) cat (file i)
-{
- cat @i stdout=@o;
-}
-
-string t = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
-string char[] = @strsplit(t, "");
-
-file out[];
-foreach j in [1:@toint(@arg("n","10"))] {
- file data<"data.txt">;
- out[j] = cat(data);
-}
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/001-catsn-pads-local-pbs-coasters.timeout
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/001-catsn-pads-local-pbs-coasters.timeout 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/001-catsn-pads-local-pbs-coasters.timeout 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1,2 +0,0 @@
-7200
-
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0001.out.expected
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0001.out.expected 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0001.out.expected 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1 +0,0 @@
-Hello world
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0002.out.expected
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0002.out.expected 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0002.out.expected 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1 +0,0 @@
-Hello world
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0003.out.expected
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0003.out.expected 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0003.out.expected 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1 +0,0 @@
-Hello world
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0004.out.expected
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0004.out.expected 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0004.out.expected 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1 +0,0 @@
-Hello world
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0005.out.expected
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0005.out.expected 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0005.out.expected 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1 +0,0 @@
-Hello world
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0006.out.expected
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0006.out.expected 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0006.out.expected 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1 +0,0 @@
-Hello world
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0007.out.expected
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0007.out.expected 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0007.out.expected 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1 +0,0 @@
-Hello world
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0008.out.expected
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0008.out.expected 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0008.out.expected 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1 +0,0 @@
-Hello world
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0009.out.expected
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0009.out.expected 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0009.out.expected 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1 +0,0 @@
-Hello world
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0010.out.expected
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0010.out.expected 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/catsn.0010.out.expected 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1 +0,0 @@
-Hello world
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/data.txt
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/data.txt 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/data.txt 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1 +0,0 @@
-Hello world
Copied: branches/release-0.93/tests/providers/local-pbs-coasters/fusion/001-catsn.check.sh (from rev 4946, trunk/tests/providers/local-pbs-coasters/fusion/001-catsn.check.sh)
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/fusion/001-catsn.check.sh (rev 0)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/fusion/001-catsn.check.sh 2011-08-05 05:14:57 UTC (rev 4947)
@@ -0,0 +1,12 @@
+#!/bin/bash
+
+set -x
+
+for count in `seq --format "%04.f" 1 1 10`
+do
+ [ -f catsn.$count.out ] || exit 1
+ CONTENTS1=$( cat catsn.$count.out.expected )
+ CONTENTS2=$( cat catsn.$count.out )
+ [[ $CONTENTS1 == $CONTENTS2 ]] || exit 1
+done
+exit 0
Copied: branches/release-0.93/tests/providers/local-pbs-coasters/fusion/001-catsn.setup.sh (from rev 4946, trunk/tests/providers/local-pbs-coasters/fusion/001-catsn.setup.sh)
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/fusion/001-catsn.setup.sh (rev 0)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/fusion/001-catsn.setup.sh 2011-08-05 05:14:57 UTC (rev 4947)
@@ -0,0 +1,4 @@
+#!/bin/bash
+
+cp -v $GROUP/data.txt . || exit 1
+cp -v $GROUP/*expected . || exit 1
Copied: branches/release-0.93/tests/providers/local-pbs-coasters/fusion/001-catsn.swift (from rev 4946, trunk/tests/providers/local-pbs-coasters/fusion/001-catsn.swift)
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/fusion/001-catsn.swift (rev 0)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/fusion/001-catsn.swift 2011-08-05 05:14:57 UTC (rev 4947)
@@ -0,0 +1,15 @@
+type file;
+
+app (file o) cat (file i)
+{
+ cat @i stdout=@o;
+}
+
+string t = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
+string char[] = @strsplit(t, "");
+
+file out[];
+foreach j in [1:@toint(@arg("n","10"))] {
+ file data<"data.txt">;
+ out[j] = cat(data);
+}
Copied: branches/release-0.93/tests/providers/local-pbs-coasters/fusion/001-catsn.timeout (from rev 4946, trunk/tests/providers/local-pbs-coasters/fusion/001-catsn.timeout)
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/fusion/001-catsn.timeout (rev 0)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/fusion/001-catsn.timeout 2011-08-05 05:14:57 UTC (rev 4947)
@@ -0,0 +1,2 @@
+7200
+
Copied: branches/release-0.93/tests/providers/local-pbs-coasters/fusion/data.txt (from rev 4946, trunk/tests/providers/local-pbs-coasters/fusion/data.txt)
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/fusion/data.txt (rev 0)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/fusion/data.txt 2011-08-05 05:14:57 UTC (rev 4947)
@@ -0,0 +1 @@
+Hello world
Copied: branches/release-0.93/tests/providers/local-pbs-coasters/fusion/sites.template.xml (from rev 4946, trunk/tests/providers/local-pbs-coasters/fusion/sites.template.xml)
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/fusion/sites.template.xml (rev 0)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/fusion/sites.template.xml 2011-08-05 05:14:57 UTC (rev 4947)
@@ -0,0 +1,17 @@
+
+
+
+
+ 1
+ 7200
+ 1
+ 1
+ 1
+ 1
+ _QUEUE_
+ 5.99
+ 10000
+ _PROJECT_
+ _WORK_
+
+
Copied: branches/release-0.93/tests/providers/local-pbs-coasters/fusion/tc.template.data (from rev 4946, trunk/tests/providers/local-pbs-coasters/fusion/tc.template.data)
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/fusion/tc.template.data (rev 0)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/fusion/tc.template.data 2011-08-05 05:14:57 UTC (rev 4947)
@@ -0,0 +1,8 @@
+local-pbs-coasters echo /bin/echo INSTALLED INTEL32::LINUX null
+local-pbs-coasters cat /bin/cat INSTALLED INTEL32::LINUX null
+local-pbs-coasters ls /bin/ls INSTALLED INTEL32::LINUX null
+local-pbs-coasters grep /bin/grep INSTALLED INTEL32::LINUX null
+local-pbs-coasters sort /bin/sort INSTALLED INTEL32::LINUX null
+local-pbs-coasters paste /bin/paste INSTALLED INTEL32::LINUX null
+local-pbs-coasters wc /usr/bin/wc INSTALLED INTEL32::LINUX null
+
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.check.sh
===================================================================
--- trunk/tests/providers/local-pbs-coasters/pads/001-catsn.check.sh 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.check.sh 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1,12 +0,0 @@
-#!/bin/bash
-
-set -x
-
-for count in `seq --format "%04.f" 1 1 10`
-do
- [ -f catsn.$count.out ] || exit 1
- CONTENTS1=$( cat catsn.$count.out.expected )
- CONTENTS2=$( cat catsn.$count.out )
- [[ $CONTENTS1 == $CONTENTS2 ]] || exit 1
-done
-exit 0
Copied: branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.check.sh (from rev 4946, trunk/tests/providers/local-pbs-coasters/pads/001-catsn.check.sh)
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.check.sh (rev 0)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.check.sh 2011-08-05 05:14:57 UTC (rev 4947)
@@ -0,0 +1,12 @@
+#!/bin/bash
+
+set -x
+
+for count in `seq --format "%04.f" 1 1 10`
+do
+ [ -f catsn.$count.out ] || exit 1
+ CONTENTS1=$( cat catsn.$count.out.expected )
+ CONTENTS2=$( cat catsn.$count.out )
+ [[ $CONTENTS1 == $CONTENTS2 ]] || exit 1
+done
+exit 0
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.setup.sh
===================================================================
--- trunk/tests/providers/local-pbs-coasters/pads/001-catsn.setup.sh 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.setup.sh 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1,4 +0,0 @@
-#!/bin/bash
-
-cp -v $GROUP/data.txt . || exit 1
-cp -v $GROUP/*expected . || exit 1
Copied: branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.setup.sh (from rev 4946, trunk/tests/providers/local-pbs-coasters/pads/001-catsn.setup.sh)
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.setup.sh (rev 0)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.setup.sh 2011-08-05 05:14:57 UTC (rev 4947)
@@ -0,0 +1,4 @@
+#!/bin/bash
+
+cp -v $GROUP/data.txt . || exit 1
+cp -v $GROUP/*expected . || exit 1
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.swift
===================================================================
--- trunk/tests/providers/local-pbs-coasters/pads/001-catsn.swift 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.swift 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1,15 +0,0 @@
-type file;
-
-app (file o) cat (file i)
-{
- cat @i stdout=@o;
-}
-
-string t = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
-string char[] = @strsplit(t, "");
-
-file out[];
-foreach j in [1:@toint(@arg("n","10"))] {
- file data<"data.txt">;
- out[j] = cat(data);
-}
Copied: branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.swift (from rev 4946, trunk/tests/providers/local-pbs-coasters/pads/001-catsn.swift)
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.swift (rev 0)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.swift 2011-08-05 05:14:57 UTC (rev 4947)
@@ -0,0 +1,15 @@
+type file;
+
+app (file o) cat (file i)
+{
+ cat @i stdout=@o;
+}
+
+string t = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
+string char[] = @strsplit(t, "");
+
+file out[];
+foreach j in [1:@toint(@arg("n","10"))] {
+ file data<"data.txt">;
+ out[j] = cat(data);
+}
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.timeout
===================================================================
--- trunk/tests/providers/local-pbs-coasters/pads/001-catsn.timeout 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.timeout 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1,2 +0,0 @@
-7200
-
Copied: branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.timeout (from rev 4946, trunk/tests/providers/local-pbs-coasters/pads/001-catsn.timeout)
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.timeout (rev 0)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/pads/001-catsn.timeout 2011-08-05 05:14:57 UTC (rev 4947)
@@ -0,0 +1,2 @@
+7200
+
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/pads/data.txt
===================================================================
--- trunk/tests/providers/local-pbs-coasters/pads/data.txt 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/pads/data.txt 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1 +0,0 @@
-Hello world
Copied: branches/release-0.93/tests/providers/local-pbs-coasters/pads/data.txt (from rev 4946, trunk/tests/providers/local-pbs-coasters/pads/data.txt)
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/pads/data.txt (rev 0)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/pads/data.txt 2011-08-05 05:14:57 UTC (rev 4947)
@@ -0,0 +1 @@
+Hello world
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/pads/sites.template.xml
===================================================================
--- trunk/tests/providers/local-pbs-coasters/pads/sites.template.xml 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/pads/sites.template.xml 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1,17 +0,0 @@
-
-
-
-
- 1
- 7200
- 1
- 1
- 1
- 1
- _QUEUE_
- 5.99
- 10000
- _PROJECT_
- _WORK_
-
-
Copied: branches/release-0.93/tests/providers/local-pbs-coasters/pads/sites.template.xml (from rev 4946, trunk/tests/providers/local-pbs-coasters/pads/sites.template.xml)
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/pads/sites.template.xml (rev 0)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/pads/sites.template.xml 2011-08-05 05:14:57 UTC (rev 4947)
@@ -0,0 +1,17 @@
+
+
+
+
+ 1
+ 7200
+ 1
+ 1
+ 1
+ 1
+ _QUEUE_
+ 5.99
+ 10000
+ _PROJECT_
+ _WORK_
+
+
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/pads/tc.template.data
===================================================================
--- trunk/tests/providers/local-pbs-coasters/pads/tc.template.data 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/pads/tc.template.data 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1,8 +0,0 @@
-local-pbs-coasters echo /bin/echo INSTALLED INTEL32::LINUX null
-local-pbs-coasters cat /bin/cat INSTALLED INTEL32::LINUX null
-local-pbs-coasters ls /bin/ls INSTALLED INTEL32::LINUX null
-local-pbs-coasters grep /bin/grep INSTALLED INTEL32::LINUX null
-local-pbs-coasters sort /bin/sort INSTALLED INTEL32::LINUX null
-local-pbs-coasters paste /bin/paste INSTALLED INTEL32::LINUX null
-local-pbs-coasters wc /usr/bin/wc INSTALLED INTEL32::LINUX null
-
Copied: branches/release-0.93/tests/providers/local-pbs-coasters/pads/tc.template.data (from rev 4946, trunk/tests/providers/local-pbs-coasters/pads/tc.template.data)
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/pads/tc.template.data (rev 0)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/pads/tc.template.data 2011-08-05 05:14:57 UTC (rev 4947)
@@ -0,0 +1,8 @@
+local-pbs-coasters echo /bin/echo INSTALLED INTEL32::LINUX null
+local-pbs-coasters cat /bin/cat INSTALLED INTEL32::LINUX null
+local-pbs-coasters ls /bin/ls INSTALLED INTEL32::LINUX null
+local-pbs-coasters grep /bin/grep INSTALLED INTEL32::LINUX null
+local-pbs-coasters sort /bin/sort INSTALLED INTEL32::LINUX null
+local-pbs-coasters paste /bin/paste INSTALLED INTEL32::LINUX null
+local-pbs-coasters wc /usr/bin/wc INSTALLED INTEL32::LINUX null
+
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/sites.template.xml
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/sites.template.xml 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/sites.template.xml 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1,17 +0,0 @@
-
-
-
-
- 1
- 7200
- 1
- 1
- 1
- 1
- _QUEUE_
- 5.99
- 10000
- _PROJECT_
- _WORK_
-
-
Deleted: branches/release-0.93/tests/providers/local-pbs-coasters/tc.template.data
===================================================================
--- branches/release-0.93/tests/providers/local-pbs-coasters/tc.template.data 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/providers/local-pbs-coasters/tc.template.data 2011-08-05 05:14:57 UTC (rev 4947)
@@ -1,8 +0,0 @@
-local-pbs-coasters echo /bin/echo INSTALLED INTEL32::LINUX null
-local-pbs-coasters cat /bin/cat INSTALLED INTEL32::LINUX null
-local-pbs-coasters ls /bin/ls INSTALLED INTEL32::LINUX null
-local-pbs-coasters grep /bin/grep INSTALLED INTEL32::LINUX null
-local-pbs-coasters sort /bin/sort INSTALLED INTEL32::LINUX null
-local-pbs-coasters paste /bin/paste INSTALLED INTEL32::LINUX null
-local-pbs-coasters wc /usr/bin/wc INSTALLED INTEL32::LINUX null
-
Modified: branches/release-0.93/tests/suite.sh
===================================================================
--- branches/release-0.93/tests/suite.sh 2011-08-05 00:18:28 UTC (rev 4946)
+++ branches/release-0.93/tests/suite.sh 2011-08-05 05:14:57 UTC (rev 4947)
@@ -97,7 +97,7 @@
HTML_COMMENTS=1
fi
-if [ $COLORIZE ]; then
+if (( $COLORIZE )); then
LGREEN="\033[1;32m"
YELLOW="\033[1;33m"
RED="\033[1;31m"
@@ -411,7 +411,7 @@
echo "| $G"
echo "|"
echo -e "\\----------------------------------------------------${GRAY}"
- echo
+ echo
if [ $TEXTREPORT == 1 ]; then
stars
printf "$G\n">>$REPORT
@@ -426,7 +426,7 @@
html "$G"
html_~th
html_~tr
-
+
html_tr group
html_td
html "#"
@@ -684,7 +684,7 @@
rm killed_test > /dev/null 2>&1 && sleep 5
verbose "killing monitor: $MONITOR_PID..."
kill $MONITOR_PID
-
+
INDIVIDUAL_TEST_TIME=$(( STOP-START ))
TOTAL_TIME=$(( INDIVIDUAL_TEST_TIME+TOTAL_TIME ))
echo -e "${YELLOW}TOOK (seconds): $INDIVIDUAL_TEST_TIME${GRAY}"
@@ -717,6 +717,7 @@
process_exec $SCRIPT
RESULT=$( result )
+ test_log
output_report test "$SYMBOL" "$LASTCMD" $RESULT
check_bailout
@@ -725,9 +726,9 @@
stage_files() {
GROUP=$1
NAME=$2
-
+
RESULT="None"
-
+
if [ -f $GROUP/$NAME.in ]; then
echo "Copying input: $NAME.in"
cp -v $GROUP/$NAME.in . 2>&1 >> $OUTPUT
@@ -749,18 +750,18 @@
RESULT="Passed"
fi
done
-
+
output_report test "s" "setup" $RESULT
-
+
check_bailout
}
check_outputs() {
GROUP=$1
NAME=$2
-
+
RESULT="None"
-
+
for EXPECTED in $GROUP/$NAME.*.expected; do
BNE=`basename $EXPECTED .expected`
echo -n "Checking output: $BNE "
@@ -775,15 +776,15 @@
RESULT="Passed"
fi
done
-
+
if [ "$RESULT" == "None" ]; then
html_td width 25
html " "
html_~td
fi
-
+
output_report test "√" "check" $RESULT
-
+
check_bailout
}
@@ -791,7 +792,7 @@
swift_test_case() {
SWIFTSCRIPT=$1
NAME=${SWIFTSCRIPT%.swift}
-
+
SETUPSCRIPT=$NAME.setup.sh
CHECKSCRIPT=$NAME.check.sh
CLEANSCRIPT=$NAME.clean.sh
@@ -804,12 +805,12 @@
else
stage_files $GROUP $NAME
fi
-
+
ARGS=""
if [ -f $GROUP/$ARGSFILE ]; then
ARGS=`cat $GROUP/$ARGSFILE`
fi
-
+
CDM=
[ -r fs.data ] && CDM="-cdm.file fs.data"
@@ -848,9 +849,9 @@
script_test_case() {
SHELLSCRIPT=$1
NAME=${SWIFTSCRIPT%.swift}
-
+
stage_files $GROUP $NAME
-
+
SETUPSCRIPT=$NAME.setup.sh
CHECKSCRIPT=$NAME.check.sh
CLEANSCRIPT=$NAME.clean.sh
From jonmon at ci.uchicago.edu Fri Aug 5 10:12:04 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Fri, 5 Aug 2011 10:12:04 -0500 (CDT)
Subject: [Swift-commit] r4948 - SwiftApps/GOSwift/pykoa/tools
Message-ID: <20110805151204.D30FF9CC9E@svn.ci.uchicago.edu>
Author: jonmon
Date: 2011-08-05 10:12:04 -0500 (Fri, 05 Aug 2011)
New Revision: 4948
Modified:
SwiftApps/GOSwift/pykoa/tools/koa_goswift_manage.py
Log:
o go_swift_manage can now stop a Swift execution by the id number.
Modified: SwiftApps/GOSwift/pykoa/tools/koa_goswift_manage.py
===================================================================
--- SwiftApps/GOSwift/pykoa/tools/koa_goswift_manage.py 2011-08-05 05:14:57 UTC (rev 4947)
+++ SwiftApps/GOSwift/pykoa/tools/koa_goswift_manage.py 2011-08-05 15:12:04 UTC (rev 4948)
@@ -45,14 +45,17 @@
return 1
processes = [ppid]
- # BROKEN: Doesn't parse the 'ps' output correctly. Fix the 'parsed_line' split delimiter
for line in output.split("\n"):
parsed_line = line.split()
- if ppid == parsed_line[2]:
- processes.append(parsed_line[1])
+ try:
+ new_ppid = int(parsed_line[2])
+ except ValueError:
+ new_ppid = -1
+ if ppid == new_ppid:
+ processes.append(int(parsed_line[1]))
ppid = int(parsed_line[1])
-
for pid in processes:
+ # Handle error in which process has already been stopped.
os.kill(pid, 9)
return 0
@@ -67,6 +70,7 @@
return 1
if options.stop:
+ sys.stdout.write("Stopping Swift run number: "+args[0]+"\n")
ret = stop(args[0])
return ret
From hategan at ci.uchicago.edu Fri Aug 5 18:31:20 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Fri, 5 Aug 2011 18:31:20 -0500 (CDT)
Subject: [Swift-commit] r4949 - in
branches/release-0.93/tests/language-behaviour: mappers procedures
Message-ID: <20110805233120.5D1299CCEC@svn.ci.uchicago.edu>
Author: hategan
Date: 2011-08-05 18:31:20 -0500 (Fri, 05 Aug 2011)
New Revision: 4949
Added:
branches/release-0.93/tests/language-behaviour/mappers/0755-ext-mapper.sh.in
branches/release-0.93/tests/language-behaviour/mappers/0756-ext-mapper-slow.timeout
branches/release-0.93/tests/language-behaviour/procedures/tc.template.mix.data
Log:
fixed tests
Added: branches/release-0.93/tests/language-behaviour/mappers/0755-ext-mapper.sh.in
===================================================================
--- branches/release-0.93/tests/language-behaviour/mappers/0755-ext-mapper.sh.in (rev 0)
+++ branches/release-0.93/tests/language-behaviour/mappers/0755-ext-mapper.sh.in 2011-08-05 23:31:20 UTC (rev 4949)
@@ -0,0 +1,4 @@
+echo "[0].l 0755-ext-mapper.AAA.out"
+echo "[1].r 0755-ext-mapper.0.3.2.1.out"
+echo "[2].l 0755-ext-mapper.____.out"
+
Property changes on: branches/release-0.93/tests/language-behaviour/mappers/0755-ext-mapper.sh.in
___________________________________________________________________
Added: svn:executable
+ *
Added: branches/release-0.93/tests/language-behaviour/mappers/0756-ext-mapper-slow.timeout
===================================================================
--- branches/release-0.93/tests/language-behaviour/mappers/0756-ext-mapper-slow.timeout (rev 0)
+++ branches/release-0.93/tests/language-behaviour/mappers/0756-ext-mapper-slow.timeout 2011-08-05 23:31:20 UTC (rev 4949)
@@ -0,0 +1 @@
+60
\ No newline at end of file
Added: branches/release-0.93/tests/language-behaviour/procedures/tc.template.mix.data
===================================================================
--- branches/release-0.93/tests/language-behaviour/procedures/tc.template.mix.data (rev 0)
+++ branches/release-0.93/tests/language-behaviour/procedures/tc.template.mix.data 2011-08-05 23:31:20 UTC (rev 4949)
@@ -0,0 +1,2 @@
+
+localhost sleep /bin/sleep INSTALLED INTEL32::LINUX null
From hategan at ci.uchicago.edu Fri Aug 5 18:31:39 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Fri, 5 Aug 2011 18:31:39 -0500 (CDT)
Subject: [Swift-commit] r4950 - branches/release-0.93/tests/groups
Message-ID: <20110805233139.60D7B9CCEC@svn.ci.uchicago.edu>
Author: hategan
Date: 2011-08-05 18:31:39 -0500 (Fri, 05 Aug 2011)
New Revision: 4950
Modified:
branches/release-0.93/tests/groups/group-all-local.sh
Log:
removed iterators from local group
Modified: branches/release-0.93/tests/groups/group-all-local.sh
===================================================================
--- branches/release-0.93/tests/groups/group-all-local.sh 2011-08-05 23:31:20 UTC (rev 4949)
+++ branches/release-0.93/tests/groups/group-all-local.sh 2011-08-05 23:31:39 UTC (rev 4950)
@@ -12,7 +12,6 @@
$TESTDIR/language-behaviour/compounds \
$TESTDIR/language-behaviour/control_structures \
$TESTDIR/language-behaviour/IO \
- $TESTDIR/language-behaviour/iterators \
$TESTDIR/language-behaviour/logic \
$TESTDIR/language-behaviour/mappers \
$TESTDIR/language-behaviour/math \
From hategan at ci.uchicago.edu Fri Aug 5 18:32:06 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Fri, 5 Aug 2011 18:32:06 -0500 (CDT)
Subject: [Swift-commit] r4951 - branches/release-0.93/tests/local
Message-ID: <20110805233206.07C529CCEC@svn.ci.uchicago.edu>
Author: hategan
Date: 2011-08-05 18:32:05 -0500 (Fri, 05 Aug 2011)
New Revision: 4951
Modified:
branches/release-0.93/tests/local/filesysmapper.check.sh
branches/release-0.93/tests/local/range.check.sh
Log:
use OUTPUT env var for script stdout
Modified: branches/release-0.93/tests/local/filesysmapper.check.sh
===================================================================
--- branches/release-0.93/tests/local/filesysmapper.check.sh 2011-08-05 23:31:39 UTC (rev 4950)
+++ branches/release-0.93/tests/local/filesysmapper.check.sh 2011-08-05 23:32:05 UTC (rev 4951)
@@ -2,8 +2,8 @@
set -x
-ls stdout.txt || exit 1
-COUNT=$( grep -c "file: [abc].dat" < stdout.txt )
+ls $OUTPUT || exit 1
+COUNT=$( grep -c "file: [abc].dat" < $OUTPUT )
(( $COUNT == 3 )) || exit 1
exit 0
Modified: branches/release-0.93/tests/local/range.check.sh
===================================================================
--- branches/release-0.93/tests/local/range.check.sh 2011-08-05 23:31:39 UTC (rev 4950)
+++ branches/release-0.93/tests/local/range.check.sh 2011-08-05 23:32:05 UTC (rev 4951)
@@ -2,9 +2,9 @@
set -x
-ls stdout.txt || exit 1
+ls $OUTPUT || exit 1
-COUNT=$( grep -c "num:" stdout.txt )
+COUNT=$( grep -c "num:" $OUTPUT )
(( $COUNT == 11 )) || exit 1
exit 0
From hategan at ci.uchicago.edu Fri Aug 5 18:32:25 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Fri, 5 Aug 2011 18:32:25 -0500 (CDT)
Subject: [Swift-commit] r4952 - branches/release-0.93/tests/bugs
Message-ID: <20110805233225.D1E309CCEC@svn.ci.uchicago.edu>
Author: hategan
Date: 2011-08-05 18:32:25 -0500 (Fri, 05 Aug 2011)
New Revision: 4952
Modified:
branches/release-0.93/tests/bugs/481-array-hang.swift
Log:
use old style array declaration
Modified: branches/release-0.93/tests/bugs/481-array-hang.swift
===================================================================
--- branches/release-0.93/tests/bugs/481-array-hang.swift 2011-08-05 23:32:05 UTC (rev 4951)
+++ branches/release-0.93/tests/bugs/481-array-hang.swift 2011-08-05 23:32:25 UTC (rev 4952)
@@ -1,4 +1,4 @@
-int[] a = [1, 2, 3];
+int a[] = [1, 2, 3];
// hangs waiting for a[4]
trace(a[4]);
\ No newline at end of file
From hategan at ci.uchicago.edu Fri Aug 5 18:33:01 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Fri, 5 Aug 2011 18:33:01 -0500 (CDT)
Subject: [Swift-commit] r4953 - branches/release-0.93/tests
Message-ID: <20110805233301.77C1A9CCEC@svn.ci.uchicago.edu>
Author: hategan
Date: 2011-08-05 18:33:01 -0500 (Fri, 05 Aug 2011)
New Revision: 4953
Modified:
branches/release-0.93/tests/suite.sh
Log:
changed output naming to match test file name rather than having some random numeric id
Modified: branches/release-0.93/tests/suite.sh
===================================================================
--- branches/release-0.93/tests/suite.sh 2011-08-05 23:32:25 UTC (rev 4952)
+++ branches/release-0.93/tests/suite.sh 2011-08-05 23:33:01 UTC (rev 4953)
@@ -369,7 +369,7 @@
printf %-10.10s "success">>$REPORT
else
echo -e "${RED}FAILED${GRAY}"
- cat $RUNDIR/$TEST_LOG < /dev/null
+ cat $RUNDIR/$OUTPUT < /dev/null
printf %-10.10s "failure">>$REPORT
fi
@@ -383,16 +383,16 @@
# WIDTH=$( width "$LABEL" )
if [ "$RESULT" == "Passed" ]; then
html_td class "success" width 25 title "$CMD"
- html_a_href $TEST_LOG "$LABEL"
+ html_a_href $OUTPUT "$LABEL"
elif [ "$RESULT" == "None" ]; then
html_td width 25
html " "
html_~td
else
echo -e "${RED}FAILED${GRAY}"
- cat $RUNDIR/$TEST_LOG < /dev/null
+ cat $RUNDIR/$OUTPUT < /dev/null
html_td class "failure" width 25 title "$CMD"
- html_a_href $TEST_LOG $LABEL
+ html_a_href $OUTPUT $LABEL
fi
html_~td
elif [ "$TYPE" == "package" ]; then
@@ -484,19 +484,6 @@
fi
}
-# Create test output_*.txt file and copy to stdout.txt
-# Rename to copy_output?
-# TEST_LOG = test log
-test_log() {
- TEST_LOG="output_$LOGCOUNT.txt"
- banner "$LASTCMD" $RUNDIR/$TEST_LOG
- if [ -f $OUTPUT ]; then
- cp $OUTPUT $RUNDIR/$TEST_LOG 2>>$LOG
- cp $OUTPUT $RUNDIR/stdout.txt
- fi
- let "LOGCOUNT=$LOGCOUNT+1"
-}
-
stars() {
for i in {1..120}
do
@@ -602,9 +589,9 @@
fi
RESULT=$( result )
- test_log
- output_report test $SEQ "$LASTCMD" $RESULT $TEST_LOG
+ output_report test $SEQ "$LASTCMD" $RESULT $OUTPUT
+
check_bailout
let "SEQ=$SEQ+1"
@@ -701,8 +688,7 @@
LASTCMD="$@"
- test_log
- output_report test $SEQ "$LASTCMD" $RESULT $TEST_LOG
+ output_report test $SEQ "$LASTCMD" $RESULT $OUTPUT
check_bailout
@@ -717,7 +703,6 @@
process_exec $SCRIPT
RESULT=$( result )
- test_log
output_report test "$SYMBOL" "$LASTCMD" $RESULT
check_bailout
@@ -800,6 +785,8 @@
ARGSFILE=$NAME.args
TEST_SHOULD_FAIL=0
+
+ OUTPUT=$NAME.setup.stdout
if [ -x $GROUP/$SETUPSCRIPT ]; then
script_exec $GROUP/$SETUPSCRIPT "S"
else
@@ -821,6 +808,7 @@
grep THIS-SCRIPT-SHOULD-FAIL $SWIFTSCRIPT > /dev/null
TEST_SHOULD_FAIL=$(( ! $? ))
+ OUTPUT=$NAME.stdout
monitored_exec $TIMEOUT swift \
-wrapperlog.always.transfer true \
-sitedir.keep true \
@@ -830,13 +818,15 @@
$CDM $SWIFTSCRIPT $ARGS
TEST_SHOULD_FAIL=0
+ OUTPUT=$NAME.check.stdout
if [ -x $GROUP/$CHECKSCRIPT ]; then
- export OUTPUT
+ export TEST_LOG=$NAME.stdout
script_exec $GROUP/$CHECKSCRIPT "√"
else
check_outputs $GROUP $NAME
fi
+ OUTPUT=$NAME.clean.stdout
if [ -x $GROUP/$CLEANSCRIPT ]; then
script_exec $GROUP/$CLEANSCRIPT "C"
else
@@ -858,6 +848,8 @@
TIMEOUTFILE=$NAME.timeout
TEST_SHOULD_FAIL=0
+
+ OUTPUT=$NAME.setup.stdout
if [ -x $GROUP/$SETUPSCRIPT ]; then
script_exec $GROUP/$SETUPSCRIPT "S"
else
@@ -879,6 +871,7 @@
html_~td
fi
+ OUTPUT=$NAME.stdout
if [ -x $GROUP/$SHELLSCRIPT ]; then
script_exec $SHELLSCRIPT "X"
else
@@ -887,6 +880,7 @@
html_~td
fi
+ OUTPUT=$NAME.check.stdout
if [ -x $GROUP/$CHECKSCRIPT ]; then
script_exec $GROUP/$CHECKSCRIPT "√"
else
@@ -895,6 +889,7 @@
html_~td
fi
+ OUTPUT=$NAME.clean.stdout
if [ -x $GROUP/$CLEANSCRIPT ]; then
script_exec $GROUP/$CLEANSCRIPT "C"
else
@@ -1171,6 +1166,7 @@
start_group "Build"
TESTLINK=
EXITONFAILURE=true
+OUTPUT=checkout.stdout
if [ "$SKIP_CHECKOUT" != "1" ]; then
TESTNAME="Checkout CoG"
start_row
@@ -1188,6 +1184,7 @@
TESTNAME="Compile"
start_row
+OUTPUT=compile.stdout
# Exit early if the Swift directory is not there
if [[ ! -d $TOPDIR/cog/modules/swift ]]
then
@@ -1206,6 +1203,7 @@
fi
SWIFT_HOME=$TOPDIR/cog/modules/swift/dist/swift-svn
+OUTPUT=build.stdout
if [ $BUILD_PACKAGE = "1" ]; then
build_package
fi
From hategan at ci.uchicago.edu Fri Aug 5 18:42:46 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Fri, 5 Aug 2011 18:42:46 -0500 (CDT)
Subject: [Swift-commit] r4954 - branches/release-0.93/tests/local
Message-ID: <20110805234246.0A02A9CCEC@svn.ci.uchicago.edu>
Author: hategan
Date: 2011-08-05 18:42:45 -0500 (Fri, 05 Aug 2011)
New Revision: 4954
Modified:
branches/release-0.93/tests/local/filesysmapper.check.sh
branches/release-0.93/tests/local/range.check.sh
Log:
oops, that should be TEST_LOG
Modified: branches/release-0.93/tests/local/filesysmapper.check.sh
===================================================================
--- branches/release-0.93/tests/local/filesysmapper.check.sh 2011-08-05 23:33:01 UTC (rev 4953)
+++ branches/release-0.93/tests/local/filesysmapper.check.sh 2011-08-05 23:42:45 UTC (rev 4954)
@@ -2,8 +2,8 @@
set -x
-ls $OUTPUT || exit 1
-COUNT=$( grep -c "file: [abc].dat" < $OUTPUT )
+ls $TEST_LOG || exit 1
+COUNT=$( grep -c "file: [abc].dat" < $TEST_LOG )
(( $COUNT == 3 )) || exit 1
exit 0
Modified: branches/release-0.93/tests/local/range.check.sh
===================================================================
--- branches/release-0.93/tests/local/range.check.sh 2011-08-05 23:33:01 UTC (rev 4953)
+++ branches/release-0.93/tests/local/range.check.sh 2011-08-05 23:42:45 UTC (rev 4954)
@@ -2,9 +2,9 @@
set -x
-ls $OUTPUT || exit 1
+ls $TEST_LOG || exit 1
-COUNT=$( grep -c "num:" $OUTPUT )
+COUNT=$( grep -c "num:" $TEST_LOG )
(( $COUNT == 11 )) || exit 1
exit 0
From hategan at ci.uchicago.edu Sat Aug 6 00:45:16 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Sat, 6 Aug 2011 00:45:16 -0500 (CDT)
Subject: [Swift-commit] r4955 -
branches/release-0.93/src/org/griphyn/vdl/karajan/lib
Message-ID: <20110806054516.AFFC79CCEC@svn.ci.uchicago.edu>
Author: hategan
Date: 2011-08-06 00:45:15 -0500 (Sat, 06 Aug 2011)
New Revision: 4955
Modified:
branches/release-0.93/src/org/griphyn/vdl/karajan/lib/RuntimeStats.java
Log:
updated to reflect changes in CopyOnWriteArrayList
Modified: branches/release-0.93/src/org/griphyn/vdl/karajan/lib/RuntimeStats.java
===================================================================
--- branches/release-0.93/src/org/griphyn/vdl/karajan/lib/RuntimeStats.java 2011-08-05 23:42:45 UTC (rev 4954)
+++ branches/release-0.93/src/org/griphyn/vdl/karajan/lib/RuntimeStats.java 2011-08-06 05:45:15 UTC (rev 4955)
@@ -185,7 +185,7 @@
}
}
finally {
- states.release();
+ states.release(stateIterator);
}
return summary;
}
From wilde at ci.uchicago.edu Sat Aug 6 17:40:14 2011
From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu)
Date: Sat, 6 Aug 2011 17:40:14 -0500 (CDT)
Subject: [Swift-commit] r4956 - trunk/bin/grid
Message-ID: <20110806224014.EF3A99CCF8@svn.ci.uchicago.edu>
Author: wilde
Date: 2011-08-06 17:40:14 -0500 (Sat, 06 Aug 2011)
New Revision: 4956
Added:
trunk/bin/grid/start-grid-service
Modified:
trunk/bin/grid/TODO
trunk/bin/grid/start-ranger-service
trunk/bin/grid/start-ranger-service~
trunk/bin/grid/start-swift-service
trunk/bin/grid/swift-workers
Log:
commit snapshot of working version - work in progress
Modified: trunk/bin/grid/TODO
===================================================================
--- trunk/bin/grid/TODO 2011-08-06 05:45:15 UTC (rev 4955)
+++ trunk/bin/grid/TODO 2011-08-06 22:40:14 UTC (rev 4956)
@@ -1,4 +1,8 @@
+BUGS TO INVESTIGATE AND/OR FILE
+- why are there two logs from the coaster service: uuid-named log and swift.log????
+
+
EXTENCI APPLICATION WORK
create modft install & test file; test under fork and work
@@ -14,11 +18,13 @@
TO RESOLVE
-- how to set swift throttles to handle a varying number of coaster workers per site?
+- how to set swift throttles to handle a varying number of coaster
+workers per site?
- why did Allan set exceptions in workdir names, eg for BNL?
-- how to dynamically grow/shrink pool and add/remove sites; dynamically take coaster services in and out of service.
+- how to dynamically grow/shrink pool and add/remove sites;
+dynamically take coaster services in and out of service.
- settings for retry and replication
@@ -27,13 +33,26 @@
- Add site selection option to foreachsite
+- Add test of larger/variable-size data transfer, and test of data
+transfer speed
+
+It should be easy to set the data size.
+
+It will be a bit harder to test the data transfer *speed*,
+though. That could perhaps be tested with a run against the fork
+jobmanager, where we can set a bound on the expected delay
+time. Otherwise the delay from the time the file staging starts to the
+time the app runs is hard to determine.
+
+
+
CLEANUP
-- Find all interim tools under swift/lab/osg and place under grid/ for development
+- Find all interim tools under swift/lab/osg and place under grid/ for
+development
ENHANCEMENTS
- Find Glen's tgsites command and integrate
- incorporate gstar (would be a good Globus Online feature)
-
Added: trunk/bin/grid/start-grid-service
===================================================================
--- trunk/bin/grid/start-grid-service (rev 0)
+++ trunk/bin/grid/start-grid-service 2011-08-06 22:40:14 UTC (rev 4956)
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+# FIXME: improve arg parsing / checking / optionals
+
+function usage ()
+{
+ echo "Usage:"
+ echo " $0 --throttle 0.01 --loglevel INFO|DEBUG|TRACE --jobspernode 1"
+}
+
+if [ $# -ne 6 ]
+then
+ usage
+ exit 1
+fi
+
+LOGLEVEL=INFO # INFO, DEBUG, TRACE for increasing detail
+THROTTLE=0.09
+
+while test "$1" != "" ; do
+ case $1 in
+ --jobspernode|-j)
+ JOBSPERNODE="$2"
+ shift
+ ;;
+ --loglevel|-l)
+ LOGLEVEL="$2"
+ shift
+ ;;
+ --throttle|-t)
+ THROTTLE="$2"
+ shift
+ ;;
+ -*)
+ echo "Error: no such option $1"
+ usage
+ exit 1
+ ;;
+ esac
+ shift
+done
+
+BIN=$(cd $(dirname $0); pwd)
+
+echo THROTTLE=$THROTTLE LOGLEVEL=$LOGLEVEL
+
+start-swift-service 1 &
+sleep 5
+SPORT=$(cat service.sports)
+cat >sites.grid-ps.xml <
+
+
+ passive
+ $JOBSPERNODE
+ $THROTTLE
+ 10000
+
+ proxy
+ /tmp/$USER
+
+
+EOF
+
+WPORT=$(cat service.wports)
+SERVICE_URL=http://$(hostname -f):$WPORT
+echo swift service started - SPORT=$(cat service.sports) WPORT=$WPORT SERVICE_URL=$SERVICE_URL
+
+# SERVICE_URL=$SERVICE_URL WORKER_LOGLEVEL=$LOGLEVEL
+
Property changes on: trunk/bin/grid/start-grid-service
___________________________________________________________________
Added: svn:executable
+ *
Modified: trunk/bin/grid/start-ranger-service
===================================================================
--- trunk/bin/grid/start-ranger-service 2011-08-06 05:45:15 UTC (rev 4955)
+++ trunk/bin/grid/start-ranger-service 2011-08-06 22:40:14 UTC (rev 4956)
@@ -3,15 +3,25 @@
function usage ()
{
echo "Usage:"
- echo " $0 --nodes nnodes --walltime hh:mm:ss --project proj-name --queue q-name --user user-name"
+ echo " $0 --nodes nnodes --walltime hh:mm:ss --project proj-name --queue q-name --user user-name" --startservice true
}
-if [ $# -ne 10 ]
+if [ $# -ne 12 ]
then
usage
exit 1
fi
+
+# NODES=${1:-1}
+# WALLTIME=${2:-00:10:00}
+# PROJECT=${3:-TG-DBS080004N}
+# QUEUE=${4:-development}
+# REMOTE_USER=${5:-$USER}
+# STARTSERVICE=false
+
+
+
while test "$1" != "" ; do
case $1 in
--nodes|-n)
@@ -34,6 +44,10 @@
REMOTE_USER="$2"
shift
;;
+ --startservice|-)
+ STARTSERVICE="$2"
+ shift
+ ;;
-*)
echo "Error: no such option $1"
usage
@@ -43,19 +57,10 @@
shift
done
-
-
-#NODES=${1:-1}
-#WALLTIME=${2:-00:10:00}
-#PROJECT=${3:-TG-DBS080004N}
-#QUEUE=${4:-development}
-#REMOTE_USER=${5:-$USER}
-
-STARTSERVICE=true
HOST=tg-login.ranger.tacc.teragrid.org
BIN=$(cd $(dirname $0); pwd)
-echo NODES=$NODES WALLTIME=$WALLTIME PROJECT=$PROJECT REMOTE_USER=$REMOTE_USER QUEUE=$QUEUE
+echo NODES=$NODES WALLTIME=$WALLTIME PROJECT=$PROJECT REMOTE_USER=$REMOTE_USER QUEUE=$QUEUE STARTSERVICE=$STARTSERVICE
LOGLEVEL=INFO # INFO, DEBUG, TRACE for increasing detail
CORESPERNODE=16
Modified: trunk/bin/grid/start-ranger-service~
===================================================================
--- trunk/bin/grid/start-ranger-service~ 2011-08-06 05:45:15 UTC (rev 4955)
+++ trunk/bin/grid/start-ranger-service~ 2011-08-06 22:40:14 UTC (rev 4956)
@@ -1,18 +1,61 @@
#! /bin/bash
-# FIXME: make these commandline keyword arguments, eg --nodes=
+function usage ()
+{
+ echo "Usage:"
+ echo " $0 --nodes nnodes --walltime hh:mm:ss --project proj-name --queue q-name --user user-name"
+}
-NODES=${1:-1}
-WALLTIME=${2:-00:10:00}
-PROJECT=${3:-TG-DBS080004N}
-QUEUE=${4:-development}
-REMOTE_USER=${5:-tg455797}
+if [ $# -ne 10 ]
+then
+ usage
+ exit 1
+fi
+while test "$1" != "" ; do
+ case $1 in
+ --nodes|-n)
+ NODES="$2"
+ shift
+ ;;
+ --walltime|-t)
+ WALLTIME="$2"
+ shift
+ ;;
+ --project|-p)
+ PROJECT="$2"
+ shift
+ ;;
+ --queue|-q)
+ QUEUE="$2"
+ shift
+ ;;
+ --user|-u)
+ REMOTE_USER="$2"
+ shift
+ ;;
+ -*)
+ echo "Error: no such option $1"
+ usage
+ exit 1
+ ;;
+ esac
+ shift
+done
+
+
+
+#NODES=${1:-1}
+#WALLTIME=${2:-00:10:00}
+#PROJECT=${3:-TG-DBS080004N}
+#QUEUE=${4:-development}
+#REMOTE_USER=${5:-$USER}
+
STARTSERVICE=true
HOST=tg-login.ranger.tacc.teragrid.org
BIN=$(cd $(dirname $0); pwd)
-echo NODES=$NODES WALLTIME=$WALLTIME PROJECT=$PROJECT REMOTE_USER=$REMOTE_USER
+echo NODES=$NODES WALLTIME=$WALLTIME PROJECT=$PROJECT REMOTE_USER=$REMOTE_USER QUEUE=$QUEUE
LOGLEVEL=INFO # INFO, DEBUG, TRACE for increasing detail
CORESPERNODE=16
@@ -39,7 +82,7 @@
10000proxy
- /tmp/wilde
+ /tmp/$USER
EOF
@@ -59,12 +102,17 @@
exit 1
fi
-echo Created remote dir
+echo Created remote dir: $rdir
scp $BIN/{worker.pl,workers.ranger.sh,workers.ranger.sub} $REMOTE_USER@$HOST:$rdir
-echo Copied grid tools to remote dir
+echo Copied grid tools to remote dir: $rdir
-ssh $REMOTE_USER@$HOST qsub -A $PROJECT -N runworkers -pe 16way $(($NODES * 16)) -l h_rt=$WALLTIME -q $QUEUE -v SERVICE_URL=$SERVICE_URL,WORKER_LOGLEVEL=$LOGLEVEL $rdir/workers.ranger.sub
+echo Submitting ...
+echo "ssh $REMOTE_USER@$HOST qsub -A $PROJECT -N runworkers -pe 16way $(($NODES * 16)) -q $QUEUE -l h_rt=$WALLTIME -v SERVICE_URL=$SERVICE_URL, WORKER_LOGLEVEL=$LOGLEVEL $rdir/workers.ranger.sub"
+
+ssh $REMOTE_USER@$HOST qsub -A $PROJECT -N runworkers -pe 16way $(($NODES * 16)) -q $QUEUE -l h_rt=$WALLTIME -v SERVICE_URL=$SERVICE_URL,WORKER_LOGLEVEL=$LOGLEVEL $rdir/workers.ranger.sub
+
echo Submitted remote worker launching script
+
Modified: trunk/bin/grid/start-swift-service
===================================================================
--- trunk/bin/grid/start-swift-service 2011-08-06 05:45:15 UTC (rev 4955)
+++ trunk/bin/grid/start-swift-service 2011-08-06 22:40:14 UTC (rev 4956)
@@ -1,36 +1,57 @@
#!/bin/bash
-NSERVICES=$1
+NSERVICES=${1:-1}
SERVICE=coaster-service # found via PATH
-ontrap() # FIXME: Not needed?
+echo $0: starting $NSERVICES services
+
+ontrap()
{
- echo '====>' in ontrap
+ echo $0: Received signal, killing coaster services
trap - 1 2 3 15
echo start_service: trapping exit or signal
- kill $(cat service-*.pid)
+ kill 0 # $(cat service-*.pid)
}
-# trap ontrap 1 2 3 15 # FIXME: Not needed?
+trap ontrap 1 2 3 15 # FIXME: Not needed?
+# Launch the requested number of services
+
rm -f service.sports service.wports
for i in `seq -w 0 $((NSERVICES - 1))`; do
- rm -f service-$i.{sport,wport,pid,log}
- $SERVICE -nosec -passive -portfile service-$i.sport -localportfile service-$i.wport &> service-$i.log &
+ rm -f service-$i.{sport,wport,pid,out}
+ $SERVICE -nosec -passive -portfile service-$i.sport -localportfile service-$i.wport &> service-$i.out &
echo $! >service-$i.pid
- sleep 3
- if [ -s service-$i.sport ]; then
- echo $(cat service-$i.sport) >> service.sports
- else
- echo service-$i.sport does not exist or is empty. exiting.
- exit 1
+done
+
+# Wait (a bit) for all services to report their port numbers
+# Record missing ones as "-"
+
+maxtries=10
+
+for (( tries=0; tries < $maxtries; tries++ )); do
+ sleep 1
+ errors=0
+ for i in `seq -w 0 $((NSERVICES - 1))`; do
+ if [ -s service-$i.sport -a -s service-$i.wport ]; then
+ echo $(cat service-$i.sport) >> service.sports
+ echo $(cat service-$i.wport) >> service.wports
+ else
+ echo 0 >> service.sports
+ echo 0 >> service.wports
+ errors=$((errors+1))
+ fi
+ done
+ if [ $errors = 0 ]; then
+ break
fi
- if [ -s service-$i.wport ]; then
- echo $(cat service-$i.wport) >> service.wports
- else
- echo service-$i.wport does not exist or is empty. exiting.
- exit 1
- fi
done
-wait
+if [ $errors != 0 ]; then
+ echo $0: $errors services failesd to report their port numbers
+fi
+echo $0: $((NSERVICES-errors)) services started successfully
+
+wait # Wait on all the started services; stop them all if signalled (ontrap)
+
+
Modified: trunk/bin/grid/swift-workers
===================================================================
--- trunk/bin/grid/swift-workers 2011-08-06 05:45:15 UTC (rev 4955)
+++ trunk/bin/grid/swift-workers 2011-08-06 22:40:14 UTC (rev 4956)
@@ -1,5 +1,11 @@
-#!/usr/bin/env ruby
+#! /usr/bin/env ruby
+$stdout.sync = true
+
+# FIXME: Is this the best way to get lib functions from bin/grid into RUBYLIB ?
+
+$:[$:.length] = File.dirname($0)
+
require 'mk_catalog'
require 'etc'
@@ -14,6 +20,10 @@
# executable = /home/wilde/swift/src/0.92/cog/modules/swift/dist/swift-svn/bin/worker.pl
# arguments = http://128.135.125.17:<%= port %> <%= name %> /tmp 14400
+# WORKER_LOGGING_LEVEL=$LOGLEVEL $HOME/swift_gridtools/worker.pl $SERVICEURL swork${worker} $LOGDIR >& /dev/null &
+
+# a mod
+
def gen_submit(count = 1)
job = %q[
universe = grid
@@ -33,11 +43,38 @@
<% } %>
]
- ERB.new(job.gsub(/^\s+/, ""), 0, "%<>", "@submit_file").result(binding)
+ ov=$VERBOSE
+ $VERBOSE=nil
+ workerExecutable = `which worker.pl`
+ $VERBOSE=ov
+# workerContact = "http://communicado.ci.uchicago.edu:36906"
+ workerContact = ARGV[2]
+
+ newjob = %Q[
+ universe = grid
+ stream_output = False
+ stream_error = False
+ transfer_executable = true
+ periodic_remove = JobStatus == 5
+ notification = Never
+
+ globus_rsl = (maxwalltime=240)
+ grid_resource = <%= @grid_resource %>
+ executable = #{workerExecutable}
+ arguments = #{workerContact} swork /tmp
+ environment = WORKER_LOGGING_LEVEL=INFO
+ log = condor.log
+
+ <% count.times { %>queue
+ <% } %>
+ ]
+
+ ERB.new(newjob.gsub(/^\s+/, ""), 0, "%<>", "@submit_file").result(binding)
end
def submit_job(count)
-# puts "Submitting #{@name} #{count} jobs"
+ puts "submit_job: Submitting #{@name} #{count} jobs"
+ count = count.to_i
output = ""
#return output
submitfile = gen_submit(count)
@@ -50,12 +87,18 @@
end
def queued
+ ov=$VERBOSE
+ $VERBOSE=nil
jobs = `condor_q #{$username} -const 'GridResource == \"#{@grid_resource}\" && JobStatus == 1' -format \"%s \" GlobalJobId`
+ $VERBOSE=ov
jobs.split(" ").size
end
def running
+ ov=$VERBOSE
+ $VERBOSE=nil
jobs = `condor_q #{$username} -const 'GridResource == \"#{@grid_resource}\" && JobStatus == 2' -format \"%s \" GlobalJobId`
+ $VERBOSE=ov
jobs.split(" ").size
end
@@ -75,13 +118,13 @@
=end
if __FILE__ == $0
- raise "No whitelist file" if !ARGV[0]
+ raise "No greenlist file" if !ARGV[0]
start_port = 61100 # FIXME
ctr = 0
threads = []
ARGV[1] = "scec" if !ARGV[1]
- whitelist = IO.readlines(ARGV[0]).map { |line| line.chomp! }
+ greenlist = IO.readlines(ARGV[0]).map { |line| line.chomp! }
$username = Etc.getlogin
puts "Username = #{$username}"
@@ -93,7 +136,7 @@
totalRunning = 0
ress_parse(ARGV[1]) do |name, value|
- next if not whitelist.index(name) and not whitelist.empty?
+ next if not greenlist.index(name) and not greenlist.empty?
totalCores += (value.throttle * 100 + 2).to_i
end
puts "totalCores for green sites = #{totalCores}"
@@ -105,14 +148,16 @@
# swiftDemand = IO.read("swiftDemand") # Replace this with sensor of Swift demand
swiftDemand = 15
paddedDemand = (swiftDemand * 1.2).to_i
+ ov=$VERBOSE;$VERBOSE=nil
totalRunning = `condor_q #{$username} -const 'JobStatus == 2' -format \"%s \" GlobalJobId`.split(" ").size
+ $VERBOSE=ov
puts "*** demandThread: swiftDemand=#{swiftDemand} paddedDemand=#{paddedDemand} totalRunning=#{totalRunning}"
sleep 60
end
end
ress_parse(ARGV[1]) do |name, value|
- next if not whitelist.index(name) and not whitelist.empty?
+ next if not greenlist.index(name) and not greenlist.empty?
site = Site.new
site.name = name
site.grid_resource = "gt2 #{value.url}/jobmanager-#{value.jm}"
From wilde at ci.uchicago.edu Sat Aug 6 17:42:04 2011
From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu)
Date: Sat, 6 Aug 2011 17:42:04 -0500 (CDT)
Subject: [Swift-commit] r4957 - trunk/bin/grid
Message-ID: <20110806224204.0C5209CCF8@svn.ci.uchicago.edu>
Author: wilde
Date: 2011-08-06 17:42:03 -0500 (Sat, 06 Aug 2011)
New Revision: 4957
Removed:
trunk/bin/grid/start-ranger-service~
Log:
Remove emacs ~ file committed in error
Deleted: trunk/bin/grid/start-ranger-service~
===================================================================
--- trunk/bin/grid/start-ranger-service~ 2011-08-06 22:40:14 UTC (rev 4956)
+++ trunk/bin/grid/start-ranger-service~ 2011-08-06 22:42:03 UTC (rev 4957)
@@ -1,118 +0,0 @@
-#! /bin/bash
-
-function usage ()
-{
- echo "Usage:"
- echo " $0 --nodes nnodes --walltime hh:mm:ss --project proj-name --queue q-name --user user-name"
-}
-
-if [ $# -ne 10 ]
-then
- usage
- exit 1
-fi
-
-while test "$1" != "" ; do
- case $1 in
- --nodes|-n)
- NODES="$2"
- shift
- ;;
- --walltime|-t)
- WALLTIME="$2"
- shift
- ;;
- --project|-p)
- PROJECT="$2"
- shift
- ;;
- --queue|-q)
- QUEUE="$2"
- shift
- ;;
- --user|-u)
- REMOTE_USER="$2"
- shift
- ;;
- -*)
- echo "Error: no such option $1"
- usage
- exit 1
- ;;
- esac
- shift
-done
-
-
-
-#NODES=${1:-1}
-#WALLTIME=${2:-00:10:00}
-#PROJECT=${3:-TG-DBS080004N}
-#QUEUE=${4:-development}
-#REMOTE_USER=${5:-$USER}
-
-STARTSERVICE=true
-HOST=tg-login.ranger.tacc.teragrid.org
-BIN=$(cd $(dirname $0); pwd)
-
-echo NODES=$NODES WALLTIME=$WALLTIME PROJECT=$PROJECT REMOTE_USER=$REMOTE_USER QUEUE=$QUEUE
-LOGLEVEL=INFO # INFO, DEBUG, TRACE for increasing detail
-
-CORESPERNODE=16
-
-THROTTLE=$(echo "scale=2; ($NODES*$CORESPERNODE)/100 -.01"|bc)
-
-echo THROTTLE=$THROTTLE
-
-# This lets user run this script to add another job full of workers to an existing coaster service
-# Must be started in the same directory where start-swift-service created the service.wports file.
-
-if [ $STARTSERVICE = true ]; then
- start-swift-service 1 &
- sleep 5
- SPORT=$(cat service.sports)
- cat >sites.pecos.xml <
-
-
- passive
- $CORESPERNODE
- $THROTTLE
- 10000
-
- proxy
- /tmp/$USER
-
-
-EOF
-fi
-
-WPORT=$(cat service.wports)
-SERVICE_URL=http://$(hostname -f):$WPORT
-echo swift service started - SPORT=$(cat service.sports) WPORT=$WPORT SERVICE_URL=$SERVICE_URL
-
-# FIXME: scp the right worker.pl, worker.sh and .sub files to the dest system (Ranger)
-
-rdir=swift_gridtools
-ssh $REMOTE_USER@$HOST mkdir -p $rdir
-
-if [ $? != 0 ]; then
- echo $0: unable to create remote directory $rdir
- exit 1
-fi
-
-echo Created remote dir: $rdir
-
-scp $BIN/{worker.pl,workers.ranger.sh,workers.ranger.sub} $REMOTE_USER@$HOST:$rdir
-
-echo Copied grid tools to remote dir: $rdir
-
-echo Submitting ...
-
-echo "ssh $REMOTE_USER@$HOST qsub -A $PROJECT -N runworkers -pe 16way $(($NODES * 16)) -q $QUEUE -l h_rt=$WALLTIME -v SERVICE_URL=$SERVICE_URL, WORKER_LOGLEVEL=$LOGLEVEL $rdir/workers.ranger.sub"
-
-ssh $REMOTE_USER@$HOST qsub -A $PROJECT -N runworkers -pe 16way $(($NODES * 16)) -q $QUEUE -l h_rt=$WALLTIME -v SERVICE_URL=$SERVICE_URL,WORKER_LOGLEVEL=$LOGLEVEL $rdir/workers.ranger.sub
-
-echo Submitted remote worker launching script
-
From hategan at ci.uchicago.edu Sat Aug 6 21:24:38 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Sat, 6 Aug 2011 21:24:38 -0500 (CDT)
Subject: [Swift-commit] r4958 - branches/release-0.93
Message-ID: <20110807022438.BC7659CCF8@svn.ci.uchicago.edu>
Author: hategan
Date: 2011-08-06 21:24:38 -0500 (Sat, 06 Aug 2011)
New Revision: 4958
Modified:
branches/release-0.93/.classpath
branches/release-0.93/.project
Log:
updated eclipse files
Modified: branches/release-0.93/.classpath
===================================================================
--- branches/release-0.93/.classpath 2011-08-06 22:42:03 UTC (rev 4957)
+++ branches/release-0.93/.classpath 2011-08-07 02:24:38 UTC (rev 4958)
@@ -16,11 +16,10 @@
-
-
-
+
+
Modified: branches/release-0.93/.project
===================================================================
--- branches/release-0.93/.project 2011-08-06 22:42:03 UTC (rev 4957)
+++ branches/release-0.93/.project 2011-08-07 02:24:38 UTC (rev 4958)
@@ -6,8 +6,13 @@
abstraction-commonkarajan
-
-
+
+
+ org.eclipse.jdt.core.javabuilder
+
+
+
+ org.eclipse.jdt.core.javanature
From hategan at ci.uchicago.edu Sat Aug 6 21:27:13 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Sat, 6 Aug 2011 21:27:13 -0500 (CDT)
Subject: [Swift-commit] r4959 - branches/release-0.93
Message-ID: <20110807022713.705309CCF8@svn.ci.uchicago.edu>
Author: hategan
Date: 2011-08-06 21:27:13 -0500 (Sat, 06 Aug 2011)
New Revision: 4959
Modified:
branches/release-0.93/build.xml
Log:
backport of chmod build optimization from trunk
Modified: branches/release-0.93/build.xml
===================================================================
--- branches/release-0.93/build.xml 2011-08-07 02:24:38 UTC (rev 4958)
+++ branches/release-0.93/build.xml 2011-08-07 02:27:13 UTC (rev 4959)
@@ -80,128 +80,145 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
From hategan at ci.uchicago.edu Sun Aug 7 18:02:27 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Sun, 7 Aug 2011 18:02:27 -0500 (CDT)
Subject: [Swift-commit] r4960 -
branches/release-0.93/src/org/griphyn/vdl/karajan
Message-ID: <20110807230227.908759CFF4@svn.ci.uchicago.edu>
Author: hategan
Date: 2011-08-07 18:02:26 -0500 (Sun, 07 Aug 2011)
New Revision: 4960
Modified:
branches/release-0.93/src/org/griphyn/vdl/karajan/HangChecker.java
Log:
disable hang checker messages if all sites are overloaded
Modified: branches/release-0.93/src/org/griphyn/vdl/karajan/HangChecker.java
===================================================================
--- branches/release-0.93/src/org/griphyn/vdl/karajan/HangChecker.java 2011-08-07 02:27:13 UTC (rev 4959)
+++ branches/release-0.93/src/org/griphyn/vdl/karajan/HangChecker.java 2011-08-07 23:02:26 UTC (rev 4960)
@@ -15,12 +15,11 @@
import java.util.TimerTask;
import org.apache.log4j.Logger;
-import org.globus.cog.karajan.scheduler.LateBindingScheduler;
+import org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler;
import org.globus.cog.karajan.stack.VariableStack;
import org.globus.cog.karajan.workflow.ExecutionException;
import org.globus.cog.karajan.workflow.events.EventBus;
import org.globus.cog.karajan.workflow.nodes.grid.SchedulerNode;
-import org.griphyn.vdl.karajan.lib.VDLFunction;
public class HangChecker extends TimerTask {
public static final Logger logger = Logger.getLogger(HangChecker.class);
@@ -41,11 +40,11 @@
public void run() {
try {
- LateBindingScheduler s = (LateBindingScheduler) stack.getGlobal(SchedulerNode.SCHEDULER);
+ WeightedHostScoreScheduler s = (WeightedHostScoreScheduler) stack.getGlobal(SchedulerNode.SCHEDULER);
if (s != null) {
int running = s.getRunning();
-
- if (running == 0 && EventBus.eventCount == lastEventCount) {
+ boolean allOverloaded = s.allOverloaded();
+ if (running == 0 && EventBus.eventCount == lastEventCount && !allOverloaded) {
logger.warn("No events in " + (CHECK_INTERVAL / 1000) + "s.");
ByteArrayOutputStream os = new ByteArrayOutputStream();
PrintStream ps = new PrintStream(os);
From hategan at ci.uchicago.edu Sun Aug 7 19:07:35 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Sun, 7 Aug 2011 19:07:35 -0500 (CDT)
Subject: [Swift-commit] r4961 -
branches/release-0.93/src/org/griphyn/vdl/karajan
Message-ID: <20110808000735.3DB179CCF8@svn.ci.uchicago.edu>
Author: hategan
Date: 2011-08-07 19:07:35 -0500 (Sun, 07 Aug 2011)
New Revision: 4961
Modified:
branches/release-0.93/src/org/griphyn/vdl/karajan/Loader.java
Log:
log arguments to swift executable
Modified: branches/release-0.93/src/org/griphyn/vdl/karajan/Loader.java
===================================================================
--- branches/release-0.93/src/org/griphyn/vdl/karajan/Loader.java 2011-08-07 23:02:26 UTC (rev 4960)
+++ branches/release-0.93/src/org/griphyn/vdl/karajan/Loader.java 2011-08-08 00:07:35 UTC (rev 4961)
@@ -16,6 +16,7 @@
import java.security.SecureRandom;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
+import java.util.Arrays;
import java.util.Date;
import java.util.Enumeration;
import java.util.Iterator;
@@ -131,7 +132,10 @@
try {
//Thread.sleep(20000);
setupLogging(ap, projectName, runID);
- logger.debug("Max heap: " + Runtime.getRuntime().maxMemory());
+ if (logger.isDebugEnabled()) {
+ logger.debug("arguments: " + Arrays.asList(argv));
+ logger.debug("Max heap: " + Runtime.getRuntime().maxMemory());
+ }
if (ap.isPresent(ARG_CDMFILE)) {
loadCDM(ap);
From hategan at ci.uchicago.edu Sun Aug 7 21:23:12 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Sun, 7 Aug 2011 21:23:12 -0500 (CDT)
Subject: [Swift-commit] r4962 - branches/release-0.93/libexec
Message-ID: <20110808022312.E1DF09CFF4@svn.ci.uchicago.edu>
Author: hategan
Date: 2011-08-07 21:23:12 -0500 (Sun, 07 Aug 2011)
New Revision: 4962
Modified:
branches/release-0.93/libexec/_swiftwrap
branches/release-0.93/libexec/_swiftwrap.staging
branches/release-0.93/libexec/vdl-int-staging.k
branches/release-0.93/libexec/vdl-int.k
Log:
use error files even when using provider status for better clarity
Modified: branches/release-0.93/libexec/_swiftwrap
===================================================================
--- branches/release-0.93/libexec/_swiftwrap 2011-08-08 00:07:35 UTC (rev 4961)
+++ branches/release-0.93/libexec/_swiftwrap 2011-08-08 02:23:12 UTC (rev 4962)
@@ -58,9 +58,13 @@
fail() {
EC=$1
shift
- if [ "$STATUSMODE" = "files" ]; then
- echo $@ >"$WFDIR/status/$JOBDIR/${ID}-error"
+
+ if [ "$STATUSMODE" != "files" ]; then
+ mkdir -p $WFDIR/status/$JOBDIR
fi
+
+ echo $@ >"$WFDIR/status/$JOBDIR/${ID}-error"
+
log $@
info
if [ "$STATUSMODE" = "files" ]; then
Modified: branches/release-0.93/libexec/_swiftwrap.staging
===================================================================
--- branches/release-0.93/libexec/_swiftwrap.staging 2011-08-08 00:07:35 UTC (rev 4961)
+++ branches/release-0.93/libexec/_swiftwrap.staging 2011-08-08 02:23:12 UTC (rev 4962)
@@ -61,19 +61,11 @@
fail() {
EC=$1
shift
- if [ "$STATUSMODE" = "files" ]; then
- echo $@ >"$WFDIR/status/$JOBDIR/${ID}-error"
- fi
+
log $@
info
- if [ "$STATUSMODE" = "files" ]; then
- exit 0
- else
- if [ "X$STDERR" != "X" ]; then
- echo "$@" >>$STDERR
- fi
- exit $EC
- fi
+ echo "$@" >> wrapper.error
+ exit $EC
}
checkError() {
@@ -308,7 +300,7 @@
fi
"$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" <"$STDIN"
fi
-checkError $? "Exit code $?"
+checkError $? "Exit code $?" <<$STDERR
if [ ! -s "$STDOUT" ]; then
log "Removing empty stdout"
Modified: branches/release-0.93/libexec/vdl-int-staging.k
===================================================================
--- branches/release-0.93/libexec/vdl-int-staging.k 2011-08-08 00:07:35 UTC (rev 4961)
+++ branches/release-0.93/libexec/vdl-int-staging.k 2011-08-08 02:23:12 UTC (rev 4962)
@@ -70,7 +70,12 @@
element(cleanups, [cleanup]
log(LOG:INFO, "START cleanups={cleanup}")
)
-
+
+ element(readErrorFile, [dir, jobid]
+ str:strip(file:read("{dir}/{jobid}.error"))
+ file:remove("{dir}/{jobid}.error")
+ )
+
element(execute2, [tr, optional(arguments, stdin, stdout, stderr), stagein, stageout, restartout,
replicationGroup, replicationChannel]
stagein := list(unique(each(stagein)))
@@ -150,6 +155,8 @@
mode = WRAPPER_TRANSFER_MODE)
//stageOut("{stdout}", "{stagingMethod}://localhost/{ddir}/{stdout}")
//stageOut("{stderr}", "{stagingMethod}://localhost/{ddir}/{stderr}")
+ stageOut("wrapper.error", "{stagingMethod}://localhost/{ddir}/{jobid}.error",
+ mode = STAGING_MODE:IF_PRESENT)
appStageouts(jobid, stageout, ".", stagingMethod)
task:cleanUp(".") //the whole job directory
@@ -163,6 +170,7 @@
)
catch("^(?!Abort$).*"
vdl:setprogress("Failed but can retry")
+ exception := try(exception(readErrorFile(ddir, jobid)), exception)
log(LOG:DEBUG, "APPLICATION_EXCEPTION jobid={jobid} - Application exception: ", exception)
throw(
Modified: branches/release-0.93/libexec/vdl-int.k
===================================================================
--- branches/release-0.93/libexec/vdl-int.k 2011-08-08 00:07:35 UTC (rev 4961)
+++ branches/release-0.93/libexec/vdl-int.k 2011-08-08 02:23:12 UTC (rev 4962)
@@ -46,24 +46,37 @@
file:remove("{wfdir}/status/{jobdir}/{jobid}-success", host=rhost)
log(LOG:INFO, "SUCCESS jobid={jobid} - Success file found")
)
- if(
- file:exists("{wfdir}/status/{jobdir}/{jobid}-error", host=rhost) then(
- log(LOG:INFO, "FAILURE jobid={jobid} - Failure file found")
- task:transfer(srchost=rhost, srcdir="{wfdir}/status/{jobdir}", srcfile="{jobid}-error")
- error := parallel(
- file:remove("{wfdir}/status/{jobdir}/{jobid}-error", host=rhost)
- sequential(
- str:strip(file:read("{jobid}-error"))
- file:remove("{jobid}-error")
- )
+ sequential(
+ try (
+ msg = checkErrorFile(rhost, wfdir, jobid, jobdir)
+ sequential (
+ log(LOG:INFO, "NO_STATUS_FILE jobid={jobid} - Both status files are missing")
+ throw("No status file was found. Check the shared filesystem on {rhost}")
)
- throw(error)
)
- else (
- log(LOG:INFO, "NO_STATUS_FILE jobid={jobid} - Both status files are missing")
- throw("No status file was found. Check the shared filesystem on {rhost}")
+ )
+ throw(checkErrorFile(rhost, wfdir, jobid, tr, jobdir))
+ )
+ )
+
+ element(checkErrorFile, [rhost, wfdir, jobid, jobdir]
+ if (
+ file:exists("{wfdir}/status/{jobdir}/{jobid}-error", host=rhost) then(
+ log(LOG:INFO, "FAILURE jobid={jobid} - Failure file found")
+ task:transfer(srchost=rhost, srcdir="{wfdir}/status/{jobdir}", srcfile="{jobid}-error")
+ error := parallel(
+ file:remove("{wfdir}/status/{jobdir}/{jobid}-error", host=rhost)
+ sequential(
+ str:strip(file:read("{jobid}-error"))
+ file:remove("{jobid}-error")
+ )
)
+ error
)
+ else (
+ log(LOG:INFO, "NO_STATUS_FILE jobid={jobid} - Error file missing")
+ throw("No status file was found. Check the shared filesystem on {rhost}")
+ )
)
)
@@ -555,6 +568,8 @@
)
catch("^(?!Abort$).*"
vdl:setprogress("Failed but can retry")
+ exception := try(exception(checkErrorFile(rhost, wfdir, jobid, jobdir)), exception)
+
log(LOG:DEBUG, "APPLICATION_EXCEPTION jobid={jobid} - Application exception: ", exception)
if(matches(exception,".*executable bit.*")
From hategan at ci.uchicago.edu Sun Aug 7 22:38:31 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Sun, 7 Aug 2011 22:38:31 -0500 (CDT)
Subject: [Swift-commit] r4963 - branches/release-0.93/libexec
Message-ID: <20110808033831.01D619CFF4@svn.ci.uchicago.edu>
Author: hategan
Date: 2011-08-07 22:38:30 -0500 (Sun, 07 Aug 2011)
New Revision: 4963
Modified:
branches/release-0.93/libexec/vdl-int-staging.k
branches/release-0.93/libexec/vdl-int.k
Log:
fixed hiding of "exception"
Modified: branches/release-0.93/libexec/vdl-int-staging.k
===================================================================
--- branches/release-0.93/libexec/vdl-int-staging.k 2011-08-08 02:23:12 UTC (rev 4962)
+++ branches/release-0.93/libexec/vdl-int-staging.k 2011-08-08 03:38:30 UTC (rev 4963)
@@ -170,7 +170,8 @@
)
catch("^(?!Abort$).*"
vdl:setprogress("Failed but can retry")
- exception := try(exception(readErrorFile(ddir, jobid)), exception)
+ prev := exception
+ exception := try(exception(readErrorFile(ddir, jobid)), prev)
log(LOG:DEBUG, "APPLICATION_EXCEPTION jobid={jobid} - Application exception: ", exception)
throw(
Modified: branches/release-0.93/libexec/vdl-int.k
===================================================================
--- branches/release-0.93/libexec/vdl-int.k 2011-08-08 02:23:12 UTC (rev 4962)
+++ branches/release-0.93/libexec/vdl-int.k 2011-08-08 03:38:30 UTC (rev 4963)
@@ -568,7 +568,8 @@
)
catch("^(?!Abort$).*"
vdl:setprogress("Failed but can retry")
- exception := try(exception(checkErrorFile(rhost, wfdir, jobid, jobdir)), exception)
+ prev := exception
+ exception := try(exception(checkErrorFile(rhost, wfdir, jobid, jobdir)), prev)
log(LOG:DEBUG, "APPLICATION_EXCEPTION jobid={jobid} - Application exception: ", exception)
From davidk at ci.uchicago.edu Mon Aug 8 11:33:47 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Mon, 8 Aug 2011 11:33:47 -0500 (CDT)
Subject: [Swift-commit] r4964 - in branches/release-0.93/tests: .
language-behaviour/math
Message-ID: <20110808163347.871D09CCB5@svn.ci.uchicago.edu>
Author: davidk
Date: 2011-08-08 11:33:46 -0500 (Mon, 08 Aug 2011)
New Revision: 4964
Added:
branches/release-0.93/tests/language-behaviour/math/006-add.out.expected
branches/release-0.93/tests/language-behaviour/math/008-add-multiply.out.expected
branches/release-0.93/tests/language-behaviour/math/009-multiply.out.expected
branches/release-0.93/tests/language-behaviour/math/010-divide.out.expected
branches/release-0.93/tests/language-behaviour/math/011-divide-float.out.expected
branches/release-0.93/tests/language-behaviour/math/013-mod.out.expected
branches/release-0.93/tests/language-behaviour/math/014-subtract.out.expected
branches/release-0.93/tests/language-behaviour/math/0145-unary-subtact.out.expected
branches/release-0.93/tests/language-behaviour/math/015-lesseq.out.expected
branches/release-0.93/tests/language-behaviour/math/016-greatereq.out.expected
branches/release-0.93/tests/language-behaviour/math/017-greater.out.expected
branches/release-0.93/tests/language-behaviour/math/018-less.out.expected
branches/release-0.93/tests/language-behaviour/math/019-equals.out.expected
branches/release-0.93/tests/language-behaviour/math/0191-not-equals.out.expected
branches/release-0.93/tests/language-behaviour/math/030-mix-float-int.out.expected
branches/release-0.93/tests/language-behaviour/math/031-add-float.out.expected
Removed:
branches/release-0.93/tests/language-behaviour/math/006-add.check.sh
branches/release-0.93/tests/language-behaviour/math/007-add-in-proc-add.check.sh
branches/release-0.93/tests/language-behaviour/math/008-add-multiply.check.sh
branches/release-0.93/tests/language-behaviour/math/009-multiply.check.sh
branches/release-0.93/tests/language-behaviour/math/010-divide.check.sh
branches/release-0.93/tests/language-behaviour/math/011-divide-float.check.sh
branches/release-0.93/tests/language-behaviour/math/013-mod.check.sh
branches/release-0.93/tests/language-behaviour/math/014-subtract.check.sh
branches/release-0.93/tests/language-behaviour/math/0145-unary-subtact.check.sh
branches/release-0.93/tests/language-behaviour/math/015-lesseq.check.sh
branches/release-0.93/tests/language-behaviour/math/016-greatereq.check.sh
branches/release-0.93/tests/language-behaviour/math/017-greater.check.sh
branches/release-0.93/tests/language-behaviour/math/018-less.check.sh
branches/release-0.93/tests/language-behaviour/math/019-equals.check.sh
branches/release-0.93/tests/language-behaviour/math/0191-not-equals.check.sh
branches/release-0.93/tests/language-behaviour/math/030-mix-float-int.check.sh
branches/release-0.93/tests/language-behaviour/math/031-add-float.check.sh
Modified:
branches/release-0.93/tests/suite.sh
Log:
Replace check scripts, which used grep, with .expected files for exact matches
Deleted: branches/release-0.93/tests/language-behaviour/math/006-add.check.sh
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/006-add.check.sh 2011-08-08 03:38:30 UTC (rev 4963)
+++ branches/release-0.93/tests/language-behaviour/math/006-add.check.sh 2011-08-08 16:33:46 UTC (rev 4964)
@@ -1,8 +0,0 @@
-#!/bin/bash
-set -x
-(cat 006-add.out | grep 135) || exit 1
-exit 0
-#!/bin/bash
-set -x
-cat 006-add.out | grep || exit 1
-exit 0
Added: branches/release-0.93/tests/language-behaviour/math/006-add.out.expected
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/006-add.out.expected (rev 0)
+++ branches/release-0.93/tests/language-behaviour/math/006-add.out.expected 2011-08-08 16:33:46 UTC (rev 4964)
@@ -0,0 +1 @@
+135
Deleted: branches/release-0.93/tests/language-behaviour/math/007-add-in-proc-add.check.sh
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/007-add-in-proc-add.check.sh 2011-08-08 03:38:30 UTC (rev 4963)
+++ branches/release-0.93/tests/language-behaviour/math/007-add-in-proc-add.check.sh 2011-08-08 16:33:46 UTC (rev 4964)
@@ -1,4 +0,0 @@
-#!/bin/bash
-set -x
-cat 007-add-in-proc-add.out | grep 189|| exit 1
-exit 0
Deleted: branches/release-0.93/tests/language-behaviour/math/008-add-multiply.check.sh
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/008-add-multiply.check.sh 2011-08-08 03:38:30 UTC (rev 4963)
+++ branches/release-0.93/tests/language-behaviour/math/008-add-multiply.check.sh 2011-08-08 16:33:46 UTC (rev 4964)
@@ -1,4 +0,0 @@
-#!/bin/bash
-set -x
-cat 008-add-multiply.out | grep 8|| exit 1
-exit 0
Added: branches/release-0.93/tests/language-behaviour/math/008-add-multiply.out.expected
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/008-add-multiply.out.expected (rev 0)
+++ branches/release-0.93/tests/language-behaviour/math/008-add-multiply.out.expected 2011-08-08 16:33:46 UTC (rev 4964)
@@ -0,0 +1 @@
+8
Deleted: branches/release-0.93/tests/language-behaviour/math/009-multiply.check.sh
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/009-multiply.check.sh 2011-08-08 03:38:30 UTC (rev 4963)
+++ branches/release-0.93/tests/language-behaviour/math/009-multiply.check.sh 2011-08-08 16:33:46 UTC (rev 4964)
@@ -1,4 +0,0 @@
-#!/bin/bash
-set -x
-cat 009-multiply.out | grep 42|| exit 1
-exit 0
Added: branches/release-0.93/tests/language-behaviour/math/009-multiply.out.expected
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/009-multiply.out.expected (rev 0)
+++ branches/release-0.93/tests/language-behaviour/math/009-multiply.out.expected 2011-08-08 16:33:46 UTC (rev 4964)
@@ -0,0 +1 @@
+42
Deleted: branches/release-0.93/tests/language-behaviour/math/010-divide.check.sh
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/010-divide.check.sh 2011-08-08 03:38:30 UTC (rev 4963)
+++ branches/release-0.93/tests/language-behaviour/math/010-divide.check.sh 2011-08-08 16:33:46 UTC (rev 4964)
@@ -1,4 +0,0 @@
-#!/bin/bash
-set -x
-cat 010-divide.out | grep 33|| exit 1
-exit 0
Added: branches/release-0.93/tests/language-behaviour/math/010-divide.out.expected
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/010-divide.out.expected (rev 0)
+++ branches/release-0.93/tests/language-behaviour/math/010-divide.out.expected 2011-08-08 16:33:46 UTC (rev 4964)
@@ -0,0 +1 @@
+33
Deleted: branches/release-0.93/tests/language-behaviour/math/011-divide-float.check.sh
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/011-divide-float.check.sh 2011-08-08 03:38:30 UTC (rev 4963)
+++ branches/release-0.93/tests/language-behaviour/math/011-divide-float.check.sh 2011-08-08 16:33:46 UTC (rev 4964)
@@ -1,4 +0,0 @@
-#!/bin/bash
-set -x
-cat 011-divide-float.out | grep 0.3333333333333333|| exit 1
-exit 0
Added: branches/release-0.93/tests/language-behaviour/math/011-divide-float.out.expected
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/011-divide-float.out.expected (rev 0)
+++ branches/release-0.93/tests/language-behaviour/math/011-divide-float.out.expected 2011-08-08 16:33:46 UTC (rev 4964)
@@ -0,0 +1 @@
+0.3333333333333333
Deleted: branches/release-0.93/tests/language-behaviour/math/013-mod.check.sh
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/013-mod.check.sh 2011-08-08 03:38:30 UTC (rev 4963)
+++ branches/release-0.93/tests/language-behaviour/math/013-mod.check.sh 2011-08-08 16:33:46 UTC (rev 4964)
@@ -1,4 +0,0 @@
-#!/bin/bash
-set -x
-cat 013-mod.out | grep 19|| exit 1
-exit 0
Added: branches/release-0.93/tests/language-behaviour/math/013-mod.out.expected
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/013-mod.out.expected (rev 0)
+++ branches/release-0.93/tests/language-behaviour/math/013-mod.out.expected 2011-08-08 16:33:46 UTC (rev 4964)
@@ -0,0 +1 @@
+19
Deleted: branches/release-0.93/tests/language-behaviour/math/014-subtract.check.sh
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/014-subtract.check.sh 2011-08-08 03:38:30 UTC (rev 4963)
+++ branches/release-0.93/tests/language-behaviour/math/014-subtract.check.sh 2011-08-08 16:33:46 UTC (rev 4964)
@@ -1,4 +0,0 @@
-#!/bin/bash
-set -x
-cat 014-subtract.out | grep '\-37'|| exit 1
-exit 0
Added: branches/release-0.93/tests/language-behaviour/math/014-subtract.out.expected
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/014-subtract.out.expected (rev 0)
+++ branches/release-0.93/tests/language-behaviour/math/014-subtract.out.expected 2011-08-08 16:33:46 UTC (rev 4964)
@@ -0,0 +1 @@
+-37
Deleted: branches/release-0.93/tests/language-behaviour/math/0145-unary-subtact.check.sh
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/0145-unary-subtact.check.sh 2011-08-08 03:38:30 UTC (rev 4963)
+++ branches/release-0.93/tests/language-behaviour/math/0145-unary-subtact.check.sh 2011-08-08 16:33:46 UTC (rev 4964)
@@ -1,4 +0,0 @@
-#!/bin/bash
-set -x
-cat 0145-unary-subtact.out | grep '\-989929'|| exit 1
-exit 0
Added: branches/release-0.93/tests/language-behaviour/math/0145-unary-subtact.out.expected
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/0145-unary-subtact.out.expected (rev 0)
+++ branches/release-0.93/tests/language-behaviour/math/0145-unary-subtact.out.expected 2011-08-08 16:33:46 UTC (rev 4964)
@@ -0,0 +1 @@
+-989929
Deleted: branches/release-0.93/tests/language-behaviour/math/015-lesseq.check.sh
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/015-lesseq.check.sh 2011-08-08 03:38:30 UTC (rev 4963)
+++ branches/release-0.93/tests/language-behaviour/math/015-lesseq.check.sh 2011-08-08 16:33:46 UTC (rev 4964)
@@ -1,4 +0,0 @@
-#!/bin/bash
-set -x
-cat 015-lesseq.out | grep true|| exit 1
-exit 0
Added: branches/release-0.93/tests/language-behaviour/math/015-lesseq.out.expected
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/015-lesseq.out.expected (rev 0)
+++ branches/release-0.93/tests/language-behaviour/math/015-lesseq.out.expected 2011-08-08 16:33:46 UTC (rev 4964)
@@ -0,0 +1 @@
+true
Deleted: branches/release-0.93/tests/language-behaviour/math/016-greatereq.check.sh
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/016-greatereq.check.sh 2011-08-08 03:38:30 UTC (rev 4963)
+++ branches/release-0.93/tests/language-behaviour/math/016-greatereq.check.sh 2011-08-08 16:33:46 UTC (rev 4964)
@@ -1,4 +0,0 @@
-#!/bin/bash
-set -x
-cat 016-greatereq.out | grep false|| exit 1
-exit 0
Added: branches/release-0.93/tests/language-behaviour/math/016-greatereq.out.expected
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/016-greatereq.out.expected (rev 0)
+++ branches/release-0.93/tests/language-behaviour/math/016-greatereq.out.expected 2011-08-08 16:33:46 UTC (rev 4964)
@@ -0,0 +1 @@
+false
Deleted: branches/release-0.93/tests/language-behaviour/math/017-greater.check.sh
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/017-greater.check.sh 2011-08-08 03:38:30 UTC (rev 4963)
+++ branches/release-0.93/tests/language-behaviour/math/017-greater.check.sh 2011-08-08 16:33:46 UTC (rev 4964)
@@ -1,4 +0,0 @@
-#!/bin/bash
-set -x
-cat 017-greater.out | grep false|| exit 1
-exit 0
Added: branches/release-0.93/tests/language-behaviour/math/017-greater.out.expected
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/017-greater.out.expected (rev 0)
+++ branches/release-0.93/tests/language-behaviour/math/017-greater.out.expected 2011-08-08 16:33:46 UTC (rev 4964)
@@ -0,0 +1 @@
+false
Deleted: branches/release-0.93/tests/language-behaviour/math/018-less.check.sh
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/018-less.check.sh 2011-08-08 03:38:30 UTC (rev 4963)
+++ branches/release-0.93/tests/language-behaviour/math/018-less.check.sh 2011-08-08 16:33:46 UTC (rev 4964)
@@ -1,4 +0,0 @@
-#!/bin/bash
-set -x
-cat 018-less.out | grep true|| exit 1
-exit 0
Added: branches/release-0.93/tests/language-behaviour/math/018-less.out.expected
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/018-less.out.expected (rev 0)
+++ branches/release-0.93/tests/language-behaviour/math/018-less.out.expected 2011-08-08 16:33:46 UTC (rev 4964)
@@ -0,0 +1 @@
+true
Deleted: branches/release-0.93/tests/language-behaviour/math/019-equals.check.sh
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/019-equals.check.sh 2011-08-08 03:38:30 UTC (rev 4963)
+++ branches/release-0.93/tests/language-behaviour/math/019-equals.check.sh 2011-08-08 16:33:46 UTC (rev 4964)
@@ -1,4 +0,0 @@
-#!/bin/bash
-set -x
-cat 019-equals.out | grep false|| exit 1
-exit 0
Added: branches/release-0.93/tests/language-behaviour/math/019-equals.out.expected
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/019-equals.out.expected (rev 0)
+++ branches/release-0.93/tests/language-behaviour/math/019-equals.out.expected 2011-08-08 16:33:46 UTC (rev 4964)
@@ -0,0 +1 @@
+false
Deleted: branches/release-0.93/tests/language-behaviour/math/0191-not-equals.check.sh
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/0191-not-equals.check.sh 2011-08-08 03:38:30 UTC (rev 4963)
+++ branches/release-0.93/tests/language-behaviour/math/0191-not-equals.check.sh 2011-08-08 16:33:46 UTC (rev 4964)
@@ -1,4 +0,0 @@
-#!/bin/bash
-set -x
-cat 0191-not-equals.out | grep true || exit 1
-exit 0
Added: branches/release-0.93/tests/language-behaviour/math/0191-not-equals.out.expected
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/0191-not-equals.out.expected (rev 0)
+++ branches/release-0.93/tests/language-behaviour/math/0191-not-equals.out.expected 2011-08-08 16:33:46 UTC (rev 4964)
@@ -0,0 +1 @@
+true
Deleted: branches/release-0.93/tests/language-behaviour/math/030-mix-float-int.check.sh
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/030-mix-float-int.check.sh 2011-08-08 03:38:30 UTC (rev 4963)
+++ branches/release-0.93/tests/language-behaviour/math/030-mix-float-int.check.sh 2011-08-08 16:33:46 UTC (rev 4964)
@@ -1,4 +0,0 @@
-#!/bin/bash
-set -x
-cat 030-mix-float-int.out | grep 54.3|| exit 1
-exit 0
Added: branches/release-0.93/tests/language-behaviour/math/030-mix-float-int.out.expected
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/030-mix-float-int.out.expected (rev 0)
+++ branches/release-0.93/tests/language-behaviour/math/030-mix-float-int.out.expected 2011-08-08 16:33:46 UTC (rev 4964)
@@ -0,0 +1 @@
+54.3
Deleted: branches/release-0.93/tests/language-behaviour/math/031-add-float.check.sh
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/031-add-float.check.sh 2011-08-08 03:38:30 UTC (rev 4963)
+++ branches/release-0.93/tests/language-behaviour/math/031-add-float.check.sh 2011-08-08 16:33:46 UTC (rev 4964)
@@ -1,4 +0,0 @@
-#!/bin/bash
-set -x
-cat 031-add-float.out | grep 135.3|| exit 1
-exit 0
Added: branches/release-0.93/tests/language-behaviour/math/031-add-float.out.expected
===================================================================
--- branches/release-0.93/tests/language-behaviour/math/031-add-float.out.expected (rev 0)
+++ branches/release-0.93/tests/language-behaviour/math/031-add-float.out.expected 2011-08-08 16:33:46 UTC (rev 4964)
@@ -0,0 +1 @@
+135.3
Modified: branches/release-0.93/tests/suite.sh
===================================================================
--- branches/release-0.93/tests/suite.sh 2011-08-08 03:38:30 UTC (rev 4963)
+++ branches/release-0.93/tests/suite.sh 2011-08-08 16:33:46 UTC (rev 4964)
@@ -1025,7 +1025,7 @@
PIECES=""
while [ "$G" != "$CRTDIR" ]; do
PIECE=`basename $G`
- PIECES="$PIECE $PIECES"
+ PIECES="$PIECE/$PIECES"
G=`dirname $G`
done
echo $PIECES
From wozniak at ci.uchicago.edu Mon Aug 8 17:04:52 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Mon, 8 Aug 2011 17:04:52 -0500 (CDT)
Subject: [Swift-commit] r4965 - usertools/plotter/src/plotter
Message-ID: <20110808220452.9EBDE9CC9A@svn.ci.uchicago.edu>
Author: wozniak
Date: 2011-08-08 17:04:52 -0500 (Mon, 08 Aug 2011)
New Revision: 4965
Modified:
usertools/plotter/src/plotter/Lines.java
Log:
Enable logarithmic axes
Modified: usertools/plotter/src/plotter/Lines.java
===================================================================
--- usertools/plotter/src/plotter/Lines.java 2011-08-08 16:33:46 UTC (rev 4964)
+++ usertools/plotter/src/plotter/Lines.java 2011-08-08 22:04:52 UTC (rev 4965)
@@ -12,6 +12,7 @@
import org.apache.xmlgraphics.java2d.GraphicContext;
import org.jfree.chart.ChartFactory;
import org.jfree.chart.JFreeChart;
+import org.jfree.chart.axis.LogarithmicAxis;
import org.jfree.chart.axis.NumberAxis;
import org.jfree.chart.plot.PlotOrientation;
import org.jfree.chart.plot.XYPlot;
@@ -28,9 +29,16 @@
{
static Properties properties;
- // defualt this is not a Black and White plot
+ static String title = null;
+ static String xlabel = "x";
+ static String ylabel = "y";
+
+ /** If true, use only black and white */
public static boolean bw = false;
+ static String axis_x_type = "normal";
+ static String axis_y_type = "normal";
+
static int width = 400;
static int height = 400;
@@ -74,9 +82,6 @@
for (int i = 2; i < args.length; i++)
names.add(args[i]);
- String title = null;
- String xlabel = null;
- String ylabel = null;
List data = new ArrayList();
List labels = new ArrayList();
@@ -165,8 +170,6 @@
(title, xlabel, ylabel, collection,
PlotOrientation.VERTICAL, withLegend, false, false);
-
-
setupPlot(chart, collection);
chart.draw(g2d, rectangle);
@@ -207,7 +210,7 @@
catch (IndexOutOfBoundsException e)
{}
- Util.verbose( "label: "+label );
+ Util.verbose("label: "+label);
XYSeries series = new XYSeries(label);
for (int i = 0; i < d.length; i++)
series.add(d[i][0], d[i][1]);
@@ -242,6 +245,8 @@
static void setAxes(XYPlot plot)
{
+ setAxisTypes(plot);
+
// Actual values: modify if necessary
double axmin, axmax, aymin, aymax;
if (xmin != null || xmax != null)
@@ -268,6 +273,27 @@
}
+ private static void setAxisTypes(XYPlot plot)
+ {
+ if (axis_x_type.equals("logarithmic"))
+ {
+ NumberAxis domainAxis = new LogarithmicAxis(xlabel);
+ plot.setDomainAxis(domainAxis);
+ }
+ else if (!axis_x_type.equals("normal"))
+ throw new RuntimeException
+ ("Invalid axis.x type: " + axis_x_type);
+
+ if (axis_y_type.equals("logarithmic"))
+ {
+ NumberAxis rangeAxis = new LogarithmicAxis(ylabel);
+ plot.setRangeAxis(rangeAxis);
+ }
+ else if (!axis_y_type.equals("normal"))
+ throw new RuntimeException
+ ("Invalid axis.y type: " + axis_y_type);
+ }
+
/**
Various plot properties. All are currently optional
@@ -314,6 +340,12 @@
tmp = properties.getProperty("legend.enabled");
if (tmp != null)
withLegend = Boolean.parseBoolean(tmp);
+ tmp = properties.getProperty("axis.x");
+ if (tmp != null)
+ axis_x_type = tmp;
+ tmp = properties.getProperty("axis.y");
+ if (tmp != null)
+ axis_y_type = tmp;
}
static void load(String propFile)
From hategan at ci.uchicago.edu Tue Aug 9 04:33:49 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Tue, 9 Aug 2011 04:33:49 -0500 (CDT)
Subject: [Swift-commit] r4966 -
branches/release-0.93/src/org/griphyn/vdl/mapping
Message-ID: <20110809093349.811499CCB5@svn.ci.uchicago.edu>
Author: hategan
Date: 2011-08-09 04:33:48 -0500 (Tue, 09 Aug 2011)
New Revision: 4966
Modified:
branches/release-0.93/src/org/griphyn/vdl/mapping/AbstractDataNode.java
branches/release-0.93/src/org/griphyn/vdl/mapping/RootArrayDataNode.java
branches/release-0.93/src/org/griphyn/vdl/mapping/RootDataNode.java
Log:
fix npe when provenance logging is enabled
Modified: branches/release-0.93/src/org/griphyn/vdl/mapping/AbstractDataNode.java
===================================================================
--- branches/release-0.93/src/org/griphyn/vdl/mapping/AbstractDataNode.java 2011-08-08 22:04:52 UTC (rev 4965)
+++ branches/release-0.93/src/org/griphyn/vdl/mapping/AbstractDataNode.java 2011-08-09 09:33:48 UTC (rev 4966)
@@ -454,14 +454,8 @@
}
}
- Mapper m;
+ Mapper m = getActualMapper();
- try {
- m = this.getMapper();
- }
- catch (FutureFault fe) {
- m = null; // no mapping info if mapper isn't initialised yet
- }
if (m != null) {
// TODO proper type here
// Not sure catching exception here is really the right thing to
@@ -518,6 +512,10 @@
}
}
}
+
+ protected Mapper getActualMapper() {
+ return null;
+ }
public boolean isClosed() {
return closed;
Modified: branches/release-0.93/src/org/griphyn/vdl/mapping/RootArrayDataNode.java
===================================================================
--- branches/release-0.93/src/org/griphyn/vdl/mapping/RootArrayDataNode.java 2011-08-08 22:04:52 UTC (rev 4965)
+++ branches/release-0.93/src/org/griphyn/vdl/mapping/RootArrayDataNode.java 2011-08-09 09:33:48 UTC (rev 4966)
@@ -102,6 +102,10 @@
assert(waitingMapperParam != null);
throw new FutureNotYetAvailable(waitingMapperParam.getFutureWrapper());
}
+
+ protected Mapper getActualMapper() {
+ return mapper;
+ }
public boolean isArray() {
return true;
Modified: branches/release-0.93/src/org/griphyn/vdl/mapping/RootDataNode.java
===================================================================
--- branches/release-0.93/src/org/griphyn/vdl/mapping/RootDataNode.java 2011-08-08 22:04:52 UTC (rev 4965)
+++ branches/release-0.93/src/org/griphyn/vdl/mapping/RootDataNode.java 2011-08-09 09:33:48 UTC (rev 4966)
@@ -212,6 +212,10 @@
assert (waitingMapperParam != null);
throw new FutureNotYetAvailable(waitingMapperParam.getFutureWrapper());
}
+
+ protected Mapper getActualMapper() {
+ return mapper;
+ }
public boolean isArray() {
return false;
From davidk at ci.uchicago.edu Tue Aug 9 06:16:34 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Tue, 9 Aug 2011 06:16:34 -0500 (CDT)
Subject: [Swift-commit] r4967 - trunk/docs/siteguide
Message-ID: <20110809111634.61A169CCBF@svn.ci.uchicago.edu>
Author: davidk
Date: 2011-08-09 06:16:34 -0500 (Tue, 09 Aug 2011)
New Revision: 4967
Removed:
trunk/docs/siteguide/siteguide.html
Modified:
trunk/docs/siteguide/pads
Log:
Fixed broken path
Modified: trunk/docs/siteguide/pads
===================================================================
--- trunk/docs/siteguide/pads 2011-08-09 09:33:48 UTC (rev 4966)
+++ trunk/docs/siteguide/pads 2011-08-09 11:16:34 UTC (rev 4967)
@@ -165,7 +165,7 @@
The swift script we will run is called catsn.swift. It simply cats a file and saves the result. This is a nice simple test to ensure jobs are running correctly. Create a file called data.txt which contains some simple input - a "hello world" will do the trick.
-----
-include::../../examples/catsn.swift[]
+include::../../examples/misc/catsn.swift[]
-----
Running Swift
Deleted: trunk/docs/siteguide/siteguide.html
===================================================================
--- trunk/docs/siteguide/siteguide.html 2011-08-09 09:33:48 UTC (rev 4966)
+++ trunk/docs/siteguide/siteguide.html 2011-08-09 11:16:34 UTC (rev 4967)
@@ -1,1160 +0,0 @@
-
-
-
-
-
-Site Configuration Guide
-
-
-
-
-
-
Site Configuration Guide
-
-
Table of Contents
-
-
-
-
-
-
1. PADS
-
-
-
1.1. Requesting Access
-
If you do not already have a Computation Institute account, you can request
-one at https://www.ci.uchicago.edu/accounts/. This page will give you a list
-of resources you can request access to. Be sure that PADS is selected. If
-you already have an existing CI account, but do not have access to PADS,
-send an email to support at ci.uchicago.edu to request access.
Once your keys are configured, you should be able to access a PADS login
-node with the following command:
-
-
-
ssh yourusername at login.pads.ci.uchicago.edu
-
-
-
-
1.4. Configuring softenv
-
Softenv is a system used for managing applications. In order to run Swift,
-the softenv environment will have to be modified slightly. Softenv is
-configured by a file in your home directory called .soft. Edit this file
-to look like this:
PADS requires that you are a member of a project. You can determine this by
-running the following command:
-
-
-
$ projects --available
-
-The following projects are available for your use
-
-Project PI Title
-
-CI-CCR000013 Michael Wilde The Swift Parallel Scripting System
You should make sure that you have a project set as default. Run
-the projects command with no arguments to determine if you have a default.
-
-
-
$ projects
-You have no default project set.
-
-
To set your default project, use projects --set
-
-
-
$ projects --set CI-CCR000013 --all
-Your default project for all CI clusters has been set to CI-CCR000013.
-
-
-
-
1.7.3. Determine your Queue
-
PADS has several different queues you can submit jobs to depending on
-the type of work you will be doing. The command "qstat -q" will print
-the most up to date list of this information.
-
-
-
Table 1. PADS Queues
-
-
-
-
-
-
-
-
-
-
-
-
Queue
-
Memory
-
CPU Time
-
Walltime
-
Node
-
Run
-
Que
-
Lm
-
State
-
-
-
-
-
route
-
—
-
—
-
—
-
—
-
0
-
0
-
—
-
E R
-
-
-
short
-
—
-
—
-
04:00:00
-
—
-
64
-
0
-
—
-
E R
-
-
-
extended
-
—
-
—
-
—
-
—
-
0
-
0
-
—
-
E R
-
-
-
fast
-
—
-
—
-
01:00:00
-
1
-
0
-
152
-
—
-
E R
-
-
-
long
-
—
-
—
-
24:00:00
-
—
-
232
-
130
-
—
-
E R
-
-
-
-
-
-
-
-
1.8. Generating Configuration Files
-
Now that you know what queue to use, your project, and your work directory, it is time to
-set up Swift. Swift uses a configuration file called sites.xml to determine how it should run.
-There are two methods you can use for creating this file. You can manually edit
-the configuration file, or generate it with a utility called gensites.
-
-
1.8.1. Manually Editing sites.xml
-
Below is the template that is used by Swift’s test suite for running on PADS.
The values to note here are the ones that are listed between underscores. In the example above, they are QUEUE, and WORK. Queue is the PADS queue to use and WORK is the swift work directory. These are placeholder values you will need to modify to fit your needs. Copy and paste this template, replace the values, and call it sites.xml.
-
-
-
-
1.9. Manually Editing tc.data
-
Below is the tc.data file used by Swift’s test suite for running on PADS.
The swift script we will run is called catsn.swift. It simply cats a file and saves the result. This is a nice simple test to ensure jobs are running correctly. Create a file called data.txt which contains some simple input - a "hello world" will do the trick.
-
-
-
-
-
-
-
1.11. Running Swift
-
Now that everything is in place, run Swift with the following command:
-
-
-
swift -sites.file sites.xml -tc.file tc.data catsn.swift -n=10
-
-
You should see several new files being created, called catsn.0001.out, catsn.0002.out, etc. Each of these
-files should contain the contents of what you placed into data.txt. If this happens, your job has run
-successfully on PADS!
-
-
-
1.12. More Help
-
The best place for additional help is the Swift user mailing list. You can subscribe to this list at
-https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user. When submitting information, please send your sites.xml file, your tc.data, and any Swift log files that were created during your attempt.
-
-
-
-
-
2. Beagle
-
-
-
2.1. Requesting Access
-
If you do not already have a Computation Institute account, you can request
-one at https://www.ci.uchicago.edu/accounts/. This page will give you a list
-of resources you can request access to.
-You already have an existing CI account, but do not have access to Beagle,
-send an email to support at ci.uchicago.edu to request access.
-
-
-
2.2. Connecting to a login node
-
Once you have account, you should be able to access a Beagle login
-node with the following command:
-
-
-
ssh yourusername at login.beagle.ci.uchicago.edu
-
-
Follow the steps outlined below to get started with Swift on Beagle:
-
step 1. Load the Swift module on Beagle as follows: module load swift
-
step 2. Create and change to a directory where your Swift related work will
-stay. (say, mkdir swift-lab, followed by, cd swift-lab)
-
step 3. To get started with a simple example running /bin/cat to read an
-input file data.txt and write to an output file f.nnn.out, copy the folder
-at /home/ketan/catsn to the above directory. (cp -r /home/ketan/catsn
-. followed by cd catsn).
-
step 4. In the sites file: beagle-coaster.xml, make the following two
-changes: 1) change the path of workdirectory to your preferred location
-(say to /lustre/beagle/$USER/swift-lab/swift.workdir) and 2) Change the
-project name to your project (CI-CCR000013) . The workdirectory will contain
-execution data related to each run, e.g. wrapper scripts, system information,
-inputs and outputs.
-
step 5. Run the example using following commandline (also found in run.sh):
-swift -config cf -tc.file tc -sites.file beagle-coaster.xml catsn.swift -n=1
-. You can further change the value of -n to any arbitrary number to run that
-many number of concurrent cat
-
step 6. Check the output in the generated outdir directory (ls outdir)
-
Note: Running from sandbox node or requesting 1 hour walltime for upto 3 nodes
-will get fast prioritized execution. Good for small tests.
If you do not already have a futuregrid account, you can follow the
-instructions at https://portal.futuregrid.org/gettingstarted to get started.
-This page provides information on how to create an account, how to join
-a project, how to set up your SSH keys, and how to create a new project.
-
-
-
3.4. Downloading Swift VM Tools
-
A set of scripts based around cloudinitd are used to easily start virtual
-machines. To download, change to your home directory and run the
-following command:
-
-
-
$ svn co https://svn.ci.uchicago.edu/svn/vdl2/usertools/swift-vm-boot
-
-
-
-
3.5. Download your Credentials
-
Run the following commands to retrieve your credentials:
-
-
-
$ scp yourusername at hotel.futuregrid.org:nimbus_creds.tar.gz .
-$ tar xvfz nimbus_creds.tar.gz
-
-
When you extract your credential file, look at the file called
-hotel.conf. Near the bottom of this file will be two settings
-called vws.repository.s3id and vws.repository.s3key. Copy these
-values for the next step.
-
-
-
3.6. Configuring Swift VM Tools
-
Change directories into the swift-vm-boot directory. Edit the file
-called env.sh. There are two settings here that you will need to
-modify: FUTUREGRID_IAAS_ACCESS_KEY and FUTUREGRID_IAAS_SECRET_KEY.
-Paste your access key (s3id) and secret key (s3key) here and save
-the file. By default, env.sh requests 2 nodes on hotel and 2 nodes
-on sierra. Change these values as needed. Run install.sh from the
-swift-vm-boot directory to complete the installation of Swift VM
-Tools. The Swift VM Tools may require a fairly recent version of
-Python. If you run into problems while running the install.sh
-script, please try a more recent version of Python and associated
-libraries.
-
-
-
3.7. Configuring coaster-service.conf
-
Within your swift distribution’s etc directory is a file called
-coaster-service.conf. There are many options here you can modify
-as needed, but these are the settings that will be required
-to run on futuregrid:
These is a brief description of these settings in the coaster-service.conf file.
-Tunneling should be used when you are restricted by a firewall. If your local
-machine has multiple network interfaces, you should also manually set the IP
-address of your machine with export IPADDR=your.ip.address.here.
-
-
-
3.8. Starting the Coaster Service Script
-
Now that everything is configured, change to a temporary directory
-and run this command to start the coaster service:
-
-
-
$ start-coaster-service
-
-
This command will start the VMs, start the required processes on the worker nodes,
-and generate Swift configuration files for you to use. The configuration files
-will be generated in your current directory. These files are sites.xml, tc.data,
-and cf.
-
-
-
3.9. Running Swift
-
Now that you have all of your configuration files generated, run the following command:
-
-
-
$ swift -sites.file sites.xml -tc.file tc.data -config cf <yourscript.swift>
-
-
If you would like to create a custom tc file for repeated use, rename it to something other
-than tc.data to prevent it from being overwritten. The sites.xml however will need to be
-regenerated every time you start the coaster service. If you need to repeatedly modify some
-sites.xml options, you may edit the template in Swift’s etc/sites/persistent-coasters. You
-may also create your own custom tc files with the hostname of persistent-coasters. More
-information about this can be found in the Swift userguide at
-http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html.
-
-
-
3.10. Stopping the Coaster Service Script
-
To stop the coaster service, run the following command:
-
-
-
$ stop-coaster-service
-
-
This will kill the coaster service, kill the worker scripts on remote systems and terminate
-the virtual machines that were created during start-coaster-service.
-
-
-
3.11. More Help
-
The best place for additional help is the Swift user mailing list. You can subscribe to this list at
-http://mail.ci.uchicago.edu/mailman/listinfo/swift-user. When submitting information, please send
-your sites.xml file, your tc.data, and any error messages you run into.
-
-
-
-
-
4. Grids, including OSG and TeraGrid
-
-
-
4.1. Overview of running on grid sites
-
-
-
-Get a DOEGrids cert. Then register it in the OSG Engage VO, and/or map it using gx-request on TeraGrid sites.
-
-
-
-
-Run GridSetup to configure Swift to use the grid sites. This tests
-for correct operation and creates a "green list" of good sites.
-
-
-
-
-Prepare an installation package for the programs you want to run on
-grid sites via Swift, and install that package using foreachsite.
-
-
-
-
-Run RunWorkers to start and maintain a pool of Swift workers on
-each site.
-
-
-
-
-Run Swift scripts that use the grid site resources.
-
-
-
-
-
-
4.2. Requesting Access
-
For OSG: Obtain a DOEGrids certificate and register the certificate in
-the OSG "Engage" VO following the procedures at:
FIXME: access to OSG wiki pages may request the user to present
-a certificate. Is this a problem from users without one? If so, make a
-copy of the page on the Swift web.
-
For TeraGrid: Obtain a DOEGrids certifcate using the OSG ENgage
-instructions above. Ask a TeraGrid PI to add you to a TeraGrid
-project. Once you obtain a login and project access (via US Mail), use
-gx-request to add your certificate
-
To run jobs using the procedures documented here you need to be logged
-in to a "submit host" on which you will run Swift and other
-grid-related utilities. This can be any host with the OSG client stack
-or equivalent tools installed. Such hosts include the OSG Engage
-submit host, and the two Swift lab servers
-{bridled,communicado}.ci.uchicago.edu.
-
Obtain a login on engage-submit.renci.org following instructions on
-the OSG URL above.
-
Obtain a CI login with access to the Swift lab servers by requesting
-"OSG Gridlab" access at:
cd $HOME
-mkdir swiftgrid
-cd swiftgrid
-gen_gridsites
-# Wait a few minutes to a few hours for Swift to validate grid sites
-get_greensites >greensites
-
-
-
-
4.8. Running Swift
-
Now that everything is in place, run Swift with the following command:
-
-
-
swift -sites.file sites.xml -tc.file tc.data catsn.swift -n=10
-
-
You should see several new files being created, called catsn.0001.out, catsn.0002.out, etc. Each of these
-files should contain the contents of what you placed into data.txt. If this happens, your job has run
-successfully on PADS!
-
-
-
4.9. More Help
-
The best place for additional help is the Swift user mailing list. You can subscribe to this list at
-http://mail.ci.uchicago.edu/mailman/listinfo/swift-user. When submitting information, please send your sites.xml file, your tc.data, and any Swift log files that were created during your attempt.
-
-
-
-
-
-
-
-
From lgadelha at ci.uchicago.edu Tue Aug 9 08:16:30 2011
From: lgadelha at ci.uchicago.edu (lgadelha at ci.uchicago.edu)
Date: Tue, 9 Aug 2011 08:16:30 -0500 (CDT)
Subject: [Swift-commit] r4968 - provenancedb
Message-ID: <20110809131630.0D5C49CCBF@svn.ci.uchicago.edu>
Author: lgadelha
Date: 2011-08-09 08:16:29 -0500 (Tue, 09 Aug 2011)
New Revision: 4968
Added:
provenancedb/ProvSQL.g
provenancedb/compare_run.py
provenancedb/compare_run.sh
provenancedb/import-run-to-datalog
provenancedb/prov-to-datalog.sh
provenancedb/provenancedb-rules.datalog
provenancedb/swift-prov-import-all-logs-datalog
Removed:
provenancedb/prov-to-swipl.sh
provenancedb/xq.xq
Modified:
provenancedb/pql_functions.sql
provenancedb/prov-init.sql
provenancedb/prov-to-sql.sh
provenancedb/swift-prov-import-all-logs
Log:
Added ProvSQL grammar. Added query templates for comparison/correlation query patterns. Added Datalog import for experimentation.
Added: provenancedb/ProvSQL.g
===================================================================
--- provenancedb/ProvSQL.g (rev 0)
+++ provenancedb/ProvSQL.g 2011-08-09 13:16:29 UTC (rev 4968)
@@ -0,0 +1,309 @@
+grammar ProvSQL;
+
+ at header {
+ import java.util.HashSet;
+ import java.util.HashMap;
+ import java.util.Iterator;
+ import org.jgrapht.*;
+ import org.jgrapht.alg.DijkstraShortestPath;
+ import org.jgrapht.graph.*;
+}
+
+ at members {
+ String selectClause = new String();
+ String fromClause = new String();
+ String whereClauseJoinExpressions = new String();
+ String whereClause = new String();
+ boolean hasWhereJoinExpression;
+ HashSet relations = new HashSet();
+ UndirectedGraph schemaGraph;
+ HashSet joinEdges;
+
+ // Ideally it could receive a DB schema in SQL and build the graph automatically
+ public static UndirectedGraph buildGraph() {
+ UndirectedGraph schemaGraph = new Multigraph(DefaultEdge.class);
+
+ schemaGraph.addVertex("a_run_t");
+ schemaGraph.addVertex("a_run_n");
+ schemaGraph.addVertex("run");
+ schemaGraph.addVertex("proc");
+ schemaGraph.addVertex("a_proc_n");
+ schemaGraph.addVertex("a_proc_t");
+ schemaGraph.addVertex("app_inv");
+ schemaGraph.addVertex("app_exec");
+ schemaGraph.addVertex("rt_info");
+ schemaGraph.addVertex("ds_in");
+ schemaGraph.addVertex("ds_out");
+ schemaGraph.addVertex("ds");
+ schemaGraph.addVertex("file");
+ schemaGraph.addVertex("in_mem");
+ schemaGraph.addVertex("a_ds_t");
+ schemaGraph.addVertex("a_ds_n");
+ schemaGraph.addVertex("ds_cont");
+ schemaGraph.addEdge("a_run_t", "run");
+ schemaGraph.addEdge("a_run_n", "run");
+ schemaGraph.addEdge("run","proc");
+ schemaGraph.addEdge("proc", "a_proc_t");
+ schemaGraph.addEdge("proc", "a_proc_n");
+ schemaGraph.addEdge("proc", "ds_out");
+ schemaGraph.addEdge("proc", "ds_in");
+ schemaGraph.addEdge("proc", "app_inv");
+ schemaGraph.addEdge("app_inv", "app_exec");
+ schemaGraph.addEdge("app_exec", "rt_info");
+ schemaGraph.addEdge("ds", "ds_in");
+ schemaGraph.addEdge("ds", "ds_out");
+ schemaGraph.addEdge("ds", "a_ds_t");
+ schemaGraph.addEdge("ds", "a_ds_n");
+ schemaGraph.addEdge("ds", "file");
+ schemaGraph.addEdge("ds", "in_mem");
+ schemaGraph.addEdge("ds", "ds_cont");
+ schemaGraph.addEdge("ds", "ds_cont");
+
+ return schemaGraph;
+ }
+
+ private static HashSet computeJoinEdges(
+ UndirectedGraph schemaGraph,
+ HashSet relations) {
+ HashSet jEdges = new HashSet();
+ Iterator i = relations.iterator();
+ String first = new String();
+ if(i.hasNext())
+ first += i.next();
+ while(i.hasNext()) {
+ DijkstraShortestPath sP = new DijkstraShortestPath(schemaGraph, first, i.next());
+ Iterator j = (sP.getPathEdgeList()).iterator();
+ while(j.hasNext())
+ jEdges.add(j.next());
+ }
+
+
+ return jEdges;
+ }
+
+
+ public static String computeFrom(UndirectedGraph schemaGraph, HashSet joinEdges, HashSet qrels) {
+ HashSet fromRels = new HashSet();
+ String fromq = " FROM ";
+ Iterator i = joinEdges.iterator();
+ while(i.hasNext()) {
+ DefaultEdge aux = i.next();
+ // If ds_in or ds_out were not in the original select clause's relations and they are on the the joinEdges
+ // then one has to make sure that both consumed and produced datasets are considered in the join so there
+ // is no loss of information. One alternative, implemented here, is to replace these occurrences by the ds
+ // view, which is an union of ds_in and ds_out.
+ if(qrels.contains("ds_in") || qrels.contains("ds_out")) {
+ fromRels.add(schemaGraph.getEdgeSource(aux));
+ fromRels.add(schemaGraph.getEdgeTarget(aux));
+ }
+ else {
+ if(aux.equals(schemaGraph.getEdge("ds_in","proc")) ||
+ aux.equals(schemaGraph.getEdge("ds_in","ds")) ||
+ aux.equals(schemaGraph.getEdge("ds_out","proc")) ||
+ aux.equals(schemaGraph.getEdge("ds_out","ds"))) {
+ fromRels.add("ds");
+ fromRels.add("ds_use");
+ fromRels.add("proc");
+ }
+ else {
+ fromRels.add(schemaGraph.getEdgeSource(aux));
+ fromRels.add(schemaGraph.getEdgeTarget(aux));
+ }
+
+ }
+ }
+ Iterator j = fromRels.iterator();
+ if(j.hasNext())
+ fromq += j.next();
+ while(j.hasNext())
+ fromq+=","+j.next();
+
+ return fromq;
+ }
+
+
+ public static String computeJoinExpressions(UndirectedGraph schemaGraph, HashSet jEdges, HashSet qrels) {
+
+ HashMap joinExpressions = new HashMap();
+ String joinExpressionsString = new String();
+
+ joinExpressions.put(schemaGraph.getEdge("a_run_t", "run"), "a_run_t.run_id=run.id");
+ joinExpressions.put(schemaGraph.getEdge("a_run_n", "run"), "a_run_n.run_id=run.id");
+ joinExpressions.put(schemaGraph.getEdge("run", "proc"), "run.id=proc.run_id");
+ joinExpressions.put(schemaGraph.getEdge("proc", "a_proc_t"), "proc.id=a_proc_t.proc_id");
+ joinExpressions.put(schemaGraph.getEdge("proc", "a_proc_n"), "proc.id=a_proc_n.proc_id");
+ joinExpressions.put(schemaGraph.getEdge("proc", "ds_out"), "proc.id=ds_out.proc_id");
+ joinExpressions.put(schemaGraph.getEdge("proc", "ds_in"), "proc.id=ds_in.proc_id");
+ joinExpressions.put(schemaGraph.getEdge("proc", "app_inv"), "proc.id=app_inv.id");
+ joinExpressions.put(schemaGraph.getEdge("app_inv", "app_exec"), "app_inv.id=app_exec.app_inv_id");
+ joinExpressions.put(schemaGraph.getEdge("app_exec", "rt_info"), "app_exec.id=rt_info.app_exec_id");
+ joinExpressions.put(schemaGraph.getEdge("ds", "ds_in"), "ds.id=ds_in.id");
+ joinExpressions.put(schemaGraph.getEdge("ds", "ds_out"), "ds.id=ds_out.id");
+ joinExpressions.put(schemaGraph.getEdge("ds", "a_ds_t"), "ds.id=a_ds_t.ds_id");
+ joinExpressions.put(schemaGraph.getEdge("ds", "a_ds_n"), "ds.id=a_ds_n.ds_id");
+ joinExpressions.put(schemaGraph.getEdge("ds", "file"), "ds.id=file.id");
+ joinExpressions.put(schemaGraph.getEdge("ds", "in_mem"), "ds.id=in_mem.id");
+ joinExpressions.put(schemaGraph.getEdge("ds", "ds_cont"), "ds.id=ds_cont.in_id");
+ joinExpressions.put(schemaGraph.getEdge("ds", "ds_cont"), "ds.id=ds_cont.out_id");
+
+ Iterator i = jEdges.iterator();
+ if(i.hasNext()) {
+ DefaultEdge aux = i.next();
+ if(qrels.contains("ds_in") || qrels.contains("ds_out")) {
+ joinExpressionsString = joinExpressions.get(aux);
+ }
+ else {
+ if(aux.equals(schemaGraph.getEdge("ds_in","proc")) || aux.equals(schemaGraph.getEdge("ds_out","proc")))
+ joinExpressionsString = "ds_use.proc_id=proc.id";
+ else if(aux.equals(schemaGraph.getEdge("ds_in","ds")) || aux.equals(schemaGraph.getEdge("ds_out","ds")))
+ joinExpressionsString = "ds_use.ds_id=ds.id";
+ else {
+ joinExpressionsString = joinExpressions.get(aux);
+ }
+
+ }
+ }
+
+
+ while(i.hasNext()) {
+ DefaultEdge aux = i.next();
+ if(qrels.contains("ds_in") || qrels.contains("ds_out")) {
+ joinExpressionsString += " AND " + joinExpressions.get(aux);
+ }
+ else {
+ if(aux.equals(schemaGraph.getEdge("ds_in","proc")) || aux.equals(schemaGraph.getEdge("ds_out","proc")))
+ joinExpressionsString += " AND " + "ds_use.proc_id=proc.id";
+ else if(aux.equals(schemaGraph.getEdge("ds_in","ds")) || aux.equals(schemaGraph.getEdge("ds_out","ds")))
+ joinExpressionsString += " AND " + "ds_use.ds_id=ds.id";
+ else {
+ joinExpressionsString += " AND " + joinExpressions.get(aux);
+ }
+
+ }
+ }
+
+ return joinExpressionsString;
+ }
+
+}
+
+query : SELECT selectExpression
+ {
+ schemaGraph = buildGraph();
+ joinEdges = computeJoinEdges(schemaGraph, relations);
+ hasWhereJoinExpression=true;
+
+ System.out.print("SELECT " + selectClause);
+ fromClause += computeFrom(schemaGraph, joinEdges, relations);
+
+ System.out.print(fromClause);
+
+ whereClauseJoinExpressions += computeJoinExpressions(schemaGraph, joinEdges, relations);
+
+ if(!whereClauseJoinExpressions.isEmpty()) {
+ hasWhereJoinExpression=true;
+ System.out.print(" WHERE " + whereClauseJoinExpressions);
+ }
+
+ }
+ (WHERE whereExpression
+ {
+ if(hasWhereJoinExpression)
+ System.out.print(",");
+ else
+ System.out.print(" WHERE ");
+ System.out.print(whereClause);
+ }
+ )?
+ SEMICOLON
+ {
+ System.out.print(";");
+ }
+ ;
+
+selectExpression
+ : a=entityAttribute
+ {
+ selectClause += $a.text;
+ relations.add($a.text.split("\\.")[0]);
+ if($a.text.split("\\.").length == 1)
+ selectClause += ".*";
+ }
+ (COLON b=entityAttribute
+ {
+ selectClause += "," + $b.text;
+ relations.add($b.text.split("\\.")[0]);
+ if($b.text.split("\\.").length == 1)
+ selectClause += ".*";
+ }
+ )*
+ ;
+
+whereExpression
+ : c=whereAtom
+ {
+ whereClause += $c.text;
+ }
+ (
+ (AND
+ {
+ whereClause += " AND ";
+ }
+ | OR
+ {
+ whereClause += " OR ";
+ }
+ ) d=whereAtom
+ {
+ whereClause += $d.text;
+ }
+ )*
+ ;
+
+whereAtom
+ : entityAttribute OP (STRING | INT | FLOAT)
+ | entityAttribute BETWEEN STRING AND STRING;
+
+entityAttribute : ID (DOT ID)?;
+
+OP : '=' | '>' | '>=' | '<' | '<=';
+
+SELECT : 's' 'e' 'l' 'e' 'c' 't';
+
+WHERE : 'w' 'h' 'e' 'r' 'e';
+
+AND : 'a' 'n' 'd';
+
+OR : 'o' 'r';
+
+DOT : '.';
+
+COLON : ',';
+
+BETWEEN : 'b' 'e' 't' 'w' 'e' 'e' 'n';
+
+SEMICOLON : ';';
+
+ID : ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'0'..'9'|'_'|'-')*
+ ;
+
+INT : '0'..'9'+
+ ;
+
+FLOAT
+ : ('0'..'9')+ '.' ('0'..'9')*
+ | '.' ('0'..'9')+
+ | ('0'..'9')+
+ ;
+
+STRING
+ : '\'' ( 'a'..'z' | 'A'..'Z' | '_' | '-' | '0'..'9' | '.')* '\''
+ ;
+
+NEWLINE : '\r' ? '\n';
+
+WS : (' ' |'\t' |'\n' |'\r' )+
+ {
+ skip();
+ }
+ ;
Added: provenancedb/compare_run.py
===================================================================
--- provenancedb/compare_run.py (rev 0)
+++ provenancedb/compare_run.py 2011-08-09 13:16:29 UTC (rev 4968)
@@ -0,0 +1,27 @@
+#!/usr/bin/env python
+
+import sys
+selectClause = 'SELECT run_id'
+fromClause = 'FROM'
+nId = 0
+for arg in sys.argv:
+ argTokens = arg.partition('=')
+
+ if argTokens[0] == 'annot_num' or argTokens[0] == 'annot_txt' or argTokens[0] == 'param':
+ key = argTokens[2]
+ nId+=1
+ sId = 'j%s' % nId
+ selectClause += ', ' + sId + '.value as ' + key
+ if nId>1:
+ fromClause += ' INNER JOIN'
+ fromClause += ' compare_run_by_' + argTokens[0] + '(\'' + key + '\') as ' + sId
+ if nId>1:
+ fromClause += ' USING (run_id)'
+
+query = selectClause + ' ' + fromClause + ';'
+
+print query
+
+
+
+
Property changes on: provenancedb/compare_run.py
___________________________________________________________________
Added: svn:executable
+ *
Added: provenancedb/compare_run.sh
===================================================================
--- provenancedb/compare_run.sh (rev 0)
+++ provenancedb/compare_run.sh 2011-08-09 13:16:29 UTC (rev 4968)
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+PROVDIR=$(dirname $0)
+pushd $PROVDIR
+PROVDIR=$(pwd)
+popd
+
+source $PROVDIR/etc/provenance.config
+export PATH=$PROVDIR:$PATH
+
+# TODO: Check python's version, should be >=2.6
+echo "DROP VIEW temp;" | $SQLCMD
+temp_view=$(python $PROVDIR/compare_run.py $@)
+echo "CREATE VIEW temp AS " $temp_view | $SQLCMD
+echo "SELECT * FROM temp;" | $SQLCMD
\ No newline at end of file
Property changes on: provenancedb/compare_run.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: provenancedb/import-run-to-datalog
===================================================================
--- provenancedb/import-run-to-datalog (rev 0)
+++ provenancedb/import-run-to-datalog 2011-08-09 13:16:29 UTC (rev 4968)
@@ -0,0 +1,13 @@
+#!/bin/bash
+
+# this should be the main driver script which can be run right after a
+# workflow has finished and will do everything necessary to import all
+# of the provenance information into the SQL provenance database.
+
+# invoke with: import-run-to-sql [logfile]
+# where [logfile] is a full path *not relative* to the log file
+# with kickstart records expected to be in the same directory as the
+# log file.
+
+version=$version prov-to-datalog.sh $1
+
Property changes on: provenancedb/import-run-to-datalog
___________________________________________________________________
Added: svn:executable
+ *
Modified: provenancedb/pql_functions.sql
===================================================================
--- provenancedb/pql_functions.sql 2011-08-09 11:16:34 UTC (rev 4967)
+++ provenancedb/pql_functions.sql 2011-08-09 13:16:29 UTC (rev 4968)
@@ -5,7 +5,6 @@
-- lists variations in a parameter's value across workflows, for parameters that are in-memory variables
drop view in_mem_in cascade;
-
create view in_mem_in as
select proc.run_id, proc.id as proc_id,
proc.name as proc_name, ds_in.ds_id,
@@ -14,7 +13,6 @@
where proc.id=ds_in.proc_id and ds_in.ds_id=in_mem.id;
drop view in_mem_out cascade;
-
create view in_mem_out as
select proc.run_id, proc.id as proc_id,
proc.name as proc_name, ds_out.ds_id,
@@ -23,14 +21,12 @@
where proc.id=ds_out.proc_id and ds_out.ds_id=in_mem.id;
drop view in_mem_use cascade;
-
create view in_mem_use as
select * from in_mem_in
union
select * from in_mem_out;
drop view file_in cascade;
-
create view file_in as
select proc.run_id, proc.id as proc_id,
proc.name as proc_name, ds_in.ds_id,
@@ -39,7 +35,6 @@
where proc.id=ds_in.proc_id and ds_in.ds_id=file.id;
drop view file_out cascade;
-
create view file_out as
select proc.run_id, proc.id as proc_id,
proc.name as proc_name, ds_out.ds_id,
@@ -48,21 +43,18 @@
where proc.id=ds_out.proc_id and ds_out.ds_id=file.id;
drop view file_use cascade;
-
create view file_use as
select * from file_in
union
select * from file_out;
-drop view ds_use cascade;
-
-create view ds_use as
+drop view ds_param_value cascade;
+create view ds_param_value as
select * from in_mem_use
union
select * from file_use;
drop view a_t cascade;
-
create view a_t as
select run.id, a_run_t.name, a_run_t.value
from run, a_run_t
@@ -77,7 +69,6 @@
where ds_use.ds_id=a_ds_t.ds_id;
drop view a_n cascade;
-
create view a_n as
select run.id, a_run_n.name, a_run_n.value
from run, a_run_n
@@ -91,18 +82,23 @@
from ds_use, a_ds_n
where ds_use.ds_id=a_ds_n.ds_id;
+drop view ds_use cascade;
+create view ds_use as
+ select * from ds_in
+ union all
+ select * from ds_out;
-drop type compare_run_by_parameter_type cascade;
-create type compare_run_by_parameter_type as (run_id varchar, param varchar, value varchar);
+drop type compare_run_by_param_type cascade;
+create type compare_run_by_param_type as (run_id varchar, param varchar, value varchar);
-create or replace function compare_run_by_parameter(param_name varchar)
-returns setof compare_run_by_parameter_type
+create or replace function compare_run_by_param(param_name varchar)
+returns setof compare_run_by_param_type
as $$
select run_id, param, value
- from ds_use
- where param=$1;
+ from ds_use,proc,in_mem
+ where proc.id=ds_use.proc_id and ds_use.ds_id=in_mem.id and param=$1;
$$ language sql;
-- PostgreSQL >= 9.0
@@ -126,19 +122,7 @@
-- GROUP BY proc.run_id, ds_in.param, in_mem.value
--$$ LANGUAGE SQL;
-DROP TYPE compare_run_by_parameter_type2;
-CREATE TYPE compare_run_by_parameter_type2 AS (run_id VARCHAR, param1 VARCHAR, value1 VARCHAR, param2 VARCHAR, value2 VARCHAR);
-CREATE OR REPLACE FUNCTION compare_run_by_parameter(param_name1 VARCHAR, param_name2 VARCHAR)
-RETURNS SETOF compare_run_by_parameter_type2
-AS $$
- SELECT *
- FROM compare_run_by_parameter($1) as t
- INNER JOIN
- compare_run_by_parameter($2) as s
- USING (run_id);
-$$ LANGUAGE SQL;
-
--CREATE OR REPLACE FUNCTION compare_run_by_parameter(param_name1 VARCHAR, param_name2 VARCHAR)
--RETURNS TABLE (
-- workflow_id VARCHAR,
@@ -156,73 +140,105 @@
--$$ LANGUAGE SQL;
-CREATE OR REPLACE FUNCTION compare_run_by_parameter(param_name1 VARCHAR, param_name2 VARCHAR, param_name3 VARCHAR)
-RETURNS TABLE (
- workflow_id VARCHAR,
- param_name1 VARCHAR,
- value1 VARCHAR,
- param_name2 VARCHAR,
- value2 VARCHAR,
- param_name3 VARCHAR,
- value3 VARCHAR
-)
-AS $$
- SELECT *
- FROM compare_run_by_parameter($1, $2) as t
- INNER JOIN
- compare_run_by_parameter($3) as s
- USING (workflow_id);
-$$ LANGUAGE SQL;
+DROP TYPE compare_run_by_annot_num_type;
+CREATE TYPE compare_run_by_annot_num_type as (run_id VARCHAR, name VARCHAR, value NUMERIC);
-
CREATE OR REPLACE FUNCTION compare_run_by_annot_num(name VARCHAR)
-RETURNS TABLE (
- workflow_id VARCHAR,
- name VARCHAR,
- value NUMERIC
-)
+RETURNS SETOF compare_run_by_annot_num_type
AS $$
- SELECT process.workflow_id, annot_ds_num.name, annot_ds_num.value
- FROM annot_ds_num,ds_usage,ds_containment,process
- WHERE annot_ds_num.id=ds_containment.in_id AND ds_containment.out_id=ds_usage.dataset_id AND
- ds_usage.process_id=process.id AND annot_ds_num.name=$1
+ SELECT proc.run_id, a_ds_n.name, a_ds_n.value
+ FROM a_ds_n,ds_use,ds_cont,proc
+ WHERE a_ds_n.ds_id=ds_cont.in_id AND ds_cont.out_id=ds_use.ds_id AND
+ ds_use.proc_id=proc.id AND a_ds_n.name=$1
UNION
- SELECT process.workflow_id, annot_ds_num.name, annot_ds_num.value
- FROM process, ds_usage, annot_ds_num
- WHERE process.id=ds_usage.process_id and ds_usage.dataset_id=annot_ds_num.id and
- annot_ds_num.name=$1
+ SELECT proc.run_id, a_ds_n.name, a_ds_n.value
+ FROM proc, ds_use, a_ds_n
+ WHERE proc.id=ds_use.proc_id and ds_use.ds_id=a_ds_n.ds_id and
+ a_ds_n.name=$1
UNION
- SELECT process.workflow_id, annot_p_num.name, annot_p_num.value
- FROM process, annot_p_num
- WHERE process.id=annot_p_num.id and annot_p_num.name=$1
+ SELECT proc.run_id, a_proc_n.name, a_proc_n.value
+ FROM proc, a_proc_n
+ WHERE proc.id=a_proc_n.proc_id and a_proc_n.name=$1
UNION
- SELECT workflow.id as workflow_id, annot_wf_num.name, annot_wf_num.value
- FROM workflow, annot_wf_num
- WHERE workflow.id=annot_wf_num.id and annot_wf_num.name=$1
+ SELECT run.id as run_id, a_run_n.name, a_run_n.value
+ FROM run, a_run_n
+ WHERE run.id=a_run_n.run_id and a_run_n.name=$1
$$ LANGUAGE SQL;
+DROP TYPE compare_run_by_annot_txt_type;
+CREATE TYPE compare_run_by_annot_txt_type as (run_id VARCHAR, name VARCHAR, value VARCHAR);
CREATE OR REPLACE FUNCTION compare_run_by_annot_txt(name VARCHAR)
-RETURNS TABLE (
- workflow_id VARCHAR,
- name VARCHAR,
- value VARCHAR)
+RETURNS SETOF compare_run_by_annot_txt_type
AS $$
- SELECT process.workflow_id, annot_ds_txt.name, annot_ds_txt.value
- FROM process, ds_usage, annot_ds_txt
- WHERE process.id=ds_usage.process_id and ds_usage.dataset_id=annot_ds_txt.id and
- annot_ds_txt.name=$1
+ SELECT proc.run_id, a_ds_t.name, a_ds_t.value
+ FROM a_ds_t,ds_use,ds_cont,proc
+ WHERE a_ds_t.ds_id=ds_cont.in_id AND ds_cont.out_id=ds_use.ds_id AND
+ ds_use.proc_id=proc.id AND a_ds_t.name=$1
UNION
- SELECT process.workflow_id, annot_p_txt.name, annot_p_txt.value
- FROM process, annot_p_txt
- WHERE process.id=annot_p_txt.id and annot_p_txt.name=$1
+ SELECT proc.run_id, a_ds_t.name, a_ds_t.value
+ FROM proc, ds_use, a_ds_t
+ WHERE proc.id=ds_use.proc_id and ds_use.ds_id=a_ds_t.ds_id and
+ a_ds_t.name=$1
UNION
- SELECT workflow.id as workflow_id, annot_wf_txt.name, annot_wf_txt.value
- FROM workflow, annot_wf_txt
- WHERE workflow.id=annot_wf_txt.id and annot_wf_txt.name=$1
+ SELECT proc.run_id, a_proc_t.name, a_proc_t.value
+ FROM proc, a_proc_t
+ WHERE proc.id=a_proc_t.proc_id and a_proc_t.name=$1
+ UNION
+ SELECT run.id as run_id, a_run_t.name, a_run_t.value
+ FROM run, a_run_t
+ WHERE run.id=a_run_t.run_id and a_run_t.name=$1
$$ LANGUAGE SQL;
+-- CREATE OR REPLACE FUNCTION compare_run_by_annot_num(name VARCHAR)
+-- RETURNS TABLE (
+-- workflow_id VARCHAR,
+-- name VARCHAR,
+-- value NUMERIC
+-- )
+-- AS $$
+-- SELECT process.workflow_id, annot_ds_num.name, annot_ds_num.value
+-- FROM annot_ds_num,ds_usage,ds_containment,process
+-- WHERE annot_ds_num.id=ds_containment.in_id AND ds_containment.out_id=ds_usage.dataset_id AND
+-- ds_usage.process_id=process.id AND annot_ds_num.name=$1
+-- UNION
+-- SELECT process.workflow_id, annot_ds_num.name, annot_ds_num.value
+-- FROM process, ds_usage, annot_ds_num
+-- WHERE process.id=ds_usage.process_id and ds_usage.dataset_id=annot_ds_num.id and
+-- annot_ds_num.name=$1
+-- UNION
+-- SELECT process.workflow_id, annot_p_num.name, annot_p_num.value
+-- FROM process, annot_p_num
+-- WHERE process.id=annot_p_num.id and annot_p_num.name=$1
+-- UNION
+-- SELECT workflow.id as workflow_id, annot_wf_num.name, annot_wf_num.value
+-- FROM workflow, annot_wf_num
+-- WHERE workflow.id=annot_wf_num.id and annot_wf_num.name=$1
+-- $$ LANGUAGE SQL;
+
+
+-- CREATE OR REPLACE FUNCTION compare_run_by_annot_txt(name VARCHAR)
+-- RETURNS TABLE (
+-- workflow_id VARCHAR,
+-- name VARCHAR,
+-- value VARCHAR)
+-- AS $$
+-- SELECT process.workflow_id, annot_ds_txt.name, annot_ds_txt.value
+-- FROM process, ds_usage, annot_ds_txt
+-- WHERE process.id=ds_usage.process_id and ds_usage.dataset_id=annot_ds_txt.id and
+-- annot_ds_txt.name=$1
+-- UNION
+-- SELECT process.workflow_id, annot_p_txt.name, annot_p_txt.value
+-- FROM process, annot_p_txt
+-- WHERE process.id=annot_p_txt.id and annot_p_txt.name=$1
+-- UNION
+-- SELECT workflow.id as workflow_id, annot_wf_txt.name, annot_wf_txt.value
+-- FROM workflow, annot_wf_txt
+-- WHERE workflow.id=annot_wf_txt.id and annot_wf_txt.name=$1
+-- $$ LANGUAGE SQL;
+
+
CREATE OR REPLACE FUNCTION compare_run_by_annot_bool(name VARCHAR)
RETURNS TABLE (
workflow_id VARCHAR,
@@ -262,16 +278,20 @@
$$ LANGUAGE SQL;
-- recursive query to find ancestor entities in a provenance graph
-CREATE OR REPLACE FUNCTION ancestors(varchar) RETURNS SETOF varchar AS $$
- WITH RECURSIVE anc(ancestor,descendant) AS
- (
- SELECT parent AS ancestor, child AS descendant FROM parent_of WHERE child=$1
- UNION
- SELECT parent_of.parent AS ancestor, anc.descendant AS descendant
- FROM anc,parent_of
- WHERE anc.ancestor=parent_of.child
- )
- SELECT ancestor FROM anc
+CREATE OR REPLACE FUNCTION ancestors(varchar)
+RETURNS SETOF varchar AS $$
+ WITH RECURSIVE anc(ancestor,descendant) AS
+ (
+ SELECT parent AS ancestor, child AS descendant
+ FROM parent_of
+ WHERE child=$1
+ UNION
+ SELECT parent_of.parent AS ancestor,
+ anc.descendant AS descendant
+ FROM anc,parent_of
+ WHERE anc.ancestor=parent_of.child
+ )
+ SELECT ancestor FROM anc
$$ LANGUAGE SQL;
Modified: provenancedb/prov-init.sql
===================================================================
--- provenancedb/prov-init.sql 2011-08-09 11:16:34 UTC (rev 4967)
+++ provenancedb/prov-init.sql 2011-08-09 13:16:29 UTC (rev 4968)
@@ -42,7 +42,7 @@
-- has an entry for this process.
-- process types: internal, rootthread, execute, function, compound, scope, operator
create table proc
- (id varchar(256) primary key,
+ (id varchar(256) primary key,
type varchar(16),
name varchar(256), -- in the case of an execute this refers to the transformation name in tc.data
run_id varchar(256) references run (id) on delete cascade -- normalize: workflow_id of sub-procedure determined
@@ -187,11 +187,17 @@
primary key (run_id, name)
);
+create table iq
+ ( idx serial primary key,
+ q varchar(2048)
+ );
+
+drop view pgraph_edge;
create view pgraph_edge as
select proc_id as parent,ds_id as child from ds_out
- union
+ union all
select ds_id as parent,proc_id as child from ds_in
- union
+ union all
select out_id as parent,in_id as child from ds_cont;
-- continue renaming from here
Added: provenancedb/prov-to-datalog.sh
===================================================================
--- provenancedb/prov-to-datalog.sh (rev 0)
+++ provenancedb/prov-to-datalog.sh 2011-08-09 13:16:29 UTC (rev 4968)
@@ -0,0 +1,186 @@
+#!/bin/bash
+
+export RUNID=$(basename $1 .log)
+
+export WFID="execute:${RUNID}:"
+
+# TODO is there already a URI form for identifying workflows?
+export WF="${RUNID}"
+
+echo Generating Datalog for $RUNID
+
+# this gives a distinction between the root process for a workflow and the
+# workflow itself. perhaps better to model the workflow as a process
+echo "isProcess('${WFID}0')." > tmp.datalog
+echo "hasType('${WFID}0', 'rootthread')." >> tmp.datalog
+echo "hasName('${WFID}0', 'name')." >> tmp.datalog
+echo "isInScriptRun('${WFID}0', '$WF')." >> tmp.datalog
+
+while read time duration thread localthread endstate tr_name scratch; do
+ echo "isProcess('$thread')." >> tmp.datalog
+ echo "hasType('$thread', 'execute')." >> tmp.datalog
+ echo "hasName('$thread', '$tr_name')." >> tmp.datalog
+ echo "hasStartTime('$thread', $time)." >> tmp.datalog
+ echo "hasDuration('$thread', $duration)." >> tmp.datalog
+ echo "hasFinalState('$thread', '$endstate')." >> tmp.datalog
+done < execute.global.event
+
+while read start_time duration globalid id endstate thread site scratch; do
+ # cut off the last component of the thread, so that we end up at the
+ # parent thread id which should correspond with the execute-level ID
+ inv_id="$WFID$(echo $thread | sed 's/-[^-]*$//')" >> tmp.datalog
+ echo "isExecutionAttempt('$globalid')." >> tmp.datalog
+ echo "attemptsToExecute('$globalid', '$inv_id')." >> tmp.datalog
+ echo "hasStartTime('$globalid', $start_time)." >> tmp.datalog
+ echo "hasDuration('$globalid', $duration)." >> tmp.datalog
+ echo "hasFinalState('$globalid', '$endstate')." >> tmp.datalog
+ echo "executedAtSite('$globalid', '$site')." >> tmp.datalog
+done < execute2.global.event
+
+while read col1 col2 col3 col4 col5 thread name lhs rhs result; do
+ thread=$(echo $thread | awk 'BEGIN { FS = "=" }; {print $2}')
+ name=$(echo $name | awk 'BEGIN { FS = "=" }; {print $2}')
+ lhs=$(echo $lhs | awk 'BEGIN { FS = "=" }; {print $2}')
+ rhs=$(echo $rhs | awk 'BEGIN { FS = "=" }; {print $2}')
+ result=$(echo $result | awk 'BEGIN { FS = "=" }; {print $2}')
+
+ operatorid="${WFID}operator:$thread"
+
+ if [ $version -le 3726 ]; then
+ lhs=$(echo $lhs | sed -e 's/tag:benc at ci.uchicago.edu,2008:swift://g')
+ rhs=$(echo $rhs | sed -e 's/tag:benc at ci.uchicago.edu,2008:swift://g')
+ result=$(echo $result | sed -e 's/tag:benc at ci.uchicago.edu,2008:swift://g')
+ fi
+
+ echo "isDataset('$lhs')." >> tmp.datalog
+ echo "isDataset('$rhs')." >> tmp.datalog
+ echo "isDataset('$result')." >> tmp.datalog
+ echo "isProcess('$operatorid')." >> tmp.datalog
+ echo "hasType('$operatorid', 'operator')." >> tmp.datalog
+ echo "hasName('$operatorid', '$name')." >> tmp.datalog
+ echo "used('$operatorid', '$lhs', 'lhs')." >> tmp.datalog
+ echo "used('$operatorid', '$rhs', 'rhs')." >> tmp.datalog
+ echo "wasGeneratedBy('$result', '$operatorid', 'result')." >> tmp.datalog
+done < operators.txt
+
+while read id name output; do
+ echo "isDataset('$output')." >> tmp.datalog
+ echo "isProcess('$id')." >> tmp.datalog
+ echo "hasType('$id', 'function')." >> tmp.datalog
+ echo "hasName('$id', '$name')." >> tmp.datalog
+ echo "isInScriptRun('$id', '$WF')." >> tmp.datalog
+ echo "wasGeneratedBy('$output', '$id', 'result')." >> tmp.datalog
+done < functions.txt
+
+while read id value; do
+ # TODO need ordering/naming
+ echo "isDataset('$value')." >> tmp.datalog
+ echo "wasGeneratedBy('$value', '$id', 'undefined')." >> tmp.datalog
+done < function-inputs.txt
+
+
+while read thread appname; do
+ echo "hasName('$thread', '$appname')." >> tmp.datalog
+done < invocation-procedure-names.txt
+
+while read outer inner; do
+ echo "isDataset('$outer')." >> tmp.datalog
+ echo "isDataset('$inner')." >> tmp.datalog
+ echo "isContainedIn('$inner', '$outer')." >> tmp.datalog
+done < tie-containers.txt
+
+while read dataset filename; do
+
+ if [ $version -le 3726 ]; then
+ dataset=$(echo $dataset | sed -e 's/tag:benc at ci.uchicago.edu,2008:swift://g')
+ fi
+ echo "isDataset('$dataset')." >> tmp.datalog
+ echo "hasFileName('$dataset', '$filename')." >> tmp.datalog
+done < dataset-filenames.txt
+
+while read dataset idtype equal value rest; do
+
+ if [ $version -le 3726 ]; then
+ dataset=$(echo $dataset | sed -e 's/tag:benc at ci.uchicago.edu,2008:swift://g')
+ fi
+
+ echo "isDataset('$dataset')." >> tmp.datalog
+ echo "hasValue('$dataset', '$value')." >> tmp.datalog
+done < dataset-values.txt
+
+while read start duration wfid rest; do
+ echo "hasStartTime('$WF', $start)." >> tmp.datalog
+ echo "hasDuration('$WF', $duration)." >> tmp.datalog
+done < workflow.event
+
+
+# TODO this could merge with other naming tables
+while read start duration thread final_state procname ; do
+ if [ "$duration" != "last-event-line" ]; then
+ compoundid=$WFID$thread
+ echo "isProcess('$compoundid')." >> tmp.datalog
+ echo "hasType('compound')." >> tmp.datalog
+ echo "hasName('$procname')." >> tmp.datalog
+ echo "isInScriptRun('$compoundid', '$WF')." >> tmp.datalog
+ fi
+done < compound.event
+
+while read start duration thread final_state procname ; do
+ if [ "$duration" != "last-event-line" ]; then
+ fqid=$WFID$thread
+ echo "isProcess('$fqid')." >> tmp.datalog
+ echo "hasType('internal')." >> tmp.datalog
+ echo "hasName('$procname')." >> tmp.datalog
+ echo "isInScriptRun('$fqid', '$WF')." >> tmp.datalog
+ fi
+done < internalproc.event
+
+while read t ; do
+ thread="${WFID}$t"
+ echo "isProcess('$thread')." >> tmp.datalog
+ echo "hasType('scope')." >> tmp.datalog
+ echo "hasName('scope')." >> tmp.datalog
+ echo "isInScriptRun('$thread', '$WF')." >> tmp.datalog
+done < scopes.txt
+
+while read thread direction dataset variable rest; do
+
+ dataset=$(echo $dataset | sed -e 's/tag:benc at ci.uchicago.edu,2008:swift://g')
+ echo "isDataset('$dataset')." >> tmp.datalog
+
+ if [ "$direction" == "input" ] ; then
+ echo "used('$thread', '$dataset', '$variable')." >> tmp.datalog
+ else
+ echo "wasGeneratedBy('$dataset', '$thread', '$variable')." >> tmp.datalog
+ fi
+
+done < tie-data-invocs.txt
+
+if [ -f extrainfo.txt ]; then
+ while read execute2_id extrainfo; do
+ echo $extrainfo | awk -F ";" '{ for (i = 1; i <= NF; i++)
+ print $i
+ }' | awk -F "=" '{ print $1 " " $2 }' | awk -F ":" '{ print $1 " " $2 }' > fields.txt
+ id=$($SQLCMD --tuples-only -c "select app_inv_id from app_exec where id='$execute2_id';" | awk '{print $1}')
+ while read name type value; do
+ # TODO: check types
+ echo "hasAnnotation('$id', '$name', '$value')." >> tmp.datalog
+ done < fields.txt
+ done < extrainfo.txt
+fi
+
+if [ -f runtime.txt ]; then
+ while read execute2_id runtime; do
+ timestamp=$(echo $runtime | awk -F "," '{print $1}' | awk -F ":" '{print $2}')
+ cpu_usage=$(echo $runtime | awk -F "," '{print $2}' | awk -F ":" '{print $2}')
+ max_phys_mem=$(echo $runtime | awk -F "," '{print $3}' | awk -F ":" '{print $2}')
+ max_virtual_mem=$(echo $runtime | awk -F "," '{print $4}' | awk -F ":" '{print $2}')
+ io_read_bytes=$(echo $runtime | awk -F "," '{print $5}' | awk -F ":" '{print $2}')
+ io_write_bytes=$(echo $runtime | awk -F "," '{print $6}' | awk -F ":" '{print $2}')
+ echo "hasRuntimeInfo('$execute2_id', $timestamp, $cpu_usage, $max_phys_mem, $max_virtual_mem, $io_read_bytes, $io_write_bytes)." >> tmp.datalog
+ done < runtime.txt
+fi
+
+cat tmp.datalog | sort | uniq >> provenancedb.datalog
+
+echo Finished writing Datalog.
\ No newline at end of file
Property changes on: provenancedb/prov-to-datalog.sh
___________________________________________________________________
Added: svn:executable
+ *
Modified: provenancedb/prov-to-sql.sh
===================================================================
--- provenancedb/prov-to-sql.sh 2011-08-09 11:16:34 UTC (rev 4967)
+++ provenancedb/prov-to-sql.sh 2011-08-09 13:16:29 UTC (rev 4968)
@@ -39,7 +39,7 @@
rhs=$(echo $rhs | sed -e 's/tag:benc at ci.uchicago.edu,2008:swift://g')
result=$(echo $result | sed -e 's/tag:benc at ci.uchicago.edu,2008:swift://g')
fi
-
+
$SQLCMD -c "INSERT INTO ds (id) VALUES ('$lhs');"
$SQLCMD -c "INSERT INTO ds (id) VALUES ('$rhs');"
$SQLCMD -c "INSERT INTO ds (id) VALUES ('$result');"
@@ -96,7 +96,7 @@
$SQLCMD -c "INSERT INTO file (id, name) VALUES ('$dataset', '$filename');"
done < dataset-filenames.txt
-while read dataset value; do
+while read dataset idtype equal value rest; do
if [ $version -le 3726 ]; then
dataset=$(echo $dataset | sed -e 's/tag:benc at ci.uchicago.edu,2008:swift://g')
Deleted: provenancedb/prov-to-swipl.sh
===================================================================
--- provenancedb/prov-to-swipl.sh 2011-08-09 11:16:34 UTC (rev 4967)
+++ provenancedb/prov-to-swipl.sh 2011-08-09 13:16:29 UTC (rev 4968)
@@ -1,32 +0,0 @@
-#!/bin/bash
-
-rm -f tmp-import.pl
-
-while read time duration thread endstate app; do
- echo "execute('$thread', $time, $duration, '$endstate', '$app')." >> tmp-import.pl
-done < $LOGDIR/execute.event
-
-while read thread direction dataset variable value rest; do
- if [ "$direction" == "input" ] ; then
- dir=I
- else
- dir=O
- fi
- echo "dataset_usage('$thread', '$dir', '$dataset', '$variable', '$value')." >> tmp-import.pl
-done < $LOGDIR/tie-data-invocs.txt
-
-while read thread appname; do
- echo "invocation_procedure_names('$thread', '$appname')." >> tmp-import.pl
-
-done < $LOGDIR/invocation-procedure-names.txt
-
-while read outer inner; do
- echo "dataset_containment('$outer', '$inner')." >> tmp-import.pl
-done < $LOGDIR/tie-containers.txt
-
-while read dataset filename; do
- echo "dataset_filenames('$dataset', '$filename')." >> tmp-import.pl
-done < $LOGDIR/dataset-filenames.txt
-
-swipl -g "['tmp-import.pl']"
-
Added: provenancedb/provenancedb-rules.datalog
===================================================================
--- provenancedb/provenancedb-rules.datalog (rev 0)
+++ provenancedb/provenancedb-rules.datalog 2011-08-09 13:16:29 UTC (rev 4968)
@@ -0,0 +1,45 @@
+isArtifact(X) :- isDataset(X).
+
+%% used(Process, Dataset, Role).
+%% wasGeneratedBy(Dataset, Process, Role).
+isProcess(X) :- used(X,_,_).
+isProcess(Y) :- wasGeneratedBy(_,Y,_).
+
+isDataset(Y) :- used(_,Y,_).
+isDataset(X) :- wasGeneratedBy(X,_,_).
+
+:- table isAncestor/2.
+isAncestor(X,Y) :- used(Y,X,_).
+isAncestor(X,Y) :- used(Y,Z,_),isAncestor(X,Z).
+isAncestor(X,Y) :- wasGeneratedBy(Y,X,_).
+isAncestor(X,Y) :- wasGeneratedBy(Y,Z,_),isAncestor(X,Z).
+%%isAncestor(X,Y) :- isContainedIn(X,Y).
+%%isAncestor(X,Y) :- isContainedIn(Z,Y),isAncestor(X,Z).
+
+:- table wasDerivedFrom/2.
+wasDerivedFrom(X,Y) :- used(Z,Y,_),wasGeneratedBy(X,Z,_).
+
+:- table wasTriggeredBy/2.
+wasTriggeredBy(X,Y) :- wasGeneratedBy(Z,Y,_),used(X,Z,_).
+
+:- table isContainedInTC/2.
+isContainedInTC(X,Y) :- isContainedIn(X,Y).
+isContainedInTC(X,Y) :- isContainedIn(X,Z),isContainedInTC(Z,Y).
+
+:- table wasDerivedFromTC/2.
+wasDerivedFromTC(X,Y) :- wasDerivedFrom(X,Y).
+wasDerivedFromTC(X,Y) :- wasDerivedFrom(X,Z),wasDerivedFromTC(Z,Y).
+
+:- table wasTriggeredByTC/2.
+wasTriggeredByTC(X,Y) :- wasTriggeredBy(X,Y).
+wasTriggeredByTC(X,Y) :- wasTriggeredBy(X,Z),wasTriggeredByTC(Z,Y).
+
+isInRun(D,X) :- wasGeneratedBy(D,P,_),isInRun(P,X).
+isInRun(D,X) :- used(P,D,_),isInRun(P,X).
+
+runCorrelation(R,D,[N|L],V) :- isRun(R),isDataset(D),isInRun(D,R),hasAnnotation(D,N,V).
+runCorrelation(R,P,[N|L],V) :- isRun(R),isProcess(P),isInRun(P,R),hasAnnotation(P,N,V).
+runCorrelation(R,P,[N|L],V) :- isRun(R),isPro
+
+
+
Modified: provenancedb/swift-prov-import-all-logs
===================================================================
--- provenancedb/swift-prov-import-all-logs 2011-08-09 11:16:34 UTC (rev 4967)
+++ provenancedb/swift-prov-import-all-logs 2011-08-09 13:16:29 UTC (rev 4968)
@@ -13,6 +13,7 @@
# this generates a file with tuples like:
#
+# This is where Swift's version is collected.
swift-plot-log $LOGREPO everylog-vs-versions.data
if [ "$?" != "0" ]; then
Added: provenancedb/swift-prov-import-all-logs-datalog
===================================================================
--- provenancedb/swift-prov-import-all-logs-datalog (rev 0)
+++ provenancedb/swift-prov-import-all-logs-datalog 2011-08-09 13:16:29 UTC (rev 4968)
@@ -0,0 +1,82 @@
+#!/bin/bash
+
+PROVDIR=$(dirname $0)
+pushd $PROVDIR
+PROVDIR=$(pwd)
+popd
+
+# we need to keep this out of the log-proceesing dir because import
+# of individual runs will clean other files.
+
+source $PROVDIR/etc/provenance.config
+export PATH=$PROVDIR:$PATH
+
+# this generates a file with tuples like:
+#
+# This is where Swift's version is collected.
+swift-plot-log $LOGREPO everylog-vs-versions.data
+
+if [ "$?" != "0" ]; then
+ echo swift-plot-log failed when building everylog-vs-versions.data
+ exit 1
+fi
+
+# TODO better tmp handling that always using the same name in a shared
+# directory
+cp everylog-vs-versions.data /tmp/
+
+echo first commandline param is $1
+if [ "$1" == "rebuild" ]; then
+ echo CLEANING DATABASE
+ cat $PROVDIR/provenancedb-rules.datalog > provenancedb.datalog
+fi
+
+while read start version filename; do
+
+ export IDIR=$(echo $filename | sed 's/\.log$/.d/')
+ echo IDIR=$IDIR
+ if [ $version -ge 1538 ]; then
+ echo -n "Log: $filename ... "
+
+ EXISTING=$(grep "hasLogFilename('$filename')." provenancedb.datalog)
+
+ if [ -z "$EXISTING" ]; then
+ PROV_ENABLED=$(grep provenanceid $filename | wc -l)
+ if [ $PROV_ENABLED -gt 0 ]; then
+ echo IMPORTING
+
+ if grep --silent "Loader Swift finished with no errors" $filename; then
+ wfstatus="SUCCESS"
+ else
+ wfstatus="FAIL"
+ fi
+
+ export RUNID=$(basename $filename .log)
+
+ export WF="${RUNID}"
+ echo WP1
+ echo "isRun('$WF')." >> provenancedb.datalog
+ echo "hasLogFilename('$WF', '$filename')." >> provenancedb.datalog
+ echo "hasSwiftVersion('$WF', '$version')." >> provenancedb.datalog
+ echo "hasFinalState('$WF', '$wfstatus')." >> provenancedb.datalog
+
+ echo version $version in log file $filename
+ echo ============= will import =============
+ prepare-for-import $filename
+ if [ "$?" != "0" ]; then
+ echo prepare-for-import failed
+ exit 2
+ fi
+ version=$version import-run-to-datalog $filename
+ if [ "$?" != "0" ]; then
+ echo import-run-to-datalog failed
+ exit 3
+ fi
+ else
+ echo SKIP: provenance.log not set to true in etc/swift.properties
+ fi
+ else
+ echo SKIP: Already known in workflow
+ fi
+ fi
+done < /tmp/everylog-vs-versions.data
Property changes on: provenancedb/swift-prov-import-all-logs-datalog
___________________________________________________________________
Added: svn:executable
+ *
Deleted: provenancedb/xq.xq
===================================================================
--- provenancedb/xq.xq 2011-08-09 11:16:34 UTC (rev 4967)
+++ provenancedb/xq.xq 2011-08-09 13:16:29 UTC (rev 4968)
@@ -1,6 +0,0 @@
-for $t in //tie
- let $dataset := //dataset[@identifier=$t/dataset]
- let $exec := //execute[thread=$t/thread]
- where $t/direction="input"
- return An invocation of {$exec/trname} took input {$dataset/filename}
-
From wozniak at ci.uchicago.edu Tue Aug 9 15:20:05 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Tue, 9 Aug 2011 15:20:05 -0500 (CDT)
Subject: [Swift-commit] r4969 - trunk/libexec/log-processing
Message-ID: <20110809202005.A713D9CCBF@svn.ci.uchicago.edu>
Author: wozniak
Date: 2011-08-09 15:20:05 -0500 (Tue, 09 Aug 2011)
New Revision: 4969
Modified:
trunk/libexec/log-processing/normalize-log.pl
Log:
Bug fix
Modified: trunk/libexec/log-processing/normalize-log.pl
===================================================================
--- trunk/libexec/log-processing/normalize-log.pl 2011-08-09 13:16:29 UTC (rev 4968)
+++ trunk/libexec/log-processing/normalize-log.pl 2011-08-09 20:20:05 UTC (rev 4969)
@@ -12,7 +12,7 @@
my $earliesttime;
open(START, $TIME) || die "$!\n";
-read(FH, $earliesttime, 64);
+read(START, $earliesttime, 64);
close START;
open(LOG, $LOG) || die "$!\n";
From wilde at ci.uchicago.edu Tue Aug 9 16:53:47 2011
From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu)
Date: Tue, 9 Aug 2011 16:53:47 -0500 (CDT)
Subject: [Swift-commit] r4970 - in trunk: bin/grid docs/siteguide
Message-ID: <20110809215347.DD0359CCBF@svn.ci.uchicago.edu>
Author: wilde
Date: 2011-08-09 16:53:47 -0500 (Tue, 09 Aug 2011)
New Revision: 4970
Added:
trunk/bin/grid/run-worker.sh
Modified:
trunk/bin/grid/TODO
trunk/bin/grid/swift-workers
trunk/docs/siteguide/grid
Log:
Snapshop of bin/grid - tested to 2M+ jobs with catsn. Adds run-worker wrapper and swiftDemand file, and interim siteguide text.
Modified: trunk/bin/grid/TODO
===================================================================
--- trunk/bin/grid/TODO 2011-08-09 20:20:05 UTC (rev 4969)
+++ trunk/bin/grid/TODO 2011-08-09 21:53:47 UTC (rev 4970)
@@ -2,7 +2,21 @@
- why are there two logs from the coaster service: uuid-named log and swift.log????
+- missing .error file???
+- new error handling strategy ?
+
+- unique sub file for each worker job
+--- log file for each
+--- worker log on $tmp
+--- cat worker log on error
+--- set worker run time
+--- option for per-site service
+--- option for gridftp if using per-site service
+--- return log file on error if possible
+
+
+
EXTENCI APPLICATION WORK
create modft install & test file; test under fork and work
Added: trunk/bin/grid/run-worker.sh
===================================================================
--- trunk/bin/grid/run-worker.sh (rev 0)
+++ trunk/bin/grid/run-worker.sh 2011-08-09 21:53:47 UTC (rev 4970)
@@ -0,0 +1,33 @@
+#! /bin/bash
+
+contact=$1
+workername=$2
+origlogdir=$3
+echo OSG_WN_TMP=$OSG_WN_TMP
+if [ _$OSG_WN_TMP = _ ]; then
+ OSG_WN_TMP=/tmp
+fi
+mkdir -p $OSG_WN_TMP
+logdir=$(mktemp -d $OSG_WN_TMP/${workername}.workerdir.XXXXXX)
+nlines=1000
+
+echo "=== contact: $contact"
+echo "=== name: $workername Running in dir $(pwd)"
+echo "=== cwd: $(pwd)"
+echo "=== logdir: $logdir"
+echo "==============================================="
+
+cat >worker.pl
+chmod +x worker.pl
+
+./worker.pl $contact $workername $logdir
+
+exitcode=$?
+
+echo "=== exit: worker.pl exited with code=$exitcode"
+
+echo "=== worker log - last $nlines lines:"
+
+echo
+
+tail -v -n $nlines $logdir/*
Property changes on: trunk/bin/grid/run-worker.sh
___________________________________________________________________
Added: svn:executable
+ *
Modified: trunk/bin/grid/swift-workers
===================================================================
--- trunk/bin/grid/swift-workers 2011-08-09 20:20:05 UTC (rev 4969)
+++ trunk/bin/grid/swift-workers 2011-08-09 21:53:47 UTC (rev 4970)
@@ -10,7 +10,7 @@
require 'etc'
class Site
- attr_accessor :grid_resource, :data_dir, :app_dir, :name, :port
+ attr_accessor :grid_resource, :gridftp, :data_dir, :app_dir, :name, :port
attr_reader :submit_file
# executable = <%= @app_dir %>/worker.pl # FIXME (below)
@@ -22,35 +22,22 @@
# WORKER_LOGGING_LEVEL=$LOGLEVEL $HOME/swift_gridtools/worker.pl $SERVICEURL swork${worker} $LOGDIR >& /dev/null &
-# a mod
-
def gen_submit(count = 1)
- job = %q[
- universe = grid
- stream_output = False
- stream_error = False
- transfer_executable = false
- periodic_remove = JobStatus == 5
- notification = Never
- globus_rsl = (maxwalltime=240)
- grid_resource = <%= @grid_resource %>
- executable = /bin/sleep
- arguments = 300
- log = condor.log
-
- <% count.times { %>queue
- <% } %>
- ]
-
ov=$VERBOSE
$VERBOSE=nil
workerExecutable = `which worker.pl`
+ workerWrapper = `which run-worker.sh`
$VERBOSE=ov
-# workerContact = "http://communicado.ci.uchicago.edu:36906"
- workerContact = ARGV[2]
+# workerContact = "http://communicado.ci.uchicago.edu:36906"
+ workerContact = ARGV[2]
- newjob = %Q[
+# for submit file: log = condor.log
+# <% count.times { %>queue
+# <% } %>
+# log = condor/$(Process).log
+
+ job = %Q[
universe = grid
stream_output = False
stream_error = False
@@ -60,23 +47,23 @@
globus_rsl = (maxwalltime=240)
grid_resource = <%= @grid_resource %>
- executable = #{workerExecutable}
- arguments = #{workerContact} swork /tmp
+ executable = #{workerWrapper}
+ arguments = #{workerContact} <%= @name.gsub(/__.*/,"") %> /tmp
environment = WORKER_LOGGING_LEVEL=INFO
- log = condor.log
+ Input = #{workerExecutable}
+ Error = condor/$(Process).err
+ Output = condor/$(Process).out
+ log = condor.log
- <% count.times { %>queue
- <% } %>
+ queue #{count}
]
-
- ERB.new(newjob.gsub(/^\s+/, ""), 0, "%<>", "@submit_file").result(binding)
+ ERB.new(job.gsub(/^\s+/, ""), 0, "%<>", "@submit_file").result(binding)
end
def submit_job(count)
puts "submit_job: Submitting #{@name} #{count} jobs"
count = count.to_i
output = ""
-#return output
submitfile = gen_submit(count)
IO.popen("condor_submit", "w+") do |submit|
submit.puts submitfile
@@ -144,9 +131,9 @@
demandThread = Thread.new("monitor-demand") do |t|
puts "starting demand thread"
while true do
- puts "in demand thread"
- # swiftDemand = IO.read("swiftDemand") # Replace this with sensor of Swift demand
- swiftDemand = 15
+ puts "in demand thread"
+ swiftDemand = IO.read("swiftDemand").to_i # Replace this with sensor of Swift demand
+ # swiftDemand = 15
paddedDemand = (swiftDemand * 1.2).to_i
ov=$VERBOSE;$VERBOSE=nil
totalRunning = `condor_q #{$username} -const 'JobStatus == 2' -format \"%s \" GlobalJobId`.split(" ").size
@@ -161,6 +148,7 @@
site = Site.new
site.name = name
site.grid_resource = "gt2 #{value.url}/jobmanager-#{value.jm}"
+ site.gridftp = "gsiftp://#{value.url}"
site.app_dir = value.app_dir
site.data_dir = value.data_dir
site.port = start_port + ctr
@@ -201,9 +189,9 @@
site.submit_job(targetQueued - queued)
end
trip += 1
- sleep 60
# puts "#{name}: #{total}"
end
+ sleep 60
end
end
Modified: trunk/docs/siteguide/grid
===================================================================
--- trunk/docs/siteguide/grid 2011-08-09 20:20:05 UTC (rev 4969)
+++ trunk/docs/siteguide/grid 2011-08-09 21:53:47 UTC (rev 4970)
@@ -4,7 +4,8 @@
Overview of running on grid sites
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-* Get a DOEGrids cert. Then register it in the *OSG Engage VO*, and/or map it using +gx-request+ on TeraGrid sites.
+* Get a DOEGrids cert. Then register it in the *OSG Engage VO*, and/or
+map it using +gx-request+ on TeraGrid sites.
* Run +GridSetup+ to configure Swift to use the grid sites. This tests
for correct operation and creates a "green list" of good sites.
@@ -12,12 +13,22 @@
* Prepare an installation package for the programs you want to run on
grid sites via Swift, and install that package using +foreachsite+.
-* Run +RunWorkers+ to start and maintain a pool of Swift workers on
+* Run +swift-workers+ to start and maintain a pool of Swift workers on
each site.
* Run Swift scripts that use the grid site resources.
+NOTE: This revision only supports a single-entry sites file which uses
+provider staging and assumes that the necessary apps are locatable
+through the same tc entries (ie either absolute or PATH-relative
+paths) on all sites.
+NOTE: This revision has been testing using the bin/grid code from
+trunk (which gets installed into trunk's bin/ directory, and the base
+swift code from the 0.93 branch. No other configurations have been
+tested at the moment. I intend to put this code in bin/grid in 0.93,
+as it should have no ill affects on other Swift usage.
+
Requesting Access
~~~~~~~~~~~~~~~~~
@@ -106,20 +117,85 @@
get_greensites >greensites
-----
+You can repeatedly try the get_greensites command, which simply
+concatenates all the site names that sucessfully resturned an output
+file from site tests.
+Starting a single coaster service
+---------------------------------
+
+This single coaster service will service all grid sites:
+
+start-grid-service --loglevel INFO --throttle 3.99 --jobspernode 1 \
+ >& start-grid-service.out
+
+Starting workers on OSG sites
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Make sure that your "greensites" file is in the current working
+directory.
+
+The swiftDemand file should be set to contain the number of workers
+you want to start across all OSG sites. Eventually this will be set
+dynamically by watching your Swift script execute. (Note that this
+number only includes jobs started by the swift-workers factory
+command, not by any workers added manually from the TeraGrid - see
+below.
+
+The condor directory must be pre-created and will be used by Condor to
+return stdout and stderr files from the Condor jobs, which will
+execute the wrapper script "run-workers.sh".
+
+NOTE: this script is current built manually, and wraps around and
+transports the worker.pl script. This needs to be automated.
+
+-----
+echo 250 >swiftDemand mkdir -p condor
+
+swift-workers greensites extenci \
+ http://communicado.ci.uchicago.edu:$(cat service-0.wport) \
+ >& swift-workers.out &
+-----
+
+Adding workers from TeraGrid sites
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The job below can be used to submit jobs to TareGrid (Ranger only at
+the moment) to add more workers to the execution pool. The same
+requirements hold there as for OSG sites, namely, that the app tools
+listed in tc for the single execution site need to be locatable on the
+TeraGrid site(s).
+
+-----
+start-ranger-service --nodes 1 --walltime 00:10:00 --project TG-DBS123456N \
+ --queue development --user tg12345 --startservice no \
+ >& start-ranger-service.out
+-----
+
+NOTE: Change the project and user names to match your TeraGrid
+parameters.
+
Running Swift
~~~~~~~~~~~~~
Now that everything is in place, run Swift with the following command:
-----
-swift -sites.file sites.xml -tc.file tc.data catsn.swift -n=10
+export SWIFT_HEAP_MAX=6000m # Add this for very large scripts
+
+swift -config cf.ps -tc.file tc -sites.file sites.grid-ps.xml \
+ catsn.swift -n=10000 >& swift.out &
-----
-You should see several new files being created, called catsn.0001.out, catsn.0002.out, etc. Each of these
-files should contain the contents of what you placed into data.txt. If this happens, your job has run
-successfully on PADS!
+You should see several new files being created, called catsn.0001.out,
+catsn.0002.out, etc. Each of these files should contain the contents
+of what you placed into data.txt. If this happens, your job has run
+successfully on the grid sites.
More Help
~~~~~~~~~
-The best place for additional help is the Swift user mailing list. You can subscribe to this list at
-http://mail.ci.uchicago.edu/mailman/listinfo/swift-user. When submitting information, please send your sites.xml file, your tc.data, and any Swift log files that were created during your attempt.
+
+The best place for additional help is the Swift user mailing list. You
+can subscribe to this list at
+http://mail.ci.uchicago.edu/mailman/listinfo/swift-user. When
+submitting information, please send your sites.xml file, your tc.data,
+and any Swift log files that were created during your attempt.
From davidk at ci.uchicago.edu Tue Aug 9 20:39:54 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Tue, 9 Aug 2011 20:39:54 -0500 (CDT)
Subject: [Swift-commit] r4971 - in branches/release-0.93/docs: . siteguide
stylesheets
Message-ID: <20110810013954.22C769CCBF@svn.ci.uchicago.edu>
Author: davidk
Date: 2011-08-09 20:39:53 -0500 (Tue, 09 Aug 2011)
New Revision: 4971
Added:
branches/release-0.93/docs/README
branches/release-0.93/docs/siteguide/images
branches/release-0.93/docs/stylesheets/
branches/release-0.93/docs/stylesheets/asciidoc.css
Removed:
branches/release-0.93/docs/siteguide/siteguide.html
Modified:
branches/release-0.93/docs/build_docs.sh
branches/release-0.93/docs/siteguide/pads
Log:
Better error handling in build_docs.sh
Link to images in siteguide
Apply stylesheets to html
Added: branches/release-0.93/docs/README
===================================================================
--- branches/release-0.93/docs/README (rev 0)
+++ branches/release-0.93/docs/README 2011-08-10 01:39:53 UTC (rev 4971)
@@ -0,0 +1,11 @@
+Prerequisites to building documentation:
+
+* asciidoc and a2x (http://www.methods.co.nz/asciidoc)
+* dblatex (http://dblatex.sourceforge.net)
+* source-highlight (http://www.gnu.org/software/src-highlite)
+
+For CI users, compiled versions of these packages can be found in ~davidk.
+
+To build, run:
+
+build_docs.sh
Modified: branches/release-0.93/docs/build_docs.sh
===================================================================
--- branches/release-0.93/docs/build_docs.sh 2011-08-09 21:53:47 UTC (rev 4970)
+++ branches/release-0.93/docs/build_docs.sh 2011-08-10 01:39:53 UTC (rev 4971)
@@ -14,12 +14,12 @@
GROUP="vdl2-svn"
# Verify correct arguments
-if [ -n "$1" ]; then
- INSTALLATION_DIRECTORY=$1
-else
- crash "Error: Must specify a directory for installation"
+if [ -z "$1" ] || [ "$1" == "-h" ] || [ "$1" == "-help" ]; then
+ crash "Usage: ./build_docs.sh /path/to/copy/output"
fi
+INSTALLATION_DIRECTORY=$1
+
# Create installation directory if needed
if [ ! -d "$INSTALLATION_DIRECTORY" ]; then
mkdir $INSTALLATION_DIRECTORY || crash "Unable to create directory $INSTALLATION_DIRECTORY"
@@ -43,7 +43,7 @@
for file in $FILES
do
echo Converting $directory"$file" to HTML
- asciidoc -a toc -a max-width=750px $file
+ asciidoc -a toc -a max-width=750px -a stylesheet=$(pwd)/../stylesheets/asciidoc.css $file
echo Converting $directory"$file" to PDF
a2x --format=pdf --no-xmllint $file
done
Added: branches/release-0.93/docs/siteguide/images
===================================================================
--- branches/release-0.93/docs/siteguide/images (rev 0)
+++ branches/release-0.93/docs/siteguide/images 2011-08-10 01:39:53 UTC (rev 4971)
@@ -0,0 +1 @@
+link ../tutorial/images
\ No newline at end of file
Property changes on: branches/release-0.93/docs/siteguide/images
___________________________________________________________________
Added: svn:special
+ *
Modified: branches/release-0.93/docs/siteguide/pads
===================================================================
--- branches/release-0.93/docs/siteguide/pads 2011-08-09 21:53:47 UTC (rev 4970)
+++ branches/release-0.93/docs/siteguide/pads 2011-08-10 01:39:53 UTC (rev 4971)
@@ -165,7 +165,7 @@
The swift script we will run is called catsn.swift. It simply cats a file and saves the result. This is a nice simple test to ensure jobs are running correctly. Create a file called data.txt which contains some simple input - a "hello world" will do the trick.
-----
-include::../../examples/catsn.swift[]
+include::../../examples/misc/catsn.swift[]
-----
Running Swift
Deleted: branches/release-0.93/docs/siteguide/siteguide.html
===================================================================
--- branches/release-0.93/docs/siteguide/siteguide.html 2011-08-09 21:53:47 UTC (rev 4970)
+++ branches/release-0.93/docs/siteguide/siteguide.html 2011-08-10 01:39:53 UTC (rev 4971)
@@ -1,964 +0,0 @@
-
-
-
-
-
-Site Configuration Guide
-
-
-
-
-
-
Site Configuration Guide
-
-
Table of Contents
-
-
-
-
-
1. PADS
-
-
1.1. Requesting Access
-
If you do not already have a Computation Institute account, you can request
-one at https://www.ci.uchicago.edu/accounts/. This page will give you a list
-of resources you can request access to. Be sure that PADS is selected. If
-you already have an existing CI account, but do not have access to PADS,
-send an email to support at ci.uchicago.edu to request access.
Once your keys are configured, you should be able to access a PADS login
-node with the following command:
-
-
-
ssh yourusername at login.pads.ci.uchicago.edu
-
-
1.4. Configuring softenv
-
Softenv is a system used for managing applications. In order to run Swift,
-the softenv environment will have to be modified slightly. Softenv is
-configured by a file in your home directory called .soft. Edit this file
-to look like this:
PADS requires that you are a member of a project. You can determine this by
-running the following command:
-
-
-
$ projects --available
-
-The following projects are available for your use
-
-Project PI Title
-
-CI-CCR000013 Michael Wilde The Swift Parallel Scripting System
You should make sure that you have a project set as default. Run
-the projects command with no arguments to determine if you have a default.
-
-
-
$ projects
-You have no default project set.
-
-
To set your default project, use projects --set
-
-
-
$ projects --set CI-CCR000013 --all
-Your default project for all CI clusters has been set to CI-CCR000013.
-
-
1.7.3. Determine your Queue
-
PADS has several different queues you can submit jobs to depending on
-the type of work you will be doing. The command "qstat -q" will print
-the most up to date list of this information.
-
-
-
Table 1. PADS Queues
-
-
-
-
-
-
-
-
-
-
-
-
Queue
-
Memory
-
CPU Time
-
Walltime
-
Node
-
Run
-
Que
-
Lm
-
State
-
-
-
-
-
route
-
—
-
—
-
—
-
—
-
0
-
0
-
—
-
E R
-
-
-
short
-
—
-
—
-
04:00:00
-
—
-
64
-
0
-
—
-
E R
-
-
-
extended
-
—
-
—
-
—
-
—
-
0
-
0
-
—
-
E R
-
-
-
fast
-
—
-
—
-
01:00:00
-
1
-
0
-
152
-
—
-
E R
-
-
-
long
-
—
-
—
-
24:00:00
-
—
-
232
-
130
-
—
-
E R
-
-
-
-
-
1.8. Generating Configuration Files
-
Now that you know what queue to use, your project, and your work directory, it is time to
-set up Swift. Swift uses a configuration file called sites.xml to determine how it should run.
-There are two methods you can use for creating this file. You can manually edit
-the configuration file, or generate it with a utility called gensites.
-
1.8.1. Manually Editing sites.xml
-
Below is the template that is used by Swift’s test suite for running on PADS.
The values to note here are the ones that are listed between underscores. In the example above, they are QUEUE, and WORK. Queue is the PADS queue to use and WORK is the swift work directory. These are placeholder values you will need to modify to fit your needs. Copy and paste this template, replace the values, and call it sites.xml.
-
1.9. Manually Editing tc.data
-
Below is the tc.data file used by Swift’s test suite for running on PADS.
The swift script we will run is called catsn.swift. It simply cats a file and saves the result. This is a nice simple test to ensure jobs are running correctly. Create a file called data.txt which contains some simple input - a "hello world" will do the trick.
Now that everything is in place, run Swift with the following command:
-
-
-
swift -sites.file sites.xml -tc.file tc.data catsn.swift -n=10
-
-
You should see several new files being created, called catsn.0001.out, catsn.0002.out, etc. Each of these
-files should contain the contents of what you placed into data.txt. If this happens, your job has run
-successfully on PADS!
-
1.12. More Help
-
The best place for additional help is the Swift user mailing list. You can subscribe to this list at
-https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user. When submitting information, please send your sites.xml file, your tc.data, and any Swift log files that were created during your attempt.
-
-
2. Beagle
-
-
2.1. Requesting Access
-
If you do not already have a Computation Institute account, you can request
-one at https://www.ci.uchicago.edu/accounts/. This page will give you a list
-of resources you can request access to.
-You already have an existing CI account, but do not have access to Beagle,
-send an email to support at ci.uchicago.edu to request access.
-
2.2. Connecting to a login node
-
Once you have account, you should be able to access a Beagle login
-node with the following command:
-
-
-
ssh yourusername at login.beagle.ci.uchicago.edu
-
-
Follow the steps outlined below to get started with Swift on Beagle:
-
step 1. Load the Swift module on Beagle as follows: module load swift
-
step 2. Create and change to a directory where your Swift related work will
-stay. (say, mkdir swift-lab, followed by, cd swift-lab)
-
step 3. To get started with a simple example running /bin/cat to read an
-input file data.txt and write to an output file f.nnn.out, copy the folder
-at /home/ketan/catsn to the above directory. (cp -r /home/ketan/catsn
-. followed by cd catsn).
-
step 4. In the sites file: beagle-coaster.xml, make the following two
-changes: 1) change the path of workdirectory to your preferred location
-(say to /lustre/beagle/$USER/swift-lab/swift.workdir) and 2) Change the
-project name to your project (CI-CCR000013) . The workdirectory will contain
-execution data related to each run, e.g. wrapper scripts, system information,
-inputs and outputs.
-
step 5. Run the example using following commandline (also found in run.sh):
-swift -config cf -tc.file tc -sites.file beagle-coaster.xml catsn.swift -n=1
-. You can further change the value of -n to any arbitrary number to run that
-many number of concurrent cat
-
step 6. Check the output in the generated outdir directory (ls outdir)
-
Note: Running from sandbox node or requesting 1 hour walltime for upto 3 nodes
-will get fast prioritized execution. Good for small tests.
If you do not already have a futuregrid account, you can follow the
-instructions at https://portal.futuregrid.org/gettingstarted to get started.
-This page provides information on how to create an account, how to join
-a project, how to set up your SSH keys, and how to create a new project.
-
3.4. Downloading Swift VM Tools
-
A set of scripts based around cloudinitd are used to easily start virtual
-machines. To download, change to your home directory and run the
-following command:
-
-
-
$ svn co https://svn.ci.uchicago.edu/svn/vdl2/usertools/swift-vm-boot
-
-
3.5. Download your Credentials
-
Run the following commands to retrieve your credentials:
-
-
-
$ scp yourusername at hotel.futuregrid.org:nimbus_creds.tar.gz .
-$ tar xvfz nimbus_creds.tar.gz
-
-
When you extract your credential file, look at the file called
-hotel.conf. Near the bottom of this file will be two settings
-called vws.repository.s3id and vws.repository.s3key. Copy these
-values for the next step.
-
3.6. Configuring Swift VM Tools
-
Change directories into the swift-vm-boot directory. Edit the file
-called env.sh. There are two settings here that you will need to
-modify: FUTUREGRID_IAAS_ACCESS_KEY and FUTUREGRID_IAAS_SECRET_KEY.
-Paste your access key (s3id) and secret key (s3key) here and save
-the file. By default, env.sh requests 2 nodes on hotel and 2 nodes
-on sierra. Change these values as needed. Run install.sh from the
-swift-vm-boot directory to complete the installation of Swift VM
-Tools. The Swift VM Tools may require a fairly recent version of
-Python. If you run into problems while running the install.sh
-script, please try a more recent version of Python and associated
-libraries.
-
3.7. Configuring coaster-service.conf
-
Within your swift distribution’s etc directory is a file called
-coaster-service.conf. There are many options here you can modify
-as needed, but these are the settings that will be required
-to run on futuregrid:
These is a brief description of these settings in the coaster-service.conf file.
-Tunneling should be used when you are restricted by a firewall. If your local
-machine has multiple network interfaces, you should also manually set the IP
-address of your machine with export IPADDR=your.ip.address.here.
-
3.8. Starting the Coaster Service Script
-
Now that everything is configured, change to a temporary directory
-and run this command to start the coaster service:
-
-
-
$ start-coaster-service
-
-
This command will start the VMs, start the required processes on the worker nodes,
-and generate Swift configuration files for you to use. The configuration files
-will be generated in your current directory. These files are sites.xml, tc.data,
-and cf.
-
3.9. Running Swift
-
Now that you have all of your configuration files generated, run the following command:
-
-
-
$ swift -sites.file sites.xml -tc.file tc.data -config cf <yourscript.swift>
-
-
If you would like to create a custom tc file for repeated use, rename it to something other
-than tc.data to prevent it from being overwritten. The sites.xml however will need to be
-regenerated every time you start the coaster service. If you need to repeatedly modify some
-sites.xml options, you may edit the template in Swift’s etc/sites/persistent-coasters. You
-may also create your own custom tc files with the hostname of persistent-coasters. More
-information about this can be found in the Swift userguide at
-http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html.
-
3.10. Stopping the Coaster Service Script
-
To stop the coaster service, run the following command:
-
-
-
$ stop-coaster-service
-
-
This will kill the coaster service, kill the worker scripts on remote systems and terminate
-the virtual machines that were created during start-coaster-service.
-
3.11. More Help
-
The best place for additional help is the Swift user mailing list. You can subscribe to this list at
-http://mail.ci.uchicago.edu/mailman/listinfo/swift-user. When submitting information, please send
-your sites.xml file, your tc.data, and any error messages you run into.
-
-
-
-
-
-
Added: branches/release-0.93/docs/stylesheets/asciidoc.css
===================================================================
--- branches/release-0.93/docs/stylesheets/asciidoc.css (rev 0)
+++ branches/release-0.93/docs/stylesheets/asciidoc.css 2011-08-10 01:39:53 UTC (rev 4971)
@@ -0,0 +1,2 @@
+a:link { color:navy; }
+a:visited { color:navy; }
From davidk at ci.uchicago.edu Tue Aug 9 20:48:34 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Tue, 9 Aug 2011 20:48:34 -0500 (CDT)
Subject: [Swift-commit] r4972 - in trunk/docs: . siteguide stylesheets
Message-ID: <20110810014834.BC0B29CCBF@svn.ci.uchicago.edu>
Author: davidk
Date: 2011-08-09 20:48:34 -0500 (Tue, 09 Aug 2011)
New Revision: 4972
Added:
trunk/docs/README
trunk/docs/siteguide/images
trunk/docs/stylesheets/
trunk/docs/stylesheets/asciidoc.css
Modified:
trunk/docs/build_docs.sh
Log:
Various doc updates
Added: trunk/docs/README
===================================================================
--- trunk/docs/README (rev 0)
+++ trunk/docs/README 2011-08-10 01:48:34 UTC (rev 4972)
@@ -0,0 +1,11 @@
+Prerequisites to building documentation:
+
+* asciidoc and a2x (http://www.methods.co.nz/asciidoc)
+* dblatex (http://dblatex.sourceforge.net)
+* source-highlight (http://www.gnu.org/software/src-highlite)
+
+For CI users, compiled versions of these packages can be found in ~davidk.
+
+To build, run:
+
+build_docs.sh
Modified: trunk/docs/build_docs.sh
===================================================================
--- trunk/docs/build_docs.sh 2011-08-10 01:39:53 UTC (rev 4971)
+++ trunk/docs/build_docs.sh 2011-08-10 01:48:34 UTC (rev 4972)
@@ -14,12 +14,12 @@
GROUP="vdl2-svn"
# Verify correct arguments
-if [ -n "$1" ]; then
- INSTALLATION_DIRECTORY=$1
-else
- crash "Error: Must specify a directory for installation"
+if [ -z "$1" ] || [ "$1" == "-h" ] || [ "$1" == "-help" ]; then
+ crash "Usage: ./build_docs.sh /path/to/copy/output"
fi
+INSTALLATION_DIRECTORY=$1
+
# Create installation directory if needed
if [ ! -d "$INSTALLATION_DIRECTORY" ]; then
mkdir $INSTALLATION_DIRECTORY || crash "Unable to create directory $INSTALLATION_DIRECTORY"
@@ -43,7 +43,7 @@
for file in $FILES
do
echo Converting $directory"$file" to HTML
- asciidoc -a toc -a max-width=750px $file
+ asciidoc -a toc -a max-width=750px -a stylesheet=$(pwd)/../stylesheets/asciidoc.css $file
echo Converting $directory"$file" to PDF
a2x --format=pdf --no-xmllint $file
done
Added: trunk/docs/siteguide/images
===================================================================
--- trunk/docs/siteguide/images (rev 0)
+++ trunk/docs/siteguide/images 2011-08-10 01:48:34 UTC (rev 4972)
@@ -0,0 +1 @@
+link ../tutorial/images
\ No newline at end of file
Property changes on: trunk/docs/siteguide/images
___________________________________________________________________
Added: svn:special
+ *
Added: trunk/docs/stylesheets/asciidoc.css
===================================================================
--- trunk/docs/stylesheets/asciidoc.css (rev 0)
+++ trunk/docs/stylesheets/asciidoc.css 2011-08-10 01:48:34 UTC (rev 4972)
@@ -0,0 +1,2 @@
+a:link { color:navy; }
+a:visited { color:navy; }
From jonmon at ci.uchicago.edu Tue Aug 9 23:30:00 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Tue, 9 Aug 2011 23:30:00 -0500 (CDT)
Subject: [Swift-commit] r4973 - in SwiftApps/SwiftMontage: . bin scripts
Message-ID: <20110810043000.B2A5A9CCB5@svn.ci.uchicago.edu>
Author: jonmon
Date: 2011-08-09 23:30:00 -0500 (Tue, 09 Aug 2011)
New Revision: 4973
Added:
SwiftApps/SwiftMontage/bin/
SwiftApps/SwiftMontage/bin/SwiftMontage
SwiftApps/SwiftMontage/setup.sh
Removed:
SwiftApps/SwiftMontage/exec/
Modified:
SwiftApps/SwiftMontage/TODO
SwiftApps/SwiftMontage/scripts/SwiftMontage_Batch.swift
Log:
o added setup.sh file that sets certain environment variables need by SwiftMontage
o Modified TODO file, removed the montage.sh entry
o removed exec directory
o added bin directory and SwiftMontage execution script. This is written in python.
Modified: SwiftApps/SwiftMontage/TODO
===================================================================
--- SwiftApps/SwiftMontage/TODO 2011-08-10 01:48:34 UTC (rev 4972)
+++ SwiftApps/SwiftMontage/TODO 2011-08-10 04:30:00 UTC (rev 4973)
@@ -1,9 +1,6 @@
o scripts/Swift_Montage_Batch.swift
-- Need to fix mProjectBatch. Currently calls mProject blindly. Need to figure out how to choose between mProject and mProjectPP
-o exec/montage.sh
- -- Figure out how to specify what site location that are going to be run on and do gensites on the templates
-
o optimizations
-- Try different sites.xml settings. Try different throttleing parameters as well as different queue sizes.
Added: SwiftApps/SwiftMontage/bin/SwiftMontage
===================================================================
--- SwiftApps/SwiftMontage/bin/SwiftMontage (rev 0)
+++ SwiftApps/SwiftMontage/bin/SwiftMontage 2011-08-10 04:30:00 UTC (rev 4973)
@@ -0,0 +1,87 @@
+#!/usr/bin/python
+
+import os
+import subprocess
+import sys
+import argparse
+
+def setup_opts():
+ parser = argparse.ArgumentParser(description="Run a Montage workflow using the Swift parallel scripting lanugage.")
+ parser.add_argument("--rawdir", dest="raw_dir", action="store",
+ help="Specify which directory to use for the raw image data.")
+
+ parser.add_argument("--unrectified", dest="unrectified", action="store_true",
+ help="Generate a mosaic with no background rectification")
+
+ parser.add_argument("--workflow", dest="workflow",
+ help="Use the specified Swiftscript instead of the default built in SwiftMontage one.")
+
+ parser.add_argument("--execution-sites", dest="sites", nargs="+",
+ help="The site list to use for execution")
+
+ args = parser.parse_args()
+
+ return args
+
+def create_work_directory():
+
+ # default is SwiftMontage_run.0, if there are other directories then adjust the SwiftMontage_run directory count
+ SwiftMontage_run_directory = "SwiftMontage_run.0"
+ last_SwiftMontage_run = 0
+ directories = os.listdir( os.getcwd() )
+
+ # This might cause a performance problem(to even begin to start Swift)
+ # Need to figure out how maybe purge these SwiftMontage_run directories
+ # For now I can manually purge
+ SwiftMontage_runs = []
+ for direc in directories:
+ ( name, sep, SwiftMontage_run_num ) = direc.partition( "." )
+ try:
+ SwiftMontage_runs.append( int(SwiftMontage_run_num) )
+ except ValueError:
+ # Don't worry about files that are not SwiftMontage_run directories
+ pass
+
+ SwiftMontage_runs.sort()
+
+ # if the directories list is not empty, create the name of the SwiftMontage_run directory
+ if SwiftMontage_runs:
+ last_SwiftMontage_run = SwiftMontage_runs[ len(SwiftMontage_runs)-1 ] + 1
+ SwiftMontage_run_directory = "SwiftMontage_run."+str(last_SwiftMontage_run)
+
+ return (SwiftMontage_run_directory, last_SwiftMontage_run)
+
+def main():
+ rc = 0
+ args = setup_opts()
+
+ (work_directory, last) = create_work_directory()
+ if args.sites:
+ execution_sites=" ".join( map ( str, args.sites ) )
+ sys.stdout.write( "Executing Swift on \"" + execution_sites + "\"\n" )
+ else:
+ sys.stdout.write("Executing Swift on \"localhost\"\n" )
+ sys.stdout.write( "Execution directory: " + work_directory + "\n\n" )
+
+ try:
+ os.mkdir( work_directory )
+ except os.error:
+ sys.stderr.write( "Could not create SwiftMontage_run directory "+work_directory+". Exiting.\n")
+ return 1
+
+ if args.workflow:
+ sys.stdout.write("Using user provided Swiftscript!\n")
+ rc = 0 # Call definition for user provided script
+ elif args.unrectified:
+ sys.stdout.write("Generating mosaic without background rectification!\n")
+ rc = 0 # Call definistion for default montage workflow(not rectification)
+ elif not args.unrectified:
+ sys.stdout.write("Generating mosaic with background rectification!\n")
+ rc = 0 # Call definistion for default montage workflow(with rectification)
+
+ return rc
+
+if __name__ == "__main__":
+ rc = main()
+ sys.exit( rc )
+
Property changes on: SwiftApps/SwiftMontage/bin/SwiftMontage
___________________________________________________________________
Added: svn:executable
+ *
Modified: SwiftApps/SwiftMontage/scripts/SwiftMontage_Batch.swift
===================================================================
--- SwiftApps/SwiftMontage/scripts/SwiftMontage_Batch.swift 2011-08-10 01:48:34 UTC (rev 4972)
+++ SwiftApps/SwiftMontage/scripts/SwiftMontage_Batch.swift 2011-08-10 04:30:00 UTC (rev 4973)
@@ -25,7 +25,7 @@
foreach d_entry, i in diffs
{
- Image img_1 ;
+ Image img_1 ;
Image img_2 ;
Image diff_img ;
@@ -42,15 +42,15 @@
Status stats[];
Table status_tbl = create_status_table( diff_tbl );
-
+
foreach d_entry, i in diffs
{
- Image img_1 ;
+ Image img_1 ;
Image img_2 ;
Image diff_img ;
Status stat_file ;
-
+
( diff_img, stat_file ) = mDiffFit( img_1, img_2, hdr );
diff_imgs[ i ] = diff_img;
Added: SwiftApps/SwiftMontage/setup.sh
===================================================================
--- SwiftApps/SwiftMontage/setup.sh (rev 0)
+++ SwiftApps/SwiftMontage/setup.sh 2011-08-10 04:30:00 UTC (rev 4973)
@@ -0,0 +1,7 @@
+SWIFTMONTAGE_HOME=`pwd`
+
+export PATH=$SWIFTMONTAGE_HOME/bin:$PATH
+export SWIFT_LIB=$SWIFTMONTAGE_HOME/scripts:$SWIFT_LIB
+
+export SWIFTMONTAGE_TEMPLATES=$SWIFTMONTAGE_HOME/templates
+export SWIFTMONTAGE_SCRIPTS=$SWIFTMONTAGE_HOME/scripts
From jonmon at ci.uchicago.edu Tue Aug 9 23:32:18 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Tue, 9 Aug 2011 23:32:18 -0500 (CDT)
Subject: [Swift-commit] r4974 - branches/release-0.93/bin
Message-ID: <20110810043218.1F4889CCB5@svn.ci.uchicago.edu>
Author: jonmon
Date: 2011-08-09 23:32:17 -0500 (Tue, 09 Aug 2011)
New Revision: 4974
Modified:
branches/release-0.93/bin/swift
Log:
o call chxml using the SWIFT_HOME environment variable.
Modified: branches/release-0.93/bin/swift
===================================================================
--- branches/release-0.93/bin/swift 2011-08-10 04:30:00 UTC (rev 4973)
+++ branches/release-0.93/bin/swift 2011-08-10 04:32:17 UTC (rev 4974)
@@ -63,7 +63,7 @@
# make sure sites.xml file is well-formed
-chxml $CMDLINE
+$SWIFT_HOME/bin/chxml $CMDLINE
### SETUP OTHER ENV VARIABLES ####
From davidk at ci.uchicago.edu Wed Aug 10 07:44:02 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Wed, 10 Aug 2011 07:44:02 -0500 (CDT)
Subject: [Swift-commit] r4975 - in wwwdev: . case_studies downloads links
main papers support usage
Message-ID: <20110810124402.0E9749CCBF@svn.ci.uchicago.edu>
Author: davidk
Date: 2011-08-10 07:44:00 -0500 (Wed, 10 Aug 2011)
New Revision: 4975
Removed:
wwwdev/main/_notes/
Modified:
wwwdev/case_studies/index.php
wwwdev/downloads/index.php
wwwdev/index.html
wwwdev/links/index.php
wwwdev/main/index.php
wwwdev/papers/index.php
wwwdev/support/index.php
wwwdev/usage/usage.php
Log:
Updated download links to 0.93
Disabled robots for wwwdev
Modified: wwwdev/case_studies/index.php
===================================================================
--- wwwdev/case_studies/index.php 2011-08-10 04:32:17 UTC (rev 4974)
+++ wwwdev/case_studies/index.php 2011-08-10 12:44:00 UTC (rev 4975)
@@ -2,6 +2,7 @@
+
Swift - Case Studies
Modified: wwwdev/downloads/index.php
===================================================================
--- wwwdev/downloads/index.php 2011-08-10 04:32:17 UTC (rev 4974)
+++ wwwdev/downloads/index.php 2011-08-10 12:44:00 UTC (rev 4975)
@@ -41,15 +41,14 @@
Checkout CoG:
-
svn co \
-https://cogkit.svn.sourceforge.net/svnroot/cogkit/tags/swift_0.92.1 cog
+
svn co https://cogkit.svn.sourceforge.net/svnroot/cogkit/branches/4.1.9/src/cog
Checkout Swift:
cd cog/modules
-
svn co https://svn.ci.uchicago.edu/svn/vdl2/tags/release-0.92.1 swift
+
svn co https://svn.ci.uchicago.edu/svn/vdl2/branches/release-0.93 swift
Modified: wwwdev/index.html
===================================================================
--- wwwdev/index.html 2011-08-10 04:32:17 UTC (rev 4974)
+++ wwwdev/index.html 2011-08-10 12:44:00 UTC (rev 4975)
@@ -2,8 +2,9 @@
Redirecting to Swift Home Page
+
Redirecting to Swift Home Page
-
\ No newline at end of file
+
Modified: wwwdev/links/index.php
===================================================================
--- wwwdev/links/index.php 2011-08-10 04:32:17 UTC (rev 4974)
+++ wwwdev/links/index.php 2011-08-10 12:44:00 UTC (rev 4975)
@@ -2,6 +2,7 @@
+
Swift: Links
Modified: wwwdev/main/index.php
===================================================================
--- wwwdev/main/index.php 2011-08-10 04:32:17 UTC (rev 4974)
+++ wwwdev/main/index.php 2011-08-10 12:44:00 UTC (rev 4975)
@@ -2,6 +2,7 @@
+
Swift - A system for rapid and reliable specification, execution, and management of large scale workflows
Modified: wwwdev/papers/index.php
===================================================================
--- wwwdev/papers/index.php 2011-08-10 04:32:17 UTC (rev 4974)
+++ wwwdev/papers/index.php 2011-08-10 12:44:00 UTC (rev 4975)
@@ -2,6 +2,7 @@
+
Swift - Papers
Modified: wwwdev/support/index.php
===================================================================
--- wwwdev/support/index.php 2011-08-10 04:32:17 UTC (rev 4974)
+++ wwwdev/support/index.php 2011-08-10 12:44:00 UTC (rev 4975)
@@ -2,6 +2,7 @@
+
Swift - Support
Modified: wwwdev/usage/usage.php
===================================================================
--- wwwdev/usage/usage.php 2011-08-10 04:32:17 UTC (rev 4974)
+++ wwwdev/usage/usage.php 2011-08-10 12:44:00 UTC (rev 4975)
@@ -1,4 +1,6 @@
+
+
@@ -9,6 +11,7 @@
var requiredRevision = 45;
-->
+
include("table.html");
From davidk at ci.uchicago.edu Wed Aug 10 14:09:55 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Wed, 10 Aug 2011 14:09:55 -0500 (CDT)
Subject: [Swift-commit] r4976 - in wwwdev: inc main
Message-ID: <20110810190955.5E0059CCBF@svn.ci.uchicago.edu>
Author: davidk
Date: 2011-08-10 14:09:55 -0500 (Wed, 10 Aug 2011)
New Revision: 4976
Modified:
wwwdev/inc/footer2.php
wwwdev/inc/header2.php
wwwdev/main/index.php
Log:
Hide some of the features that aren't ready to use yet, like case studies and supported platforms
Modified: wwwdev/inc/footer2.php
===================================================================
--- wwwdev/inc/footer2.php 2011-08-10 12:44:00 UTC (rev 4975)
+++ wwwdev/inc/footer2.php 2011-08-10 19:09:55 UTC (rev 4976)
@@ -2,10 +2,12 @@
-
\ No newline at end of file
+
Modified: wwwdev/inc/header2.php
===================================================================
--- wwwdev/inc/header2.php 2011-08-10 12:44:00 UTC (rev 4975)
+++ wwwdev/inc/header2.php 2011-08-10 19:09:55 UTC (rev 4976)
@@ -13,7 +13,9 @@
\ No newline at end of file
+
Modified: wwwdev/main/index.php
===================================================================
--- wwwdev/main/index.php 2011-08-10 12:44:00 UTC (rev 4975)
+++ wwwdev/main/index.php 2011-08-10 19:09:55 UTC (rev 4976)
@@ -80,17 +80,21 @@
read the papers
-
+
+
+
What's New
From davidk at ci.uchicago.edu Wed Aug 10 14:17:44 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Wed, 10 Aug 2011 14:17:44 -0500 (CDT)
Subject: [Swift-commit] r4977 - wwwdev
Message-ID: <20110810191744.718A39CCBF@svn.ci.uchicago.edu>
Author: davidk
Date: 2011-08-10 14:17:44 -0500 (Wed, 10 Aug 2011)
New Revision: 4977
Modified:
wwwdev/push_to.sh
Log:
Add some of the new files to the push list
Modified: wwwdev/push_to.sh
===================================================================
--- wwwdev/push_to.sh 2011-08-10 19:09:55 UTC (rev 4976)
+++ wwwdev/push_to.sh 2011-08-10 19:17:44 UTC (rev 4977)
@@ -19,6 +19,7 @@
# Long list of files to explicitly include follows
FILES="
inc/footer.php
+inc/footer2.php
inc/home_sidebar.php
inc/papers_sidebar.php
inc/docs_sidebar.php
@@ -26,6 +27,7 @@
inc/downloads_sidebar.php
inc/support_sidebar.php
inc/header.php
+inc/header2.php
inc/links_sidebar.php
inc/nav.php
inc/apps_sidebar.php
From davidk at ci.uchicago.edu Wed Aug 10 15:23:00 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Wed, 10 Aug 2011 15:23:00 -0500 (CDT)
Subject: [Swift-commit] r4978 - wwwdev/main
Message-ID: <20110810202300.117619CCBF@svn.ci.uchicago.edu>
Author: davidk
Date: 2011-08-10 15:22:59 -0500 (Wed, 10 Aug 2011)
New Revision: 4978
Modified:
wwwdev/main/index.php
Log:
Google analytics code for new site
Modified: wwwdev/main/index.php
===================================================================
--- wwwdev/main/index.php 2011-08-10 19:17:44 UTC (rev 4977)
+++ wwwdev/main/index.php 2011-08-10 20:22:59 UTC (rev 4978)
@@ -18,6 +18,20 @@
+
+
+
+
+
+
+
+
+
Swift v0.8 is a development release intended to release functionality
-and fixes that have gone in to trunk since v0.7.
-[swift-0.8.tar.gz]
-[swift-0.8-stripped.tar.gz - for installation on systems with existing grid stacks]
-[release-notes-0.8.txt]
-
-
-
-
-
-
Swift 0.7 - 2008/11/11
-
Swift v0.7 is a development release intended to release functionality
-and fixes that have gone in to trunk since v0.7.
-[vdsk-0.7.tar.gz]
-[release-notes-0.7.txt]
-
-
-As an alternative to the above traditional Swift packaging, Swift can be
-downloaded and installed using
-pacman, a package
-manager commonly used on the Open Science Grid.
-
-
-There are two installation targets:
-The first will install Swift alongside an existing VDT installation
-
-The second will install Swift as well as a number
-of supporting packages from the VDT software release to support use of the
-DOE CA and use of VOMS. (These packages are also available in a regular OSG or
-VDT installation):
-
Swift v0.6 is a development release intended to release functionality
-and fixes that have gone in to trunk since v0.5.
-[vdsk-0.6.tar.gz]
-[release-notes-0.6.txt]
-
-
-As an alternative to the above traditional Swift packaging, Swift can be
-downloaded and installed using
-pacman, a package
-manager commonly used on the Open Science Grid.
-
-
-This will install Swift alongside an existing VDT installation:
-
Swift v0.5 is a development release intended to release functionality
-and fixes that have gone in to trunk since v0.4.
-[vdsk-0.5.tar.gz]
-[release-notes-0.5.txt]
-
-
Swift 0.4 - 2008/03/18
-
Swift v0.4 is a development release intended to release functionality
-and fixes that have gone in to trunk since v0.3. More details are contained
-in the release notes.
-[vdsk-0.4.tar.gz]
-[release-notes-0.4.txt]
-
-
-
Swift 0.3 - 2007/10/04
-
Swift v0.3 is a development release intended to release functionality
-and fixes that have gone in to trunk since v0.2.
-[vdsk-0.3.tar.gz]
-
-
-
Swift 0.2 - 2007/07/19
-
Swift v0.2 is a development release intended to release functionality
-and fixes that have gone in to trunk since v0.1.
-[vdsk-0.2.tar.gz]
-
-
-
Swift 0.1 - 2007/03/02
-
Swift v0.1 is a development release intended to bring functionality
-and fixes that have gone in to trunk since version 0 in a known release.
-[vdsk-0.1.tar.gz]
-
-
-
Swift 0 RC3
-
- The 3rd release candidate of Swift version 0:
- [tar.gz] A bug that caused applications
- correctly declared in tc.data to not be recognized was fixed For getting
- started with Swift, please take a look at the documentation page
-
-
-
Swift 0 RC2
-
-
- The 2nd release candidate of Swift version 0:
- [tar.gz] This release candidate
- features improvements in startup speed and the addition of ENV:: and
- GLOBUS:: profiles.
-
-
-
Swift 0 RC1
-
-
- The 1st release candidate of Swift version 0:
- [tar.gz]
-
-
-
Swift 0 RC0
-
-
- The 0th release candidate of Swift version 0:
- [tar.gz]
-
Read our Quick Start Guide
and start using Swift today!
From davidk at ci.uchicago.edu Sun Aug 28 23:27:45 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Sun, 28 Aug 2011 23:27:45 -0500 (CDT)
Subject: [Swift-commit] r5016 - in wwwdev: css downloads inc support
Message-ID: <20110829042745.AD97F9CC9D@svn.ci.uchicago.edu>
Author: davidk
Date: 2011-08-28 23:27:45 -0500 (Sun, 28 Aug 2011)
New Revision: 5016
Modified:
wwwdev/css/style3.css
wwwdev/downloads/index.php
wwwdev/inc/downloads_sidebar.php
wwwdev/support/index.php
Log:
Added details on support page
Edited instructions on how to build
Added description of differences between source/binary/stable/trunk
Increased font size on support page
Modified: wwwdev/css/style3.css
===================================================================
--- wwwdev/css/style3.css 2011-08-28 03:34:06 UTC (rev 5015)
+++ wwwdev/css/style3.css 2011-08-29 04:27:45 UTC (rev 5016)
@@ -166,13 +166,14 @@
.mailinglists {
width: 85%;
margin: 0 0 0 35px;
- font-size:10px;
+ font-size:12px;
}
.mailinglists td {
padding: 5px 5px;
line-height: 16px;
background-color: #f0f0f0;
+ font-size:15px;
}
.mailinglists th {
@@ -183,6 +184,7 @@
font-family: myriad-pro-1,myriad-pro-2, Arial, Helvetica, sans-serif;
font-weight:600;
font-style: normal;
+ font-size:15px;
background-color: #e0e0e0;
}
Modified: wwwdev/downloads/index.php
===================================================================
--- wwwdev/downloads/index.php 2011-08-28 03:34:06 UTC (rev 5015)
+++ wwwdev/downloads/index.php 2011-08-29 04:27:45 UTC (rev 5016)
@@ -26,6 +26,10 @@
Latest Release
Swift 0.93 RC - 2011/8/17
+For the majority of users, this is the version you will want to download. This package
+contains the latest Stable release of Swift. Since Swift is written in Java, this
+package will run on all supported platforms with Java Runtime Environment 1.5 or greater.
+
- You need
- Apache Ant
- for the actual build, and a Java JDK ≥ 1.5.
- So here are the steps:
+ Here are the steps:
-
- Checkout CoG:
-
svn co https://cogkit.svn.sourceforge.net/svnroot/cogkit/branches/4.1.9/src/cog
+
+ $ mkdir swift-0.93
+ $ cd swift-0.93
+ $ svn co https://cogkit.svn.sourceforge.net/svnroot/cogkit/branches/4.1.9/src/cog
+ $ cd cog/modules
+ $ svn co https://svn.ci.uchicago.edu/svn/vdl2/branches/release-0.93 swift
+ $ cd swift
+ $ ant redist
+
-
-
-
- Checkout Swift:
-
cd cog/modules
-
svn co https://svn.ci.uchicago.edu/svn/vdl2/branches/release-0.93 swift
-
-
-
-
- Change directory to the swift module:
-
cd swift
-
-
-
-
- Build the thing:
-
ant dist
-
-
-
-
The dist directory will contain the complete build.
-
+
Development Version
- Note: The development code has the highest chance of containing buggy and
- untested code. If you need stability please use the latest release.
+ The development version of Swift is aimed at developers and testers. The development
+ code has the highest chance of containing buggy and untested code. If you need
+ stability please use the latest release.
-
-
- Checkout CoG:
-
svn co \
-https://cogkit.svn.sourceforge.net/svnroot/cogkit/trunk/current/src/cog
-
-
-
-
- Checkout Swift:
-
cd cog/modules
-
svn co https://svn.ci.uchicago.edu/svn/vdl2/trunk swift
-
-
-
-
- Change directory to the swift module:
-
cd swift
-
-
-
-
- Build the thing:
-
ant dist
-
-
-
-
- The dist directory will contain the complete build.
-
+
+ $ mkdir swift-devel
+ $ cd swift-devel
+ $ svn co https://cogkit.svn.sourceforge.net/svnroot/cogkit/trunk/current/src/cog
+ $ cd cog/modules
+ $ svn co https://svn.ci.uchicago.edu/svn/vdl2/trunk swift
+ $ cd swift
+ $ ant redist
+
For previous release, see Older Downloads
Modified: wwwdev/inc/downloads_sidebar.php
===================================================================
--- wwwdev/inc/downloads_sidebar.php 2011-08-28 03:34:06 UTC (rev 5015)
+++ wwwdev/inc/downloads_sidebar.php 2011-08-29 04:27:45 UTC (rev 5016)
@@ -1,6 +1,6 @@
+ The best place to report problems and receive assistance is through the Swift mailing lists.
+ When reporting an issue, please be sure to include the Swift log files that were generated
+ as a part of your run. If the log files are too large to include in an email, please provide
+ access to the logs in a publicly readable directory or website. More information on these
+ mailing lists are provided below.
+
+
Mailing Lists
From davidk at ci.uchicago.edu Mon Aug 29 09:40:16 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Mon, 29 Aug 2011 09:40:16 -0500 (CDT)
Subject: [Swift-commit] r5017 - branches/release-0.93/docs/quickstart
Message-ID: <20110829144016.C994C9CC9D@svn.ci.uchicago.edu>
Author: davidk
Date: 2011-08-29 09:40:16 -0500 (Mon, 29 Aug 2011)
New Revision: 5017
Modified:
branches/release-0.93/docs/quickstart/quickstart.txt
Log:
Updates to quickstart guide
Modified: branches/release-0.93/docs/quickstart/quickstart.txt
===================================================================
--- branches/release-0.93/docs/quickstart/quickstart.txt 2011-08-29 04:27:45 UTC (rev 5016)
+++ branches/release-0.93/docs/quickstart/quickstart.txt 2011-08-29 14:40:16 UTC (rev 5017)
@@ -8,38 +8,29 @@
and run the basic examples for Swift. If you are using a pre-installed
version of Swift, you can skip directly to the configuration section.
+Stable Releases vs. Development Releases
+----------------------------------------
+Stable releases of Swift have undergone more extensive testing than development releases.
+In general, they are more stable, have fewer bugs, and have been tested on a variety of
+systems.
+
+The development version of Swift is aimed at developers and testers. The development
+code has the highest chance of containing buggy and untested code. If you need stability
+please use the latest stable release.
+
Downloading a Swift Distribution
--------------------------------
-There are three main ways of getting the Swift implementation: stable
-releases, nightly builds, and the source code repository.
+There are two main ways of getting the Swift implementation: binary
+releases and the source repository.
-
-Stable Releases
+Binary Releases
~~~~~~~~~~~~~~~
-Stable releases can be obtained from the Swift download page: Swift
-Downloads Page. Once you downloaded the package, please move to the
-install section.
+For the majority of users, downloading and installing binary releases is recommended.
+Since Swift is written in Java, the binary packages will run on all supported platforms with
+Java Runtime Environment 1.5 or greater. Binary releases can be obtained from the
+http://www.ci.uchicago.edu/swift/downloads/index.php[Swift downloads page].
-
-Nightly Builds
-~~~~~~~~~~~~~~
-Swift builds and tests are being run every day. The Swift downloads page
-contains links to the latest build and test page. The nightly builds
-reflect a development version of the Swift code and should not be used in
-production mode. After downloading a nightly build package, please
-continue to the install section.
-
-Source Repository
-~~~~~~~~~~~~~~~~~
-Details about accessing the Swift source repository together with build
-instructions are available on the Swift downloads page. Once
-built, the dist/swift-svn directory will contain a self-contained
-build which can be used in place or moved to a different location. You
-should then proceed to the configuration section.
-
-Installing a Swift Binary Package
----------------------------------
-Simply unpack the downloaded package (swift-.tar.gz) into a
+Once downloaded, simply unpack the downloaded package (swift-.tar.gz) into a
directory of your choice:
-----
@@ -48,91 +39,63 @@
This will create a swift- directory containing the build.
+Source Repository
+~~~~~~~~~~~~~~~~~
+The source code for Swift is available to developers who have an interest in contributing
+new features. To build Swift from source code, you will need http://ant.apache.org/[Apache Ant]
+and http://www.oracle.com/technetwork/java/javase/downloads/index.html[Java JDK]. Once
+built, the dist/swift-svn directory will contain your build.
-Configuring Swift
------------------
-This section describes configuration steps that need to be taken in
-order to get Swift running. Since all command line tools provided with
-Swift can be found in the |bin/| directory of the Swift distribution, it
-may be a good idea to add this directory to your |PATH| environment
-variable:
-
+To download and build Swift 0.93, follow these instructions:
-----
-export PATH=/path/to/swift/bin:$PATH
+$ mkdir swift-0.93
+$ cd swift-0.93
+$ svn co https://cogkit.svn.sourceforge.net/svnroot/cogkit/branches/4.1.9/src/cog
+$ cd cog/modules
+$ svn co https://svn.ci.uchicago.edu/svn/vdl2/branches/release-0.93 swift
+$ cd swift
+$ ant redist
-----
-Grid Security
-~~~~~~~~~~~~~
-For local execution of jobs, no grid security configuration is necessary.
+Setting your PATH
+-----------------
+Once Swift is installed, it is useful to add the swift binary to your PATH
+environment variable. To do this, first determine where the Swift bin
+directory is located. If you installed Swift from a binary release, it will
+be in the swift-0.93/bin directory where you installed it. If you followed
+the instructions above for installing Swift from a source repository, it
+will be located in swift-0.93/cog/modules/swift/dist/swift-svn/bin.
-However, when submitting jobs to a remote machine using Globus Toolkit
-services, Swift makes use of the Grid Security Infrastructure (GSI)
-at http://www.globus.org/toolkit/docs/4.0/security/key-index.html for
-authentication and authorization. The requirements for this are detailed
-in the following sections. Note that GSI is not required to be
-configured for local execution (which will usually be the case when
-first starting with Swift).
+Add the following line to the bottom of ~/.bashrc:
-User Certificate
-~~~~~~~~~~~~~~~~
-GSI requires a certificate/private key pair for authentication to Globus
-Toolkit services. The certificate and private key should be placed into
-the ~/.globus/usercert.pem and ~/.globus/userkey.pem files, respectively.
+-----
+export PATH=$PATH:/full/path/to/swift
+-----
-
-Certificate Authorities Root Certificates
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The Swift client libraries are generally required to authenticate the
-services to which they connect. This process requires the presence on
-the Swift submit site of the root certificates used to sign the host
-certificates of services used. These root certificates need to be
-installed in either (or both) the ~/.globus/certificates and
-/etc/grid-security/certificates directories. A package with the root
-certificates of the certificate authorities used in the
-http://www.teragrid.org[TeraGrid] can be found
-http://security.teragrid.org/TG-CAs.html[here].
-
-Swift Properties
-~~~~~~~~~~~~~~~~
-A Swift properties file (named swift.properties) can be used to
-customize certain configuration aspects of Swift. A shared version of
-this file, etc/swift.properties in the installation directory can be
-used to provide installation-wide defaults. A per-user properties file,
-~/.swift/swift.properties can be used for user specific settings.
-Swift first loads the shared configuration file and, if present, the
-user configuration file. Any properties not explicitly set in the user
-configuration file will be inherited from the shared configuration file.
-Properties are specified in the following format:
-
+When you login, test this out by typing the command
-----
-name=value
+$ which swift
-----
-For details about the various properties Swift accepts, please take a
-look at the http://www.ci.uchicago.edu/swift/guides/userguide.php#properties[Swift Properties Section]
-in the http://www.ci.uchicago.edu/swift/guides/userguide.php[Swift User Guide].
+This should point you to the path of the Swift binary.
-
Running Swift Examples
----------------------
The Swift examples can be found in the examples directory in the Swift
distribution. The examples are written in the SwiftScript language, and
-have |.swift| as a file extension.
+have .swift as a file extension.
-The Grid Security Infrastructure, which Swift uses, works with limited
-time certificates called proxies. These proxies can be generated from
-your user certificate and private key using one of grid-proxy-init or
-cog-proxy-init (the latter being a Java Swing interface to the former).
-
Execution of a Swift workflow is done using the swift command, which
takes the Swift workflow file name as an argument:
-----
-cd examples/swift
-swift first.swift
+cd examples/tutorial
+swift hello.swift
-----
-The
-http://www.ci.uchicago.edu/swift/guides/userguide.php#swiftcommand[Swift Command Options Section]
-in the http://www.ci.uchicago.edu/swift/guides/userguide.php[Swift User Guide] contains details
-about the various options of swift.
+When you run this application, it should create a file called hello.txt.
+If this file gets created, you have successfully ran your first
+Swift script!
+
+More documentation on how to run Swift can be found at
+http://www.ci.uchicago.edu/swift/docs/index.php,
From davidk at ci.uchicago.edu Mon Aug 29 10:40:14 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Mon, 29 Aug 2011 10:40:14 -0500 (CDT)
Subject: [Swift-commit] r5018 - branches/release-0.93/examples/tutorial
Message-ID: <20110829154014.AC5559CCBE@svn.ci.uchicago.edu>
Author: davidk
Date: 2011-08-29 10:40:14 -0500 (Mon, 29 Aug 2011)
New Revision: 5018
Added:
branches/release-0.93/examples/tutorial/README
Log:
Added README to examples/tutorial
Added: branches/release-0.93/examples/tutorial/README
===================================================================
--- branches/release-0.93/examples/tutorial/README (rev 0)
+++ branches/release-0.93/examples/tutorial/README 2011-08-29 15:40:14 UTC (rev 5018)
@@ -0,0 +1,42 @@
+The files in this directory are examples from the
+http://www.ci.uchicago.edu/swift/guides/release-0.93/tutorial/tutorial.html[Swift tutorial].
+
+hello.swift
+ The Swift hello world program. It outputs a hello world message into a file
+ called hello.txt
+
+parameter.swift
+ Shows how to specify parameters to procedures
+
+manyparam.swift
+ Similar to parameter.swift, but called multiple times with different
+ parameters
+
+capitalise.swift
+ Gets input from a file, capitalizes the contents, and writes to a new file
+
+types.swift
+ How to use user-defined structures
+
+arrays.swift
+ Introduction to arrays
+
+regexp_mapper.swift
+ Example of how to use the regular expression mapper
+
+fixed_array_mapper.swift
+ Example of how to use the fixed array mapper
+
+foreach.swift
+ Demonstrates the usage of foreach by counting words in multiple files
+
+if.swift
+ Handling decisions with if
+
+sequential_iteration.swift
+ Demonstrates the usage of iterate
+
+default.swift
+ Shows named parameters and default values
+
+
From davidk at ci.uchicago.edu Mon Aug 29 13:32:10 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Mon, 29 Aug 2011 13:32:10 -0500 (CDT)
Subject: [Swift-commit] r5019 - in branches/release-0.93/tests: groups
providers providers/PADS providers/PADS/coasters providers/PADS/pbs
Message-ID: <20110829183210.C93C19CC9D@svn.ci.uchicago.edu>
Author: davidk
Date: 2011-08-29 13:32:10 -0500 (Mon, 29 Aug 2011)
New Revision: 5019
Added:
branches/release-0.93/tests/groups/providers-pads.sh
branches/release-0.93/tests/providers/PADS/
branches/release-0.93/tests/providers/PADS/coasters/
branches/release-0.93/tests/providers/PADS/coasters/README
branches/release-0.93/tests/providers/PADS/coasters/catsn.0001.out.expected
branches/release-0.93/tests/providers/PADS/coasters/catsn.0002.out.expected
branches/release-0.93/tests/providers/PADS/coasters/catsn.0003.out.expected
branches/release-0.93/tests/providers/PADS/coasters/catsn.0004.out.expected
branches/release-0.93/tests/providers/PADS/coasters/catsn.0005.out.expected
branches/release-0.93/tests/providers/PADS/coasters/catsn.0006.out.expected
branches/release-0.93/tests/providers/PADS/coasters/catsn.0007.out.expected
branches/release-0.93/tests/providers/PADS/coasters/catsn.0008.out.expected
branches/release-0.93/tests/providers/PADS/coasters/catsn.0009.out.expected
branches/release-0.93/tests/providers/PADS/coasters/catsn.0010.out.expected
branches/release-0.93/tests/providers/PADS/coasters/catsn.setup.sh
branches/release-0.93/tests/providers/PADS/coasters/catsn.swift
branches/release-0.93/tests/providers/PADS/coasters/catsn.timeout
branches/release-0.93/tests/providers/PADS/coasters/data.txt
branches/release-0.93/tests/providers/PADS/coasters/sites.template.xml
branches/release-0.93/tests/providers/PADS/coasters/tc.template.data
branches/release-0.93/tests/providers/PADS/pbs/
branches/release-0.93/tests/providers/PADS/pbs/catsn.0001.out.expected
branches/release-0.93/tests/providers/PADS/pbs/catsn.0002.out.expected
branches/release-0.93/tests/providers/PADS/pbs/catsn.0003.out.expected
branches/release-0.93/tests/providers/PADS/pbs/catsn.0004.out.expected
branches/release-0.93/tests/providers/PADS/pbs/catsn.0005.out.expected
branches/release-0.93/tests/providers/PADS/pbs/catsn.0006.out.expected
branches/release-0.93/tests/providers/PADS/pbs/catsn.0007.out.expected
branches/release-0.93/tests/providers/PADS/pbs/catsn.0008.out.expected
branches/release-0.93/tests/providers/PADS/pbs/catsn.0009.out.expected
branches/release-0.93/tests/providers/PADS/pbs/catsn.0010.out.expected
branches/release-0.93/tests/providers/PADS/pbs/catsn.setup.sh
branches/release-0.93/tests/providers/PADS/pbs/catsn.swift
branches/release-0.93/tests/providers/PADS/pbs/catsn.timeout
branches/release-0.93/tests/providers/PADS/pbs/data.txt
branches/release-0.93/tests/providers/PADS/pbs/sites.template.xml
branches/release-0.93/tests/providers/PADS/pbs/tc.template.data
Removed:
branches/release-0.93/tests/groups/group-pads.sh
Log:
Updated provider tests for PADS
Deleted: branches/release-0.93/tests/groups/group-pads.sh
===================================================================
--- branches/release-0.93/tests/groups/group-pads.sh 2011-08-29 15:40:14 UTC (rev 5018)
+++ branches/release-0.93/tests/groups/group-pads.sh 2011-08-29 18:32:10 UTC (rev 5019)
@@ -1,7 +0,0 @@
-
-# GROUPLIST definition to run pbs tests
-
-GROUPLIST=(
- $TESTDIR/providers/local \
- $TESTDIR/providers/local-pbs/pads \
-)
Copied: branches/release-0.93/tests/groups/providers-pads.sh (from rev 5018, branches/release-0.93/tests/groups/group-pads.sh)
===================================================================
--- branches/release-0.93/tests/groups/providers-pads.sh (rev 0)
+++ branches/release-0.93/tests/groups/providers-pads.sh 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1,7 @@
+
+# GROUPLIST definition of providers that run on PADS
+
+GROUPLIST=(
+ #$TESTDIR/providers/PADS/coasters \
+ $TESTDIR/providers/PADS/pbs \
+)
Added: branches/release-0.93/tests/providers/PADS/coasters/README
===================================================================
--- branches/release-0.93/tests/providers/PADS/coasters/README (rev 0)
+++ branches/release-0.93/tests/providers/PADS/coasters/README 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1,2 @@
+Provider test for PADS with coasters and pbs
+Make sure you have your default project set up (run 'projects' or 'projects -h' for more info)
Added: branches/release-0.93/tests/providers/PADS/coasters/catsn.0001.out.expected
===================================================================
--- branches/release-0.93/tests/providers/PADS/coasters/catsn.0001.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/PADS/coasters/catsn.0001.out.expected 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/coasters/catsn.0002.out.expected
===================================================================
--- branches/release-0.93/tests/providers/PADS/coasters/catsn.0002.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/PADS/coasters/catsn.0002.out.expected 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/coasters/catsn.0003.out.expected
===================================================================
--- branches/release-0.93/tests/providers/PADS/coasters/catsn.0003.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/PADS/coasters/catsn.0003.out.expected 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/coasters/catsn.0004.out.expected
===================================================================
--- branches/release-0.93/tests/providers/PADS/coasters/catsn.0004.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/PADS/coasters/catsn.0004.out.expected 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/coasters/catsn.0005.out.expected
===================================================================
--- branches/release-0.93/tests/providers/PADS/coasters/catsn.0005.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/PADS/coasters/catsn.0005.out.expected 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/coasters/catsn.0006.out.expected
===================================================================
--- branches/release-0.93/tests/providers/PADS/coasters/catsn.0006.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/PADS/coasters/catsn.0006.out.expected 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/coasters/catsn.0007.out.expected
===================================================================
--- branches/release-0.93/tests/providers/PADS/coasters/catsn.0007.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/PADS/coasters/catsn.0007.out.expected 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/coasters/catsn.0008.out.expected
===================================================================
--- branches/release-0.93/tests/providers/PADS/coasters/catsn.0008.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/PADS/coasters/catsn.0008.out.expected 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/coasters/catsn.0009.out.expected
===================================================================
--- branches/release-0.93/tests/providers/PADS/coasters/catsn.0009.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/PADS/coasters/catsn.0009.out.expected 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/coasters/catsn.0010.out.expected
===================================================================
--- branches/release-0.93/tests/providers/PADS/coasters/catsn.0010.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/PADS/coasters/catsn.0010.out.expected 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/coasters/catsn.setup.sh
===================================================================
--- branches/release-0.93/tests/providers/PADS/coasters/catsn.setup.sh (rev 0)
+++ branches/release-0.93/tests/providers/PADS/coasters/catsn.setup.sh 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1,3 @@
+#!/bin/bash
+
+cp -v $GROUP/data.txt . || exit 1
Property changes on: branches/release-0.93/tests/providers/PADS/coasters/catsn.setup.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: branches/release-0.93/tests/providers/PADS/coasters/catsn.swift
===================================================================
--- branches/release-0.93/tests/providers/PADS/coasters/catsn.swift (rev 0)
+++ branches/release-0.93/tests/providers/PADS/coasters/catsn.swift 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1,15 @@
+type file;
+
+app (file o) cat (file i)
+{
+ cat @i stdout=@o;
+}
+
+string t = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
+string char[] = @strsplit(t, "");
+
+file out[];
+foreach j in [1:@toint(@arg("n","10"))] {
+ file data<"data.txt">;
+ out[j] = cat(data);
+}
Added: branches/release-0.93/tests/providers/PADS/coasters/catsn.timeout
===================================================================
--- branches/release-0.93/tests/providers/PADS/coasters/catsn.timeout (rev 0)
+++ branches/release-0.93/tests/providers/PADS/coasters/catsn.timeout 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1,2 @@
+7200
+
Added: branches/release-0.93/tests/providers/PADS/coasters/data.txt
===================================================================
--- branches/release-0.93/tests/providers/PADS/coasters/data.txt (rev 0)
+++ branches/release-0.93/tests/providers/PADS/coasters/data.txt 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/coasters/sites.template.xml
===================================================================
--- branches/release-0.93/tests/providers/PADS/coasters/sites.template.xml (rev 0)
+++ branches/release-0.93/tests/providers/PADS/coasters/sites.template.xml 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1,16 @@
+
+
+
+
+ 2
+ 300
+ 1
+ 1
+ 1
+ 1
+ fast
+ 5.99
+ 10000
+ _WORK_
+
+
Added: branches/release-0.93/tests/providers/PADS/coasters/tc.template.data
===================================================================
--- branches/release-0.93/tests/providers/PADS/coasters/tc.template.data (rev 0)
+++ branches/release-0.93/tests/providers/PADS/coasters/tc.template.data 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1,7 @@
+PADS-coasters echo /bin/echo INSTALLED INTEL32::LINUX null
+PADS-coasters cat /bin/cat INSTALLED INTEL32::LINUX null
+PADS-coasters ls /bin/ls INSTALLED INTEL32::LINUX null
+PADS-coasters grep /bin/grep INSTALLED INTEL32::LINUX null
+PADS-coasters sort /bin/sort INSTALLED INTEL32::LINUX null
+PADS-coasters paste /bin/paste INSTALLED INTEL32::LINUX null
+PADS-coasters wc /usr/bin/wc INSTALLED INTEL32::LINUX null
Added: branches/release-0.93/tests/providers/PADS/pbs/catsn.0001.out.expected
===================================================================
--- branches/release-0.93/tests/providers/PADS/pbs/catsn.0001.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/PADS/pbs/catsn.0001.out.expected 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/pbs/catsn.0002.out.expected
===================================================================
--- branches/release-0.93/tests/providers/PADS/pbs/catsn.0002.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/PADS/pbs/catsn.0002.out.expected 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/pbs/catsn.0003.out.expected
===================================================================
--- branches/release-0.93/tests/providers/PADS/pbs/catsn.0003.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/PADS/pbs/catsn.0003.out.expected 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/pbs/catsn.0004.out.expected
===================================================================
--- branches/release-0.93/tests/providers/PADS/pbs/catsn.0004.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/PADS/pbs/catsn.0004.out.expected 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/pbs/catsn.0005.out.expected
===================================================================
--- branches/release-0.93/tests/providers/PADS/pbs/catsn.0005.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/PADS/pbs/catsn.0005.out.expected 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/pbs/catsn.0006.out.expected
===================================================================
--- branches/release-0.93/tests/providers/PADS/pbs/catsn.0006.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/PADS/pbs/catsn.0006.out.expected 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/pbs/catsn.0007.out.expected
===================================================================
--- branches/release-0.93/tests/providers/PADS/pbs/catsn.0007.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/PADS/pbs/catsn.0007.out.expected 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/pbs/catsn.0008.out.expected
===================================================================
--- branches/release-0.93/tests/providers/PADS/pbs/catsn.0008.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/PADS/pbs/catsn.0008.out.expected 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/pbs/catsn.0009.out.expected
===================================================================
--- branches/release-0.93/tests/providers/PADS/pbs/catsn.0009.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/PADS/pbs/catsn.0009.out.expected 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/pbs/catsn.0010.out.expected
===================================================================
--- branches/release-0.93/tests/providers/PADS/pbs/catsn.0010.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/PADS/pbs/catsn.0010.out.expected 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/pbs/catsn.setup.sh
===================================================================
--- branches/release-0.93/tests/providers/PADS/pbs/catsn.setup.sh (rev 0)
+++ branches/release-0.93/tests/providers/PADS/pbs/catsn.setup.sh 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1,3 @@
+#!/bin/bash
+
+cp -v $GROUP/data.txt . || exit 1
Property changes on: branches/release-0.93/tests/providers/PADS/pbs/catsn.setup.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: branches/release-0.93/tests/providers/PADS/pbs/catsn.swift
===================================================================
--- branches/release-0.93/tests/providers/PADS/pbs/catsn.swift (rev 0)
+++ branches/release-0.93/tests/providers/PADS/pbs/catsn.swift 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1,15 @@
+type file;
+
+app (file o) cat (file i)
+{
+ cat @i stdout=@o;
+}
+
+string t = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
+string char[] = @strsplit(t, "");
+
+file out[];
+foreach j in [1:@toint(@arg("n","10"))] {
+ file data<"data.txt">;
+ out[j] = cat(data);
+}
Added: branches/release-0.93/tests/providers/PADS/pbs/catsn.timeout
===================================================================
--- branches/release-0.93/tests/providers/PADS/pbs/catsn.timeout (rev 0)
+++ branches/release-0.93/tests/providers/PADS/pbs/catsn.timeout 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1,2 @@
+7200
+
Added: branches/release-0.93/tests/providers/PADS/pbs/data.txt
===================================================================
--- branches/release-0.93/tests/providers/PADS/pbs/data.txt (rev 0)
+++ branches/release-0.93/tests/providers/PADS/pbs/data.txt 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/PADS/pbs/sites.template.xml
===================================================================
--- branches/release-0.93/tests/providers/PADS/pbs/sites.template.xml (rev 0)
+++ branches/release-0.93/tests/providers/PADS/pbs/sites.template.xml 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1,16 @@
+
+
+
+
+ 2
+ 300
+ 1
+ 1
+ 1
+ 1
+ fast
+ 5.99
+ 10000
+ _WORK_
+
+
Added: branches/release-0.93/tests/providers/PADS/pbs/tc.template.data
===================================================================
--- branches/release-0.93/tests/providers/PADS/pbs/tc.template.data (rev 0)
+++ branches/release-0.93/tests/providers/PADS/pbs/tc.template.data 2011-08-29 18:32:10 UTC (rev 5019)
@@ -0,0 +1,7 @@
+PADS-pbs echo /bin/echo INSTALLED INTEL32::LINUX null
+PADS-pbs cat /bin/cat INSTALLED INTEL32::LINUX null
+PADS-pbs ls /bin/ls INSTALLED INTEL32::LINUX null
+PADS-pbs grep /bin/grep INSTALLED INTEL32::LINUX null
+PADS-pbs sort /bin/sort INSTALLED INTEL32::LINUX null
+PADS-pbs paste /bin/paste INSTALLED INTEL32::LINUX null
+PADS-pbs wc /usr/bin/wc INSTALLED INTEL32::LINUX null
From lgadelha at ci.uchicago.edu Mon Aug 29 15:14:28 2011
From: lgadelha at ci.uchicago.edu (lgadelha at ci.uchicago.edu)
Date: Mon, 29 Aug 2011 15:14:28 -0500 (CDT)
Subject: [Swift-commit] r5020 - provenancedb
Message-ID: <20110829201428.828DB9CCBE@svn.ci.uchicago.edu>
Author: lgadelha
Date: 2011-08-29 15:14:28 -0500 (Mon, 29 Aug 2011)
New Revision: 5020
Modified:
provenancedb/ProvSQL.g
Log:
Minor fixes.
Modified: provenancedb/ProvSQL.g
===================================================================
--- provenancedb/ProvSQL.g 2011-08-29 18:32:10 UTC (rev 5019)
+++ provenancedb/ProvSQL.g 2011-08-29 20:14:28 UTC (rev 5020)
@@ -198,7 +198,7 @@
INTERSECT { System.out.println(" INTERSECT "); }
|
EXCEPT { System.out.println(" EXCEPT "); }
- )
+ )
(
ALL { System.out.println(" ALL "); }
)?
@@ -210,7 +210,13 @@
}
;
-squery : SELECT selectExpression
+squery : SELECT (
+ DISTINCT
+ {
+ System.out.print(" DISTINCT ");
+ }
+ )?
+ selectExpression
{
System.out.print("SELECT " + selectClause);
@@ -266,10 +272,7 @@
;
whereExpression
- : c=whereAtom
- {
- whereClause += $c.text;
- }
+ : whereAtom
(
(AND
{
@@ -279,10 +282,7 @@
{
whereClause += " OR ";
}
- ) d=whereAtom
- {
- whereClause += $d.text;
- }
+ ) whereAtom
)*
;
@@ -290,23 +290,62 @@
: a=entityAttribute
{
relations.add($a.text.split("\\.")[0]);
+ whereClause += $a.text;
}
- OP
- (STRING | INT | FLOAT)
- | b=entityAttribute
+ b=OP
+ {
+ whereClause += $b.text;
+ }
+ (
+ c=STRING
+ {
+ whereClause += $c.text;
+ }
+ |
+ d=INT
+ {
+ whereClause += $d.text;
+ }
+ |
+ e=FLOAT
+ {
+ whereClause += $e.text;
+ }
+ )
+ | f=entityAttribute
{
- relations.add($b.text.split("\\.")[0]);
+ relations.add($f.text.split("\\.")[0]);
+ whereClause += $f.text;
}
- BETWEEN STRING AND STRING;
-
+ BETWEEN
+ {
+ whereClause += " BETWEEN ";
+ }
+ g=STRING
+ {
+ whereClause += $g.text;
+ }
+ AND
+ {
+ whereClause += " AND ";
+ }
+ h=STRING
+ {
+ whereClause += $h.text;
+ }
+ ;
+
entityAttribute : ID (DOT ID)?;
OP : '=' | '>' | '>=' | '<' | '<=';
SELECT : 's' 'e' 'l' 'e' 'c' 't';
+DISTINCT
+ : 'd' 'i' 's' 't' 'i' 'n' 'c' 't';
+
WHERE : 'w' 'h' 'e' 'r' 'e';
AND : 'a' 'n' 'd';
From davidk at ci.uchicago.edu Mon Aug 29 16:33:40 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Mon, 29 Aug 2011 16:33:40 -0500 (CDT)
Subject: [Swift-commit] r5021 - branches/release-0.93/tests/groups
Message-ID: <20110829213340.9F6689CC9D@svn.ci.uchicago.edu>
Author: davidk
Date: 2011-08-29 16:33:40 -0500 (Mon, 29 Aug 2011)
New Revision: 5021
Modified:
branches/release-0.93/tests/groups/providers-pads.sh
Log:
Modified: branches/release-0.93/tests/groups/providers-pads.sh
===================================================================
--- branches/release-0.93/tests/groups/providers-pads.sh 2011-08-29 20:14:28 UTC (rev 5020)
+++ branches/release-0.93/tests/groups/providers-pads.sh 2011-08-29 21:33:40 UTC (rev 5021)
@@ -2,6 +2,6 @@
# GROUPLIST definition of providers that run on PADS
GROUPLIST=(
- #$TESTDIR/providers/PADS/coasters \
+ $TESTDIR/providers/PADS/coasters \
$TESTDIR/providers/PADS/pbs \
)
From hategan at ci.uchicago.edu Mon Aug 29 19:23:33 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Mon, 29 Aug 2011 19:23:33 -0500 (CDT)
Subject: [Swift-commit] r5022 - trunk/src/org/griphyn/vdl/engine
Message-ID: <20110830002333.891519CCBE@svn.ci.uchicago.edu>
Author: hategan
Date: 2011-08-29 19:23:33 -0500 (Mon, 29 Aug 2011)
New Revision: 5022
Modified:
trunk/src/org/griphyn/vdl/engine/Karajan.java
Log:
patch from Yadu to complain when procedures are re-defined
Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java
===================================================================
--- trunk/src/org/griphyn/vdl/engine/Karajan.java 2011-08-29 21:33:40 UTC (rev 5021)
+++ trunk/src/org/griphyn/vdl/engine/Karajan.java 2011-08-30 00:23:33 UTC (rev 5022)
@@ -250,12 +250,20 @@
private void processProcedures(Program prog, VariableScope scope) throws CompilationException {
// Keep track of declared procedures
+ // Check for redefinitions of existing procedures
+ Set procsDefined = new HashSet() ;
for (int i = 0; i < prog.sizeOfProcedureArray(); i++) {
Procedure proc = prog.getProcedureArray(i);
- ProcedureSignature ps = new ProcedureSignature(proc.getName());
+ String procName = proc.getName().toLowerCase();
+ if (procsDefined.contains(procName)){
+ // We have a redefinition error
+ throw new CompilationException("Illegal redefinition of procedure attempted for " + procName );
+ }
+ procsDefined.add(procName);
+ ProcedureSignature ps = new ProcedureSignature(procName);
ps.setInputArgs(proc.getInputArray());
ps.setOutputArgs(proc.getOutputArray());
- proceduresMap.put(proc.getName(), ps);
+ proceduresMap.put(procName, ps);
}
for (int i = 0; i < prog.sizeOfProcedureArray(); i++) {
From wozniak at ci.uchicago.edu Mon Aug 29 21:19:40 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Mon, 29 Aug 2011 21:19:40 -0500 (CDT)
Subject: [Swift-commit] r5023 - trunk/libexec
Message-ID: <20110830021940.3A7199CC9D@svn.ci.uchicago.edu>
Author: wozniak
Date: 2011-08-29 21:19:40 -0500 (Mon, 29 Aug 2011)
New Revision: 5023
Modified:
trunk/libexec/_swiftwrap
Log:
Minor fix for MPI jobs
Modified: trunk/libexec/_swiftwrap
===================================================================
--- trunk/libexec/_swiftwrap 2011-08-30 00:23:33 UTC (rev 5022)
+++ trunk/libexec/_swiftwrap 2011-08-30 02:19:40 UTC (rev 5023)
@@ -232,6 +232,8 @@
if [ -z $MPI_RANK ]; then
INFOFILE="$INFODIR/${ID}-info"
else
+ # Just touch this file so the *.k stuff does not fail
+ [[ $MPI_RANK == 0 ]] && touch $INFODIR/${ID}-info
INFOFILE="$INFODIR/${ID}-${MPI_RANK}-info"
fi
rm -f $INFOFILE
From wozniak at ci.uchicago.edu Mon Aug 29 21:35:44 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Mon, 29 Aug 2011 21:35:44 -0500 (CDT)
Subject: [Swift-commit] r5024 - trunk/tests/mpi/crow/long
Message-ID: <20110830023544.95B6D9CC9D@svn.ci.uchicago.edu>
Author: wozniak
Date: 2011-08-29 21:35:44 -0500 (Mon, 29 Aug 2011)
New Revision: 5024
Modified:
trunk/tests/mpi/crow/long/100-mci.clean.sh
trunk/tests/mpi/crow/long/100-mci.swift
Log:
Test now works with no errors on Crow
Modified: trunk/tests/mpi/crow/long/100-mci.clean.sh
===================================================================
--- trunk/tests/mpi/crow/long/100-mci.clean.sh 2011-08-30 02:19:40 UTC (rev 5023)
+++ trunk/tests/mpi/crow/long/100-mci.clean.sh 2011-08-30 02:35:44 UTC (rev 5024)
@@ -2,6 +2,6 @@
set -x
-# rm -v transform-*.txt || exit 1
+rm -v transform-*.txt || exit 1
exit 0
Modified: trunk/tests/mpi/crow/long/100-mci.swift
===================================================================
--- trunk/tests/mpi/crow/long/100-mci.swift 2011-08-30 02:19:40 UTC (rev 5023)
+++ trunk/tests/mpi/crow/long/100-mci.swift 2011-08-30 02:35:44 UTC (rev 5024)
@@ -3,7 +3,7 @@
app (file o) transform(file i)
{
- mpi_sleep "-i" @i "-o" @o ;
+ mpi_sleep "-i" @i "-o" @o 3;
}
file input<"100-input.txt">;
From wozniak at ci.uchicago.edu Mon Aug 29 21:36:19 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Mon, 29 Aug 2011 21:36:19 -0500 (CDT)
Subject: [Swift-commit] r5025 - in trunk/tests/mpi: . beagle/long
Message-ID: <20110830023619.77C5B9CC9D@svn.ci.uchicago.edu>
Author: wozniak
Date: 2011-08-29 21:36:19 -0500 (Mon, 29 Aug 2011)
New Revision: 5025
Added:
trunk/tests/mpi/beagle/
Modified:
trunk/tests/mpi/beagle/long/100-mci.clean.sh
trunk/tests/mpi/beagle/long/100-mci.swift
Log:
Copy in Beagle tests from Crow
Modified: trunk/tests/mpi/beagle/long/100-mci.clean.sh
===================================================================
--- trunk/tests/mpi/crow/long/100-mci.clean.sh 2011-08-29 15:40:14 UTC (rev 5018)
+++ trunk/tests/mpi/beagle/long/100-mci.clean.sh 2011-08-30 02:36:19 UTC (rev 5025)
@@ -2,6 +2,6 @@
set -x
-# rm -v transform-*.txt || exit 1
+rm -v transform-*.txt || exit 1
exit 0
Modified: trunk/tests/mpi/beagle/long/100-mci.swift
===================================================================
--- trunk/tests/mpi/crow/long/100-mci.swift 2011-08-29 15:40:14 UTC (rev 5018)
+++ trunk/tests/mpi/beagle/long/100-mci.swift 2011-08-30 02:36:19 UTC (rev 5025)
@@ -3,7 +3,7 @@
app (file o) transform(file i)
{
- mpi_sleep "-i" @i "-o" @o ;
+ mpi_sleep "-i" @i "-o" @o 3;
}
file input<"100-input.txt">;
From wozniak at ci.uchicago.edu Mon Aug 29 22:54:10 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Mon, 29 Aug 2011 22:54:10 -0500 (CDT)
Subject: [Swift-commit] r5026 - trunk/tests/mpi/beagle/short
Message-ID: <20110830035410.3F02A9CC9D@svn.ci.uchicago.edu>
Author: wozniak
Date: 2011-08-29 22:54:10 -0500 (Mon, 29 Aug 2011)
New Revision: 5026
Added:
trunk/tests/mpi/beagle/short/100-mci.timeout
Modified:
trunk/tests/mpi/beagle/short/sites.template.xml
trunk/tests/mpi/beagle/short/tc.template.data
Log:
Fixes: test works on Beagle
Added: trunk/tests/mpi/beagle/short/100-mci.timeout
===================================================================
--- trunk/tests/mpi/beagle/short/100-mci.timeout (rev 0)
+++ trunk/tests/mpi/beagle/short/100-mci.timeout 2011-08-30 03:54:10 UTC (rev 5026)
@@ -0,0 +1 @@
+1000
Modified: trunk/tests/mpi/beagle/short/sites.template.xml
===================================================================
--- trunk/tests/mpi/beagle/short/sites.template.xml 2011-08-30 02:36:19 UTC (rev 5025)
+++ trunk/tests/mpi/beagle/short/sites.template.xml 2011-08-30 03:54:10 UTC (rev 5026)
@@ -5,7 +5,7 @@
-
+1
Modified: trunk/tests/mpi/beagle/short/tc.template.data
===================================================================
--- trunk/tests/mpi/beagle/short/tc.template.data 2011-08-30 02:36:19 UTC (rev 5025)
+++ trunk/tests/mpi/beagle/short/tc.template.data 2011-08-30 03:54:10 UTC (rev 5026)
@@ -19,4 +19,4 @@
# hydra-tests-2
-crow mpi_cp _DIR_/mpi-cp INSTALLED INTEL32::LINUX globus::hostCount=2
+beagle mpi_cp _DIR_/mpi-cp INSTALLED INTEL32::LINUX globus::hostCount=2
From wozniak at ci.uchicago.edu Mon Aug 29 22:58:00 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Mon, 29 Aug 2011 22:58:00 -0500 (CDT)
Subject: [Swift-commit] r5027 - trunk/tests/mpi/beagle/long
Message-ID: <20110830035800.A91709CC9D@svn.ci.uchicago.edu>
Author: wozniak
Date: 2011-08-29 22:58:00 -0500 (Mon, 29 Aug 2011)
New Revision: 5027
Modified:
trunk/tests/mpi/beagle/long/sites.template.xml
trunk/tests/mpi/beagle/long/tc.template.data
Log:
Fixes: test works on Beagle
Modified: trunk/tests/mpi/beagle/long/sites.template.xml
===================================================================
--- trunk/tests/mpi/beagle/long/sites.template.xml 2011-08-30 03:54:10 UTC (rev 5026)
+++ trunk/tests/mpi/beagle/long/sites.template.xml 2011-08-30 03:58:00 UTC (rev 5027)
@@ -5,14 +5,14 @@
-
+17200
- pbs.aprun;pbs.mpp;depth=6
+ pbs.aprun;pbs.mpp;depth=1
11
Modified: trunk/tests/mpi/beagle/long/tc.template.data
===================================================================
--- trunk/tests/mpi/beagle/long/tc.template.data 2011-08-30 03:54:10 UTC (rev 5026)
+++ trunk/tests/mpi/beagle/long/tc.template.data 2011-08-30 03:58:00 UTC (rev 5027)
@@ -19,5 +19,5 @@
# hydra-tests-2
-crow mpi_cp _DIR_/mpi-cp INSTALLED INTEL32::LINUX globus::hostCount=2
-crow mpi_sleep _DIR_/mpi-sleep INSTALLED INTEL32::LINUX globus::hostCount=2
+beagle mpi_cp _DIR_/mpi-cp INSTALLED INTEL32::LINUX globus::hostCount=2
+beagle mpi_sleep _DIR_/mpi-sleep INSTALLED INTEL32::LINUX globus::hostCount=2
From lgadelha at ci.uchicago.edu Tue Aug 30 09:28:16 2011
From: lgadelha at ci.uchicago.edu (lgadelha at ci.uchicago.edu)
Date: Tue, 30 Aug 2011 09:28:16 -0500 (CDT)
Subject: [Swift-commit] r5028 - provenancedb
Message-ID: <20110830142816.9964A9CD07@svn.ci.uchicago.edu>
Author: lgadelha
Date: 2011-08-30 09:28:16 -0500 (Tue, 30 Aug 2011)
New Revision: 5028
Modified:
provenancedb/ProvSQL.g
Log:
Minor fixes.
Modified: provenancedb/ProvSQL.g
===================================================================
--- provenancedb/ProvSQL.g 2011-08-30 03:58:00 UTC (rev 5027)
+++ provenancedb/ProvSQL.g 2011-08-30 14:28:16 UTC (rev 5028)
@@ -15,6 +15,7 @@
String whereClauseJoinExpressions = new String();
String whereClause = new String();
boolean hasWhereJoinExpression;
+ boolean hasWhereExpression = false;
HashSet relations = new HashSet();
UndirectedGraph schemaGraph;
HashSet joinEdges;
@@ -218,13 +219,14 @@
)?
selectExpression
{
-
System.out.print("SELECT " + selectClause);
-
-
}
(WHERE whereExpression
{
+ hasWhereExpression=true;
+ }
+ )?
+ {
schemaGraph = buildGraph();
joinEdges = computeJoinEdges(schemaGraph, relations);
hasWhereJoinExpression=false;
@@ -240,13 +242,15 @@
System.out.print(" WHERE " + whereClauseJoinExpressions);
}
- if(hasWhereJoinExpression)
- System.out.print(" AND ");
- else
- System.out.print(" WHERE ");
- System.out.print(whereClause);
+ if(hasWhereExpression) {
+ if(hasWhereJoinExpression)
+ System.out.print(" AND ");
+ else
+ System.out.print(" WHERE ");
+ System.out.print(whereClause);
+ }
}
- )?
+
|
'(' { System.out.print("("); }
squery
From wozniak at ci.uchicago.edu Tue Aug 30 13:28:40 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Tue, 30 Aug 2011 13:28:40 -0500 (CDT)
Subject: [Swift-commit] r5029 - in trunk/tests/mpi: . fusion/long
fusion/short
Message-ID: <20110830182840.771869CD07@svn.ci.uchicago.edu>
Author: wozniak
Date: 2011-08-30 13:28:40 -0500 (Tue, 30 Aug 2011)
New Revision: 5029
Added:
trunk/tests/mpi/fusion/
trunk/tests/mpi/fusion/long/sites.template.xml
trunk/tests/mpi/fusion/long/tc.template.data
trunk/tests/mpi/fusion/short/100-mci.timeout
trunk/tests/mpi/fusion/short/sites.template.xml
trunk/tests/mpi/fusion/short/tc.template.data
Removed:
trunk/tests/mpi/fusion/long/sites.template.xml
trunk/tests/mpi/fusion/long/tc.template.data
trunk/tests/mpi/fusion/short/sites.template.xml
trunk/tests/mpi/fusion/short/tc.template.data
Log:
Copy Beagle test for Fusion
Deleted: trunk/tests/mpi/fusion/long/sites.template.xml
===================================================================
--- trunk/tests/mpi/beagle/long/sites.template.xml 2011-08-30 02:36:19 UTC (rev 5025)
+++ trunk/tests/mpi/fusion/long/sites.template.xml 2011-08-30 18:28:40 UTC (rev 5029)
@@ -1,31 +0,0 @@
-
-
-
-
-
-
-
-
-
-
- 1
- 7200
-
-
- pbs.aprun;pbs.mpp;depth=6
-
- 1
- 1
- 10
- 20
- DEBUG
- {wdir}
- 5.99
- 10000
- {wdir}
-
-
-
-
Copied: trunk/tests/mpi/fusion/long/sites.template.xml (from rev 5027, trunk/tests/mpi/beagle/long/sites.template.xml)
===================================================================
--- trunk/tests/mpi/fusion/long/sites.template.xml (rev 0)
+++ trunk/tests/mpi/fusion/long/sites.template.xml 2011-08-30 18:28:40 UTC (rev 5029)
@@ -0,0 +1,31 @@
+
+
+
+
+
+
+
+
+
+
+ 1
+ 7200
+
+
+ pbs.aprun;pbs.mpp;depth=1
+
+ 1
+ 1
+ 10
+ 20
+ DEBUG
+ {wdir}
+ 5.99
+ 10000
+ {wdir}
+
+
+
+
Deleted: trunk/tests/mpi/fusion/long/tc.template.data
===================================================================
--- trunk/tests/mpi/beagle/long/tc.template.data 2011-08-30 02:36:19 UTC (rev 5025)
+++ trunk/tests/mpi/fusion/long/tc.template.data 2011-08-30 18:28:40 UTC (rev 5029)
@@ -1,23 +0,0 @@
-#This is the transformation catalog.
-#
-#It comes pre-configured with a number of simple transformations with
-#paths that are likely to work on a linux box. However, on some systems,
-#the paths to these executables will be different (for example, sometimes
-#some of these programs are found in /usr/bin rather than in /bin)
-#
-#NOTE WELL: fields in this file must be separated by tabs, not spaces; and
-#there must be no trailing whitespace at the end of each line.
-#
-# sitename transformation path INSTALLED platform profiles
-localhost echo /bin/echo INSTALLED INTEL32::LINUX null
-localhost cat /bin/cat INSTALLED INTEL32::LINUX null
-localhost ls /bin/ls INSTALLED INTEL32::LINUX null
-localhost grep /bin/grep INSTALLED INTEL32::LINUX null
-localhost sort /bin/sort INSTALLED INTEL32::LINUX null
-localhost paste /bin/paste INSTALLED INTEL32::LINUX null
-localhost cp /bin/cp INSTALLED INTEL32::LINUX null
-
-# hydra-tests-2
-
-crow mpi_cp _DIR_/mpi-cp INSTALLED INTEL32::LINUX globus::hostCount=2
-crow mpi_sleep _DIR_/mpi-sleep INSTALLED INTEL32::LINUX globus::hostCount=2
Copied: trunk/tests/mpi/fusion/long/tc.template.data (from rev 5027, trunk/tests/mpi/beagle/long/tc.template.data)
===================================================================
--- trunk/tests/mpi/fusion/long/tc.template.data (rev 0)
+++ trunk/tests/mpi/fusion/long/tc.template.data 2011-08-30 18:28:40 UTC (rev 5029)
@@ -0,0 +1,23 @@
+#This is the transformation catalog.
+#
+#It comes pre-configured with a number of simple transformations with
+#paths that are likely to work on a linux box. However, on some systems,
+#the paths to these executables will be different (for example, sometimes
+#some of these programs are found in /usr/bin rather than in /bin)
+#
+#NOTE WELL: fields in this file must be separated by tabs, not spaces; and
+#there must be no trailing whitespace at the end of each line.
+#
+# sitename transformation path INSTALLED platform profiles
+localhost echo /bin/echo INSTALLED INTEL32::LINUX null
+localhost cat /bin/cat INSTALLED INTEL32::LINUX null
+localhost ls /bin/ls INSTALLED INTEL32::LINUX null
+localhost grep /bin/grep INSTALLED INTEL32::LINUX null
+localhost sort /bin/sort INSTALLED INTEL32::LINUX null
+localhost paste /bin/paste INSTALLED INTEL32::LINUX null
+localhost cp /bin/cp INSTALLED INTEL32::LINUX null
+
+# hydra-tests-2
+
+beagle mpi_cp _DIR_/mpi-cp INSTALLED INTEL32::LINUX globus::hostCount=2
+beagle mpi_sleep _DIR_/mpi-sleep INSTALLED INTEL32::LINUX globus::hostCount=2
Copied: trunk/tests/mpi/fusion/short/100-mci.timeout (from rev 5026, trunk/tests/mpi/beagle/short/100-mci.timeout)
===================================================================
--- trunk/tests/mpi/fusion/short/100-mci.timeout (rev 0)
+++ trunk/tests/mpi/fusion/short/100-mci.timeout 2011-08-30 18:28:40 UTC (rev 5029)
@@ -0,0 +1 @@
+1000
Deleted: trunk/tests/mpi/fusion/short/sites.template.xml
===================================================================
--- trunk/tests/mpi/beagle/short/sites.template.xml 2011-08-30 02:36:19 UTC (rev 5025)
+++ trunk/tests/mpi/fusion/short/sites.template.xml 2011-08-30 18:28:40 UTC (rev 5029)
@@ -1,31 +0,0 @@
-
-
-
-
-
-
-
-
-
-
- 1
- 7200
-
-
- pbs.aprun;pbs.mpp;depth=6
-
- 1
- 1
- 2
- 4
- DEBUG
- {wdir}
- 5.99
- 10000
- {wdir}
-
-
-
-
Copied: trunk/tests/mpi/fusion/short/sites.template.xml (from rev 5026, trunk/tests/mpi/beagle/short/sites.template.xml)
===================================================================
--- trunk/tests/mpi/fusion/short/sites.template.xml (rev 0)
+++ trunk/tests/mpi/fusion/short/sites.template.xml 2011-08-30 18:28:40 UTC (rev 5029)
@@ -0,0 +1,31 @@
+
+
+
+
+
+
+
+
+
+
+ 1
+ 7200
+
+
+ pbs.aprun;pbs.mpp;depth=6
+
+ 1
+ 1
+ 2
+ 4
+ DEBUG
+ {wdir}
+ 5.99
+ 10000
+ {wdir}
+
+
+
+
Deleted: trunk/tests/mpi/fusion/short/tc.template.data
===================================================================
--- trunk/tests/mpi/beagle/short/tc.template.data 2011-08-30 02:36:19 UTC (rev 5025)
+++ trunk/tests/mpi/fusion/short/tc.template.data 2011-08-30 18:28:40 UTC (rev 5029)
@@ -1,22 +0,0 @@
-#This is the transformation catalog.
-#
-#It comes pre-configured with a number of simple transformations with
-#paths that are likely to work on a linux box. However, on some systems,
-#the paths to these executables will be different (for example, sometimes
-#some of these programs are found in /usr/bin rather than in /bin)
-#
-#NOTE WELL: fields in this file must be separated by tabs, not spaces; and
-#there must be no trailing whitespace at the end of each line.
-#
-# sitename transformation path INSTALLED platform profiles
-localhost echo /bin/echo INSTALLED INTEL32::LINUX null
-localhost cat /bin/cat INSTALLED INTEL32::LINUX null
-localhost ls /bin/ls INSTALLED INTEL32::LINUX null
-localhost grep /bin/grep INSTALLED INTEL32::LINUX null
-localhost sort /bin/sort INSTALLED INTEL32::LINUX null
-localhost paste /bin/paste INSTALLED INTEL32::LINUX null
-localhost cp /bin/cp INSTALLED INTEL32::LINUX null
-
-# hydra-tests-2
-
-crow mpi_cp _DIR_/mpi-cp INSTALLED INTEL32::LINUX globus::hostCount=2
Copied: trunk/tests/mpi/fusion/short/tc.template.data (from rev 5026, trunk/tests/mpi/beagle/short/tc.template.data)
===================================================================
--- trunk/tests/mpi/fusion/short/tc.template.data (rev 0)
+++ trunk/tests/mpi/fusion/short/tc.template.data 2011-08-30 18:28:40 UTC (rev 5029)
@@ -0,0 +1,22 @@
+#This is the transformation catalog.
+#
+#It comes pre-configured with a number of simple transformations with
+#paths that are likely to work on a linux box. However, on some systems,
+#the paths to these executables will be different (for example, sometimes
+#some of these programs are found in /usr/bin rather than in /bin)
+#
+#NOTE WELL: fields in this file must be separated by tabs, not spaces; and
+#there must be no trailing whitespace at the end of each line.
+#
+# sitename transformation path INSTALLED platform profiles
+localhost echo /bin/echo INSTALLED INTEL32::LINUX null
+localhost cat /bin/cat INSTALLED INTEL32::LINUX null
+localhost ls /bin/ls INSTALLED INTEL32::LINUX null
+localhost grep /bin/grep INSTALLED INTEL32::LINUX null
+localhost sort /bin/sort INSTALLED INTEL32::LINUX null
+localhost paste /bin/paste INSTALLED INTEL32::LINUX null
+localhost cp /bin/cp INSTALLED INTEL32::LINUX null
+
+# hydra-tests-2
+
+beagle mpi_cp _DIR_/mpi-cp INSTALLED INTEL32::LINUX globus::hostCount=2
From wozniak at ci.uchicago.edu Tue Aug 30 13:42:23 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Tue, 30 Aug 2011 13:42:23 -0500 (CDT)
Subject: [Swift-commit] r5030 - trunk/tests/mpi/fusion/short
Message-ID: <20110830184223.2104C9CD07@svn.ci.uchicago.edu>
Author: wozniak
Date: 2011-08-30 13:42:22 -0500 (Tue, 30 Aug 2011)
New Revision: 5030
Modified:
trunk/tests/mpi/fusion/short/sites.template.xml
trunk/tests/mpi/fusion/short/tc.template.data
Log:
Modifications for MPI Fusion short test: works
Modified: trunk/tests/mpi/fusion/short/sites.template.xml
===================================================================
--- trunk/tests/mpi/fusion/short/sites.template.xml 2011-08-30 18:28:40 UTC (rev 5029)
+++ trunk/tests/mpi/fusion/short/sites.template.xml 2011-08-30 18:42:22 UTC (rev 5030)
@@ -5,15 +5,12 @@
-
+17200
-
- pbs.aprun;pbs.mpp;depth=6
- 112
Modified: trunk/tests/mpi/fusion/short/tc.template.data
===================================================================
--- trunk/tests/mpi/fusion/short/tc.template.data 2011-08-30 18:28:40 UTC (rev 5029)
+++ trunk/tests/mpi/fusion/short/tc.template.data 2011-08-30 18:42:22 UTC (rev 5030)
@@ -19,4 +19,4 @@
# hydra-tests-2
-beagle mpi_cp _DIR_/mpi-cp INSTALLED INTEL32::LINUX globus::hostCount=2
+fusion mpi_cp _DIR_/mpi-cp INSTALLED INTEL32::LINUX globus::hostCount=2
From wozniak at ci.uchicago.edu Tue Aug 30 13:47:01 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Tue, 30 Aug 2011 13:47:01 -0500 (CDT)
Subject: [Swift-commit] r5031 - trunk/tests/mpi/beagle/long
Message-ID: <20110830184701.417039CD07@svn.ci.uchicago.edu>
Author: wozniak
Date: 2011-08-30 13:47:01 -0500 (Tue, 30 Aug 2011)
New Revision: 5031
Added:
trunk/tests/mpi/beagle/long/100-mci.timeout
Log:
Bring in timeout file
Copied: trunk/tests/mpi/beagle/long/100-mci.timeout (from rev 5026, trunk/tests/mpi/beagle/short/100-mci.timeout)
===================================================================
--- trunk/tests/mpi/beagle/long/100-mci.timeout (rev 0)
+++ trunk/tests/mpi/beagle/long/100-mci.timeout 2011-08-30 18:47:01 UTC (rev 5031)
@@ -0,0 +1 @@
+1000
From ketan at ci.uchicago.edu Tue Aug 30 17:01:14 2011
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Tue, 30 Aug 2011 17:01:14 -0500 (CDT)
Subject: [Swift-commit] r5032 - trunk/bin/grid
Message-ID: <20110830220114.161679CD07@svn.ci.uchicago.edu>
Author: ketan
Date: 2011-08-30 17:01:13 -0500 (Tue, 30 Aug 2011)
New Revision: 5032
Added:
trunk/bin/grid/start-grid-services
Modified:
trunk/bin/grid/foreachsite
trunk/bin/grid/run-worker.sh
Log:
start multiple grid services
Modified: trunk/bin/grid/foreachsite
===================================================================
--- trunk/bin/grid/foreachsite 2011-08-30 18:47:01 UTC (rev 5031)
+++ trunk/bin/grid/foreachsite 2011-08-30 22:01:13 UTC (rev 5032)
@@ -1,3 +1,5 @@
+#!/bin/bash
+
# Default settings:
resource=work # fork or work(er)
@@ -9,7 +11,6 @@
usage="$0 foreachsite [-resource fork|worker ] scriptname"
# Process command line arguments
-
while [ $# -gt 0 ]; do
case $1 in
-resource) resource=$2; shift 2 ;;
Modified: trunk/bin/grid/run-worker.sh
===================================================================
--- trunk/bin/grid/run-worker.sh 2011-08-30 18:47:01 UTC (rev 5031)
+++ trunk/bin/grid/run-worker.sh 2011-08-30 22:01:13 UTC (rev 5032)
@@ -3,6 +3,7 @@
contact=$1
workername=$2
origlogdir=$3
+export PATH=${OSG_APP}/scec:$PATH
echo OSG_WN_TMP=$OSG_WN_TMP
if [ _$OSG_WN_TMP = _ ]; then
OSG_WN_TMP=/tmp
@@ -31,3 +32,4 @@
echo
tail -v -n $nlines $logdir/*
+
Added: trunk/bin/grid/start-grid-services
===================================================================
--- trunk/bin/grid/start-grid-services (rev 0)
+++ trunk/bin/grid/start-grid-services 2011-08-30 22:01:13 UTC (rev 5032)
@@ -0,0 +1,138 @@
+#! /bin/bash
+
+# FIXME: improve arg parsing / checking / optionals
+
+function usage ()
+{
+ echo "Usage:"
+ echo " $0 --throttle 0.01 --loglevel INFO|DEBUG|TRACE --jobspernode 1"
+}
+
+if [ $# -ne 6 ]
+then
+ usage
+ exit 1
+fi
+
+LOGLEVEL=INFO # INFO, DEBUG, TRACE for increasing detail
+THROTTLE=0.09
+
+while test "$1" != "" ; do
+ case $1 in
+ --jobspernode|-j)
+ JOBSPERNODE="$2"
+ shift
+ ;;
+ --loglevel|-l)
+ LOGLEVEL="$2"
+ shift
+ ;;
+ --throttle|-t)
+ THROTTLE="$2"
+ shift
+ ;;
+ -*)
+ echo "Error: no such option $1"
+ usage
+ exit 1
+ ;;
+ esac
+ shift
+done
+
+BIN=$(cd $(dirname $0); pwd)
+
+echo THROTTLE=$THROTTLE LOGLEVEL=$LOGLEVEL
+
+### Start n services; where n is the number of sites in the greensites
+#sed '$d' < greensites > file1 ; mv file1 greensites
+n=`cat greensites|wc -l`
+
+echo "$n services will start"
+#start-swift-service 1 &
+start-swift-service $n &
+
+sleep 60
+
+#SPORT=$(cat service.sports)
+#just temporary
+#SPORT=100
+rm -f service.sports
+rm -f service.wports
+m=`expr $n - 1`
+for i in `seq -w 00 $m`
+do
+ cat service-$i.sport >> service.sports
+ echo "" >> service.sports
+ cat service-$i.wport >> service.wports
+ echo "" >> service.wports
+done
+#sed '$d' < greensites > file1 ; mv file1 greensites
+sed '$d' < service.sports > file2 ; mv file2 service.sports
+sed '$d' < service.wports > file3 ; mv file3 service.wports
+
+#Merge the ports and greensites
+paste -d' ' service.sports service.wports greensites > merged_ports_and_sites
+
+#remove any old sites.xml files
+rm -f sites.grid-ps.xml
+rm -f tc.factory
+
+cat >tc.factory <
+#sportcount=0
+#while
+
+#for aline in `cat greensites`
+#for aline in `cat merged_ports_and_sites`
+cat merged_ports_and_sites | while read aline
+do
+ bline=`echo $aline | awk -F" " '{print $3}'`
+ cursport=`echo $aline | awk -F" " '{print $1}'`
+ curwport=`echo $aline | awk -F" " '{print $2}'`
+ curosgsite=`echo $bline|awk -F"__" '{print $2}'`
+ echo $curosgsite
+cat >>sites.grid-ps.xml <
+
+
+ passive
+ $JOBSPERNODE
+ $THROTTLE
+ 10000
+
+
+
+ /tmp/$USER
+
+
+
+EOF
+
+cat >>tc.factory <
Author: davidk
Date: 2011-08-31 12:37:41 -0500 (Wed, 31 Aug 2011)
New Revision: 5033
Added:
branches/release-0.93/docs/documentation/
branches/release-0.93/docs/documentation/automation
branches/release-0.93/docs/documentation/building
branches/release-0.93/docs/documentation/building.html
branches/release-0.93/docs/documentation/documentation.txt
branches/release-0.93/docs/documentation/overview
branches/release-0.93/docs/documentation/structure
Log:
Description of how the new documentation system works
Added: branches/release-0.93/docs/documentation/automation
===================================================================
--- branches/release-0.93/docs/documentation/automation (rev 0)
+++ branches/release-0.93/docs/documentation/automation 2011-08-31 17:37:41 UTC (rev 5033)
@@ -0,0 +1,26 @@
+Document Automation
+-------------------
+All .txt files in the docs directory get converted to HTML and PDF on a nightly
+basis. These documents then get copied to the Swift web site.
+
+Cron Job
+~~~~~~~~
+The job that processes these documents runs on Bridled at 7pm nightly. The script is
+/home/davidk/docscripts/update_and_build.sh. The script calls svn update on several
+different swift directories in ~davidk, and then builds the the documents. The
+documents should be built for each Swift branch later than 0.93, including trunk. As
+new versions of Swift are created, this script will need to be updated.
+
+Website
+-------
+Once the cron job runs, documents can be accessed at:
+
+\http://www.ci.uchicago.edu/swift/guides/release-/documentname/documentname.html
+\http://www.ci.uchicago.edu/swift/guides/release-/documentname/documentname.pdf
+
+For example, assume that you create a document called docs/newdoc. You check this
+guide into Swift 0.93. The AsciiDoc .txt file is called newdoc.txt. The URLs
+that would be generated would be:
+
+\http://www.ci.uchicago.edu/swift/guides/release-0.93/newdoc/newdoc.html
+\http://www.ci.uchicago.edu/swift/guides/release-0.93/newdoc/newdoc.pdf
Added: branches/release-0.93/docs/documentation/building
===================================================================
--- branches/release-0.93/docs/documentation/building (rev 0)
+++ branches/release-0.93/docs/documentation/building 2011-08-31 17:37:41 UTC (rev 5033)
@@ -0,0 +1,26 @@
+Building Documentation Manually
+-------------------------------
+Before committing anything to SVN, it is a good idea manually
+build the documentation for testing purposes.
+
+Prerequisites
+~~~~~~~~~~~~~
+There are several prerequisites for building documentation:
+
+* asciidoc and a2x (http://www.methods.co.nz/asciidoc)
+* dblatex (http://dblatex.sourceforge.net)
+* source-highlight (http://www.gnu.org/software/src-highlite)
+
+For CI users, compiled versions of these packages can be found in ~davidk.
+
+build_docs.sh
+~~~~~~~~~~~~~
+
+The script to build the documentation is docs/build_docs.sh. This script
+will convert the documents into HTML and PDF, and then copy the
+files to a given output directory.
+
+.Usage
+-----
+./build_docs.sh /path/to/copy/output
+-----
Added: branches/release-0.93/docs/documentation/building.html
===================================================================
--- branches/release-0.93/docs/documentation/building.html (rev 0)
+++ branches/release-0.93/docs/documentation/building.html 2011-08-31 17:37:41 UTC (rev 5033)
@@ -0,0 +1,576 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Prerequisites
+
+
There are several prerequisites for building documentation:
For CI users, compiled versions of these packages can be found in ~davidk.
+
+
Build Docs Script
+
Before committing any changes to documentation, it is a good idea manually
+build the documentation. The script to build the documentation is
+docs/build_docs.sh.
+
This script will convert the documents into HTML and PDF, and then copy the
+files to a given output directory.
+
+
+
+
+
+
+
+
Added: branches/release-0.93/docs/documentation/documentation.txt
===================================================================
--- branches/release-0.93/docs/documentation/documentation.txt (rev 0)
+++ branches/release-0.93/docs/documentation/documentation.txt 2011-08-31 17:37:41 UTC (rev 5033)
@@ -0,0 +1,11 @@
+Swift Document Generation
+=========================
+
+include::overview[]
+
+include::structure[]
+
+include::building[]
+
+include::automation[]
+
Added: branches/release-0.93/docs/documentation/overview
===================================================================
--- branches/release-0.93/docs/documentation/overview (rev 0)
+++ branches/release-0.93/docs/documentation/overview 2011-08-31 17:37:41 UTC (rev 5033)
@@ -0,0 +1,5 @@
+Overview
+--------
+This guide provides a brief overview of how Swift Document Generation works. It
+should explain the structure of a docs directory, how to manually build documentation,
+and explain how document automation works.
Added: branches/release-0.93/docs/documentation/structure
===================================================================
--- branches/release-0.93/docs/documentation/structure (rev 0)
+++ branches/release-0.93/docs/documentation/structure 2011-08-31 17:37:41 UTC (rev 5033)
@@ -0,0 +1,60 @@
+Structure
+---------
+A specific file and directory structure is required for each document in order to ease the
+process of automation, conversion, and copying.
+
+Directories
+~~~~~~~~~~~
+Each subdirectory within docs/ should correspond to the name of the document.
+This name will also later be used in the URL used to point to the
+document.
+
+.Example Directory Names
+-----
+docs/userguide
+docs/tutorial
+-----
+
+AsciiDoc File
+~~~~~~~~~~~~~
+Every document directory should contain one file with a .txt
+extension. This .txt file is the AsciiDoc document that later gets converted
+to various formats.
+
+.Example AsciiDoc Documents
+-----
+docs/userguide/userguide.txt
+docs/tutorial/tutorial.txt
+-----
+
+Chapters
+~~~~~~~~
+Each document directory should contain individual chapters
+that make up the document. These chapters should not have any
+file extensions.
+
+.Example Chapters
+-----
+docs/userguide/overview
+docs/userguide/language
+docs/userguide/mappers
+docs/userguide/commands
+-----
+
+Including Chapters in a Document
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The .txt file should then include the individual chapters
+with the AsciiDoc include command.
+
+.userguide.txt
+-----
+\include::overview[]
+
+\include::language[]
+
+\include::mappers[]
+
+\include::commands[]
+-----
+
+WARNING: Be sure to include spaces between includes. Putting them together with no spaces can cause formatting/compilation errors.
From davidk at ci.uchicago.edu Wed Aug 31 13:43:39 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Wed, 31 Aug 2011 13:43:39 -0500 (CDT)
Subject: [Swift-commit] r5034 - in branches/release-0.93/docs: . siteguide
tutorial userguide utils
Message-ID: <20110831184339.BB0E19CCF6@svn.ci.uchicago.edu>
Author: davidk
Date: 2011-08-31 13:43:39 -0500 (Wed, 31 Aug 2011)
New Revision: 5034
Added:
branches/release-0.93/docs/siteguide/prereqs
branches/release-0.93/docs/utils/
branches/release-0.93/docs/utils/gensites.txt
Modified:
branches/release-0.93/docs/siteguide/pads
branches/release-0.93/docs/siteguide/siteguide.txt
branches/release-0.93/docs/tutorial/runtime_features
branches/release-0.93/docs/userguide/coasters
Log:
Added gensites documentation to 0.93
Added prereqs page to siteguide
Added a PADS quickstart
Some tutorial and userguide cleanup
Modified: branches/release-0.93/docs/siteguide/pads
===================================================================
--- branches/release-0.93/docs/siteguide/pads 2011-08-31 17:37:41 UTC (rev 5033)
+++ branches/release-0.93/docs/siteguide/pads 2011-08-31 18:43:39 UTC (rev 5034)
@@ -1,10 +1,61 @@
PADS
----
+PADS Quickstart
+~~~~~~~~~~~~~~~
+This section will walk you through running a simple Swift script
+on PADS.
+
+In your home directory, edit a file called .soft and add these
+lines (in this order):
+-----
++java-sun
++maui
++torque
+ at default
+-----
+
+Log out of PADS, and log back in.
+
+Next, paste the example below into a file called sites.xml.
+
+-----
+include::../../tests/providers/PADS/coasters/sites.template.xml[]
+-----
+
+Create a directory called swiftwork. Modify _WORK_ in sites.xml
+to point to this new directory. For example
+-----
+/home/myhome/swiftwork
+-----
+
+Next, paste the following example into a file called tc.data
+
+-----
+include::../../tests/providers/PADS/coasters/tc.template.data[]
+-----
+
+Copy the catsn Swift script and input text file into your current directory
+-----
+$ cp ~/swift-0.93/examples/misc/catsn.swift .
+$ cp ~/swift-0.93/examples/misc/data.txt .
+-----
+TIP: The location of your swift directory may vary depending on how you installed it. Change this to the examples/misc directory of your installation as needed.
+
+And finally, run the script
+-----
+$ swift -sites.file sites.xml -tc.file tc.data catsn.swift
+-----
+
+You should see 10 new text files get created, named catsn*.out. If
+you see these files, then you have succesfully run Swift on PADS!
+
+TIP: Make sure your default project is defined. Read on for more information.
+
Requesting Access
~~~~~~~~~~~~~~~~~
If you do not already have a Computation Institute account, you can request
-one at https://www.ci.uchicago.edu/accounts/. This page will give you a list
+one at https://www.ci.uchicago.edu/accounts. This page will give you a list
of resources you can request access to. Be sure that PADS is selected. If
you already have an existing CI account, but do not have access to PADS,
send an email to support at ci.uchicago.edu to request access.
@@ -39,34 +90,6 @@
@default
-----
-Downloading and building Swift
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The most recent versions of Swift can be found at
-http://www.ci.uchicago.edu/swift/downloads/index.php. Follow the instructions
-provided on that site to download and build Swift.
-
-Adding Swift to your PATH
-~~~~~~~~~~~~~~~~~~~~~~~~~
-Once you have installed Swift, add the Swift binary to your PATH so you can
-easily run it from any directory.
-
-In your home directory, edit the file ".bashrc".
-
-If you have installed Swift via a source repository, add the following line
-at the bottom of .bashrc.
-
------
-export PATH=$PATH:$HOME/cog/modules/swift/dist/swift-svn/bin
------
-
-If you have installed Swift via a binary package, add this line:
-
------
-export PATH=$PATH:$HOME/swift-/bin
------
-
-Replace with the actual name of the swift directory in the example above.
-
What You Need To Know Before Running Swift
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Before you can create a Swift configuration file, there are some things
@@ -145,17 +168,18 @@
Below is the template that is used by Swift's test suite for running on PADS.
-----
-include::../../tests/providers/local-pbs/pads/sites.template.xml[]
+include::../../tests/providers/PADS/coasters/sites.template.xml[]
-----
-The values to note here are the ones that are listed between underscores. In the example above, they are _QUEUE_, and _WORK_. Queue is the PADS queue to use and WORK is the swift work directory. These are placeholder values you will need to modify to fit your needs. Copy and paste this template, replace the values, and call it sites.xml.
+Before you can use this configuration file, you will need to modify _WORK_ to a directory writable by you,
+and have your default project defined. Copy this template, replace _WORK_, and then save as sites.xml.
Manually Editing tc.data
~~~~~~~~~~~~~~~~~~~~~~~~
Below is the tc.data file used by Swift's test suite for running on PADS.
-----
-include::../../tests/providers/local-pbs/pads/tc.template.data[]
+include::../../tests/providers/PADS/coasters/tc.template.data[]
------
Copy these commands and save it as tc.data.
Added: branches/release-0.93/docs/siteguide/prereqs
===================================================================
--- branches/release-0.93/docs/siteguide/prereqs (rev 0)
+++ branches/release-0.93/docs/siteguide/prereqs 2011-08-31 18:43:39 UTC (rev 5034)
@@ -0,0 +1,8 @@
+Prerequisites
+-------------
+
+This guide assumes that you have already downloaded and installed Swift.
+It assumes that Swift is in your PATH and that you have a working
+version of Sun Java 1.5+. For more information on downloading and
+installing Swift, please see the
+http://www.ci.uchicago.edu/swift/guides/release-0.93/quickstart/quickstart.html[Swift Quickstart Guide].
Modified: branches/release-0.93/docs/siteguide/siteguide.txt
===================================================================
--- branches/release-0.93/docs/siteguide/siteguide.txt 2011-08-31 17:37:41 UTC (rev 5033)
+++ branches/release-0.93/docs/siteguide/siteguide.txt 2011-08-31 18:43:39 UTC (rev 5034)
@@ -6,6 +6,8 @@
:website: http://www.ci.uchicago.edu/swift/guides/siteguide.php
:numbered:
+include::prereqs[]
+
include::pads[]
include::beagle[]
Modified: branches/release-0.93/docs/tutorial/runtime_features
===================================================================
--- branches/release-0.93/docs/tutorial/runtime_features 2011-08-31 17:37:41 UTC (rev 5033)
+++ branches/release-0.93/docs/tutorial/runtime_features 2011-08-31 18:43:39 UTC (rev 5034)
@@ -22,167 +22,10 @@
examples, we've invoked echo and tr executables from our SwiftScript
program. These have been run on the local system (the same computer on
which you ran swift). We can also make our computations run on a
-remote resource.
+remote resource. For more information on running Swift on a remote site
+please see the http://www.ci.uchicago.edu/swift/guides/release-0.93/siteguide/siteguide.html[Site Configuration Guide].
-WARNING: This example is necessarily more vague than previous ones,
-because it requires access to remote resources. You should ensure that
-you can submit a job using the globus-job-run (or globusrun-ws)
-command(s).
-We do not need to modify any SwiftScript code to run on another
-resource. Instead, we must modify another catalog, the site catalog.
-This catalog provides details of the location that applications will be
-run, with the default settings referring to the local machine. We will
-modify it to refer to a remote resource - the UC Teraport cluster. If
-you are not a UC Teraport user, you should use details of a different
-resource that you do have access to.
-
-The site catalog is located in etc/sites.xml and is a relatively
-straightforward XML format file. We must modify each of the following
-three settings: gridftp (which indicates how and where data can be
-transferred to the remote resource), jobmanager (which indicates how
-applications can be run on the remote resource) and workdirectory (which
-indicates where working storage can be found on the remote resource).
-
-
-Writing a mapper
-~~~~~~~~~~~~~~~~
-
-This section will introduce writing a custom mapper so that Swift is
-able to access data files laid out in application-specific ways.
-
-An application-specific mapper must take the form of a Java class that
-implements the http://www.ci.uchicago.edu/swift/javadoc/vdsk/org/griphyn/vdl/mapping/Mapper.html[Mapper
-interface].
-
-Usually you don't need to implement this interface directly, because
-Swift provides a number of more concrete classes with some functionality
-already implemented.
-
-The hierarchy of helper classes is:
-
-http://www.ci.uchicago.edu/swift/javadoc/vdsk/org/griphyn/vdl/mapping/Mapper.html[Mapper]
-
-This is the abstract interface for mappers in Swift. You must
-implement methods to provide access to mapper properties, to map from a
-SwiftScript dataset path (such as foo[1].bar) to a file name, to check
-whether a file exists. None of the default Swift mappers implement this
-interface directly - instead they use one of the following helper classes.
-
-http://www.ci.uchicago.edu/swift/javadoc/vdsk/org/griphyn/vdl/mapping/AbstractMapper.html[Abstract Mapper]
-
-This provides helper methods to manage mapper properties and to handle
-existance checking. Examples of mappers which use this class are:
-http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.array_mapper[Array Mapper],
-http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.csv_mapper[CSV Mapper],
-http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.fixed_array_mapper[Fixed Array Mapper],
-http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.regexp_mapper[Regexp Mapper],
-and http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.single_file_mapper[Single File Mapper].
-
-http://www.ci.uchicago.edu/swift/javadoc/vdsk/org/griphyn/vdl/mapping/file/AbstractFileMapper.html[AbstractFileMapper]
-This provides a helper class for mappers which select files based on
-selecting files from a directory listing. It is necessary to write some
-helper methods that are different from the above mapper methods.
-Examples of mappers which use this class are:
-http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.simple_mapper[simple_mapper],
-http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.filesys_mapper[filesys_mapper],
-and the StructuredRegularExpressionMapper.
-
-In general, to write a mapper, choose either the AbstractMapper or the
-AbstractFileMapper and extend those. If your mapper will generally
-select the files it returns based on a directory listing and will
-convert paths to filenames using some regular conversion (for example,
-in the way that simple_mapper maps files in a directory that match a
-particular pattern), then you should probably use the
-AbstractFileMapper. If your mapper will produce a list of files in some
-other way (for example, in the way that csv_mapper maps based on
-filenames given in a CSV file rather than looking at which files are in
-a directory), then you should probably use the AbstractMapper.
-
-
-Writing a very basic mapper
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-In this section, we will write a very basic (almost useless) mapper that
-will map a SwiftScript dataset into a hardcoded file called
-myfile.txt, like this:
-
-
- Swift variable Filename
-
- var <-----------------------------> myfile.txt
-
-We should be able to use the mapper we write in a SwiftScript program
-like this:
-
-
-----
-type file;
-file f ;
-----
-
-First we must choose a base class - AbstractMapper or
-AbstractFileMapper. We aren't going to use a directory listing to decide
-on our mapping - we are getting the mapping from some other source (in
-fact, it will be hard coded). So we will use AbstractMapper.
-
-So now onto the source code. We must define a subclass of AbstractMapper
-and implement several mapper methods: isStatic, existing, and map. These
-methods are documented in the javadoc for the Mapper interface.
-
-Here is the code implementing this mapper. Put this in your source
-vdsk directory, make a directory src/tutorial/|and put this file in
-src/tutorial/MyFirstMapper.java.
-
-.MyFirstMapper.java
-*******************
-----
-include::../../examples/tutorial/MyFirstMapper.java[]
-----
-*******************
-
-Now we need to inform the Swift engine about the existence of this
-mapper. We do that by editing the MapperFactory class definition, in
-src/org/griphyn/vdl/mapping/MapperFactory.java and adding a
-registerMapper call alongside the existing registerMapper calls, like this:
-
-
-----
-registerMapper("my_first_mapper", tutorial.MyFirstMapper.class);
-----
-
-The first parameter is the name of the mapper that will be used in
-SwiftScript program. The second parameter is the new Mapper class that
-we just wrote.
-
-Now rebuild Swift using the "ant redist" target.
-
-This new Swift build will be aware of your new mapper. We can test it
-out with a hello world program:
-
-.my_first_mapper.swift
-**********************
------
-include::../../examples/tutorial/my_first_mapper.swift[]
------
-**********************
-
-Run this program, and hopefully you will find the "hello" string has
-been output into the hard coded output file myfile.txt:
-
-
-----
-$ cat myfile.txt
-hello
-----
-
-So that's a first very simple mapper implemented. Compare the source
-code to the single_file_mapper in
-http://www.ci.uchicago.edu/trac/swift/browser/trunk/src/org/griphyn/vdl/mapping/file/SingleFileMapper.java[SingleFileMapper.java].
-There is not much more code to the single_file_mapper - mostly code to
-deal with the file parameter.
-
-
Starting and restarting
~~~~~~~~~~~~~~~~~~~~~~~
Modified: branches/release-0.93/docs/userguide/coasters
===================================================================
--- branches/release-0.93/docs/userguide/coasters 2011-08-31 17:37:41 UTC (rev 5033)
+++ branches/release-0.93/docs/userguide/coasters 2011-08-31 18:43:39 UTC (rev 5034)
@@ -1,7 +1,5 @@
Coasters
--------
-Coasters were introduced in Swift v0.6 as an experimental feature.
-
In many applications, Swift performance can be greatly enhanced by the
use of CoG coasters. CoG coasters provide a low-overhead job submission
and file transfer mechanism suited for the execution of short jobs (on
Added: branches/release-0.93/docs/utils/gensites.txt
===================================================================
--- branches/release-0.93/docs/utils/gensites.txt (rev 0)
+++ branches/release-0.93/docs/utils/gensites.txt 2011-08-31 18:43:39 UTC (rev 5034)
@@ -0,0 +1,235 @@
+Gensites
+==========
+:toc:
+:icons:
+:website: http://www.ci.uchicago.edu/swift
+:numbered:
+
+Overview
+--------
+To simplify this configuration process, versions of Swift starting with
+0.92 include a utility called gensites. The gensites command is used
+to generate a sites.xml file for running a swift workflow on a given
+site. It accomplishes this by using a series of templates. The templates
+used by gensites are the same templates used for internal testing, so
+they are likely up to date and known to work on a given site.
+
+
+Viewing All Available Site Templates
+------------------------------------
+To view a list of all available templates, run the following command:
+
+-----
+$ gensites -T
+-----
+
+You should see output similar to this:
+-----
+intrepid
+local
+local-pbs-coasters
+pads
+queenbee
+sge-local
+ssh
+ssh-pbs-coasters
+surveyor
+-----
+
+You will notice that the templates can be specific to a particular set
+of machines like Intrepid and Queenbee, or they may be more general and
+aim to work across a variety of machines, as in the case of
+local-pbs-coasters. Gensites will look in three directories for available
+templates: your current directory, $SWIFT_HOME/etc/sites and $HOME/.swift/sites.
+
+Listing the Template
+--------------------
+To view the contents of a template, type:
+
+-----
+$ gensites -l templatename
+-----
+
+Running this command will print some information about the template and give
+you an idea of what settings you will need to specify. You should see
+something like this:
+
+-----
+Description: Template for Intrepid. More information about this system can be found at http://www.alcf.anl.gov/support/gettingstarted/index.php
+Required Tokens: host work project queue
+-----
+
+The required tokens are required to properly use the templates. These are placeholder
+values you will need to specify in the
+following steps.
+
+Providing Site Specific Values
+------------------------------
+
+The gensites script needs to know how to replace the placeholder values
+in the template. This is done by configuring the swift.properties file.
+Gensites will first look for a swift.properties file in the current
+directory. If it does not exist, it will next look in $HOME/.swift.
+
+To add site specific values to swift.properties, add a line in the
+follow format:
+
+-----
+#site templatename setting=value
+-----
+
+Continuing the previous example, here is what you could add to
+swift.properties to replace the values of project, queue and work for
+the surveyor template:
+
+-----
+#site surveyor project=MyProject
+#site surveyor queue=MyQueue
+#site surveyor work=/path/to/workdir
+-----
+
+Now, running the command 'gensites surveyor' will produce the following
+valid configuration file:
+
+-----
+
+
+
+
+ MyProject
+ MyQueue
+ zeptoos
+ true
+ 21
+ 10000
+ 1
+ DEBUG
+ 1
+ 900
+ 64
+ 64
+ /path/to/workdir
+
+
+-----
+
+Generating Application Configurations with Gensites
+---------------------------------------------------
+Gensites can also be used to create a valid application catalog, commonly
+called tc.data. Here are some examples of how to specify applications within
+your swift.properties file:
+
+-----
+#app intrepid echo=/usr/bin/echo
+-----
+
+This first example shows a site specific application. The #app definition
+tells gensites this is related to an application rather than a #site
+definition. In the second part, echo=/usr/bin/echo, the left hand side
+is the name of the application that will be called from within swift. The
+right hand site is the path name which points to the binary.
+
+-----
+#app intrepid echo=$HOME/bin/echo
+-----
+
+Environment variables will be interpreted and converted to full path names
+for Swift.
+
+-----
+#app intrepid echo=bin/echo
+-----
+
+Gensites can take relative paths (relative to your current directory) and
+translate them to full path names for Swift.
+
+WARNING: Running gensites with #app definitions will replace any file
+called tc.data in your current directory. If a file called tc.data exists,
+it will be renamed to tc.data.old. If you run gensites twice, the original
+contents of your tc.data will be lost. Please either rename your tc file
+or copy to a different location.
+
+Running Swift With the New Configuration
+----------------------------------------
+
+Now that the gensites is configured and producing a valid configuration
+file, Swift needs to know to use it. The first step is to create a
+unique config file based on the preferences you specified.
+
+-----
+$ gensites surveyor > myconfig.xml
+-----
+
+This will send the output of gensites to myconfig.xml. This example will
+use a swift.properties location in the default directories (your current
+directory, ~/.swift/swift.properties). To specify a different location
+to the swift.properties, use:
+
+-----
+$ gensites surveyor -p myswift.properties > myconfig.xml
+-----
+
+Next, provide the configuration filename to swift:
+-----
+$ swift -sites.file myconfig.xml myscript.swift
+-----
+
+Alternatively, if you have specified applications, be sure to load that into
+Swift
+-----
+$ swift -sites.file myconfig.xml -tc.file tc.data mycript.swift
+-----
+
+Providing Default Values for All Templates
+------------------------------------------
+It is also possible to specify a default value for a setting, regardless
+of template you use. If you want to set your queue to default to "fast"
+across all templates, you can do this by omitting the template name.
+Consider the following swift.properties:
+
+-----
+#site queue=fast
+#site surveyor project=MyProject
+#site surveyor work=/path/to/workdir
+-----
+By omitting the template name, the default value for queue on surveyor
+(and any other template you use) will be set to "fast". One thing to
+keep in mind when setting default values is that order matters. Be sure
+to set your default values first before setting template specific values.
+
+Just like the #site definitions, when a site name is not specified, an app
+will be created for every site that is defined in your template.
+-----
+#app echo=/bin/echo
+-----
+
+More Help
+---------
+The gensites script provides additional options not discussed here, such
+as using templates and swift.properties in non-standard directories. For
+more information, run gensites -h. Here is a full list of all options
+available.
+
+-----
+$ gensites -help
+
+ usage: gensites template [-p properties.file] [-L template_directory] [-h] [-T] [-l]
+
+ template Name of template to use
+ -p properties.file Specify a swift.properties to use
+ -L template_directory Specify a non-standard template directory
+ -T List all templates available
+ -h Help / usage information
+ -l List the contents of a specific template
+
+ Examples:
+
+ Create a site configuration file for sites.xml using default properties.file in current directory
+ $ gensites pads > sites.xml
+
+ Use a specific properties file for a site
+ $ gensites -p sites.properties pads > sites.xml
+
+ Specify a non-standard directory where templates are located
+ $ gensites -L template.dir pads > sites.xml
+-----
From davidk at ci.uchicago.edu Wed Aug 31 15:46:21 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Wed, 31 Aug 2011 15:46:21 -0500 (CDT)
Subject: [Swift-commit] r5035 - in branches/release-0.93/tests: groups
providers providers/fusion providers/local-pbs
Message-ID: <20110831204621.4C30B9CCC1@svn.ci.uchicago.edu>
Author: davidk
Date: 2011-08-31 15:46:21 -0500 (Wed, 31 Aug 2011)
New Revision: 5035
Added:
branches/release-0.93/tests/groups/providers-fusion.sh
branches/release-0.93/tests/providers/fusion/
branches/release-0.93/tests/providers/fusion/catsn.0001.out.expected
branches/release-0.93/tests/providers/fusion/catsn.0002.out.expected
branches/release-0.93/tests/providers/fusion/catsn.0003.out.expected
branches/release-0.93/tests/providers/fusion/catsn.0004.out.expected
branches/release-0.93/tests/providers/fusion/catsn.0005.out.expected
branches/release-0.93/tests/providers/fusion/catsn.0006.out.expected
branches/release-0.93/tests/providers/fusion/catsn.0007.out.expected
branches/release-0.93/tests/providers/fusion/catsn.0008.out.expected
branches/release-0.93/tests/providers/fusion/catsn.0009.out.expected
branches/release-0.93/tests/providers/fusion/catsn.0010.out.expected
branches/release-0.93/tests/providers/fusion/catsn.setup.sh
branches/release-0.93/tests/providers/fusion/catsn.swift
branches/release-0.93/tests/providers/fusion/catsn.timeout
branches/release-0.93/tests/providers/fusion/data.txt
branches/release-0.93/tests/providers/fusion/sites.template.xml
branches/release-0.93/tests/providers/fusion/tc.template.data
Removed:
branches/release-0.93/tests/providers/local-pbs/fusion/
branches/release-0.93/tests/providers/local-pbs/pads/
Log:
Updates to fusion provider test
Added: branches/release-0.93/tests/groups/providers-fusion.sh
===================================================================
--- branches/release-0.93/tests/groups/providers-fusion.sh (rev 0)
+++ branches/release-0.93/tests/groups/providers-fusion.sh 2011-08-31 20:46:21 UTC (rev 5035)
@@ -0,0 +1,6 @@
+
+# GROUPLIST definition of providers that run on PADS
+
+GROUPLIST=(
+ $TESTDIR/providers/fusion
+)
Added: branches/release-0.93/tests/providers/fusion/catsn.0001.out.expected
===================================================================
--- branches/release-0.93/tests/providers/fusion/catsn.0001.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/fusion/catsn.0001.out.expected 2011-08-31 20:46:21 UTC (rev 5035)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/fusion/catsn.0002.out.expected
===================================================================
--- branches/release-0.93/tests/providers/fusion/catsn.0002.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/fusion/catsn.0002.out.expected 2011-08-31 20:46:21 UTC (rev 5035)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/fusion/catsn.0003.out.expected
===================================================================
--- branches/release-0.93/tests/providers/fusion/catsn.0003.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/fusion/catsn.0003.out.expected 2011-08-31 20:46:21 UTC (rev 5035)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/fusion/catsn.0004.out.expected
===================================================================
--- branches/release-0.93/tests/providers/fusion/catsn.0004.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/fusion/catsn.0004.out.expected 2011-08-31 20:46:21 UTC (rev 5035)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/fusion/catsn.0005.out.expected
===================================================================
--- branches/release-0.93/tests/providers/fusion/catsn.0005.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/fusion/catsn.0005.out.expected 2011-08-31 20:46:21 UTC (rev 5035)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/fusion/catsn.0006.out.expected
===================================================================
--- branches/release-0.93/tests/providers/fusion/catsn.0006.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/fusion/catsn.0006.out.expected 2011-08-31 20:46:21 UTC (rev 5035)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/fusion/catsn.0007.out.expected
===================================================================
--- branches/release-0.93/tests/providers/fusion/catsn.0007.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/fusion/catsn.0007.out.expected 2011-08-31 20:46:21 UTC (rev 5035)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/fusion/catsn.0008.out.expected
===================================================================
--- branches/release-0.93/tests/providers/fusion/catsn.0008.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/fusion/catsn.0008.out.expected 2011-08-31 20:46:21 UTC (rev 5035)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/fusion/catsn.0009.out.expected
===================================================================
--- branches/release-0.93/tests/providers/fusion/catsn.0009.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/fusion/catsn.0009.out.expected 2011-08-31 20:46:21 UTC (rev 5035)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/fusion/catsn.0010.out.expected
===================================================================
--- branches/release-0.93/tests/providers/fusion/catsn.0010.out.expected (rev 0)
+++ branches/release-0.93/tests/providers/fusion/catsn.0010.out.expected 2011-08-31 20:46:21 UTC (rev 5035)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/fusion/catsn.setup.sh
===================================================================
--- branches/release-0.93/tests/providers/fusion/catsn.setup.sh (rev 0)
+++ branches/release-0.93/tests/providers/fusion/catsn.setup.sh 2011-08-31 20:46:21 UTC (rev 5035)
@@ -0,0 +1,6 @@
+#!/bin/bash
+
+cp -v $GROUP/data.txt . || exit 1
+ipaddr=$( ifconfig | grep inet | head -1 | cut -d ':' -f 2 | awk '{print $1}' )
+export GLOBUS_HOSTNAME=$ipaddr
+
Property changes on: branches/release-0.93/tests/providers/fusion/catsn.setup.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: branches/release-0.93/tests/providers/fusion/catsn.swift
===================================================================
--- branches/release-0.93/tests/providers/fusion/catsn.swift (rev 0)
+++ branches/release-0.93/tests/providers/fusion/catsn.swift 2011-08-31 20:46:21 UTC (rev 5035)
@@ -0,0 +1,15 @@
+type file;
+
+app (file o) cat (file i)
+{
+ cat @i stdout=@o;
+}
+
+string t = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
+string char[] = @strsplit(t, "");
+
+file out[];
+foreach j in [1:@toint(@arg("n","10"))] {
+ file data<"data.txt">;
+ out[j] = cat(data);
+}
Added: branches/release-0.93/tests/providers/fusion/catsn.timeout
===================================================================
--- branches/release-0.93/tests/providers/fusion/catsn.timeout (rev 0)
+++ branches/release-0.93/tests/providers/fusion/catsn.timeout 2011-08-31 20:46:21 UTC (rev 5035)
@@ -0,0 +1 @@
+30
Added: branches/release-0.93/tests/providers/fusion/data.txt
===================================================================
--- branches/release-0.93/tests/providers/fusion/data.txt (rev 0)
+++ branches/release-0.93/tests/providers/fusion/data.txt 2011-08-31 20:46:21 UTC (rev 5035)
@@ -0,0 +1 @@
+Hello world
Added: branches/release-0.93/tests/providers/fusion/sites.template.xml
===================================================================
--- branches/release-0.93/tests/providers/fusion/sites.template.xml (rev 0)
+++ branches/release-0.93/tests/providers/fusion/sites.template.xml 2011-08-31 20:46:21 UTC (rev 5035)
@@ -0,0 +1,16 @@
+
+
+
+
+ 750
+ 1
+ 1
+ 1
+ 2
+ shared
+ 5.99
+ 10000
+ _WORK_
+
+
+
Added: branches/release-0.93/tests/providers/fusion/tc.template.data
===================================================================
--- branches/release-0.93/tests/providers/fusion/tc.template.data (rev 0)
+++ branches/release-0.93/tests/providers/fusion/tc.template.data 2011-08-31 20:46:21 UTC (rev 5035)
@@ -0,0 +1,7 @@
+fusion echo /bin/echo INSTALLED INTEL32::LINUX
+fusion cat /bin/cat INSTALLED INTEL32::LINUX
+fusion ls /bin/ls INSTALLED INTEL32::LINUX
+fusion grep /bin/grep INSTALLED INTEL32::LINUX
+fusion sort /bin/sort INSTALLED INTEL32::LINUX
+fusion paste /bin/paste INSTALLED INTEL32::LINUX
+fusion wc /usr/bin/wc INSTALLED INTEL32::LINUX
From wozniak at ci.uchicago.edu Wed Aug 31 17:48:43 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Wed, 31 Aug 2011 17:48:43 -0500 (CDT)
Subject: [Swift-commit] r5036 - trunk/libexec
Message-ID: <20110831224843.0BBE19CCC1@svn.ci.uchicago.edu>
Author: wozniak
Date: 2011-08-31 17:48:42 -0500 (Wed, 31 Aug 2011)
New Revision: 5036
Modified:
trunk/libexec/vdl-int.k
Log:
More logging for CDM outputs
Modified: trunk/libexec/vdl-int.k
===================================================================
--- trunk/libexec/vdl-int.k 2011-08-31 20:46:21 UTC (rev 5035)
+++ trunk/libexec/vdl-int.k 2011-08-31 22:48:42 UTC (rev 5036)
@@ -293,6 +293,7 @@
//make sure we do have the directory on the client side
dir:make(ldir, host=dhost, provider=provider)
policy := cdm:query(query=file)
+ log(LOG:DEBUG, "CDM: {file} : {policy}")
if (sys:or(policy == "DEFAULT", policy == "BROADCAST")
then(
restartOnError(".*", 2
From wozniak at ci.uchicago.edu Wed Aug 31 22:30:27 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Wed, 31 Aug 2011 22:30:27 -0500 (CDT)
Subject: [Swift-commit] r5037 -
trunk/tests/providers/local-pbs-coasters/fusion
Message-ID: <20110901033027.7CEFB9CCF6@svn.ci.uchicago.edu>
Author: wozniak
Date: 2011-08-31 22:30:27 -0500 (Wed, 31 Aug 2011)
New Revision: 5037
Modified:
trunk/tests/providers/local-pbs-coasters/fusion/sites.template.xml
trunk/tests/providers/local-pbs-coasters/fusion/tc.template.data
Log:
Update configuration
Modified: trunk/tests/providers/local-pbs-coasters/fusion/sites.template.xml
===================================================================
--- trunk/tests/providers/local-pbs-coasters/fusion/sites.template.xml 2011-08-31 22:48:42 UTC (rev 5036)
+++ trunk/tests/providers/local-pbs-coasters/fusion/sites.template.xml 2011-09-01 03:30:27 UTC (rev 5037)
@@ -1,5 +1,5 @@
-
+1
Modified: trunk/tests/providers/local-pbs-coasters/fusion/tc.template.data
===================================================================
--- trunk/tests/providers/local-pbs-coasters/fusion/tc.template.data 2011-08-31 22:48:42 UTC (rev 5036)
+++ trunk/tests/providers/local-pbs-coasters/fusion/tc.template.data 2011-09-01 03:30:27 UTC (rev 5037)
@@ -1,8 +1,7 @@
-local-pbs-coasters echo /bin/echo INSTALLED INTEL32::LINUX null
-local-pbs-coasters cat /bin/cat INSTALLED INTEL32::LINUX null
-local-pbs-coasters ls /bin/ls INSTALLED INTEL32::LINUX null
-local-pbs-coasters grep /bin/grep INSTALLED INTEL32::LINUX null
-local-pbs-coasters sort /bin/sort INSTALLED INTEL32::LINUX null
-local-pbs-coasters paste /bin/paste INSTALLED INTEL32::LINUX null
-local-pbs-coasters wc /usr/bin/wc INSTALLED INTEL32::LINUX null
-
+fusion echo /bin/echo INSTALLED INTEL32::LINUX
+fusion cat /bin/cat INSTALLED INTEL32::LINUX
+fusion ls /bin/ls INSTALLED INTEL32::LINUX
+fusion grep /bin/grep INSTALLED INTEL32::LINUX
+fusion sort /bin/sort INSTALLED INTEL32::LINUX
+fusion paste /bin/paste INSTALLED INTEL32::LINUX
+fusion wc /usr/bin/wc INSTALLED INTEL32::LINUX
From wozniak at ci.uchicago.edu Wed Aug 31 22:31:29 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Wed, 31 Aug 2011 22:31:29 -0500 (CDT)
Subject: [Swift-commit] r5038 -
trunk/tests/providers/local-pbs-coasters/fusion
Message-ID: <20110901033129.79BD69CCF6@svn.ci.uchicago.edu>
Author: wozniak
Date: 2011-08-31 22:31:29 -0500 (Wed, 31 Aug 2011)
New Revision: 5038
Modified:
trunk/tests/providers/local-pbs-coasters/fusion/001-catsn.check.sh
trunk/tests/providers/local-pbs-coasters/fusion/001-catsn.setup.sh
trunk/tests/providers/local-pbs-coasters/fusion/001-catsn.swift
Log:
Minor improvements
Modified: trunk/tests/providers/local-pbs-coasters/fusion/001-catsn.check.sh
===================================================================
--- trunk/tests/providers/local-pbs-coasters/fusion/001-catsn.check.sh 2011-09-01 03:30:27 UTC (rev 5037)
+++ trunk/tests/providers/local-pbs-coasters/fusion/001-catsn.check.sh 2011-09-01 03:31:29 UTC (rev 5038)
@@ -2,11 +2,12 @@
set -x
-for count in `seq --format "%04.f" 1 1 10`
+FILES=( catsn.*.out )
+[[ ${#FILES[@]} == 10 ]] || exit 1
+
+for FILE in ${FILES}
do
- [ -f catsn.$count.out ] || exit 1
- CONTENTS1=$( cat catsn.$count.out.expected )
- CONTENTS2=$( cat catsn.$count.out )
- [[ $CONTENTS1 == $CONTENTS2 ]] || exit 1
+ grep -q "Hello world" ${FILE} || exit 1
done
+
exit 0
Modified: trunk/tests/providers/local-pbs-coasters/fusion/001-catsn.setup.sh
===================================================================
--- trunk/tests/providers/local-pbs-coasters/fusion/001-catsn.setup.sh 2011-09-01 03:30:27 UTC (rev 5037)
+++ trunk/tests/providers/local-pbs-coasters/fusion/001-catsn.setup.sh 2011-09-01 03:31:29 UTC (rev 5038)
@@ -1,4 +1,4 @@
#!/bin/bash
cp -v $GROUP/data.txt . || exit 1
-cp -v $GROUP/*expected . || exit 1
+# cp -v $GROUP/*expected . || exit 1
Modified: trunk/tests/providers/local-pbs-coasters/fusion/001-catsn.swift
===================================================================
--- trunk/tests/providers/local-pbs-coasters/fusion/001-catsn.swift 2011-09-01 03:30:27 UTC (rev 5037)
+++ trunk/tests/providers/local-pbs-coasters/fusion/001-catsn.swift 2011-09-01 03:31:29 UTC (rev 5038)
@@ -5,9 +5,6 @@
cat @i stdout=@o;
}
-string t = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
-string char[] = @strsplit(t, "");
-
file out[];
foreach j in [1:@toint(@arg("n","10"))] {
file data<"data.txt">;
From vytas at ci.uchicago.edu Mon Aug 1 08:18:59 2011
From: vytas at ci.uchicago.edu (vytas at ci.uchicago.edu)
Date: Mon, 01 Aug 2011 13:18:59 -0000
Subject: [Swift-commit] r4889 - /
Message-ID: <20110801131858.B442D9CC9E@svn.ci.uchicago.edu>
Author: vytas
Date: 2011-08-01 08:18:58 -0500 (Mon, 01 Aug 2011)
New Revision: 4889
Added:
wwwdev/
Log:
new look and feel
From vytas at ci.uchicago.edu Mon Aug 1 09:15:47 2011
From: vytas at ci.uchicago.edu (vytas at ci.uchicago.edu)
Date: Mon, 01 Aug 2011 14:15:47 -0000
Subject: [Swift-commit] r4890 - in wwwdev: . _notes
Message-ID: <20110801141546.03D2D9CC9E@svn.ci.uchicago.edu>
Author: vytas
Date: 2011-08-01 09:15:45 -0500 (Mon, 01 Aug 2011)
New Revision: 4890
Added:
wwwdev/_notes/
wwwdev/_notes/dwsync.xml
wwwdev/index.php
Log:
new look and feel
Added: wwwdev/_notes/dwsync.xml
===================================================================
--- wwwdev/_notes/dwsync.xml (rev 0)
+++ wwwdev/_notes/dwsync.xml 2011-08-01 14:15:45 UTC (rev 4890)
@@ -0,0 +1,4 @@
+
+
+
+
\ No newline at end of file
Added: wwwdev/index.php
===================================================================
--- wwwdev/index.php (rev 0)
+++ wwwdev/index.php 2011-08-01 14:15:45 UTC (rev 4890)
@@ -0,0 +1,62 @@
+
+
+
+
+Swift - About
+
+
+
+
+
+
+
+
+
+
+
+
+
+
About
+
+
+
+
+
+
Swift incorporates several existing toolkits and engines to achieve its results. What can Swift do for you?
+
+
Globus Toolkit
+
Swift uses the Globus Toolkit as middleware to talk to various resources.
+
+
Java CoG Kit
+
The CoG Kit provides a high level uniform interface to different versions of the Globus Toolkit. It also contains the Karajan language and interpreter which is used as an underlying execution engine for Swift.
+
+
TeraGrid
+
+"TeraGrid is an open scientific discovery infrastructure combining leadership class resources at nine partner sites to create an integrated, persistent computational resource."
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
From vytas at ci.uchicago.edu Mon Aug 1 09:17:10 2011
From: vytas at ci.uchicago.edu (vytas at ci.uchicago.edu)
Date: Mon, 01 Aug 2011 14:17:10 -0000
Subject: [Swift-commit] r4891 - wwwdev
Message-ID: <20110801141654.5A6D69CC9E@svn.ci.uchicago.edu>
Author: vytas
Date: 2011-08-01 09:16:54 -0500 (Mon, 01 Aug 2011)
New Revision: 4891
Removed:
wwwdev/_notes/
wwwdev/index.php
Log:
wrong directory
Deleted: wwwdev/index.php
===================================================================
--- wwwdev/index.php 2011-08-01 14:15:45 UTC (rev 4890)
+++ wwwdev/index.php 2011-08-01 14:16:54 UTC (rev 4891)
@@ -1,62 +0,0 @@
-
-
-
-
-Swift - About
-
-
-
-
-
-
-
-
-
-
-
-
-
-
About
-
-
-
-
-
-
Swift incorporates several existing toolkits and engines to achieve its results. What can Swift do for you?
-
-
Globus Toolkit
-
Swift uses the Globus Toolkit as middleware to talk to various resources.
-
-
Java CoG Kit
-
The CoG Kit provides a high level uniform interface to different versions of the Globus Toolkit. It also contains the Karajan language and interpreter which is used as an underlying execution engine for Swift.
-
-
TeraGrid
-
-"TeraGrid is an open scientific discovery infrastructure combining leadership class resources at nine partner sites to create an integrated, persistent computational resource."
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
From vytas at ci.uchicago.edu Mon Aug 1 09:17:41 2011
From: vytas at ci.uchicago.edu (vytas at ci.uchicago.edu)
Date: Mon, 01 Aug 2011 14:17:41 -0000
Subject: [Swift-commit] r4892 - wwwdev
Message-ID: <20110801141740.585EC9CC9E@svn.ci.uchicago.edu>
Author: vytas
Date: 2011-08-01 09:17:40 -0500 (Mon, 01 Aug 2011)
New Revision: 4892
Added:
wwwdev/about/
Log:
new look & feel
From vytas at ci.uchicago.edu Mon Aug 1 09:18:32 2011
From: vytas at ci.uchicago.edu (vytas at ci.uchicago.edu)
Date: Mon, 01 Aug 2011 14:18:32 -0000
Subject: [Swift-commit] r4893 - in wwwdev/about: . _notes
Message-ID: <20110801141831.366B29CC9E@svn.ci.uchicago.edu>
Author: vytas
Date: 2011-08-01 09:18:31 -0500 (Mon, 01 Aug 2011)
New Revision: 4893
Added:
wwwdev/about/_notes/
wwwdev/about/_notes/dwsync.xml
wwwdev/about/index.php
Log:
new look & feel
Added: wwwdev/about/_notes/dwsync.xml
===================================================================
--- wwwdev/about/_notes/dwsync.xml (rev 0)
+++ wwwdev/about/_notes/dwsync.xml 2011-08-01 14:18:31 UTC (rev 4893)
@@ -0,0 +1,4 @@
+
+
+
+
\ No newline at end of file
Added: wwwdev/about/index.php
===================================================================
--- wwwdev/about/index.php (rev 0)
+++ wwwdev/about/index.php 2011-08-01 14:18:31 UTC (rev 4893)
@@ -0,0 +1,62 @@
+
+
+
+
+Swift - About
+
+
+
+
+
+
+
+
+
+
+
+
+
+
About
+
+
+
+
+
+
Swift incorporates several existing toolkits and engines to achieve its results. What can Swift do for you?
+
+
Globus Toolkit
+
Swift uses the Globus Toolkit as middleware to talk to various resources.
+
+
Java CoG Kit
+
The CoG Kit provides a high level uniform interface to different versions of the Globus Toolkit. It also contains the Karajan language and interpreter which is used as an underlying execution engine for Swift.
+
+
TeraGrid
+
+"TeraGrid is an open scientific discovery infrastructure combining leadership class resources at nine partner sites to create an integrated, persistent computational resource."
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
From vytas at ci.uchicago.edu Mon Aug 1 09:22:09 2011
From: vytas at ci.uchicago.edu (vytas at ci.uchicago.edu)
Date: Mon, 01 Aug 2011 14:22:09 -0000
Subject: [Swift-commit] r4894 - in wwwdev: . apps apps/_notes
Message-ID: <20110801142155.158739CC9E@svn.ci.uchicago.edu>
Author: vytas
Date: 2011-08-01 09:21:54 -0500 (Mon, 01 Aug 2011)
New Revision: 4894
Added:
wwwdev/apps/
wwwdev/apps/_notes/
wwwdev/apps/_notes/dwsync.xml
wwwdev/apps/index.php
wwwdev/apps/style2.css
Log:
new look & feel
Added: wwwdev/apps/_notes/dwsync.xml
===================================================================
--- wwwdev/apps/_notes/dwsync.xml (rev 0)
+++ wwwdev/apps/_notes/dwsync.xml 2011-08-01 14:21:54 UTC (rev 4894)
@@ -0,0 +1,5 @@
+
+
+
+
+
\ No newline at end of file
Added: wwwdev/apps/index.php
===================================================================
--- wwwdev/apps/index.php (rev 0)
+++ wwwdev/apps/index.php 2011-08-01 14:21:54 UTC (rev 4894)
@@ -0,0 +1,54 @@
+
+
+
+
+Swift
+
+
+
+
+
+
This cookbook covers various recipes involving setting up and running Swift under diverse
+configurations based on the application requirements and the underlying
+infrastructures. The Swift system comprises of SwiftScript language and the
+Swift runtime system. For
+introductory material, consult the Swift tutorial found
+here.
+
+
+
+
2. Swift Basics
+
+
+
2.1. Installation
+
This section takes you through the installation of the Swift system on your
+computer. We will start with the prerequisites as explained in the subsequent
+section.
+
+
2.1.1. Prerequisites
+
Check your Java
Swift is a Java application. Make sure you are running Java version 5 or higher. You
+can make sure you have Java in your $PATH (or $HOME/.soft file depending upon
+your environment)
+
Following are the possible ways to detect and run Java:
+
+
+
$ grep java $HOME/.soft
+#+java-sun # Gives you Java 5
++java-1.6.0_03-sun-r1
+$ which java
+/soft/java-1.6.0_11-sun-r1/bin/java
+$ java -version
+java version "1.6.0_11"
+Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
+Java HotSpot(TM) 64-Bit Server VM (build 11.0-b16, mixed mode)
+
+
+
+
+
2.2. Setting up to run Swift
+
This is simple. We will be using a pre-compiled version of Swift that can be
+downloaded from here. Download and untar the latest precompiled version as follows:
+
+
+
$ tar xf swift-0.92.1.tar.gz
+
+
+
2.2.1. Environment Setup
+
The examples were tested with Java version 1.6. Make sure you do not already
+have Swift in your PATH. If you do, remove it, or remove any +swift or @swift
+lines from your $HOME/.soft or $HOME/.bash_profile file. Then do:
+
+
+
PATH=$PATH:/path/to/swift/bin
+
+
Note that the environment will be different when using Swift from prebuilt distribution (as above) and trunk. The PATH setup when using swift from trunk would be as follows:
+
+
+
PATH=$PATH:/path/to/swift/dist/swift-svn/bin
+
+
+
+
+
Warning
+
+
Do NOT set SWIFT_HOME or CLASSPATH in your environment unless you
+fully understand how these will affect Swift’s execution.
+
+
+
To execute your Swift script on a login host (or "localhost") use
+the following command:
+
+
+
swift -tc.file tc somescript.swift
+
+
+
+
2.2.2. Setting transformation catalog
+
The transformation catalog lists where application executables are located on
+remote sites.
+
By default, the site catalog is stored in etc/tc.data. This path can be
+overridden with the tc.file configuration property, either in the Swift
+configuration file or on the command line.
+
The format is one line per executable per site, with fields separated by tabs
+or spaces.
The fields are: site, transformation-name, executable-path, installation-status, platform, and profile entries.
+
The site field should correspond to a site name listed in the sites catalog.
+
The transformation-name should correspond to the transformation name used in a
+SwiftScript app procedure.
+
The executable-path should specify where the particular executable is located
+on that site.
+
The installation-status and platform fields are not used. Set them to
+INSTALLED and INTEL32::LINUX respectively.
+
The profiles field should be set to null if no profile entries are to be
+specified, or should contain the profile entries separated by semicolons.
+
+
+
2.2.3. Setting swift configuration
+
Many configuration properties could be set using the Swift configuration file.
+We will not cover them all in this section. see
+here for details. In this section we will cover a simple configuration file with the most basic properties.
sites.xml specifies details of the sites that Swift can run on. Following is
+an example of a simple sites.xml file entry for running Swift on local
+environment:
Also includes a description of Swift inputs and outputs.
+
+
+
2.6. What if Swift hangs
+
Owing to its highly multithreaded architecture it is often the case that the
+underlying java virtual machine gets into deadlock situations or Swift hangs
+because of other complications in its threaded operations. Under such
+situations, Swift hang-checker chips in and resolves the situation.
+
+
+
+how to use the information to identify and correct the deadlock.
+
+
+
+
+How close to the Swift source code can we make the hang-checker messages, so that the user can relate it to Swift functions, expressions, and ideally source code lines?
+
+
+
+
+The current Hang Checker output is actually very nice and useful already:
+
+
+
+
+
+
Registered futures:
+Rupture[] rups Closed, 1 elements, 0 listeners
+Variation vars - Closed, no listeners
+SgtDim sub - Open, 1 listeners
+string site Closed, no listeners
+Variation[] vars Closed, 72 elements, 0 listeners
+
+
+
+
2.7. Resuming a stopped or crashed Swift Run
+
I had a .rlog file from a Swift run that ran out of time. I kicked it off
+using the -resume flag described in section 16.2 of the Swift User Guide and
+it picked up where it left off. Then I killed it because I wanted to make
+changes to my sites file.
No new rlog file was emitted but it did recognize the progress that had been
+made, the 96 tasks that finished sucessfully above and resumed from 2558 tasks
+finished.
From Neil: A comment about that section of the user guide: It says "In order
+to restart from a restart log file, the -resume logfile argument can be used
+after the SwiftScript? program file name." and then puts the -resume logfile
+argument before the script file name. I’m sure the order doesn’t matter but
+the contradiction is confusing.
+
Notes to add (from Mike):
+
+
+
+explain what aspects of a Swift script make it restartable, and which
+ aspects are notrestartable. Eg, if your mappers can return different data at
+different times, what happens? What other non-determinsitc behavior would
+cause unpredictable, unexpected, or undesired behavior on resumption?
+
+
+
+
+explain what changes you can make in the execution environment (eg
+ increasing or reducing CPUs to run on or throttles, etc); fixing tc.data
+entries, env vars, or apps, etc.
+
+
+
+
+note that resume will again retry failed app() calls. Explain if the retry
+ count starts over or not.
+
+
+
+
+explain how to resume after multiple failures and resumes - i.e. if a .rlog
+ is generated on each run, which one should you resume from? Do you have a
+choice of resuming from any of them, and what happens if you go backwards to
+an older resume file?
+
+
+
+
+whap happens when you kill (eg with C) a running swift script? Is the
+ signal caught, and the resume file written out at that point? Or written out
+all along? (Note case in which script ws running for hours, then hit C, but
+resume fie was short (54 bbytes) and swift shows no sign of doing a resume?
+(It silently ignored resume file instead of acknowleging that it found one
+with not useful resume state in it???) Swift should clearly state that its
+resuming and what its resume state is.
+
+
+
+
swift -resume ftdock-[id].0.rlog \[rest of the exact command line from initial
+run\]
+
+
+
2.8. Passing an array to swift?
+
Arrays can be passed to Swift in one of the following ways:
+
+
+
+You can write the array to a file and read in in swift using
+readData (or readData2).
+
+
+
+
+Direct the array into a file (possibly with a "here document" which expands the array) and then read the file in Swift with readData() or process it with a Swift app() function?
+
+
+
+
+You can use @strsplit on a comma separated command line arg and that works well for me.
+
how the .'s match
+prefix and suffix dont span dirs
+intervening pattern must be digits
+these digits become the array indices
+explain how padding= arg works & helps (including padding=0)
+figure out and explain differences between simple_mapper and
+filesys_mapper
+FIXME: Use the "filesys_mapper" and its "location=" parameter to map the
+input data from /home/wilde/bigdata/*
+
Abbreviations for SingleFileMapper
+Notes:
+
within <> you can only have a literal string as in <"filename">, not an
+expression. Someday we will fix this to make <> accept a general expression.
+you can use @filenames( ) (note: plural) to pull off a list of filenames.
+
writeData()
+
example here
+
+
+
$ cat writedata.swift
+
+
+
+
type file;
+
+file f <"filea">;
+file nf <"filenames">;
+nf =writeData(@f);
StructuredRegexpMapper
+IN PROGRESS This mapper can be used to base the mapped filenames of an output
+array on the mapped filenames of an existing array. landuse outputfiles[]
+<structured_regexp_mapper; source=inputfiles,
+location="./output",match="(.)*tif", transform="\\1histogram">;
+
Use the undocumented "structured_regexp_mapper" to name the output
+filenames based on the input filenames:
Coasters were introduced in Swift v0.6 as an experimental feature. In many
+applications, Swift performance can be greatly enhanced by the use of CoG
+coasters. CoG coasters provide a low-overhead job submission and file transfer
+mechanism suited for the execution of short jobs (on the order of a few
+seconds). A detailed information on coasters can be found at http://www.ci.uchicago.edu/swift/guides/userguide.php#coasters.
+
Following is a coasters setup case-study for a PBS underlying provider where sites.xml coaster settings were:
+
+
+
<executionprovider="coaster"jobmanager="local:pbs"/>
+<profilenamespace="globus"key="project">CI-CCR000013</profile>
+
+<!-- Note that the following is going to be defunct in the new version (0.93+) and replaced by
+"ProviderAttributes" key and may not work in the future Swift versions-->
+
+<!--<profile namespace="globus" key="ppn">24:cray:pack</profile>-->
+
+<profilenamespace="globus"key="providerAttributes">
+pbs.aprun
+pbs.mpp=true
+</profile>
+
+<profilenamespace="globus"key="workersPerNode">24</profile>
+<profilenamespace="globus"key="maxTime">100000</profile>
+
+<profilenamespace="globus"key="lowOverallocation">100</profile>
+<profilenamespace="globus"key="highOverallocation">100</profile>
+
+<profilenamespace="globus"key="slots">20</profile>
+<profilenamespace="globus"key="nodeGranularity">5</profile>
+<profilenamespace="globus"key="maxNodes">5</profile>
+<profilenamespace="karajan"key="jobThrottle">20.00</profile>
+<profilenamespace="karajan"key="initialScore">10000</profile>
+
The following table briefly describes the elements on the coasters setup:
+
+
+
+
+
+
+
profile key
+
brief description
+
+
+
+
+
slots
+
How many maximum LRM jobs/worker blocks are allowed
+
+
+
workersPerNode
+
How many coaster workers to run per execution node
+
+
+
nodeGranularity
+
Each worker block uses a number of nodes that is a multiple of this number
+
+
+
lowOverallocation
+
How many times larger than the job walltime should a block’s walltime be if all jobs are 1s long
+
+
+
highOverallocation
+
How many times larger than the job walltime should a block’s walltime be if all jobs are infinitely long
+
+
+
workersPerNode
+
How many coaster workers to run per execution node reserve How many seconds to reserve in a block’s walltime for starting/shutdown operations
+
+
+
maxnodes
+
The maximum number of nodes allowed in a block
+
+
+
maxtime
+
The maximum number of walltime allowed for a block coaster service
+
+
+
jobThrottle
+
the number of concurrent jobs allowed on a site
+
+
+
+
+
+
3.1. For Advanced Users
+
One of the main reason that one would initially deviate from coaster
+defaults into more complex pool entries is to force jobs to fit into some
+site-imposed constraint. For instance a typical submission to the experimental
+queue requires a user to request upto 3 nodes for under 1 hour. This setup
+could be achieved with a careful tuning of coaters parameters.
+
+
+
3.2. Coaster providers: local, ssh, pbs
+
Settings and examples for different coaster providers mechanisms.
+
+
+
+
+
4. Swift on Diverse Infrastructures
+
+
+
4.1. Beagle
+
Swift is now installed on Beagle as a module. Swift supports a Coasters based,
+computing environment for Beagle. A detailed Swift documentation is maintained
+[[here]. To get started
+with Swift on Beagle follow the steps outlined below:
+
step 1. Load the Swift module on Beagle as follows: module load swift
+
step 2. Create and change to a directory where your Swift related work will
+stay. (say, mkdir swift-lab, followed by, cd swift-lab)
+
step 3. To get started with a simple example running /bin/cat to read an input file data.txt and write to an output file f.nnn.out, copy the folder at /home/ketan/labs/catsn to the above directory. (cp -r /home/ketan/catsn . followed by cd catsn).
+
step 4. In the sites file: beagle-coaster.xml, make the following two
+changes: 1) change the path of workdirectory to your preferred location
+(say to /lustre/beagle/$USER/swift-lab/swift.workdir) and 2) Change the
+project name to your project (CI-CCR000013) . The workdirectory will contain
+execution data related to each run, e.g. wrapper scripts, system information,
+inputs and outputs.
+
step 5. Run the example using following commandline (also found in run.sh):
+swift -config cf -tc.file tc -sites.file beagle-coaster.xml catsn.swift -n=1
+. You can further change the value of -n to any arbitrary number to run that
+many number of concurrent cat
+
step 6. Check the output in the generated outdir directory (ls outdir)
+
Note: Running from sandbox node or requesting 1 hour walltime for upto 3 nodes
+will get fast prioritized execution. Good for small tests
+
+
+
4.2. PADS
+
Swift on PADS
+To execute your Swift script on the PADS cluster use this command:
+
+
+
swift -tc.file tc -sites.file pbs.xml modis.swift
+
+
where the contents of a simple pbs.xml sites file could be:
This section describes how to get Swift running on the OSG Grid. We will use a
+manual coaster setup to get Swift running on OSG.
+
Coaster setup on OSG
The following figure shows an abstract scheme for the manual coasters setup on
+OSG.
+
+
+
+
+
+
In the following steps, we will go through the process of manually setting
+
+
+
4.4. Bionimbus
+
This section explains a step by step procedure on getting Swift running on the
+Bionimbus cloud. We will use the manualcoasters configuration on the
+Bionimbus cloud.
+
step1. Connect to the gateway (ssh gatewayx.lac.uic.edu)
WHERE:
+*=network interface, should remain the same on all cases
+
localhost=the gateway host, should remain the same
+
5000(LEFT OF localhost)=the port number on localhost to listen to **THIS WILL
+vary depending upon which port you want to listen to
+
5000(RIGHT OF localhost)=the port on target host that you want to forward
+
root at 10.101.8.50=the ip of the Virtual Machine on Bionimbus cloud, this will
+vary based on what ip you get for your Virtual Machine instance
+
#On anywhere as long as provide the correct callback uri: here the
+"http://140.221.9.110:35753" is the callback uri of previous ones
+
step6. Copy the worker.pl from gatewayx to the VM: scp
+swift-0.92/bin/worker.pl root at 10.101.8.50:/bin/, you will be prompted for the
+root password of the VM.
+
step7. Start the worker from the virtual machine
+worker.pl http://localhost:35753 tmp /tmp
+where 35753 is the port where the coaster service is listening to the workers+
+All text from standard output and standard error produced by running the
+swift command
+
+
+
+
+The .log file from this run. It will be named swiftscript.uniqueID.log
+where "swiftscript" is the name of your *.swift script source file, and
+uniqueID is a long unique id which starts with the date and time you ran the
+swift command.
+
+
+
+
+$HOME/.globus/coasters directory on remote machines on which you are
+running coasters
+
+
+
+
+$HOME/.globus/scripts directory on the host on which you run the Swift
+command, when swift is submitting to a local scheduler (Condor, PBS, SGE,
+Cobalt)
+
+
+
+
+$HOME/.globus/??? on remote systems that you access via Globus
+
+
+
+
+
+
+
6. Log Processing
+
+
To properly generate log plots, you must enable VDL/Karajan logging. Make sure
+log4.properties contains:
+tc.data and sites.xml (or whatever you named these files)
+
+
+
+
+your .swift source file and any .swift files it imports
+
+
+
+
+any external mapper scripts called by your .swift script
+
+
+
+
+all text from standard output and standard error produced by running the
+swift command
+
+
+
+
+The .log file from this run. It will be named swiftscript.uniqueID.log
+
+
+
+
+where "swiftscript" is the name of your *.swift script source file, and
+uniqueID is a long unique id which starts with the date and time you ran the
+swift command.
+
+
+
+
+The swift command line you invoked
+
+
+
+
+Any swift.properties entries you over-rode ($HOME/.swift/swift.properties,
+-config.file argument properties file, any changes to etc/swift.proerties from
+your swift distribution)
+
+
+
+
+Which swift distribution you are running (release; svn revisions; other
+local changes you mave have made or included)
+
+
+
+
+
+
+
+
+
+
+
Added: wwwdev/cookbook/cookbook-asciidoc.txt
===================================================================
--- wwwdev/cookbook/cookbook-asciidoc.txt (rev 0)
+++ wwwdev/cookbook/cookbook-asciidoc.txt 2011-08-01 14:23:26 UTC (rev 4896)
@@ -0,0 +1,895 @@
+// To compile use: asciidoc -a toc -n cookbook-asciidoc.txt
+
+
+Swift Cookbook
+==============
+Swift team
+v0.92, March 2011
+
+Overview
+--------
+This cookbook covers various recipes involving setting up and running Swift under diverse
+configurations based on the application requirements and the underlying
+infrastructures. The Swift system comprises of SwiftScript language and the
+Swift runtime system. For
+introductory material, consult the Swift tutorial found
+link:http://www.ci.uchicago.edu/swift/guides/tutorial.php[here].
+
+Swift Basics
+------------
+
+Installation
+~~~~~~~~~~~~
+
+This section takes you through the installation of the Swift system on your
+computer. We will start with the prerequisites as explained in the subsequent
+section.
+
+Prerequisites
+^^^^^^^^^^^^^^
+.Check your Java
+Swift is a Java application. Make sure you are running Java version 5 or higher. You
+can make sure you have Java in your $PATH (or $HOME/.soft file depending upon
+your environment)
+
+Following are the possible ways to detect and run Java:
+
+----
+$ grep java $HOME/.soft
+#+java-sun # Gives you Java 5
++java-1.6.0_03-sun-r1
+$ which java
+/soft/java-1.6.0_11-sun-r1/bin/java
+$ java -version
+java version "1.6.0_11"
+Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
+Java HotSpot(TM) 64-Bit Server VM (build 11.0-b16, mixed mode)
+----
+
+Setting up to run Swift
+~~~~~~~~~~~~~~~~~~~~~~~~
+This is simple. We will be using a pre-compiled version of Swift that can be
+downloaded from link:http://www.ci.uchicago.edu/swift/downloads/index.php[here]. Download and untar the latest precompiled version as follows:
+
+----
+$ tar xf swift-0.92.1.tar.gz
+----
+
+Environment Setup
+^^^^^^^^^^^^^^^^^^
+
+The examples were tested with Java version 1.6. Make sure you do not already
+have Swift in your PATH. If you do, remove it, or remove any +swift or @swift
+lines from your $HOME/.soft or $HOME/.bash_profile file. Then do:
+
+----
+PATH=$PATH:/path/to/swift/bin
+----
+
+Note that the environment will be different when using Swift from prebuilt distribution (as above) and trunk. The PATH setup when using swift from trunk would be as follows:
+
+----
+PATH=$PATH:/path/to/swift/dist/swift-svn/bin
+----
+
+WARNING: Do NOT set SWIFT_HOME or CLASSPATH in your environment unless you
+fully understand how these will affect Swift's execution.
+
+To execute your Swift script on a login host (or "localhost") use
+the following command:
+
+----
+swift -tc.file tc somescript.swift
+----
+
+Setting transformation catalog
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The transformation catalog lists where application executables are located on
+remote sites.
+
+By default, the site catalog is stored in etc/tc.data. This path can be
+overridden with the tc.file configuration property, either in the Swift
+configuration file or on the command line.
+
+The format is one line per executable per site, with fields separated by tabs
+or spaces.
+
+Some example entries:
+
+----
+localhost echo /bin/echo INSTALLED INTEL32::LINUX null
+TGUC touch /usr/bin/touch INSTALLED INTEL32::LINUX GLOBUS::maxwalltime="00:00:10"
+----
+The fields are: _site_, _transformation-name_, _executable-path_, _installation-status_, _platform_, and _profile_ entries.
+
+The _site_ field should correspond to a site name listed in the sites catalog.
+
+The _transformation-name_ should correspond to the transformation name used in a
+SwiftScript app procedure.
+
+The _executable-path_ should specify where the particular executable is located
+on that site.
+
+The _installation-status_ and _platform_ fields are not used. Set them to
+**INSTALLED** and **INTEL32::LINUX** respectively.
+
+The _profiles_ field should be set to null if no profile entries are to be
+specified, or should contain the profile entries separated by semicolons.
+
+Setting swift configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Many configuration properties could be set using the Swift configuration file.
+We will not cover them all in this section. see
+link:http://www.ci.uchicago.edu/swift/guides/userguide.php#engineconfiguration[here] for details. In this section we will cover a simple configuration file with the most basic properties.
+
+----
+# A comment
+wrapperlog.always.transfer=true
+sitedir.keep=true
+execution.retries=1
+lazy.errors=true
+status.mode=provider
+use.provider.staging=true
+provider.staging.pin.swiftfiles=false
+clustering.enabled=false
+clustering.queue.delay=10
+clustering.min.time=86400
+foreach.max.threads=100
+provenance.log=true
+----
+
+Setting sites.xml
+^^^^^^^^^^^^^^^^^^
+sites.xml specifies details of the sites that Swift can run on. Following is
+an example of a simple sites.xml file entry for running Swift on local
+environment:
+
+[xml]
+source~~~~~~
+
+
+
+/var/tmp
+.07
+100000
+
+source~~~~~~
+
+First SwiftScript
+~~~~~~~~~~~~~~~~~
+Your first SwiftScript
+Hello Swift-World!
+
+A good sanity check that Swift is set up and running OK locally is this:
+
+----
+$ which swift
+
+/home/wilde/swift/src/stable/cog/modules/swift/dist/swift-svn/bin/swift
+
+$ echo 'trace("Hello, Swift world!");' >hello.swift
+
+$ swift hello.swift
+
+Swift svn swift-r3202 cog-r2682
+
+RunID: 20100115-1240-6xhzxuz3
+
+Progress:
+
+SwiftScript trace: Hello, Swift world!
+
+Final status:
+
+$
+----
+A good first tutorial in using Swift is at:
+http://www.ci.uchicago.edu/swift/guides/tutorial.php. Follow the steps in that
+tutorial to learn how to run a few simple scripts on the login host.
+
+second SwiftScript
+~~~~~~~~~~~~~~~~~~~
+Following is a more involved Swift script.
+[java]
+source~~~~~~~
+type file;
+
+app (file o) cat (file i)
+{
+ cat @i stdout=@o;
+}
+
+file out[];
+
+foreach j in [1:@toint(@arg("n","1"))] {
+
+ file data<"data.txt">;
+
+ out[j] = cat(data);
+}
+source~~~~~~~
+
+Swift Commandline Options
+~~~~~~~~~~~~~~~~~~~~~~~~~
+A description of Swift Commandline Options
+
+Also includes a description of Swift inputs and outputs.
+
+What if Swift hangs
+~~~~~~~~~~~~~~~~~~~
+Owing to its highly multithreaded architecture it is often the case that the
+underlying java virtual machine gets into deadlock situations or Swift hangs
+because of other complications in its threaded operations. Under such
+situations, Swift _hang-checker_ chips in and resolves the situation.
+
+. how to use the information to identify and correct the deadlock.
+
+. How close to the Swift source code can we make the hang-checker messages, so that the user can relate it to Swift functions, expressions, and ideally source code lines?
+
+. The current Hang Checker output is actually *very* nice and useful already:
+
+----
+Registered futures:
+Rupture[] rups Closed, 1 elements, 0 listeners
+Variation vars - Closed, no listeners
+SgtDim sub - Open, 1 listeners
+string site Closed, no listeners
+Variation[] vars Closed, 72 elements, 0 listeners
+----
+
+Resuming a stopped or crashed Swift Run
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+I had a .rlog file from a Swift run that ran out of time. I kicked it off
+using the -resume flag described in section 16.2 of the Swift User Guide and
+it picked up where it left off. Then I killed it because I wanted to make
+changes to my sites file.
+
+----
+. . .
+Progress: Selecting site:1150 Stage in:55 Active:3 Checking status:1
+Stage out:37 Finished in previous run:2462 Finished successfully:96
+Progress: Selecting site:1150 Stage in:55 Active:2 Checking status:1
+Stage out:38 Finished in previous run:2462 Finished successfully:96
+Cleaning up...
+Shutting down service at https://192.5.86.6:54813
+Got channel MetaChannel: 1293358091 -> null
++ Done
+Canceling job 9297.svc.pads.ci.uchicago.edu
+----
+
+No new rlog file was emitted but it did recognize the progress that had been
+made, the 96 tasks that finished sucessfully above and resumed from 2558 tasks
+finished.
+
+----
+[nbest at login2 files]$ pwd
+/home/nbest/bigdata/files
+[nbest at login2 files]$
+~wilde/swift/src/stable/cog/modules/swift/dist/swift-svn/bin/swift \
+> -tc.file tc -sites.file pbs.xml ~/scripts/mcd12q1.swift -resume
+> mcd12q1-20100310-1326-ptxe1x1d.0.rlog
+Swift svn swift-r3255 (swift modified locally) cog-r2723 (cog modified
+locally)
+RunID: 20100311-1027-148caf0a
+Progress:
+Progress: uninitialized:4
+Progress: Selecting site:671 Initializing site shared directory:1 Finished
+in previous run:1864
+Progress: uninitialized:1 Selecting site:576 Stage in:96 Finished in
+previous run:1864
+Progress: Selecting site:1150 Stage in:94 Submitting:2 Finished in
+previous run:2558
+Progress: Selecting site:1150 Stage in:94 Submitted:2 Finished in previous
+run:2558
+Progress: Selecting site:1150 Stage in:93 Submitting:1 Submitted:2
+Finished in previous run:2558
+Progress: Selecting site:1150 Stage in:90 Submitting:1 Submitted:5
+Finished in previous run:2558
+Progress: Selecting site:1150 Stage in:90 Submitted:5 Active:1 Finished
+in previous run:2558
+----
+
+
+From Neil: A comment about that section of the user guide: It says "In order
+to restart from a restart log file, the -resume logfile argument can be used
+after the SwiftScript? program file name." and then puts the -resume logfile
+argument before the script file name. I'm sure the order doesn't matter but
+the contradiction is confusing.
+
+Notes to add (from Mike):
+
+- explain what aspects of a Swift script make it restartable, and which
+ aspects are notrestartable. Eg, if your mappers can return different data at
+different times, what happens? What other non-determinsitc behavior would
+cause unpredictable, unexpected, or undesired behavior on resumption?
+
+- explain what changes you can make in the execution environment (eg
+ increasing or reducing CPUs to run on or throttles, etc); fixing tc.data
+entries, env vars, or apps, etc.
+
+- note that resume will again retry failed app() calls. Explain if the retry
+ count starts over or not.
+
+- explain how to resume after multiple failures and resumes - i.e. if a .rlog
+ is generated on each run, which one should you resume from? Do you have a
+choice of resuming from any of them, and what happens if you go backwards to
+an older resume file?
+
+- whap happens when you kill (eg with ^C) a running swift script? Is the
+ signal caught, and the resume file written out at that point? Or written out
+all along? (Note case in which script ws running for hours, then hit ^C, but
+resume fie was short (54 bbytes) and swift shows no sign of doing a resume?
+(It silently ignored resume file instead of acknowleging that it found one
+with not useful resume state in it???) Swift should clearly state that its
+resuming and what its resume state is.
+
++swift -resume ftdock-[id].0.rlog \[rest of the exact command line from initial
+run\]+
+
+
+Passing an array to swift?
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+Arrays can be passed to Swift in one of the following ways:
+
+. You can write the array to a file and read in in swift using
+readData (or readData2).
+. Direct the array into a file (possibly with a "here document" which expands the array) and then read the file in Swift with readData() or process it with a Swift app() function?
+. You can use @strsplit on a comma separated command line arg and that works well for me.
+
+Mappers
+^^^^^^^^
+SimpleMapper
+
+----
+$ cat swiftapply.swift
+----
+
+[java]
+source~~~~
+type RFile;
+trace("hi 1");
+app (RFile result) RunR (RFile rcall)
+{
+ RunR @rcall @result;
+}
+trace("hi 2");
+RFile rcalls[] ;
+RFile results[] ;
+trace("start");
+foreach c, i in rcalls {
+ trace("c",i, at c);
+ trace("r",i, at filename(results[i]));
+ results[i] = RunR(c);
+}
+source~~~~
+
+----
+$ ls calldir resdir
+calldir:
+rcall.1.Rdata rcall.2.Rdata rcall.3.Rdata rcall.4.Rdata
+resdir:
+result.1.Rdata result.2.Rdata result.3.Rdata result.4.Rdata
+$
+----
+
+Notes:
+
+how the .'s match
+prefix and suffix dont span dirs
+intervening pattern must be digits
+these digits become the array indices
+explain how padding= arg works & helps (including padding=0)
+figure out and explain differences between simple_mapper and
+filesys_mapper
+FIXME: Use the "filesys_mapper" and its "location=" parameter to map the
+input data from /home/wilde/bigdata/*
+
+Abbreviations for SingleFileMapper
+Notes:
+
+within <> you can only have a literal string as in <"filename">, not an
+expression. Someday we will fix this to make <> accept a general expression.
+you can use @filenames( ) (note: plural) to pull off a list of filenames.
+
+writeData()
+
+example here
+
+----
+$ cat writedata.swift
+----
+
+[java]
+source~~~~
+type file;
+
+file f <"filea">;
+file nf <"filenames">;
+nf = writeData(@f);
+source~~~~
+
+----
+$ swift writedata.swift
+Swift svn swift-r3264 (swift modified locally) cog-r2730 (cog modified
+locally)
+RunID: 20100319-2002-s9vpo0pe
+Progress:
+Final status:
+$ cat filenames
+filea$
+----
+
+StructuredRegexpMapper
+IN PROGRESS This mapper can be used to base the mapped filenames of an output
+array on the mapped filenames of an existing array. landuse outputfiles[]
+;
+
+Use the undocumented "structured_regexp_mapper" to name the output
+filenames based on the input filenames:
+
+For example:
+
+----
+login2$ ls /home/wilde/bigdata/data/sample
+h11v04.histogram h11v05.histogram h12v04.histogram h32v08.histogram
+h11v04.tif h11v05.tif h12v04.tif h32v08.tif
+login2$
+
+login2$ cat regexp2.swift
+type tif;
+type mytype;
+
+tif images[];
+
+mytype of[] ;
+
+foreach image, i in images {
+ trace(i, at filename(images));
+ trace(i, at filename(of[i]));
+}
+login2$
+
+login1$ swift regexp2.swift
+Swift svn swift-r3255 (swift modified locally) cog-r2723 (cog modified
+locally)
+
+RunID: 20100310-1105-4okarq08
+Progress:
+SwiftScript trace: 1, output/myfile.h11v04.mytype
+SwiftScript trace: 2, home/wilde/bigdata/data/sample/h11v05.tif
+SwiftScript trace: 3, home/wilde/bigdata/data/sample/h12v04.tif
+SwiftScript trace: 0, output/myfile.h32v08.mytype
+SwiftScript trace: 0, home/wilde/bigdata/data/sample/h32v08.tif
+SwiftScript trace: 3, output/myfile.h12v04.mytype
+SwiftScript trace: 1, home/wilde/bigdata/data/sample/h11v04.tif
+SwiftScript trace: 2, output/myfile.h11v05.mytype
+Final status:
+login1$
+----
+
+Coasters
+--------
+Coasters were introduced in Swift v0.6 as an experimental feature. In many
+applications, Swift performance can be greatly enhanced by the use of CoG
+coasters. CoG coasters provide a low-overhead job submission and file transfer
+mechanism suited for the execution of short jobs (on the order of a few
+seconds). A detailed information on coasters can be found at http://www.ci.uchicago.edu/swift/guides/userguide.php#coasters.
+//**Include neat diagrams.**
+
+Following is a coasters setup case-study for a PBS underlying provider where sites.xml coaster settings were:
+
+[xml]
+source~~~~
+
+CI-CCR000013
+
+
+
+
+
+
+pbs.aprun
+pbs.mpp=true
+
+
+24
+100000
+
+100
+100
+
+20
+5
+5
+20.00
+10000
+source~~~~
+
+The following table briefly describes the elements on the coasters setup:
+[width="70%", cols="^3,10", options="header"]
+|=============================================================================================
+|profile key | brief description
+|slots | How many maximum LRM jobs/worker blocks are allowed
+|workersPerNode | How many coaster workers to run per execution node
+|nodeGranularity | Each worker block uses a number of nodes that is a multiple of this number
+|lowOverallocation | How many times larger than the job walltime should a block's walltime be if all jobs are 1s long
+|highOverallocation | How many times larger than the job walltime should a block's walltime be if all jobs are infinitely long
+|workersPerNode | How many coaster workers to run per execution node reserve How many seconds to reserve in a block's walltime for starting/shutdown operations
+|maxnodes | The maximum number of nodes allowed in a block
+|maxtime | The maximum number of walltime allowed for a block coaster service
+|jobThrottle |the number of concurrent jobs allowed on a site
+|==============================================================================================
+
+//
+//For Beginners
+//~~~~~~~~~~~~~~
+//Coasters for beginners. Usage of existing, prebuilt templates.
+//
+//For Intermediate Users
+//~~~~~~~~~~~~~~~~~~~~~~~
+//Coasters for intermediate users.
+//
+//Using gensites
+//^^^^^^^^^^^^^^^
+//Usage of gensites to generate your own sites
+//configurations.
+//
+//
+//
+
+For Advanced Users
+~~~~~~~~~~~~~~~~~~
+
+//Coasters for advanced users. Getting your hands dirty.
+
+One of the main reason that one would initially deviate from coaster
+defaults into more complex pool entries is to force jobs to fit into some
+site-imposed constraint. For instance a typical submission to the experimental
+queue requires a user to request upto 3 nodes for under 1 hour. This setup
+could be achieved with a careful tuning of coaters parameters.
+
+//How to run Swift under different Coasters configurations
+//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+//
+//Manual
+//^^^^^^^
+//Todo
+//
+//Passive
+//^^^^^^^^
+//Todo
+//
+//Persistent
+//^^^^^^^^^^^
+//Todo
+//
+//Summary of Differences Between different Coaster types
+//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+//A tabular representations of highlights of different coaster setups
+//
+//
+//Data Management
+//
+
+Coaster providers: local, ssh, pbs
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Settings and examples for different coaster providers mechanisms.
+
+
+
+Swift on Diverse Infrastructures
+---------------------------------
+
+Beagle
+~~~~~~
+Swift is now installed on Beagle as a module. Swift supports a Coasters based,
+computing environment for Beagle. A detailed Swift documentation is maintained
+[[http://www.ci.uchicago.edu/swift/docs/index.php][here]]. To get started
+with Swift on Beagle follow the steps outlined below:
+
+*step 1.* Load the Swift module on Beagle as follows: +module load swift+
+
+*step 2.* Create and change to a directory where your Swift related work will
+stay. (say, +mkdir swift-lab+, followed by, +cd swift-lab+)
+
+*step 3.* To get started with a simple example running +/bin/cat+ to read an input file +data.txt+ and write to an output file +f.nnn.out+, copy the folder at +/home/ketan/labs/catsn+ to the above directory. (+cp -r /home/ketan/catsn .+ followed by +cd catsn+).
+
+*step 4.* In the sites file: +beagle-coaster.xml+, make the following two
+changes: *1)* change the path of +workdirectory+ to your preferred location
+(say to +/lustre/beagle/$USER/swift-lab/swift.workdir+) and *2)* Change the
+project name to your project (+CI-CCR000013+) . The workdirectory will contain
+execution data related to each run, e.g. wrapper scripts, system information,
+inputs and outputs.
+
+*step 5.* Run the example using following commandline (also found in run.sh):
++swift -config cf -tc.file tc -sites.file beagle-coaster.xml catsn.swift -n=1+
+. You can further change the value of +-n+ to any arbitrary number to run that
+many number of concurrent +cat+
+
+*step 6.* Check the output in the generated +outdir+ directory (+ls outdir+)
+
+Note: Running from sandbox node or requesting 1 hour walltime for upto 3 nodes
+will get fast prioritized execution. Good for small tests
+
+//
+//Intrepid-BG/P
+//~~~~~~~~~~~~~
+//Swift on Intrepid-BG/P
+//
+
+PADS
+~~~~
+Swift on PADS
+To execute your Swift script on the PADS cluster use this command:
+
+----
+swift -tc.file tc -sites.file pbs.xml modis.swift
+----
+where the contents of a simple pbs.xml sites file could be:
+[xml]
+source~~~~
+
+
+
+ fast
+ 00:05:00
+ 10000
+ .10
+
+ /home/you/swiftwork
+
+
+source~~~~
+
+OSG
+~~~
+This section describes how to get Swift running on the OSG Grid. We will use a
+manual coaster setup to get Swift running on OSG.
+
+.Coaster setup on OSG
+The following figure shows an abstract scheme for the manual coasters setup on
+OSG.
+
+image::figures/coaster_setup.png[Coaster setup]
+
+In the following steps, we will go through the process of manually setting
+//Swift on OSG
+//
+
+Bionimbus
+~~~~~~~~~
+This section explains a step by step procedure on getting Swift running on the
+Bionimbus cloud. We will use the _manual_ _coasters_ configuration on the
+Bionimbus cloud.
+
+**step1.** Connect to the gateway (ssh gatewayx.lac.uic.edu)
+
+**step2.** Start a virtual machine (euca-run-instances -n 1 -t m1.small
+emi-17EB1170)
+
+**step3.** Start the coaster-service on gateway
++coaster-service -port 1984 -localport 35753 -nosec+
+
+**step4.** Start the Swift-script from the gateway using normal Swift commandline
+
++swift -config cf -tc.file tc -sites.file sites.xml yourscript.swift -aparam=999+
+
+.cf
+----
+wrapperlog.always.transfer=true
+sitedir.keep=true
+execution.retries=1
+lazy.errors=true
+status.mode=provider
+use.provider.staging=true
+provider.staging.pin.swiftfiles=false
+foreach.max.threads=100
+provenance.log=true
+----
+
+.tc
+----
+localhost modftdock /home/ketan/swift-labs/bionimbus-coaster-modftdock/app/modftdock.sh null null GLOBUS::maxwalltime="1:00:00"
+----
+
+(See below for a sample sites.xml for this run)
+
+**step5.** Connect back to gateway from virtual machines using reverse ssh tunneling as follows:
+
+.From the gateway prompt
+
++ssh -R *:5000:localhost:5000 root at 10.101.8.50 sleep 999+
+
+WHERE:
+*=network interface, should remain the same on all cases
+
+localhost=the gateway host, should remain the same
+
+5000(LEFT OF localhost)=the port number on localhost to listen to **THIS WILL
+vary depending upon which port you want to listen to
+
+5000(RIGHT OF localhost)=the port on target host that you want to forward
+
+root at 10.101.8.50=the ip of the Virtual Machine on Bionimbus cloud, this will
+vary based on what ip you get for your Virtual Machine instance
+
+#On anywhere as long as provide the correct callback uri: here the
+"http://140.221.9.110:35753" is the callback uri of previous ones
+
+**step6.** Copy the +worker.pl+ from gatewayx to the VM: +scp
+swift-0.92/bin/worker.pl root at 10.101.8.50:/bin/+, you will be prompted for the
+root password of the VM.
+
+**step7.** Start the worker from the virtual machine
++worker.pl http://localhost:35753 tmp /tmp+
+where 35753 is the port where the coaster service is listening to the workers+
+
+.sites.xml for the above run
+
+[xml]
+source~~~~
+
+
+
+ passive
+
+ 4
+ 10000
+ 100
+ 100
+ 100
+ 1
+ 10
+ 25.00
+ 10000
+ proxy
+
+ /home/ketan/swift-labs/bionimbus-coaster-modftdock/swift.workdir
+
+
+source~~~~
+
+//Magellan
+//~~~~~~~~
+//Swift on Magellan
+//
+
+Debugging Swift
+---------------
+Swift errors are logged in several places:
+
+. All text from standard output and standard error produced by running the
+swift command
+. The .log file from this run. It will be named swiftscript.uniqueID.log
+where "swiftscript" is the name of your *.swift script source file, and
+uniqueID is a long unique id which starts with the date and time you ran the
+swift command.
+. $HOME/.globus/coasters directory on remote machines on which you are
+running coasters
+. $HOME/.globus/scripts directory on the host on which you run the Swift
+command, when swift is submitting to a local scheduler (Condor, PBS, SGE,
+Cobalt)
+. $HOME/.globus/??? on remote systems that you access via Globus
+
+
+Log Processing
+---------------
+
+To properly generate log plots, you must enable VDL/Karajan logging. Make sure
+log4.properties contains:
+----
+log4j.logger.swift=DEBUG
+log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Cpu=DEBUG
+----
+
+Make a basic load plot from Coasters Cpu log lines
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.Generate the log (may set log4j.logger.swift=INFO for this one)
+(assuming the log is titled swift-run.log)
+
+.Convert the log times to Unix time
+
+----
+./iso-to-secs < swift-run.log > swift-run.time
+----
+
+.Make the start time file (this contains the earliest timestamp)
+
+----
+make LOG=swift-run.log start-time.tmp
+----
+
+or
+
+----
+extract-start-time swift-run.log > start-time.tmp
+----
+
+.Normalize the transition times
+
+----
+./normalise-event-start-time < swift-run.time > swift-run.norm
+----
+
+.Build up a load data file:
+
+----
+./cpu-job-load.pl < swift-run.norm > load.data
+----
+
+.Plot with the JFreeChart-based plotter in usertools/plotter:
+
+----
+lines.zsh load.cfg load.eps load.data
+----
+
+Make a basic job completion plot from Coasters Cpu log lines
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.Same as above, but, build up a completed data file:
+
+----
+./cpu-job-completed.pl < swift-run.norm > completed.data
+----
+
+.Plot with the JFreeChart-based plotter in usertools/plotter:
+
+----
+lines.zsh completed.cfg completed.eps completed.data
+----
+
+Make a basic Block allocation plot from Coasters Block log lines
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Same as above, but:
+
+.Build up a block allocation data file:
+
+----
+./block-level.pl < swift-run.norm > blocks.data
+----
+
+.Plot with the JFreeChart-based plotter in usertools/plotter:
+
+----
+lines.zsh blocks.{cfg,eps,data}
+----
+
+Problem Reporting
+~~~~~~~~~~~~~~~~~
+When reporting problems to swift-user at ci.uchicago.edu, please attach the
+following files and information:
+
+. tc.data and sites.xml (or whatever you named these files)
+
+. your .swift source file and any .swift files it imports
+
+. any external mapper scripts called by your .swift script
+
+. all text from standard output and standard error produced by running the
+swift command
+
+. The .log file from this run. It will be named swiftscript.uniqueID.log
+
+. where "swiftscript" is the name of your *.swift script source file, and
+uniqueID is a long unique id which starts with the date and time you ran the
+swift command.
+
+. The swift command line you invoked
+
+. Any swift.properties entries you over-rode ($HOME/.swift/swift.properties,
+-config.file argument properties file, any changes to etc/swift.proerties from
+your swift distribution)
+
+. Which swift distribution you are running (release; svn revisions; other
+local changes you mave have made or included)
+
+
Added: wwwdev/cookbook/figures/_notes/dwsync.xml
===================================================================
--- wwwdev/cookbook/figures/_notes/dwsync.xml (rev 0)
+++ wwwdev/cookbook/figures/_notes/dwsync.xml 2011-08-01 14:23:26 UTC (rev 4896)
@@ -0,0 +1,4 @@
+
+
+
+
\ No newline at end of file
Added: wwwdev/cookbook/figures/coaster_setup.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/cookbook/figures/coaster_setup.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
From vytas at ci.uchicago.edu Mon Aug 1 09:23:42 2011
From: vytas at ci.uchicago.edu (vytas at ci.uchicago.edu)
Date: Mon, 01 Aug 2011 14:23:42 -0000
Subject: [Swift-commit] r4897 - in wwwdev: . css css/_notes
Message-ID: <20110801142338.B332E9CC9E@svn.ci.uchicago.edu>
Author: vytas
Date: 2011-08-01 09:23:38 -0500 (Mon, 01 Aug 2011)
New Revision: 4897
Added:
wwwdev/css/
wwwdev/css/960_12_col.css
wwwdev/css/_notes/
wwwdev/css/_notes/dwsync.xml
wwwdev/css/orbit-1.2.3.css
wwwdev/css/reset.css
wwwdev/css/style.css
wwwdev/css/style1col.css
wwwdev/css/style2.css
wwwdev/css/style3.css
Log:
new look & feel
Added: wwwdev/css/960_12_col.css
===================================================================
--- wwwdev/css/960_12_col.css (rev 0)
+++ wwwdev/css/960_12_col.css 2011-08-01 14:23:38 UTC (rev 4897)
@@ -0,0 +1,357 @@
+/*
+ 960 Grid System ~ Core CSS.
+ Learn more ~ http://960.gs/
+
+ Licensed under GPL and MIT.
+*/
+
+/*
+ Forces backgrounds to span full width,
+ even if there is horizontal scrolling.
+ Increase this if your layout is wider.
+
+ Note: IE6 works fine without this fix.
+*/
+
+body {
+ min-width: 960px;
+}
+
+/* `Container
+----------------------------------------------------------------------------------------------------*/
+
+.container_12 {
+ margin-left: auto;
+ margin-right: auto;
+ width: 960px;
+}
+
+/* `Grid >> Global
+----------------------------------------------------------------------------------------------------*/
+
+.grid_1,
+.grid_2,
+.grid_3,
+.grid_4,
+.grid_5,
+.grid_6,
+.grid_7,
+.grid_8,
+.grid_9,
+.grid_10,
+.grid_11,
+.grid_12 {
+ display: inline;
+ float: left;
+ margin-left: 10px;
+ margin-right: 10px;
+}
+
+.push_1, .pull_1,
+.push_2, .pull_2,
+.push_3, .pull_3,
+.push_4, .pull_4,
+.push_5, .pull_5,
+.push_6, .pull_6,
+.push_7, .pull_7,
+.push_8, .pull_8,
+.push_9, .pull_9,
+.push_10, .pull_10,
+.push_11, .pull_11 {
+ position: relative;
+}
+
+/* `Grid >> Children (Alpha ~ First, Omega ~ Last)
+----------------------------------------------------------------------------------------------------*/
+
+.alpha {
+ margin-left: 0;
+}
+
+.omega {
+ margin-right: 0;
+}
+
+/* `Grid >> 12 Columns
+----------------------------------------------------------------------------------------------------*/
+
+.container_12 .grid_1 {
+ width: 60px;
+}
+
+.container_12 .grid_2 {
+ width: 140px;
+}
+
+.container_12 .grid_3 {
+ width: 220px;
+}
+
+.container_12 .grid_4 {
+ width: 300px;
+}
+
+.container_12 .grid_5 {
+ width: 380px;
+}
+
+.container_12 .grid_6 {
+ width: 460px;
+}
+
+.container_12 .grid_7 {
+ width: 540px;
+}
+
+.container_12 .grid_8 {
+ width: 620px;
+}
+
+.container_12 .grid_9 {
+ width: 700px;
+}
+
+.container_12 .grid_10 {
+ width: 780px;
+}
+
+.container_12 .grid_11 {
+ width: 860px;
+}
+
+.container_12 .grid_12 {
+ width: 940px;
+}
+
+/* `Prefix Extra Space >> 12 Columns
+----------------------------------------------------------------------------------------------------*/
+
+.container_12 .prefix_1 {
+ padding-left: 80px;
+}
+
+.container_12 .prefix_2 {
+ padding-left: 160px;
+}
+
+.container_12 .prefix_3 {
+ padding-left: 240px;
+}
+
+.container_12 .prefix_4 {
+ padding-left: 320px;
+}
+
+.container_12 .prefix_5 {
+ padding-left: 400px;
+}
+
+.container_12 .prefix_6 {
+ padding-left: 480px;
+}
+
+.container_12 .prefix_7 {
+ padding-left: 560px;
+}
+
+.container_12 .prefix_8 {
+ padding-left: 640px;
+}
+
+.container_12 .prefix_9 {
+ padding-left: 720px;
+}
+
+.container_12 .prefix_10 {
+ padding-left: 800px;
+}
+
+.container_12 .prefix_11 {
+ padding-left: 880px;
+}
+
+/* `Suffix Extra Space >> 12 Columns
+----------------------------------------------------------------------------------------------------*/
+
+.container_12 .suffix_1 {
+ padding-right: 80px;
+}
+
+.container_12 .suffix_2 {
+ padding-right: 160px;
+}
+
+.container_12 .suffix_3 {
+ padding-right: 240px;
+}
+
+.container_12 .suffix_4 {
+ padding-right: 320px;
+}
+
+.container_12 .suffix_5 {
+ padding-right: 400px;
+}
+
+.container_12 .suffix_6 {
+ padding-right: 480px;
+}
+
+.container_12 .suffix_7 {
+ padding-right: 560px;
+}
+
+.container_12 .suffix_8 {
+ padding-right: 640px;
+}
+
+.container_12 .suffix_9 {
+ padding-right: 720px;
+}
+
+.container_12 .suffix_10 {
+ padding-right: 800px;
+}
+
+.container_12 .suffix_11 {
+ padding-right: 880px;
+}
+
+/* `Push Space >> 12 Columns
+----------------------------------------------------------------------------------------------------*/
+
+.container_12 .push_1 {
+ left: 80px;
+}
+
+.container_12 .push_2 {
+ left: 160px;
+}
+
+.container_12 .push_3 {
+ left: 240px;
+}
+
+.container_12 .push_4 {
+ left: 320px;
+}
+
+.container_12 .push_5 {
+ left: 400px;
+}
+
+.container_12 .push_6 {
+ left: 480px;
+}
+
+.container_12 .push_7 {
+ left: 560px;
+}
+
+.container_12 .push_8 {
+ left: 640px;
+}
+
+.container_12 .push_9 {
+ left: 720px;
+}
+
+.container_12 .push_10 {
+ left: 800px;
+}
+
+.container_12 .push_11 {
+ left: 880px;
+}
+
+/* `Pull Space >> 12 Columns
+----------------------------------------------------------------------------------------------------*/
+
+.container_12 .pull_1 {
+ left: -80px;
+}
+
+.container_12 .pull_2 {
+ left: -160px;
+}
+
+.container_12 .pull_3 {
+ left: -240px;
+}
+
+.container_12 .pull_4 {
+ left: -320px;
+}
+
+.container_12 .pull_5 {
+ left: -400px;
+}
+
+.container_12 .pull_6 {
+ left: -480px;
+}
+
+.container_12 .pull_7 {
+ left: -560px;
+}
+
+.container_12 .pull_8 {
+ left: -640px;
+}
+
+.container_12 .pull_9 {
+ left: -720px;
+}
+
+.container_12 .pull_10 {
+ left: -800px;
+}
+
+.container_12 .pull_11 {
+ left: -880px;
+}
+
+/* `Clear Floated Elements
+----------------------------------------------------------------------------------------------------*/
+
+/* http://sonspring.com/journal/clearing-floats */
+
+.clear {
+ clear: both;
+ display: block;
+ overflow: hidden;
+ visibility: hidden;
+ width: 0;
+ height: 0;
+}
+
+/* http://www.yuiblog.com/blog/2010/09/27/clearfix-reloaded-overflowhidden-demystified */
+
+.clearfix:before,
+.clearfix:after,
+.container_12:before,
+.container_12:after {
+ content: '.';
+ display: block;
+ overflow: hidden;
+ visibility: hidden;
+ font-size: 0;
+ line-height: 0;
+ width: 0;
+ height: 0;
+}
+
+.clearfix:after,
+.container_12:after {
+ clear: both;
+}
+
+/*
+ The following zoom:1 rule is specifically for IE6 + IE7.
+ Move to separate stylesheet if invalid CSS is a problem.
+*/
+
+.clearfix,
+.container_12 {
+ zoom: 1;
+}
\ No newline at end of file
Property changes on: wwwdev/css/960_12_col.css
___________________________________________________________________
Added: svn:executable
+
Added: wwwdev/css/_notes/dwsync.xml
===================================================================
--- wwwdev/css/_notes/dwsync.xml (rev 0)
+++ wwwdev/css/_notes/dwsync.xml 2011-08-01 14:23:38 UTC (rev 4897)
@@ -0,0 +1,7 @@
+
+
+
+
+
+
+
\ No newline at end of file
Added: wwwdev/css/orbit-1.2.3.css
===================================================================
--- wwwdev/css/orbit-1.2.3.css (rev 0)
+++ wwwdev/css/orbit-1.2.3.css 2011-08-01 14:23:38 UTC (rev 4897)
@@ -0,0 +1,208 @@
+/* CSS for jQuery Orbit Plugin 1.2.3
+ * www.ZURB.com/playground
+ * Copyright 2010, ZURB
+ * Free to use under the MIT license.
+ * http://www.opensource.org/licenses/mit-license.php */
+
+.container {
+ padding-left:20px;
+ padding-bottom: 40px;
+}
+
+
+
+/* PUT IN YOUR SLIDER ID AND SIZE TO MAKE LOAD BEAUTIFULLY
+ ================================================== */
+#featured {
+ width: 674px;
+ height: 260px;
+ background: #f8f8f8 url('../images/orbit/spinner.gif') no-repeat center center;
+ overflow: hidden;
+ border: 1px solid #eaeaea;
+ }
+#featured>img,
+#featured>div,
+#featured>a { display: none; }
+
+
+
+
+/* CONTAINER
+ ================================================== */
+
+div.orbit-wrapper {
+ width: 1px;
+ height: 1px;
+ position: relative; }
+
+div.orbit {
+ width: 1px;
+ height: 1px;
+ position: relative;
+ overflow: hidden }
+
+div.orbit>img {
+ position: absolute;
+ top: 0;
+ left: 0;
+ display: none; }
+
+div.orbit>a {
+ border: none;
+ position: absolute;
+ top: 0;
+ left: 0;
+ line-height: 0;
+ display: none; }
+
+.orbit>div {
+ position: absolute;
+ top: 0;
+ left: 0;
+ width: 100%;
+ height: 100%; }
+
+/* Note: If your slider only uses content or anchors, you're going to want to put the width and height declarations on the ".orbit>div" and "div.orbit>a" tags in addition to just the .orbit-wrapper */
+
+
+/* TIMER
+ ================================================== */
+
+div.timer {
+ width: 40px;
+ height: 40px;
+ overflow: hidden;
+ position: absolute;
+ top: 10px;
+ right: 10px;
+ opacity: .6;
+ cursor: pointer;
+ z-index: 1001; }
+
+span.rotator {
+ display: block;
+ width: 40px;
+ height: 40px;
+ position: absolute;
+ top: 0;
+ left: -20px;
+ background: url(../images/orbit/rotator-black.png) no-repeat;
+ z-index: 3; }
+
+span.mask {
+ display: block;
+ width: 20px;
+ height: 40px;
+ position: absolute;
+ top: 0;
+ right: 0;
+ z-index: 2;
+ overflow: hidden; }
+
+span.rotator.move {
+ left: 0 }
+
+span.mask.move {
+ width: 40px;
+ left: 0;
+ background: url(../images/orbit/timer-black.png) repeat 0 0; }
+
+span.pause {
+ display: block;
+ width: 40px;
+ height: 40px;
+ position: absolute;
+ top: 0;
+ left: 0;
+ background: url(../images/orbit/pause-black.png) no-repeat;
+ z-index: 4;
+ opacity: 0; }
+
+span.pause.active {
+ background: url(../images/orbit/pause-black.png) no-repeat 0 -40px }
+
+div.timer:hover span.pause,
+span.pause.active {
+ opacity: 1 }
+
+
+/* CAPTIONS
+ ================================================== */
+
+.orbit-caption {
+ display: none;
+ font-family: "HelveticaNeue", "Helvetica-Neue", Helvetica, Arial, sans-serif; }
+
+.orbit-wrapper .orbit-caption {
+ background: #000;
+ background: rgba(0,0,0,.6);
+ z-index: 1000;
+ color: #fff;
+ text-align: center;
+ padding: 7px 0;
+ font-size: 13px;
+ position: absolute;
+ right: 0;
+ bottom: 0;
+ width: 100%; }
+
+
+/* DIRECTIONAL NAV
+ ================================================== */
+
+div.slider-nav {
+ display: block }
+
+div.slider-nav span {
+ width: 32px;
+ height: 32px;
+ text-indent: -9999px;
+ position: absolute;
+ z-index: 1000;
+ top: 50%;
+ margin-top: -16px;
+ cursor: pointer; }
+
+div.slider-nav span.right {
+ background: url(../images/orbit/right-arrow.png);
+ right: -15px; }
+
+div.slider-nav span.left {
+ background: url(../images/orbit/left-arrow.png);
+ left: -15px; }
+
+/* BULLET NAV
+ ================================================== */
+
+.orbit-bullets {
+ position: absolute;
+ z-index: 1000;
+ list-style: none;
+ bottom: -30px;
+ left: 50%;
+ margin-left: -30px;
+ padding: 0; }
+
+.orbit-bullets li {
+ float: left;
+ margin-left: 5px;
+ cursor: pointer;
+ color: #999;
+ text-indent: -9999px;
+ background: url(../images/orbit/bullets.jpg) no-repeat 4px 0;
+ width: 13px;
+ height: 12px;
+ overflow: hidden; }
+
+.orbit-bullets li.active {
+ color: #222;
+ background-position: -8px 0; }
+
+.orbit-bullets li.has-thumb {
+ background: none;
+ width: 100px;
+ height: 75px; }
+
+.orbit-bullets li.active.has-thumb {
+ background-position: 0 0;
+ border-top: 2px solid #000; }
\ No newline at end of file
Added: wwwdev/css/reset.css
===================================================================
--- wwwdev/css/reset.css (rev 0)
+++ wwwdev/css/reset.css 2011-08-01 14:23:38 UTC (rev 4897)
@@ -0,0 +1,202 @@
+/* `XHTML, HTML4, HTML5 Reset
+----------------------------------------------------------------------------------------------------*/
+
+a,
+abbr,
+acronym,
+address,
+applet,
+article,
+aside,
+audio,
+b,
+big,
+blockquote,
+body,
+canvas,
+caption,
+center,
+cite,
+code,
+dd,
+del,
+details,
+dfn,
+dialog,
+div,
+dl,
+dt,
+em,
+embed,
+fieldset,
+figcaption,
+figure,
+font,
+footer,
+form,
+h1,
+h2,
+h3,
+h4,
+h5,
+h6,
+header,
+hgroup,
+hr,
+html,
+i,
+iframe,
+img,
+ins,
+kbd,
+label,
+legend,
+li,
+mark,
+menu,
+meter,
+nav,
+object,
+ol,
+output,
+p,
+pre,
+progress,
+q,
+rp,
+rt,
+ruby,
+s,
+samp,
+section,
+small,
+span,
+strike,
+strong,
+sub,
+summary,
+sup,
+table,
+tbody,
+td,
+tfoot,
+th,
+thead,
+time,
+tr,
+tt,
+u,
+ul,
+var,
+video,
+xmp {
+ border: 0;
+ margin: 0;
+ padding: 0;
+ font-size: 100%;
+}
+
+html,
+body {
+ height: 100%;
+}
+
+article,
+aside,
+details,
+figcaption,
+figure,
+footer,
+header,
+hgroup,
+menu,
+nav,
+section {
+/*
+ Override the default (display: inline) for
+ browsers that do not recognize HTML5 tags.
+
+ IE8 (and lower) requires a shiv:
+ http://ejohn.org/blog/html5-shiv
+*/
+ display: block;
+}
+
+b,
+strong {
+/*
+ Makes browsers agree.
+ IE + Opera = font-weight: bold.
+ Gecko + WebKit = font-weight: bolder.
+*/
+ font-weight: bold;
+}
+
+img {
+ color: transparent;
+ font-size: 0;
+ vertical-align: middle;
+/*
+ For IE.
+ http://css-tricks.com/ie-fix-bicubic-scaling-for-images
+*/
+ -ms-interpolation-mode: bicubic;
+}
+
+li {
+/*
+ For IE6 + IE7.
+*/
+ display: list-item;
+}
+
+table {
+ border-collapse: collapse;
+ border-spacing: 0;
+}
+
+th,
+td,
+caption {
+ font-weight: normal;
+ vertical-align: top;
+ text-align: left;
+}
+
+q {
+ quotes: none;
+}
+
+q:before,
+q:after {
+ content: '';
+ content: none;
+}
+
+sub,
+sup,
+small {
+ font-size: 75%;
+}
+
+sub,
+sup {
+ line-height: 0;
+ position: relative;
+ vertical-align: baseline;
+}
+
+sub {
+ bottom: -0.25em;
+}
+
+sup {
+ top: -0.5em;
+}
+
+svg {
+/*
+ For IE9.
+*/
+ overflow: hidden;
+}
\ No newline at end of file
Property changes on: wwwdev/css/reset.css
___________________________________________________________________
Added: svn:executable
+
Added: wwwdev/css/style.css
===================================================================
--- wwwdev/css/style.css (rev 0)
+++ wwwdev/css/style.css 2011-08-01 14:23:38 UTC (rev 4897)
@@ -0,0 +1,319 @@
+#header {
+ background-color: #172B17;
+ /*background-image: url(../images/graphics/h_backgrnd.gif);*/
+ background-image: url(../images/graphics/h_backgrnd2.gif);
+ background-repeat: repeat-x;
+ width: 100%;
+}
+body {
+ margin: 0px auto;
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ background: #fff no-repeat center top;
+}
+
+body#home a.home,
+body#downloads a.downloads,
+body#about a.about,
+body#docs a.docs,
+body#apps a.apps,
+body#support a.support{
+ background-color: #666;
+ color:#FFFFFF;
+}
+
+
+a:link {
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ font-weight: normal;
+ color: #3366cc;
+ text-decoration: none;
+}
+a:visited {
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ font-weight: normal;
+ color: #3366cc;
+ text-decoration: none;
+}
+a:hover {
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ font-weight: normal;
+ color: #0099cc;
+ text-decoration: underline;
+}
+/************* MENU ************/
+
+#nav {
+font-family: Arial, Helvetica, sans-serif;
+ width: 100%;
+ font-size: 12px;
+ font-weight: normal;
+ margin: 0px auto;
+ padding: 0px 0px 0px 0px;
+ background-color: #000000;
+ border-bottom: #ccc 2px solid;
+}
+
+#nav ul {
+width: 850px;
+ margin: 0px;
+ padding: 0px;
+ list-style: none;
+ text-align: center;
+ letter-spacing: 1px;
+}
+
+#nav li {
+ display: inline;
+}
+
+#nav a {
+font-size: 12px;
+ padding-right: 6px;
+ padding-left: 6px;
+ text-transform: uppercase;
+ text-decoration: none;
+ color: #FFFFFF;
+}
+
+#nav a:hover {
+ background-color: #666;
+}
+#content {
+ width: 100%;
+ margin: 0px 0px 40px 0px;
+ padding: 20px 0px 0px 0px;
+ border-top: thin #000 solid;
+ color: #333;
+}
+.image {
+ background-color: #eee;
+ margin:10px auto;
+ padding:3px;
+ border: 1px solid #ccc;
+ width: 419px;
+
+}
+.highlight{
+ border: 1px #ccc solid;
+ padding:5px;
+ background-color: #FCFAFA;
+ font-size: 11px;
+ margin-left:10px;
+}
+
+.mailinglists { width: 85%; margin: 0 0 0 35px; font-size:10px; }
+.mailinglists td { padding: 5px 5px; line-height: 16px;}
+.mailinglists th { font-size: 12px; text-align: left; color:#000; padding: 7px; letter-spacing: 1px; font-family:"Trebuchet MS", Arial, sans-serif;}
+
+#left {
+ float: left;
+ width: 65%;
+ padding: 0px 10px 70px 5px;
+ border-right: 1px #ccc dashed;
+ margin-bottom: 20px;
+ margin-top: 10px;
+}
+
+#right {
+ float: left;
+ width: 250px;
+ color: #FFF;
+ margin: 25px 15px 70px 5px;
+/* border: 1px #ddd solid;
+ background-color: #FCFAFA;*/
+ padding: 5px;
+
+}
+#right h3 {
+ margin: 0px;
+ font-size: 13px;
+ font-weight: bold;
+ color: #cc0000;
+ letter-spacing: 1px;
+ padding-top: 5px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 0px;
+ font-family:"Trebuchet MS", Arial, sans-serif;
+ text-transform: uppercase;
+
+}
+#right h4 {
+ margin: 0px;
+ font-size: 12px;
+ font-weight: bold;
+ color: #333;
+ letter-spacing: 1px;
+ padding-top: 10px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 5px;
+ font-family:"Trebuchet MS", Arial, sans-serif;
+
+}
+#right p{
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 11px;
+ line-height: 15px;
+ color: #333333;
+ padding: 0px 5px 0px 5px;
+ margin: 0px;
+}
+#right a:link {
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 11px;
+ font-weight: normal;
+ color: #3366cc;
+ text-decoration: none;
+}
+#right a:visited {
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 11px;
+ font-weight: normal;
+ color: #3366cc;
+ text-decoration: none;
+}
+ #right a:hover {
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 11px;
+ font-weight: normal;
+ color: #0099cc;
+ text-decoration: underline;
+}
+
+.featurebox_side {
+ background-color: #eee;
+ margin-left:5px;
+ margin-right:5px;
+ margin-bottom:5px;
+ padding:4px;
+ border: 1px dotted #4E8787;
+ font-family: verdana, arial, sans-serif;
+ font-size: 11px;
+ line-height: 16px;
+ color: #333333;
+}
+
+#left p{
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ line-height: 17px;
+ color: #333333;
+ padding: 0px 30px 0px 30px;
+
+ margin: 0px 0px 8px 0px;
+}
+
+#left ul{
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ line-height: 17px;
+}
+
+#left li{
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ line-height: 17px;
+ list-style:circle;
+ margin: 10px 10px 10px 35px;
+}
+
+h1 {
+ margin: 0px;
+ font-size: 16px;
+ font-weight: bold;
+ color: #172B17;
+ letter-spacing: 1px;
+ padding-top: 0px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 10px;
+ font-family:"Trebuchet MS", Arial, sans-serif;
+
+}
+h2 {
+ margin: 0px;
+ font-size: 14px;
+ font-weight: bold;
+ color: #CC0000;
+ letter-spacing: 1px;
+ padding-top: 5px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 20px;
+ font-family:"Trebuchet MS", Arial, sans-serif;
+
+}
+h3 {
+ margin: 0px;
+ font-size: 13px;
+ font-weight: bold;
+ color: #333;
+ letter-spacing: 1px;
+ padding-top: 5px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 20px;
+ font-family:"Trebuchet MS", Arial, sans-serif;
+
+}
+h4 {
+ margin: 0px;
+ font-size: 12px;
+ font-weight: bold;
+ color: #666;
+ letter-spacing: 1px;
+ padding-top: 5px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 20px;
+ font-family:"Trebuchet MS", Arial, sans-serif;
+ text-transform: uppercase;
+
+}
+#container {
+ background-color: #FFFFFF;
+ font-size: 10px;
+ font-weight: normal;
+ /*border: 1px #ccc solid;*/
+ width: 100%;
+ background-repeat: no-repeat;
+ background-position: center;
+ margin-top: 3px;
+ margin-right: auto;
+ margin-bottom: 0px;
+ margin-left: 3px;
+}
+/************* FOOTER ************/
+
+#footer {
+ clear: both;
+ width: 850;
+ margin: 0px;
+ border-top: 1px solid #ddd;
+ border-bottom: 25px #FFF solid;
+ background-color: #FFF;
+ text-align: center;
+ padding: 3px;
+ color: #999;
+}
+
+#footer a:link {
+ font-size: 9px;
+ font-weight: normal;
+ color: #3366cc;
+ text-decoration: none;
+}
+#footer a:visited {
+ font-size: 9px;
+ color: #3366cc;
+ text-decoration: none;
+}
+ #footer a:hover {
+ font-size: 9px;
+ color: #0099cc;
+ text-decoration: underline;
+}
Added: wwwdev/css/style1col.css
===================================================================
--- wwwdev/css/style1col.css (rev 0)
+++ wwwdev/css/style1col.css 2011-08-01 14:23:38 UTC (rev 4897)
@@ -0,0 +1,310 @@
+#header {
+ background-color: #172B17;
+ /*background-image: url(../images/graphics/h_backgrnd.gif);*/
+ background-image: url(../images/graphics/h_backgrnd2.gif);
+ background-repeat: repeat-x;
+ width: 100%;
+ background-position: center;
+}
+body {
+ margin: 0px auto;
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ background: #fff no-repeat center top;
+}
+
+body#home a.home,
+body#downloads a.downloads,
+body#about a.about,
+body#docs a.docs,
+body#apps a.apps,
+body#support a.support{
+ background-color: #666;
+ color:#FFFFFF;
+}
+
+
+a:link {
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ font-weight: normal;
+ color: #3366cc;
+ text-decoration: none;
+}
+a:visited {
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ font-weight: normal;
+ color: #3366cc;
+ text-decoration: none;
+}
+a:hover {
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ font-weight: normal;
+ color: #0099cc;
+ text-decoration: underline;
+}
+
+pre, pre *, div.dp-highlighter * {
+ font-family: Consolas, Andale Mono, monospaced;
+}
+
+/************* MENU ************/
+
+#nav {
+font-family: Arial, Helvetica, sans-serif;
+ width: 100%;
+ font-size: 12px;
+ font-weight: normal;
+ margin: 0px auto;
+ padding: 0px 0px 0px 0px;
+ background-color: #000000;
+ border-bottom: #ccc 2px solid;
+}
+
+#nav ul {
+width: 100%s;
+ margin: 0px;
+ padding: 0px;
+ list-style: none;
+ text-align: center;
+ letter-spacing: 1px;
+}
+
+#nav li {
+ display: inline;
+}
+
+#nav a {
+font-size: 12px;
+ padding-right: 6px;
+ padding-left: 6px;
+ text-transform: uppercase;
+ text-decoration: none;
+ color: #FFFFFF;
+}
+
+#nav a:hover {
+ background-color: #666;
+}
+#page {
+ width: 850px;
+ margin: 0px 0px 40px 0px;
+ padding: 20px 0px 0px 0px;
+ border-top: thin #000 solid;
+ color: #333;
+}
+.image {
+ background-color: #eee;
+ margin:10px auto;
+ padding:3px;
+ border: 1px solid #ccc;
+ width: 419px;
+
+}
+.highlight{
+ border: 1px #ccc solid;
+ padding:5px;
+ background-color: #FCFAFA;
+ font-size: 11px;
+ margin-left:10px;
+}
+
+pre.programlisting {
+ padding: 10pt;
+ background: #FCFAFA;
+ border: 1px #ccc solid;
+ white-space: normal;
+}
+
+pre.shell, pre.screen {
+ padding: 10pt;
+ background: #FCFAFA;
+ border: 1px #ccc solid;
+ white-space: pre;
+}
+
+.mailinglists { width: 85%; margin: 0 0 0 35px; font-size:10px; }
+.mailinglists td { padding: 5px 5px; line-height: 16px;}
+.mailinglists th { font-size: 12px; text-align: left; color:#000; padding: 7px; letter-spacing: 1px; font-family:"Trebuchet MS", Arial, sans-serif;}
+
+#content {
+ float: left;
+ width: 100%;
+ padding: 0px 10px 70px 5px;
+ margin-bottom: 20px;
+ margin-top: 10px;
+}
+
+
+#content p{
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ line-height: 17px;
+ color: #333333;
+ padding: 5px 30px 0px 30px;
+
+ margin: 0px 0px 8px 0px;
+}
+
+#content ul{
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ line-height: 17px;
+}
+
+#content li{
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ line-height: 17px;
+ list-style:circle;
+ margin: 10px 10px 10px 35px;
+}
+
+h1 {
+ margin: 0px;
+ font-size: 16px;
+ font-weight: bold;
+ color: #172B17;
+ letter-spacing: 1px;
+ padding-top: 0px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 10px;
+ font-family:"Trebuchet MS", Arial, sans-serif;
+
+}
+h2 {
+ margin: 0px;
+ font-size: 14px;
+ font-weight: bold;
+ color: #CC0000;
+ letter-spacing: 1px;
+ padding-top: 5px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 20px;
+ font-family:"Trebuchet MS", Arial, sans-serif;
+
+}
+h3 {
+ margin: 0px;
+ font-size: 13px;
+ font-weight: bold;
+ color: #333;
+ letter-spacing: 1px;
+ padding-top: 5px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 20px;
+ font-family:"Trebuchet MS", Arial, sans-serif;
+
+}
+h4 {
+ margin: 0px;
+ font-size: 12px;
+ font-weight: bold;
+ color: #666;
+ letter-spacing: 1px;
+ padding-top: 5px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 20px;
+ font-family:"Trebuchet MS", Arial, sans-serif;
+ text-transform: uppercase;
+
+}
+h5 {
+ margin: 0px;
+ font-size: 12px;
+ font-weight: bold;
+ color: #666;
+ letter-spacing: 1px;
+ padding-top: 5px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 20px;
+ font-family:"Trebuchet MS", Arial, sans-serif;
+ text-transform: uppercase;
+
+}
+#container {
+ background-color: #FFFFFF;
+ font-size: 10px;
+ font-weight: normal;
+ border: 1px #ccc solid;
+ width: 850px;
+ background-repeat: no-repeat;
+ background-position: center;
+ margin-top: 3px;
+ margin-right: auto;
+ margin-bottom: 0px;
+ margin-left: auto;
+}
+/************* FOOTER ************/
+
+#footer {
+ clear: both;
+ width: 100%;
+ margin: 0px;
+ border-top: 1px solid #ddd;
+ border-bottom: 25px #FFF solid;
+ background-color: #FFF;
+ text-align: center;
+ padding: 0px;
+ color: #999;
+}
+
+#footer a:link {
+ font-size: 9px;
+ font-weight: normal;
+ color: #3366cc;
+ text-decoration: none;
+}
+#footer a:visited {
+ font-size: 9px;
+ color: #3366cc;
+ text-decoration: none;
+}
+ #footer a:hover {
+ font-size: 9px;
+ color: #0099cc;
+ text-decoration: underline;
+}
+
+/********* DOCBOOK STUFF ********/
+
+.article {
+ margin: 3px 20px;
+ font-size: 10px;
+}
+
+.article p {
+ margin: 2px 25px;
+}
+
+/* Syntax highlighting stuff */
+
+div.dp-highlighter {
+ background: #FCFAFA;
+ border: 1px #ccc solid;
+ white-space: normal;
+ margin: 10px;
+}
+
+span.comment {
+ color: #505050;
+}
+
+span.keyword {
+ color: #202080;
+ font-weight: bold;
+}
+
+span.string {
+ color: #209020;
+}
+
+span.function {
+ color: #800040;
+}
Added: wwwdev/css/style2.css
===================================================================
--- wwwdev/css/style2.css (rev 0)
+++ wwwdev/css/style2.css 2011-08-01 14:23:38 UTC (rev 4897)
@@ -0,0 +1,404 @@
+#header {
+ background-color: #172B17;
+ /*background-image: url(../images/graphics/h_backgrnd.gif);*/
+ background-image: url(../images/graphics/h_backgrnd2.gif);
+ background-repeat: repeat-x;
+ width: 100%;
+ background-position: center;
+}
+body {
+ margin: 0px auto;
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ background: #fff no-repeat center top;
+}
+
+body#home a.home,
+body#downloads a.downloads,
+body#about a.about,
+body#docs a.docs,
+body#papers a.papers,
+body#apps a.apps,
+body#support a.support{
+ background-color: #666;
+ color:#FFFFFF;
+}
+
+
+a:link {
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ font-weight: normal;
+ color: #3366cc;
+ text-decoration: none;
+}
+a:visited {
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ font-weight: normal;
+ color: #3366cc;
+ text-decoration: none;
+}
+a:hover {
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ font-weight: normal;
+ color: #0099cc;
+ text-decoration: underline;
+}
+
+pre, pre *, div.dp-highlighter * {
+ font-family: Consolas, Andale Mono, monospaced;
+}
+
+/************* MENU ************/
+
+#nav {
+font-family: Arial, Helvetica, sans-serif;
+ width: 100%;
+ font-size: 12px;
+ font-weight: normal;
+ margin: 0px auto;
+ padding: 0px 0px 0px 0px;
+ background-color: #000000;
+ border-bottom: #ccc 2px solid;
+}
+
+#nav ul {
+ width: 100%s;
+ margin: 0px;
+ padding: 0px;
+ list-style: none;
+ text-align: center;
+ letter-spacing: 1px;
+}
+
+#nav li {
+ display: inline;
+}
+
+#nav a {
+font-size: 12px;
+ padding-right: 6px;
+ padding-left: 6px;
+ text-transform: uppercase;
+ text-decoration: none;
+ color: #FFFFFF;
+}
+
+#nav a:hover {
+ background-color: #666;
+}
+
+#content {
+ width: 100%;
+ margin: 0px 0px 40px 0px;
+ padding: 20px 0px 0px 0px;
+ border-top: thin #000 solid;
+ color: #333;
+}
+
+.image {
+ background-color: #eee;
+ margin: 3px auto;
+ padding: 3px;
+ border: 1px solid #ccc;
+ width: 525px;
+
+}
+
+.caption{
+ border: 1px #ccc solid;
+ padding:3px;
+ background-color: #FCFAFA;
+ font-size: 11px;
+ margin: 3px auto;
+ width: 525px;
+}
+
+
+.highlight{
+ border: 1px #ccc solid;
+ padding:5px;
+ background-color: #FCFAFA;
+ font-size: 11px;
+ margin-left: 10px;
+ white-space: pre;
+ position: relative;
+ left: 28px;
+ width: 90%;
+}
+
+pre.shell, pre.programlisting {
+ padding: 10pt;
+ background: #FCFAFA;
+ border: 1px #ccc solid;
+ white-space: normal;
+}
+
+div.dp-highlighter {
+ background: #FCFAFA;
+ border: 1px #ccc solid;
+ white-space: normal;
+ margin: 10px;
+}
+
+
+.mailinglists {
+ width: 85%;
+ margin: 0 0 0 35px;
+ font-size:10px;
+}
+
+.mailinglists td {
+ padding: 5px 5px;
+ line-height: 16px;
+ background-color: #f0f0f0;
+}
+
+.mailinglists th {
+ font-size: 12px;
+ text-align: left;
+ color:#000;
+ padding: 7px;
+ letter-spacing: 1px;
+ font-family:"Trebuchet MS", Arial, sans-serif;
+ background-color: #e0e0e0;
+}
+
+#left {
+ float: left;
+ width: 70%;
+ padding: 0px 10px 70px 5px;
+ border-right: 1px #ccc dashed;
+ margin-bottom: 20px;
+ margin-top: 10px;
+}
+
+#right {
+ float: left;
+ width: 185px;
+ color: #FFF;
+ margin: 30px 2px 70px 5px;
+/* border: 1px #ddd solid;
+ background-color: #FCFAFA;*/
+ padding: 5px;
+
+}
+#right h3 {
+ margin: 0px;
+ font-size: 13px;
+ font-weight: bold;
+ color: #cc0000;
+ letter-spacing: 1px;
+ padding-top: 5px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 0px;
+ font-family:"Trebuchet MS", Arial, sans-serif;
+ text-transform: uppercase;
+
+}
+#right h4 {
+ margin: 0px;
+ font-size: 12px;
+ font-weight: bold;
+ color: #333;
+ letter-spacing: 1px;
+ padding-top: 10px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 5px;
+ font-family:"Trebuchet MS", Arial, sans-serif;
+
+}
+#right p{
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 11px;
+ line-height: 15px;
+ color: #333333;
+ padding: 0px 5px 0px 5px;
+ margin: 0px;
+}
+#right a:link {
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 11px;
+ font-weight: normal;
+ color: #3366cc;
+ text-decoration: none;
+}
+#right a:visited {
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 11px;
+ font-weight: normal;
+ color: #3366cc;
+ text-decoration: none;
+}
+ #right a:hover {
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 11px;
+ font-weight: normal;
+ color: #0099cc;
+ text-decoration: underline;
+}
+
+.featurebox_side {
+ background-color: #eee;
+ margin-left:5px;
+ margin-right:5px;
+ margin-bottom:5px;
+ padding:4px;
+ border: 1px dotted #4E8787;
+ font-family: verdana, arial, sans-serif;
+ font-size: 11px;
+ line-height: 16px;
+ color: #333333;
+}
+
+#left p{
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ line-height: 17px;
+ color: #333333;
+ padding: 5px 30px 0px 30px;
+
+ margin: 0px 0px 8px 0px;
+}
+
+#left ul{
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ line-height: 17px;
+}
+
+#left li{
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ line-height: 17px;
+ list-style:circle;
+ margin: 10px 10px 10px 35px;
+}
+
+h1 {
+ margin: 0px;
+ font-size: 16px;
+ font-weight: bold;
+ color: #172B17;
+ letter-spacing: 1px;
+ padding-top: 0px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 10px;
+ font-family:"Trebuchet MS", Arial, sans-serif;
+
+}
+h2 {
+ margin: 0px;
+ font-size: 14px;
+ font-weight: bold;
+ color: #CC0000;
+ letter-spacing: 1px;
+ padding-top: 5px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 20px;
+ font-family:"Trebuchet MS", Arial, sans-serif;
+
+}
+h3 {
+ margin: 0px;
+ font-size: 13px;
+ font-weight: bold;
+ color: #333;
+ letter-spacing: 1px;
+ padding-top: 5px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 20px;
+ font-family:"Trebuchet MS", Arial, sans-serif;
+
+}
+h4 {
+ margin: 0px;
+ font-size: 12px;
+ font-weight: bold;
+ color: #666;
+ letter-spacing: 1px;
+ padding-top: 5px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 20px;
+ font-family:"Trebuchet MS", Arial, sans-serif;
+ text-transform: uppercase;
+
+}
+#container {
+ background-color: #FFFFFF;
+ font-size: 10px;
+ font-weight: normal;
+ border: 1px #ccc solid;
+ width: 850px;
+ background-repeat: no-repeat;
+ background-position: center;
+ margin-top: 3px;
+ margin-right: auto;
+ margin-bottom: 0px;
+ margin-left: auto;
+}
+/************* FOOTER ************/
+
+#footer {
+ clear: both;
+ width: 850px;
+ margin: 0px;
+ border-top: 1px solid #ddd;
+ border-bottom: 25px #FFF solid;
+ background-color: #FFF;
+ text-align: center;
+ padding: 0px;
+ color: #999;
+}
+
+#footer a:link {
+ font-size: 9px;
+ font-weight: normal;
+ color: #3366cc;
+ text-decoration: none;
+}
+#footer a:visited {
+ font-size: 9px;
+ color: #3366cc;
+ text-decoration: none;
+}
+ #footer a:hover {
+ font-size: 9px;
+ color: #0099cc;
+ text-decoration: underline;
+}
+
+/* Syntax highlighting stuff */
+
+span.comment {
+ color: #505050;
+}
+
+span.keyword {
+ color: #202080;
+ font-weight: bold;
+}
+
+span.string {
+ color: #209020;
+}
+
+span.function {
+ color: #800040;
+}
+
+.publication {
+ padding-bottom: 9px;
+ padding-left: 30px;
+}
+
+.publication .source {
+ font-style: italic;
+}
Added: wwwdev/css/style3.css
===================================================================
--- wwwdev/css/style3.css (rev 0)
+++ wwwdev/css/style3.css 2011-08-01 14:23:38 UTC (rev 4897)
@@ -0,0 +1,589 @@
+#header {
+ width: 100%;
+ background-position: center;
+}
+body {
+ margin: 0px auto;
+ font-family: Arial, Helvetica, sans-serif;
+ color: #333;
+ background: #fff no-repeat center top;
+ background:url(../images/bg_page.png) repeat-x;
+ font-size:16px;
+ line-height: 1.3em;
+}
+
+body#home a.home,
+body#downloads a.downloads,
+body#about a.about,
+body#docs a.docs,
+body#papers a.papers,
+body#apps a.apps,
+body#support a.support{
+ background-color: #666;
+ color:#FFFFFF;
+}
+
+
+a:link {
+ font-family: myriad-pro-1,myriad-pro-2, Arial, Helvetica, sans-serif;
+ font-weight:400;
+ font-style: normal;
+ font-weight: normal;
+ color: #3366cc;
+ text-decoration: none;
+}
+a:visited {
+ font-family: myriad-pro-1,myriad-pro-2, Arial, Helvetica, sans-serif;
+ font-weight:400;
+ font-style: normal;
+ color: #3366cc;
+ text-decoration: none;
+}
+a:hover {
+ font-family: myriad-pro-1,myriad-pro-2, Arial, Helvetica, sans-serif;
+ font-weight:400;
+ font-style: normal;
+ color: #0099cc;
+ text-decoration: underline;
+}
+
+pre, pre *, div.dp-highlighter * {
+ font-family: Consolas, "Andale Mono", Monaco, monospaced;
+ padding-left:30px;
+ color:#699;
+}
+
+/************* MENU ************/
+
+#nav {
+font-family: Arial, Helvetica, sans-serif;
+ width: 100%;
+ font-size: 18px;
+ margin: 0px auto;
+ padding: 60px 0px 0px 0px;
+ color: #fff;
+}
+
+#nav ul {
+ width: 100%s;
+ margin: 0px;
+ padding: 0px;
+ list-style: none;
+ text-align: center;
+}
+
+#nav li {
+ display: inline;
+ border-right: 1px solid #0066cc;
+}
+
+#nav .last {
+ border-right: 0px;
+}
+
+#nav a {
+ font-size: 18px;
+ padding-right: 16px;
+ padding-left: 12px;
+ text-decoration: none;
+ color: #66ccff;
+ text-shadow: 0px 2px 3px #333;
+}
+
+#nav a:hover {
+ color: #fff;
+}
+
+.tagline {
+ font-family: myriad-pro-1,myriad-pro-2, Arial, Helvetica, sans-serif;
+ font-weight: 300;
+ font-style: normal;
+ text-align:center;
+ text-shadow: 1px 1px 1px #ccc;
+ padding: 30px 0px 30px 0px;
+ color:#333;
+ font-size: 20px;
+}
+
+#content {
+ width: 100%;
+ margin: 0px 0px 40px 0px;
+ padding: 20px 0px 0px 0px;
+ border-top: thin #000 solid;
+ color: #333;
+}
+
+p {
+ padding: 0px 0px 15px 20px;
+}
+
+.image {
+ background-color: #eee;
+ margin: 3px auto;
+ padding: 3px;
+ border: 1px solid #ccc;
+ width: 525px;
+
+}
+
+.caption{
+ border: 1px #ccc solid;
+ padding:3px;
+ background-color: #FCFAFA;
+ font-size: 11px;
+ margin: 3px auto;
+ width: 525px;
+}
+
+
+.highlight{
+ border: 1px #ccc solid;
+ padding: 5px;
+ background-color: #FCFAFA;
+ font-size: 0.9em;
+ margin-left: 10px;
+ white-space: pre;
+ position: relative;
+ left: 28px;
+ width: 90%;
+}
+
+pre.shell, pre.programlisting {
+ padding: 10pt;
+ background: #FCFAFA;
+ border: 1px #ccc solid;
+ white-space: normal;
+}
+
+div.dp-highlighter {
+ background: #FCFAFA;
+ border: 1px #ccc solid;
+ white-space: normal;
+ margin: 10px;
+}
+
+
+.mailinglists {
+ width: 85%;
+ margin: 0 0 0 35px;
+ font-size:10px;
+}
+
+.mailinglists td {
+ padding: 5px 5px;
+ line-height: 16px;
+ background-color: #f0f0f0;
+}
+
+.mailinglists th {
+ font-size: 12px;
+ text-align: left;
+ color:#333;
+ padding: 7px;
+ font-family: myriad-pro-1,myriad-pro-2, Arial, Helvetica, sans-serif;
+ font-weight:600;
+ font-style: normal;
+ background-color: #e0e0e0;
+}
+
+#left {
+ float: left;
+ width: 70%;
+ padding: 0px 10px 70px 5px;
+ border-right: 1px #ccc dashed;
+ margin-bottom: 20px;
+ margin-top: 10px;
+}
+
+#right {
+ float: left;
+ width: 185px;
+ color: #FFF;
+ margin: 30px 2px 70px 5px;
+/* border: 1px #ddd solid;
+ background-color: #FCFAFA;*/
+ padding: 5px;
+
+}
+#right h3 {
+ margin: 0px;
+ font-size: 1.1em;
+ color: #333;
+ padding-top: 5px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 0px;
+ font-family: myriad-pro-1,myriad-pro-2, Arial, Helvetica, sans-serif;
+ font-weight:600;
+ font-style: normal;
+
+}
+#right h4 {
+ margin: 0px;
+ font-size: 1.0em;
+ color: #333;
+ padding-top: 10px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 5px;
+ font-family: myriad-pro-1,myriad-pro-2, Arial, Helvetica, sans-serif;
+ font-weight:600;
+ font-style: normal;
+
+}
+#right p{
+ font-family: Arial, Helvetica, sans-serif;
+ font-size: 11px;
+ line-height: 15px;
+ color: #333333;
+ padding: 0px 5px 0px 5px;
+ margin: 0px;
+}
+#right a:link {
+ font-family: Arial, Helvetica, sans-serif;
+ font-size: 11px;
+ font-weight: normal;
+ color: #3366cc;
+ text-decoration: none;
+}
+#right a:visited {
+ font-family: Arial, Helvetica, sans-serif;
+ font-size: 11px;
+ font-weight: normal;
+ color: #3366cc;
+ text-decoration: none;
+}
+ #right a:hover {
+ font-family: Arial, Helvetica, sans-serif;
+ font-size: 11px;
+ font-weight: normal;
+ color: #0099cc;
+ text-decoration: underline;
+}
+
+.featurebox_side {
+ background-color: #eee;
+ margin-left:5px;
+ margin-right:5px;
+ margin-bottom:5px;
+ padding:4px;
+ border: 1px dotted #4E8787;
+ font-family: arial, Helvetica, sans-serif;
+ font-size: 11px;
+ line-height: 16px;
+ color: #333333;
+}
+
+#left p{
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ line-height: 17px;
+ color: #333333;
+ padding: 5px 30px 0px 30px;
+
+ margin: 0px 0px 8px 0px;
+}
+
+#left ul{
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ line-height: 17px;
+}
+
+#left li{
+ font-family: Verdana, Arial, Helvetica, sans-serif;
+ font-size: 12px;
+ line-height: 17px;
+ list-style:circle;
+ margin: 10px 10px 10px 35px;
+}
+
+h1 {
+ margin: 0px;
+ font-size: 2.0em;
+ color: #333;
+ padding-top: 0px;
+ padding-bottom: 10px;
+ font-family: myriad-pro-1,myriad-pro-2, Arial, Helvetica, sans-serif;
+ font-weight:400;
+ font-style: normal;
+ text-shadow: 1px 1px 1px #dedede;
+
+}
+h2 {
+ margin: 0px;
+ font-size: 1.4em;
+ color: #333;
+ padding:15px 0px 10px 0px;
+ font-family: myriad-pro-1,myriad-pro-2, Arial, Helvetica, sans-serif;
+ font-weight:600;
+ font-style: normal;
+ text-shadow: 1px 1px 1px #dedede;
+ text-transform:uppercase;
+}
+h3 {
+ margin: 0px;
+ font-size: 1.3em;
+ color: #333;
+ padding-bottom: 10px;
+ font-family: myriad-pro-1,myriad-pro-2, Arial, Helvetica, sans-serif;
+ font-weight:600;
+ font-style: normal;
+ text-shadow: 1px 1px 1px #dedede;
+}
+h4 {
+ margin: 0px;
+ font-size: 1.2em;
+ color: #333;
+ padding-top: 5px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 20px;
+ font-family: myriad-pro-1,myriad-pro-2, Arial, Helvetica, sans-serif;
+ font-weight:600;
+ font-style: normal;
+}
+h5 {
+ margin: 0px;
+ font-size: 1.0em;
+ color: #666;
+ padding-top: 5px;
+ padding-right: 5px;
+ padding-bottom: 5px;
+ padding-left: 20px;
+ font-family: myriad-pro-1,myriad-pro-2, Arial, Helvetica, sans-serif;
+ font-weight:600;
+ font-style: normal;
+ text-transform:uppercase;
+}
+#container {
+ background-color: #FFFFFF;
+ font-size: 10px;
+ font-weight: normal;
+ border: 1px #ccc solid;
+ width: 850px;
+ background-repeat: no-repeat;
+ background-position: center;
+ margin-top: 3px;
+ margin-right: auto;
+ margin-bottom: 0px;
+ margin-left: auto;
+}
+/************* FOOTER ************/
+
+#footer {
+ background-color: #f2f5f8;
+ text-align: center;
+ padding: 20px 20px 50px 20px;
+ color: #999;
+}
+#footer .footer-blurb {
+ width: 550px;
+ font-size: 0.8em;
+ padding-top: 20px;
+}
+
+.footer-blurb {
+ text-align: center;
+ margin-left: auto;
+ margin-right: auto;
+ line-height: 1.2em;
+}
+#footer a {
+ font-size: 1.0em;
+ padding-right: 12px;
+ padding-left: 10px;
+ text-decoration: none;
+ color: #66ccff;
+}
+
+#footer a:hover {
+ color: #66ccff;
+ text-decoration:underline;
+}
+
+#footer a:link {
+ font-weight: normal;
+ color: #3366cc;
+ text-decoration: none;
+}
+#footer a:visited {
+ color: #3366cc;
+ text-decoration: none;
+}
+#footer a:hover {
+ color: #3366cc;
+ text-decoration: underline;
+}
+#footer ul {
+ width: 100%s;
+ margin: 0px;
+ padding: 0px;
+ list-style: none;
+ text-align: center;
+}
+
+#footer li {
+ display: inline;
+ border-right: 1px solid #999;
+}
+
+#footer .last {
+ border-right: 0px;
+}
+
+/* Syntax highlighting stuff */
+
+span.comment {
+ color: #505050;
+}
+
+span.keyword {
+ color: #202080;
+ font-weight: bold;
+}
+
+span.string {
+ color: #209020;
+}
+
+span.function {
+ color: #800040;
+}
+
+
+#feature {
+ padding: 0px 0px 40px 0px;
+}
+.trailingLink {
+ display:block;
+ padding: 0px 0px 26px 52px;
+ height:24px;
+ line-height: 1.3em;
+ background:url(../images/bullet.png) 30px 1px no-repeat ;
+ float: left;
+ font-size: 1.0em;
+}
+
+.trailingLinkPDF {
+ display:block;
+ padding: 0px 0px 26px 54px;
+ height:24px;
+ line-height: 1.3em;
+ background:url(../images/bullet_pdf.png) 30px 1px no-repeat ;
+ float: left;
+ font-size: 1.0em;
+}
+
+
+ul {
+ list-style-type: none;
+ padding: 0px 0px 15px 10px;
+ margin-left: 0px;
+ line-height: 1.5em;
+}
+.swift-bullet li{
+ background: url(../images/bullet_swift.png) 0px 3px no-repeat;
+ padding-left:22px;
+}
+.indented {
+ padding-left: 20px;
+}
+.indented-half {
+ padding-left: 10px;
+}
+#box-prime {
+ text-align:center;
+ padding: 15px;
+ -moz-border-radius: 6px;
+ border-radius: 6px;
+ border: 1px solid #eaeaea;
+ font-size: 0.9em;
+ background: url(../images/bg_box_shaded.png) 0px 0px repeat-x;
+ -webkit-box-shadow: 0 1px 50px rgba(0,0,0,.1);
+ -moz-box-shadow: 0 1px 5px rgba(0,0,0,.1);
+ box-shadow: 0 1px 5px rgba(0,0,0,.1);
+ line-height: 1.2em;
+ margin-bottom: 15px;
+}
+.button_175 {
+ background: url(../images/button_bg_175_sp.png) 2px 0px no-repeat;
+ width: 180px;
+ height: 40px;
+ font-family: myriad-pro-1,myriad-pro-2, Arial, Helvetica, sans-serif;
+ font-weight:600;
+ font-style: normal;
+ font-size: 1.4em;
+ line-height: 1.8em;
+ color: #fff;
+ padding: 10px 0px 0px 0x;
+ margin-left: auto;
+ margin-right: auto;
+ margin-bottom: 8px
+}
+
+.button_175:hover {
+ background: url(../images/button_bg_175_sp.png) 2px -40px no-repeat;
+}
+.button_175 a {
+ color: #fff;
+ text-shadow: 0 0 2px rgba(56,115,183,.7);
+}
+.button_175 a:hover {
+ color: #fff;
+ text-decoration:none;
+ text-shadow: 0 0 3px rgba(52,103,165,.9);
+}
+.highlight-A {
+ padding: 10px 40px 15px 120px;
+ background: url(../images/icon_special_A.png) 60px 13px no-repeat;
+}
+.highlight-B {
+ padding: 10px 40px 25px 120px;
+ background: url(../images/icon_special_B.png) 60px 13px no-repeat;
+}
+#nav .current {
+ color: #fff;
+}
+#holder {
+ position:absolute;
+}
+#mover {
+ position:relative;
+ top: 92px;
+}
+.page-header {
+ padding:40px 0px 20px 0px;
+}
+#box-prime h2 {
+ text-align:left;
+ font-size: 1.5em;
+ padding: 0px 0px 10px 0px;
+}
+.authors {
+}
+.title{
+ font-weight:600;
+}
+.source{
+ font-style:italic;
+}
+.date{
+ color:#666;
+ font-size:0.8em;
+}
+.publication {
+ padding-bottom: 15px;
+ padding-left: 30px;
+}
+.quote {
+ padding: 20px 15px 20px 20px;
+ background: url(../images/bg_quotes.png) 0px 0px no-repeat;
+ font-style:italic;
+ font-size: 0.9em;
+}
+.quote-speaker {
+ padding-left: 30px;
+ font-size: 0.7em;
+ line-height: 1.2em;
+}
\ No newline at end of file
From vytas at ci.uchicago.edu Mon Aug 1 09:23:46 2011
From: vytas at ci.uchicago.edu (vytas at ci.uchicago.edu)
Date: Mon, 01 Aug 2011 14:23:46 -0000
Subject: [Swift-commit] r4898 - wwwdev
Message-ID: <20110801142345.CBA899CC9E@svn.ci.uchicago.edu>
Author: vytas
Date: 2011-08-01 09:23:45 -0500 (Mon, 01 Aug 2011)
New Revision: 4898
Added:
wwwdev/dhtml.js
Log:
new look & feel
Added: wwwdev/dhtml.js
===================================================================
--- wwwdev/dhtml.js (rev 0)
+++ wwwdev/dhtml.js 2011-08-01 14:23:45 UTC (rev 4898)
@@ -0,0 +1,40 @@
+function initjs() {
+ selectCurrent();
+ //decorateLinks();
+}
+
+function decorateLinks() {
+ for (i in document.links) {
+ link = document.links[i];
+ if (link.className == "menulink") {
+ ld = link.parentNode.innerHTML;
+ index = ld.indexOf(">");
+ link.parentNode.innerHTML = ld.substr(0, index) +
+ " onMouseOver=\"highlight(" + i + ");\"" +
+ " onMouseOut=\"unhighlight(" + i + ");\"" +
+ ld.substr(index);
+ }
+ }
+}
+
+function highlight(i) {
+ link = document.links[i];
+ link.style.color = "#6060ff";
+}
+
+function unhighlight(i) {
+ link = document.links[i];
+ link.style.color = "#000000";
+}
+
+function selectCurrent() {
+ docname = document.URL;
+ for (i in document.links) {
+ link = document.links[i];
+ if (link.className == "menulink") {
+ if (docname.indexOf(link.href) != -1) {
+ link.className = "selected";
+ }
+ }
+ }
+}
From vytas at ci.uchicago.edu Mon Aug 1 09:23:55 2011
From: vytas at ci.uchicago.edu (vytas at ci.uchicago.edu)
Date: Mon, 01 Aug 2011 14:23:55 -0000
Subject: [Swift-commit] r4899 - in wwwdev: . docs docs/_notes
Message-ID: <20110801142354.735919CC9E@svn.ci.uchicago.edu>
Author: vytas
Date: 2011-08-01 09:23:54 -0500 (Mon, 01 Aug 2011)
New Revision: 4899
Added:
wwwdev/docs/
wwwdev/docs/_notes/
wwwdev/docs/_notes/dwsync.xml
wwwdev/docs/index copy.php
wwwdev/docs/index.php
wwwdev/docs/license.php
wwwdev/docs/tracking_policy.php
wwwdev/docs/usage_data.php
Log:
new look & feel
Added: wwwdev/docs/_notes/dwsync.xml
===================================================================
--- wwwdev/docs/_notes/dwsync.xml (rev 0)
+++ wwwdev/docs/_notes/dwsync.xml 2011-08-01 14:23:54 UTC (rev 4899)
@@ -0,0 +1,7 @@
+
+
+
+
+
+
+
\ No newline at end of file
Added: wwwdev/docs/index copy.php
===================================================================
--- wwwdev/docs/index copy.php (rev 0)
+++ wwwdev/docs/index copy.php 2011-08-01 14:23:54 UTC (rev 4899)
@@ -0,0 +1,149 @@
+
+
+
+
+Swift - A system for rapid and reliable specification, execution, and management of large scale workflows
+
+
+
+
+
+
+
+
+
+ This Swift tutorial carries on where the Quickstart Guide leaves off,
+ introducing you to Swift environment and the SwiftScript language in more
+ depth.
+
+ This tutorial (part of the Open Science Grid teaching lab) introduces
+ you to the Swift environment and the SwiftScript language in more
+ depth using a provided training account where installation of the
+ software has already been performed.
+
+
+
+
Swift User Guide
+
+
+ The User Guide provides more detailed reference documentation and background
+ information on swift. It is assumed that the reader is already familiar with
+ the material in the Quickstart and Tutorial documents.
+
+ The Swift log processing tools can be used to analyse the log files of
+ Swift runs - as a webpage, and in several processed text formats. This
+ document gives some usage information for those tools.
+
+
+
Elsewhere
+
+
+We've taught a hands on Swift tutorial on a number of occasions. This is
+harder to do alone because it relies on a number of brain image processing
+applications which you probably don't have installed.
+[html]
+
+
+
Historical - SwiftScript Language
+
+
+
+ The SwiftScript language reference was a specification for an
+ earlier version of the language, and is probably of little interest to the majority of users.
+
+
+
SwiftScript Language Reference - latest work in progress [html] [pdf]
+ This Swift tutorial carries on where the Quickstart Guide leaves off,
+ introducing you to Swift environment and the SwiftScript language in more
+ depth.
+
+ This tutorial (part of the Open Science Grid teaching lab) introduces
+ you to the Swift environment and the SwiftScript language in more
+ depth using a provided training account where installation of the
+ software has already been performed.
+
+
+
+
+
Swift User Guide
+
+
+ The User Guide provides more detailed reference documentation and background
+ information on swift. It is assumed that the reader is already familiar with
+ the material in the Quickstart and Tutorial documents.
+
+ The Swift log processing tools can be used to analyse the log files of
+ Swift runs - as a webpage, and in several processed text formats. This
+ document gives some usage information for those tools.
+
+
+
Elsewhere
+
+
+We've taught a hands on Swift tutorial on a number of occasions. This is
+harder to do alone because it relies on a number of brain image processing
+applications which you probably don't have installed.
+[html]
+
+
+
+
Historical - SwiftScript Language
+
+
+
+ The SwiftScript language reference was a specification for an
+ earlier version of the language, and is probably of little interest to the majority of users.
+
+
+
SwiftScript Language Reference - latest work in progress [html] [pdf]
Swift v0.8 is a development release intended to release functionality
+and fixes that have gone in to trunk since v0.7.
+[swift-0.8.tar.gz]
+[swift-0.8-stripped.tar.gz - for installation on systems with existing grid stacks]
+[release-notes-0.8.txt]
+
+
+
+
+
+
Swift 0.7 - 2008/11/11
+
Swift v0.7 is a development release intended to release functionality
+and fixes that have gone in to trunk since v0.7.
+[vdsk-0.7.tar.gz]
+[release-notes-0.7.txt]
+
+
+As an alternative to the above traditional Swift packaging, Swift can be
+downloaded and installed using
+pacman, a package
+manager commonly used on the Open Science Grid.
+
+
+There are two installation targets:
+The first will install Swift alongside an existing VDT installation
+
+The second will install Swift as well as a number
+of supporting packages from the VDT software release to support use of the
+DOE CA and use of VOMS. (These packages are also available in a regular OSG or
+VDT installation):
+
Swift v0.6 is a development release intended to release functionality
+and fixes that have gone in to trunk since v0.5.
+[vdsk-0.6.tar.gz]
+[release-notes-0.6.txt]
+
+
+As an alternative to the above traditional Swift packaging, Swift can be
+downloaded and installed using
+pacman, a package
+manager commonly used on the Open Science Grid.
+
+
+This will install Swift alongside an existing VDT installation:
+
Swift v0.5 is a development release intended to release functionality
+and fixes that have gone in to trunk since v0.4.
+[vdsk-0.5.tar.gz]
+[release-notes-0.5.txt]
+
+
Swift 0.4 - 2008/03/18
+
Swift v0.4 is a development release intended to release functionality
+and fixes that have gone in to trunk since v0.3. More details are contained
+in the release notes.
+[vdsk-0.4.tar.gz]
+[release-notes-0.4.txt]
+
+
+
Swift 0.3 - 2007/10/04
+
Swift v0.3 is a development release intended to release functionality
+and fixes that have gone in to trunk since v0.2.
+[vdsk-0.3.tar.gz]
+
+
+
Swift 0.2 - 2007/07/19
+
Swift v0.2 is a development release intended to release functionality
+and fixes that have gone in to trunk since v0.1.
+[vdsk-0.2.tar.gz]
+
+
+
Swift 0.1 - 2007/03/02
+
Swift v0.1 is a development release intended to bring functionality
+and fixes that have gone in to trunk since version 0 in a known release.
+[vdsk-0.1.tar.gz]
+
+
+
Swift 0 RC3
+
+ The 3rd release candidate of Swift version 0:
+ [tar.gz] A bug that caused applications
+ correctly declared in tc.data to not be recognized was fixed For getting
+ started with Swift, please take a look at the documentation page
+
+
+
Swift 0 RC2
+
+
+ The 2nd release candidate of Swift version 0:
+ [tar.gz] This release candidate
+ features improvements in startup speed and the addition of ENV:: and
+ GLOBUS:: profiles.
+
+
+
Swift 0 RC1
+
+
+ The 1st release candidate of Swift version 0:
+ [tar.gz]
+
+
+
Swift 0 RC0
+
+
+ The 0th release candidate of Swift version 0:
+ [tar.gz]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Added: wwwdev/downloads/release-notes-0.5.txt
===================================================================
--- wwwdev/downloads/release-notes-0.5.txt (rev 0)
+++ wwwdev/downloads/release-notes-0.5.txt 2011-08-01 14:24:04 UTC (rev 4900)
@@ -0,0 +1,29 @@
+These are release notes for the Swift 0.4 release. Items
+listed below are changes since Swift 0.3 - both enhancements
+and known issues.
+
+*** Fixed data channel reuse behavior - this was broken in Swift 0.4 for
+ a number of sites.
+
+*** String literals now use \ as escape character - for example, use \"
+ to insert a quote into a string literal. Where \ was previously used
+ in a string literal, it is now necessary to escape it as \\
+
+ Use of quotes in string literals is not fully implemented, however,
+ so attempts to use it are likely to cause problems.
+
+*** Logging performance improvement in wrapper.sh
+
+*** The WeightedHost scheduler now has a lower bound (of -10)
+ on scoring to reduce the effect of penalising a host in the case of a
+ large number of transient errors. Previously, a large number of
+ transient errors would push the score so low that the score would
+ take far too long to return to a more appropriate value.
+
+*** Unused lrc field in site catalog removed.
+
+*** url field in site catalog execute element is now optional.
+
+*** jobmanager attribute to execution element in sites.xml does not work
+ when used with provider="gt2". Such execution elements can be expressed
+ using the jobmanager element syntax instead.
Added: wwwdev/downloads/release-notes-0.6.txt
===================================================================
--- wwwdev/downloads/release-notes-0.6.txt (rev 0)
+++ wwwdev/downloads/release-notes-0.6.txt 2011-08-01 14:24:04 UTC (rev 4900)
@@ -0,0 +1,46 @@
+These are release notes for the Swift 0.6 release. Items
+listed below are changes since Swift 0.5 - both enhancements
+and known issues.
+
+Swift 0.6 is built from Swift SVN r2177 and CoG SVN r2127.
+(0.5 was swift r1783 cog r1962)
+
+A more detailed change log can be found in CHANGES.txt in the distribution
+tarball.
+
+This release contains the following new features:
+
+* job replication (experimental)
+
+When a job has been queued on a site for too long, Swift can resubmit the
+job (possibly to a different site). Replication is controlled by
+replication.* parameters in the swift.properties configuration file. Those
+parameters are documented inline in that configuration file.
+
+* cog coasters (experimental)
+
+Swift can use cog coasters which allow a pilot job to be submitted to
+a site which will pull in actual Swift jobs for execution. This has
+several advantages:
+ . when a large number of jobs need to be submitted, using coasters
+ reduces the job submission overhead in (eg) GRAM to one job per
+ worker node.
+ . once a node is allocated to a coaster, it can be used to run
+ many swift jobs, avoiding the need for each job to be queued
+ separately.
+
+* rate limiting for bad sites
+
+Previous versions of Swift would try to ensure that each site had at
+least two jobs queued or running at once, no matter how poorly performing.
+Swift v0.6 will substantially reduce the number of jobs sent to poorly
+performing sites; when a site score goes below 1 (job at once) then
+Swift will delay sending more jobs to that site, with the delay increasing
+exponentially as the score lowers.
+
+* more compile time type checking
+
+More type checking is done at compile time. This will catch some errors at the
+start of execution which would previously have been detected later on in a run,
+and will give more useful error messages.
+
Added: wwwdev/downloads/release-notes-0.7.txt
===================================================================
--- wwwdev/downloads/release-notes-0.7.txt (rev 0)
+++ wwwdev/downloads/release-notes-0.7.txt 2011-08-01 14:24:04 UTC (rev 4900)
@@ -0,0 +1,52 @@
+These are the release notes for Swift 0.7
+
+More information about Swift can be found at http://www.ci.uchicago.edu/swift/
+
+Swift 0.7 is built from Swift SVN r2318 and cog SVN r2255.
+
+The following are significant changes since Swift 0.6 was released.
+
+Language changes:
+* readData2 procedure allows reading of data from a file in a format similar
+ to that used by the external mapper. This can sometimes be easier and more
+ expressive than using the plain readData format. See:
+ http://www.ci.uchicago.edu/swift/guides/userguide/procedures.php#procedure.readdata2
+* Empty statements are no longer allowed in SwiftScript programs.
+* A simpler syntax has been introduced for application procedures. The
+ previous syntax will continue to work for a number of future releases.
+ The new syntax is shown here:
+ http://www.ci.uchicago.edu/swift/guides/userguide/language.php#procedures.atomic
+
+Coasters:
+* Coasters can now transfer files. This can be used when there are a
+ large number of small files to transfer during a run. See:
+ http://www.ci.uchicago.edu/swift/guides/userguide/coasters.php
+* A large amount of testing and bugfixing on the coaster code should make
+ runs with coasters substantially more reliable.
+
+Configuration changes:
+* Swift will find executables on the remote system $PATH if no explicit
+ path is specified for an application executable in tc.data
+* Configuration files in the ~/.vdl2 directory are now ignored. Previously,
+ attempting to use a ~/.vdl2 configuration file would give an error message.
+
+Replication:
+* The expected queue time for the first job executed is now guestimated for
+ replication purposes, so that job can be replicated if necessary.
+ Previously, the first job would never be replicated as there was
+ insufficient information to estimate its expected queue time.
+
+Minor bugfixes:
+* Uninitialised state was missing from list of known states in runtime
+ ticker, causing it to appear at the end of status list rather than
+ its more natural position at the start.
+* The stageout destination directory is created using the same destination
+ host and provider as the actual stageout transfer. Previously, the
+ stageout directory was always created locally, which was incorrect
+ when using remote stageout URLs.
+* Better walltime handling to work with more LRM backends in GRAM
+
+Packaging:
+* Swift is now available in VDT-friendly pacman form to facilitate use
+ alongside the OSG software stack.
+
Added: wwwdev/downloads/release-notes-0.8.txt
===================================================================
--- wwwdev/downloads/release-notes-0.8.txt (rev 0)
+++ wwwdev/downloads/release-notes-0.8.txt 2011-08-01 14:24:04 UTC (rev 4900)
@@ -0,0 +1,53 @@
+These are the release notes for Swift 0.8
+
+More information about Swift can be found at http://www.ci.uchicago.edu/swift/
+
+Swift 0.8 is built from Swift SVN r2448 and cog SVN r2261
+
+The following are significant changes since Swift 0.7 was released:
+
+Log plotting:
+ * The swift log-processing package is now included in the release. This
+ can be used to plot graphs of Swift logs using the swift-plot-log
+ command:
+
+ $ swift-plot-log first-200901010000-abcdefg.log
+
+ which will create a report in report-first-200901010000-abcdefg/index.html
+
+Execution:
+ * Some job execution systems do not set the initial job working directory
+ as specified by Swift. Previously Swift was unable to execute on such
+ systems. As of 0.8, Swift sets the initial job working directory in a
+ more robust fashion. This problem commonly affected OSG sites running
+ Condor.
+
+ * Application success/failure status can now be taken from the CoG
+ provider layer, instead of using success/failure files on the remote
+ file system. This can reduce the load on the remote file system, but
+ does not work with all job execution mechanisms. Specifically, this
+ mechanism will not work with GRAM2 or clustering.
+
+Local execution:
+ * A deadlock caused when jobs output large amounts of data to stderr has been
+ fixed.
+
+Commandline client:
+ * A number of error messages have been improved.
+ * Console output is much less verbose: the progress ticker appears more often,
+ and individual application start and end messages are no longer shown. This
+ improves the quality of console output when jobs are executed at a high rate.
+
+Language:
+ * Handling of assignment statements in declarations and out of declarations
+ has been made more homogenous. Previously, some assignments could only
+ be made in a declaration statement (such as assigning arrays); and some
+ assignments could only be made away from a declaration statement (such as
+ expressions whose values were computed by some slow process). These two
+ forms of assignment should now be interchangeable.
+
+ * Mapper parameters can now be results of slow computations, rather than
+ needing to be known by the time that declarations are first encountered.
+ This remedies previous unintuitive behaviour.
+
+
Added: wwwdev/downloads/release-notes-0.9.txt
===================================================================
--- wwwdev/downloads/release-notes-0.9.txt (rev 0)
+++ wwwdev/downloads/release-notes-0.9.txt 2011-08-01 14:24:04 UTC (rev 4900)
@@ -0,0 +1,146 @@
+These are the release notes for Swift 0.9
+
+More information about Swift can be found at http://www.ci.uchicago.edu/swift/
+
+Swift 0.9 is built from Swift SVN r2860 and cog SVN r2388
+
+The following are significant changes since Swift 0.8 was released:
+
+User interface
+==============
+
+*** Added console text interface to provide live information about swift
+ runs, which can be enabled with the -tui commandline parameter
+
+*** when replication is enabled, swift will locally kill jobs that have
+ run for twice their specified walltime
+
+Execution modes
+===============
+
+*** Support for Condor-G submissions, by setting a job specification attribute
+ of "grid" and specifying a gridResource attribute containing the string
+ to be placed into the Condor-G grid_resource classad.
+
+*** Support for submissions to a local condor pool
+
+*** Coasters: substantial ongoing development work
+
+Configuration parameters
+========================
+
+*** Environment variable SWIFT_EXTRA_INFO, when set in an environment profile,
+ is executed with the result being recorded in wrapper logs. This allows
+ arbitrary information about the remote environment to be gathered and
+ returned to the submit side.
+
+*** New configuration option wrapper.invocation.mode, specifiable either
+ globally in the configuration file or per-site as a profile key,
+ that configures whether wrapper script invocations are made with an
+ absolute path (as was the behaviour in Swift 0.8) or with a relative
+ path (as was behaviour in previous versions of Swift).
+
+*** New profile key coasterWorkerMaxwalltime - a coaster parameter to
+ explicitly set worker maxwalltime, overriding the default computation
+ based on job walltimes. this should be useful when it is desirable
+ to specify worker parameters based on the known properties of the
+ target queue rather than on properties of the jobs to be executed
+ inside coasters.
+
+*** New profile key coasterInternalIP parameter that allows the IP address
+ used by coaster workers to communicate to the coaster head job to be set
+ explicitly. This can be used when running on a cluster which an internal
+ network which cannot access the IP address that the head node picks
+
+*** configuration property ticker.disable to disable runtime ticker display
+
+New commands
+============
+
+*** The log-processing code, primarily exposed as the swift-plot-log command,
+ has been merged into the main Swift release, rather than being a separate
+ download.
+
+*** There is a new command swift-osg-ress-site-catalog which will generate
+ a site catalog based on data gathered from OSG's ReSS information
+ system. This can be used by OSG users to easily generate a large number
+ of site catalog entries for sites that are likely to work.
+
+Language changes
+================
+
+*** Procedure invocations can be made in any expression context now, rather
+ than only directly in an assignment.
+
+*** Mappings can now be made in any declaration, whether it has an assignment
+ or not. Previous a procedure call assignment and a mapping could not be
+ made in the same statement.
+
+*** Handling of [*] and . has changed. [*] has been broken for several
+ Swift releases, but now has some approximation of its former behaviour,
+ in a cleaner implementation.
+ [*] is now an identity operation, that is array[*] == array.
+ The structure access operator . can now take an array on the left
+ hand side. In situations where a[*].img was used, it is permissible
+ to continue to write a[*].img, or to write a.img - both of these will
+ return an array of img elements.
+
+*** Tighter type checking on app blocks. Primitive types and arrays of
+ primitive types are permitted. Other types are prohibited.
+
+*** Arrays of primitive types can be passed to app blocks, and will be
+ expanded into multiple command-line parameters, one parameter per
+ array element.
+
+*** != operator had been broken forever. It works now.
+
+*** output of trace() is changed for non-primitive datasets:
+
+ * Array handling:
+ Previous to this, trace would show the internal Java object representation
+ of an arrays, which is fairly useless for a user. This commit makes
+ trace show more about the array, and only emit the trace when the array
+ is closed.
+
+ * Other datasets:
+ Trace will wait for those datasets to be closed, and emit their default
+ string representation (including the variable name and path used in
+ SwiftScript)
+
+*** loop condition in iterate statements can now refer to variables declared
+ within the iteration body (this was bug 177)
+
+*** Mappings can now be made in any declaration, whether it has an assignment
+ or not.
+
+*** strsplit function which will split the input string based on separators
+ that match the given pattern and return a string array.
+
+
+Deprecations and removal of old functionality
+=============================================
+
+*** Removed support for .dtm file extension which was deprecated in Swift 0.4
+
+Internal changes
+================
+
+*** The wrapper.sh and seq.sh scripts that are deployed to remote sites to
+ help with execution have been renamed to more Swift specific names, to
+ avoid collision with user-supplied files of the same name. The new names
+ are _swiftwrap and _swiftseq
+
+*** Recompilation will happen if a .kml file was compiled with a different
+ version of Swift to the version being invoked. This is in addition to the
+ existing behaviour where a .swift file will be recompiled if it is newer
+ than the corresponding .kml file.
+
+*** Added a throttling parallelFor and changed the swift foreach underlying
+ implementation to it. The throttling parallelFor limits the number of
+ concurrent iterations allowed, which should allow swift to scale better
+ in certain cases. This is controlled by the configuration property
+ foreach.max.threads
+
+*** Default JVM heap size is now 256M
+
+
From vytas at ci.uchicago.edu Mon Aug 1 09:24:53 2011
From: vytas at ci.uchicago.edu (vytas at ci.uchicago.edu)
Date: Mon, 01 Aug 2011 14:24:53 -0000
Subject: [Swift-commit] r4901 - in wwwdev: . images images/_notes
images/graphics images/orbit images/orbit/_notes
images/rotator images/rotator/_notes
Message-ID: <20110801142451.7AF279CC9E@svn.ci.uchicago.edu>
Author: vytas
Date: 2011-08-01 09:24:51 -0500 (Mon, 01 Aug 2011)
New Revision: 4901
Added:
wwwdev/images/
wwwdev/images/_notes/
wwwdev/images/_notes/dwsync.xml
wwwdev/images/bg_box_shaded.png
wwwdev/images/bg_page.png
wwwdev/images/bg_quotes.png
wwwdev/images/bullet.png
wwwdev/images/bullet_next_32.png
wwwdev/images/bullet_pdf.png
wwwdev/images/bullet_prev_32.png
wwwdev/images/bullet_swift.png
wwwdev/images/button_bg_175.png
wwwdev/images/button_bg_175_ro.png
wwwdev/images/button_bg_175_sp.png
wwwdev/images/graphics/
wwwdev/images/graphics/banner.gif
wwwdev/images/graphics/banner2.gif
wwwdev/images/graphics/h_backgrnd.gif
wwwdev/images/graphics/h_backgrnd2.gif
wwwdev/images/graphics/h_ckgrnd.gif
wwwdev/images/graphics/swift_sketch.gif
wwwdev/images/icon_special_A.png
wwwdev/images/icon_special_B.png
wwwdev/images/logo_swift.png
wwwdev/images/orbit/
wwwdev/images/orbit/_notes/
wwwdev/images/orbit/_notes/dwsync.xml
wwwdev/images/orbit/bullets.jpg
wwwdev/images/orbit/left-arrow.png
wwwdev/images/orbit/loading.gif
wwwdev/images/orbit/mask-black.png
wwwdev/images/orbit/pause-black.png
wwwdev/images/orbit/right-arrow.png
wwwdev/images/orbit/rotator-black.png
wwwdev/images/orbit/spinner.gif
wwwdev/images/orbit/timer-black.png
wwwdev/images/rotator.png
wwwdev/images/rotator/
wwwdev/images/rotator/2011_08_neuroscience.jpg
wwwdev/images/rotator/_notes/
wwwdev/images/rotator/_notes/dwsync.xml
wwwdev/images/rotator/bg_rotator1.jpg
wwwdev/images/rotator/bg_rotator1.png
wwwdev/images/rotator/bg_rotator2.jpg
wwwdev/images/rotator/bg_rotator2.png
wwwdev/images/rotator/bg_rotator3.jpg
wwwdev/images/rotator/bg_rotator3.png
wwwdev/images/rotator_shadow.png
wwwdev/images/uvula.png
wwwdev/images/workflow.gif
wwwdev/images/workflow.jpg
Log:
new look & feel
Added: wwwdev/images/_notes/dwsync.xml
===================================================================
--- wwwdev/images/_notes/dwsync.xml (rev 0)
+++ wwwdev/images/_notes/dwsync.xml 2011-08-01 14:24:51 UTC (rev 4901)
@@ -0,0 +1,18 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
Added: wwwdev/images/bg_box_shaded.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/bg_box_shaded.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/bg_page.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/bg_page.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/bg_quotes.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/bg_quotes.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/bullet.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/bullet.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/bullet_next_32.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/bullet_next_32.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/bullet_pdf.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/bullet_pdf.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/bullet_prev_32.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/bullet_prev_32.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/bullet_swift.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/bullet_swift.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/button_bg_175.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/button_bg_175.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/button_bg_175_ro.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/button_bg_175_ro.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/button_bg_175_sp.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/button_bg_175_sp.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/graphics/banner.gif
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/graphics/banner.gif
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/graphics/banner2.gif
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/graphics/banner2.gif
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/graphics/h_backgrnd.gif
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/graphics/h_backgrnd.gif
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/graphics/h_backgrnd2.gif
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/graphics/h_backgrnd2.gif
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/graphics/h_ckgrnd.gif
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/graphics/h_ckgrnd.gif
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/graphics/swift_sketch.gif
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/graphics/swift_sketch.gif
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/icon_special_A.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/icon_special_A.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/icon_special_B.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/icon_special_B.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/logo_swift.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/logo_swift.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/orbit/_notes/dwsync.xml
===================================================================
--- wwwdev/images/orbit/_notes/dwsync.xml (rev 0)
+++ wwwdev/images/orbit/_notes/dwsync.xml 2011-08-01 14:24:51 UTC (rev 4901)
@@ -0,0 +1,12 @@
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
Added: wwwdev/images/orbit/bullets.jpg
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/orbit/bullets.jpg
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/orbit/left-arrow.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/orbit/left-arrow.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/orbit/loading.gif
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/orbit/loading.gif
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/orbit/mask-black.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/orbit/mask-black.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/orbit/pause-black.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/orbit/pause-black.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/orbit/right-arrow.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/orbit/right-arrow.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/orbit/rotator-black.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/orbit/rotator-black.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/orbit/spinner.gif
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/orbit/spinner.gif
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/orbit/timer-black.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/orbit/timer-black.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/rotator/2011_08_neuroscience.jpg
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/rotator/2011_08_neuroscience.jpg
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/rotator/_notes/dwsync.xml
===================================================================
--- wwwdev/images/rotator/_notes/dwsync.xml (rev 0)
+++ wwwdev/images/rotator/_notes/dwsync.xml 2011-08-01 14:24:51 UTC (rev 4901)
@@ -0,0 +1,10 @@
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
Added: wwwdev/images/rotator/bg_rotator1.jpg
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/rotator/bg_rotator1.jpg
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/rotator/bg_rotator1.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/rotator/bg_rotator1.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/rotator/bg_rotator2.jpg
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/rotator/bg_rotator2.jpg
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/rotator/bg_rotator2.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/rotator/bg_rotator2.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/rotator/bg_rotator3.jpg
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/rotator/bg_rotator3.jpg
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/rotator/bg_rotator3.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/rotator/bg_rotator3.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/rotator.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/rotator.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/rotator_shadow.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/rotator_shadow.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/uvula.png
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/uvula.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/workflow.gif
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/workflow.gif
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: wwwdev/images/workflow.jpg
===================================================================
(Binary files differ)
Property changes on: wwwdev/images/workflow.jpg
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
From vytas at ci.uchicago.edu Mon Aug 1 09:25:04 2011
From: vytas at ci.uchicago.edu (vytas at ci.uchicago.edu)
Date: Mon, 01 Aug 2011 14:25:04 -0000
Subject: [Swift-commit] r4902 - in wwwdev: . inc inc/_notes
Message-ID: <20110801142503.A3BE69CC9E@svn.ci.uchicago.edu>
Author: vytas
Date: 2011-08-01 09:25:03 -0500 (Mon, 01 Aug 2011)
New Revision: 4902
Added:
wwwdev/inc/
wwwdev/inc/_notes/
wwwdev/inc/_notes/dwsync.xml
wwwdev/inc/about_sidebar.php
wwwdev/inc/apps_sidebar.php
wwwdev/inc/docs_sidebar.php
wwwdev/inc/downloads_sidebar.php
wwwdev/inc/footer.php
wwwdev/inc/footer2.php
wwwdev/inc/header.php
wwwdev/inc/header2.php
wwwdev/inc/home_sidebar.php
wwwdev/inc/links_sidebar.php
wwwdev/inc/nav.php
wwwdev/inc/papers_sidebar.php
wwwdev/inc/rotator.php
wwwdev/inc/side_content.php
wwwdev/inc/support_sidebar.php
Log:
new look & feel
Added: wwwdev/inc/_notes/dwsync.xml
===================================================================
--- wwwdev/inc/_notes/dwsync.xml (rev 0)
+++ wwwdev/inc/_notes/dwsync.xml 2011-08-01 14:25:03 UTC (rev 4902)
@@ -0,0 +1,18 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
Added: wwwdev/inc/about_sidebar.php
===================================================================
--- wwwdev/inc/about_sidebar.php (rev 0)
+++ wwwdev/inc/about_sidebar.php 2011-08-01 14:25:03 UTC (rev 4902)
@@ -0,0 +1,11 @@
+
+ Swift uses the Globus Toolkit as
+ middleware to talk to various resources.
+
+
+
Java CoG Kit
+
+ The CoG
+ Kit provides a high level uniform interface to different versions of the
+ Globus Toolkit. It also contains the Karajan language and interpreter which
+ is used as an underlying execution engine for Swift.
+
+
+
TeraGrid
+
+ "TeraGrid is an open scientific
+ discovery infrastructure combining leadership class resources at nine
+ partner sites to create an integrated, persistent computational resource."
+
\ No newline at end of file
Added: wwwdev/inc/home_sidebar.php
===================================================================
--- wwwdev/inc/home_sidebar.php (rev 0)
+++ wwwdev/inc/home_sidebar.php 2011-08-01 14:25:03 UTC (rev 4902)
@@ -0,0 +1,15 @@
+
+
What's New?
+
Paper published
+
Swift: A Language for Distributed Parallel Scripting [pdf]
+
Paper published
+
Parallel Scripting for Applications at the Petascale and Beyond [pdf]
+
SWIFT 0.92.1 RELEASE - 2011/04/13
+
+The latest release of Swift, v0.92.1, is available from the
+downloads page.
+
+
+
+
You need to run an analysis pipeline on a thousand functional magnetic resonance imaging (fMRI) files. The files are in some funny directory structure, the TeraGrid is miles away, and neither writing an MPI program nor performing the many analysis tasks manually is attractive. Swift addresses such challenges. A few lines of SwiftScript describe how your analysis procedures should be applied to the many fMRI images. Swift tools dispatch the thousands of tasks to TeraGrid nodes. Your computation is performed quickly and reliably, and the results are documented with data about how they were computed. Life is good.
+
+ I'm A Caption: I can have links, style or anything that is valid markup :)
+
+
+
\ No newline at end of file
Added: wwwdev/inc/side_content.php
===================================================================
--- wwwdev/inc/side_content.php (rev 0)
+++ wwwdev/inc/side_content.php 2011-08-01 14:25:03 UTC (rev 4902)
@@ -0,0 +1,14 @@
+
Read our Quick Start Guide
+and start using Swift today!
+
+
+
+
+
+
+ Swift is a system for the rapid and reliable specification, execution, and management of large-scale science and engineering workflows. It supports applications that execute many tasks coupled by disk-resident datasets?as is common, for example, when analyzing large quantities of data or performing parameter studies or ensemble simulations.
+
A simple scripting language to enable the concise, high-level specifications of complex parallel computations, and mappers for accessing diverse data formats in a convenient manner.
+
An execution engine that can manage the dispatch of many (10,000) tasks to many (100) processors, whether on parallel computers, campus grids, or multi-site grids.
+
Swift users have achieved multiple-order-of-magnitude savings in program development and execution time.
Cillum brisket ad veniam exercitation andouille. Cillum turkey ut jowl in tail. Reprehenderit nisi pig ex culpa qui, ea meatball ball tip tenderloin pancetta pork.
A few lines of SwiftScript described how our analysis procedures should be applied to thousands of fMRI images. Our computation was performed quickly and reliably. Life is good.
+
Dr. Marielle Goodwin
+Director of Brain Trauma Research
+University of Chicago Hospitals
Read our Quick Start Guide
+and start using Swift today!
+
+
+
+
+
+
+ Swift is a system for the rapid and reliable specification, execution, and management of large-scale science and engineering workflows. It supports applications that execute many tasks coupled by disk-resident datasets?as is common, for example, when analyzing large quantities of data or performing parameter studies or ensemble simulations.
+
A simple scripting language to enable the concise, high-level specifications of complex parallel computations, and mappers for accessing diverse data formats in a convenient manner.
+
An execution engine that can manage the dispatch of many (10,000) tasks to many (100) processors, whether on parallel computers, campus grids, or multi-site grids.
+
Swift users have achieved multiple-order-of-magnitude savings in program development and execution time.
Cillum brisket ad veniam exercitation andouille. Cillum turkey ut jowl in tail. Reprehenderit nisi pig ex culpa qui, ea meatball ball tip tenderloin pancetta pork.
A few lines of SwiftScript described how our analysis procedures should be applied to thousands of fMRI images. Our computation was performed quickly and reliably. Life is good.
+
Dr. Marielle Goodwin
+Director of Brain Trauma Research
+University of Chicago Hospitals