From davidk at ci.uchicago.edu  Tue Nov  1 00:05:10 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Tue,  1 Nov 2011 00:05:10 -0500 (CDT)
Subject: [Swift-commit] r5267 - branches/release-0.93/docs/userguide
Message-ID: <20111101050510.C6D4B9CCAA@svn.ci.uchicago.edu>

Author: davidk
Date: 2011-11-01 00:05:10 -0500 (Tue, 01 Nov 2011)
New Revision: 5267

Modified:
   branches/release-0.93/docs/userguide/transformation_catalog
Log:
Explain how to set environment variables


Modified: branches/release-0.93/docs/userguide/transformation_catalog
===================================================================
--- branches/release-0.93/docs/userguide/transformation_catalog	2011-10-31 19:32:08 UTC (rev 5266)
+++ branches/release-0.93/docs/userguide/transformation_catalog	2011-11-01 05:05:10 UTC (rev 5267)
@@ -8,7 +8,7 @@
 Swift configuration file or on the command line.
 
 The format is one line per executable per site, with fields separated by
-tabs. Spaces cannot be used to separate fields.
+tabs.
 
 Some example entries:
 ----
@@ -17,7 +17,7 @@
 ----
 
 The fields are: site, transformation name, executable path, installation
-status, platform, and profile entrys.
+status, platform, and profile entries.
 
 The site field should correspond to a site name listed in the sites catalog.
 
@@ -31,5 +31,23 @@
 INSTALLED and INTEL32::LINUX respectively.
 
 The profiles field should be set to null if no profile entries are to
-be specified, or should contain the profile entries separated by
-semicolons.
+be specified.
+
+Setting Environment Variables
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+It is often useful to set environment variables when running an application.
+This can be accomplished using *env* in the profile entry. For example,
+the following application sets an environment variable called R_LIBS to
+/home/user/R_libs.
+-----
+localhost	R	/usr/bin/R	INSTALLED	INTEL32::LINUX	env::R_LIBS=/home/user/r_libs
+-----
+
+Setting Multiple Profiles
+~~~~~~~~~~~~~~~~~~~~~~~~~
+Multiple profile entries can be added by using a semicolon. The example below
+sets two environment variables: R_LIBS and R_HOME.
+
+-----
+localhost       R       /usr/bin/R      INSTALLED       INTEL32::LINUX  env::R_LIBS=/home/user/r_libs;env::R_HOME=/home/user/r
+-----



From davidk at ci.uchicago.edu  Tue Nov  1 00:09:56 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Tue,  1 Nov 2011 00:09:56 -0500 (CDT)
Subject: [Swift-commit] r5268 - trunk/docs/userguide
Message-ID: <20111101050956.2D2F69CCAA@svn.ci.uchicago.edu>

Author: davidk
Date: 2011-11-01 00:09:55 -0500 (Tue, 01 Nov 2011)
New Revision: 5268

Modified:
   trunk/docs/userguide/transformation_catalog
Log:
Examples of setting environment variables


Modified: trunk/docs/userguide/transformation_catalog
===================================================================
--- trunk/docs/userguide/transformation_catalog	2011-11-01 05:05:10 UTC (rev 5267)
+++ trunk/docs/userguide/transformation_catalog	2011-11-01 05:09:55 UTC (rev 5268)
@@ -8,7 +8,7 @@
 Swift configuration file or on the command line.
 
 The format is one line per executable per site, with fields separated by
-tabs. Spaces cannot be used to separate fields.
+tabs.
 
 Some example entries:
 ----
@@ -17,7 +17,7 @@
 ----
 
 The fields are: site, transformation name, executable path, installation
-status, platform, and profile entrys.
+status, platform, and profile entries.
 
 The site field should correspond to a site name listed in the sites catalog.
 
@@ -31,5 +31,23 @@
 INSTALLED and INTEL32::LINUX respectively.
 
 The profiles field should be set to null if no profile entries are to
-be specified, or should contain the profile entries separated by
-semicolons.
+be specified.
+
+Setting Environment Variables
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+It is often useful to set environment variables when running an application.
+This can be accomplished using *env* in the profile entry. For example,
+the following application sets an environment variable called R_LIBS to
+/home/user/R_libs.
+-----
+localhost	R	/usr/bin/R	INSTALLED	INTEL32::LINUX	env::R_LIBS=/home/user/r_libs
+-----
+
+Setting Multiple Profiles
+~~~~~~~~~~~~~~~~~~~~~~~~~
+Multiple profile entries can be added by using a semicolon. The example below
+sets two environment variables: R_LIBS and R_HOME.
+
+-----
+localhost       R       /usr/bin/R      INSTALLED       INTEL32::LINUX  env::R_LIBS=/home/user/r_libs;env::R_HOME=/home/user/r
+-----



From lgadelha at ci.uchicago.edu  Tue Nov  1 10:57:01 2011
From: lgadelha at ci.uchicago.edu (lgadelha at ci.uchicago.edu)
Date: Tue,  1 Nov 2011 10:57:01 -0500 (CDT)
Subject: [Swift-commit] r5269 - provenancedb
Message-ID: <20111101155701.AE2D69CCA5@svn.ci.uchicago.edu>

Author: lgadelha
Date: 2011-11-01 10:57:01 -0500 (Tue, 01 Nov 2011)
New Revision: 5269

Modified:
   provenancedb/ProvSQL.g
Log:
Minor changes.


Modified: provenancedb/ProvSQL.g
===================================================================
--- provenancedb/ProvSQL.g	2011-11-01 05:09:55 UTC (rev 5268)
+++ provenancedb/ProvSQL.g	2011-11-01 15:57:01 UTC (rev 5269)
@@ -330,6 +330,8 @@
 				selectClause +=  ".*";
 		}
 		')' 	{ selectClause+=")"; }
+		|
+		builtInAttribute
 	;
 
 selectExpression
@@ -558,6 +560,12 @@
 
 entityAttribute	:	ID (DOT ID)?;
 
+builtInAttribute
+	:	COMPARERUN '(' builtInAtom (COLON builtInAtom)* ')' (DOT ID)?;
+
+builtInAtom
+	:	('parameter' | 'key' ) '=' STRING ;
+	
 entityAndAttribute
 	:	ID DOT ID;
 
@@ -570,6 +578,8 @@
 COMPARERUN
 	:	'compare_run';
 
+ANCESTOR:	'ancestor';
+
 BY	:	'by';
 
 AGGRFUN	:	'avg' | 'max' | 'min' | 'count' | 'sum';



From tga at ci.uchicago.edu  Tue Nov  1 11:27:23 2011
From: tga at ci.uchicago.edu (tga at ci.uchicago.edu)
Date: Tue,  1 Nov 2011 11:27:23 -0500 (CDT)
Subject: [Swift-commit] r5270 - SwiftApps/SwiftR/Swift/exec
Message-ID: <20111101162723.033B19CCA5@svn.ci.uchicago.edu>

Author: tga
Date: 2011-11-01 11:27:22 -0500 (Tue, 01 Nov 2011)
New Revision: 5270

Modified:
   SwiftApps/SwiftR/Swift/exec/rserver.swift
Log:
Add log message when batch completes

Modified: SwiftApps/SwiftR/Swift/exec/rserver.swift
===================================================================
--- SwiftApps/SwiftR/Swift/exec/rserver.swift	2011-11-01 15:57:01 UTC (rev 5269)
+++ SwiftApps/SwiftR/Swift/exec/rserver.swift	2011-11-01 16:27:22 UTC (rev 5270)
@@ -23,6 +23,7 @@
   wait = apply(runDir);
 
   string resultPipeName = @strcat(runDir,"/resultpipe");
+  tracef("%k%s done\n", wait, resultPipeName);
   fprintf(resultPipeName, "%kdone\n", wait);
 }
 



From davidk at ci.uchicago.edu  Wed Nov  2 00:32:14 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Wed,  2 Nov 2011 00:32:14 -0500 (CDT)
Subject: [Swift-commit] r5271 - in wwwdev: docs support
Message-ID: <20111102053214.CD9779CCAA@svn.ci.uchicago.edu>

Author: davidk
Date: 2011-11-02 00:32:13 -0500 (Wed, 02 Nov 2011)
New Revision: 5271

Modified:
   wwwdev/docs/index.php
   wwwdev/support/index.php
Log:
Add searches to mailing lists and documentation


Modified: wwwdev/docs/index.php
===================================================================
--- wwwdev/docs/index.php	2011-11-01 16:27:22 UTC (rev 5270)
+++ wwwdev/docs/index.php	2011-11-02 05:32:13 UTC (rev 5271)
@@ -85,6 +85,18 @@
 </ul>
 <p>&nbsp;</p>
 
+<h3>Search All Documentation</h3>
+<div id="cse" style="width: 100%;">Loading</div>
+<script src="//www.google.com/jsapi" type="text/javascript"></script>
+<script type="text/javascript"> 
+  google.load('search', '1', {language : 'en', style : google.loader.themes.SHINY});
+  google.setOnLoadCallback(function() {
+    var customSearchControl = new google.search.CustomSearchControl('007067124251691692991:pbx7ijbjeea');
+    customSearchControl.setResultSetSize(google.search.Search.FILTERED_CSE_RESULTSET);
+    customSearchControl.draw('cse');
+  }, true);
+</script>
+
 <!-- 
 <h3>Swift Cookbook</h3>
         
@@ -107,6 +119,7 @@
 				
   </div>
 <!-- end .container_12 -->
+
 <!-- footer -->
 <?php require('../inc/footer2.php') ?> 
 <!-- end footer -->
@@ -119,5 +132,6 @@
 var pageTracker = _gat._getTracker("UA-106257-5");
 pageTracker._trackPageview();
 } catch(err) {}</script>
+
 </body>
 </html>

Modified: wwwdev/support/index.php
===================================================================
--- wwwdev/support/index.php	2011-11-01 16:27:22 UTC (rev 5270)
+++ wwwdev/support/index.php	2011-11-02 05:32:13 UTC (rev 5271)
@@ -29,7 +29,7 @@
         as a part of your run. If the log files are too large to include in an email, please provide 
         access to the logs in a publicly readable directory or website. More information on these
         mailing lists are provided below.
-        <br><br>
+        <br><br><br>
 
 	<h3>Mailing Lists</h3>
 	<table class="mailinglists">
@@ -72,17 +72,28 @@
 	</table>
 	<p>&nbsp;</p>
 	<p>&nbsp;</p>
+          <h3>Search Mailing List Archive</h3>
+           <div id="cse" style="width: 100%;">Loading</div>
+           <script src="//www.google.com/jsapi" type="text/javascript"></script>
+           <script type="text/javascript"> 
+              google.load('search', '1', {language : 'en', style : google.loader.themes.SHINY});
+              google.setOnLoadCallback(function() {
+              var customSearchControl = new google.search.CustomSearchControl('007067124251691692991:zxjheipt4is');
+              customSearchControl.setResultSetSize(google.search.Search.FILTERED_CSE_RESULTSET);
+              customSearchControl.draw('cse');
+              }, true);
+           </script>
+</div> 
 	</div>
   <!-- end .grid_9 -->
-  <div class="grid_3">
+  <!-- 
+    <div class="grid_3">
   	<h3>Bug Reports</h3>
     <p>We have a Bugzilla, at <a href="http://bugzilla.mcs.anl.gov/swift/">bugzilla.mcs.anl.gov/swift/</a>. </p>
     <p>&nbsp;</p>
   </div>
+  -->
   <!-- end .grid_3 -->
-  
-  <div class="clear"></div>
-				
   </div>
 <!-- end .container_12 -->
 <!-- footer -->



From davidk at ci.uchicago.edu  Wed Nov  2 00:41:18 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Wed,  2 Nov 2011 00:41:18 -0500 (CDT)
Subject: [Swift-commit] r5272 - wwwdev/docs
Message-ID: <20111102054118.327899CCAA@svn.ci.uchicago.edu>

Author: davidk
Date: 2011-11-02 00:41:17 -0500 (Wed, 02 Nov 2011)
New Revision: 5272

Modified:
   wwwdev/docs/index.php
Log:
Disable doc searching until site is live and google can index


Modified: wwwdev/docs/index.php
===================================================================
--- wwwdev/docs/index.php	2011-11-02 05:32:13 UTC (rev 5271)
+++ wwwdev/docs/index.php	2011-11-02 05:41:17 UTC (rev 5272)
@@ -84,7 +84,7 @@
         </li>
 </ul>
 <p>&nbsp;</p>
-
+<!--
 <h3>Search All Documentation</h3>
 <div id="cse" style="width: 100%;">Loading</div>
 <script src="//www.google.com/jsapi" type="text/javascript"></script>
@@ -96,6 +96,7 @@
     customSearchControl.draw('cse');
   }, true);
 </script>
+-->
 
 <!-- 
 <h3>Swift Cookbook</h3>



From ketan at ci.uchicago.edu  Mon Nov  7 13:46:44 2011
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Mon,  7 Nov 2011 13:46:44 -0600 (CST)
Subject: [Swift-commit] r5273 - branches/release-0.93/docs/siteguide
Message-ID: <20111107194644.5DF589CC9D@svn.ci.uchicago.edu>

Author: ketan
Date: 2011-11-07 13:46:44 -0600 (Mon, 07 Nov 2011)
New Revision: 5273

Modified:
   branches/release-0.93/docs/siteguide/beagle
Log:
corrected sites terminology and meanings

Modified: branches/release-0.93/docs/siteguide/beagle
===================================================================
--- branches/release-0.93/docs/siteguide/beagle	2011-11-02 05:41:17 UTC (rev 5272)
+++ branches/release-0.93/docs/siteguide/beagle	2011-11-07 19:46:44 UTC (rev 5273)
@@ -53,8 +53,8 @@
 The following sites.xml parameters must be set to scale that is intended for a large run:
 
  * *maxTime* : The expected walltime for completion of your run. This parameter is accepted in seconds.
- * *slots* : Number of qsub jobs needs to be submitted by swift. This number will determine how many qsubs swift will submit for your run. Typical values range between 40 and 80 for large runs.
- * *nodeGranularity* : Determines the number of nodes per job. Total nodes will thus be slots times nodegranularity. This may vary for advanced configurations though.
+ * *slots* : This parameter specifies the maximum number of jobs/blocks that the coaster scheduler will have running at any given time. On Beagle, this number will determine how many qsubs swift will submit for your run. Typical values range between 40 and 60 for large runs.
+ * *nodeGranularity* : Determines the number of nodes per job. It restricts the number of nodes in a job to a multiple of this value. The total number of workers will then be a multiple of jobsPerNode * nodeGranularity. For Beagle, jobsPerNode value is 24 corresponding to its 24 cores per node. 
  * *maxNodes* : Determines the maximum number of nodes a job must pack into its qsub. This parameter determines the largest single job that your run will submit.
  * *jobThrottle* : A factor that determines the number of tasks dispatched simultaneously. The intended number of simultaneous tasks must match the number of cores targeted. The number of tasks is calculated from the jobThrottle factor is as follows:
 



From lgadelha at ci.uchicago.edu  Tue Nov  8 18:10:54 2011
From: lgadelha at ci.uchicago.edu (lgadelha at ci.uchicago.edu)
Date: Tue,  8 Nov 2011 18:10:54 -0600 (CST)
Subject: [Swift-commit] r5274 - provenancedb
Message-ID: <20111109001054.248929CCA3@svn.ci.uchicago.edu>

Author: lgadelha
Date: 2011-11-08 18:10:53 -0600 (Tue, 08 Nov 2011)
New Revision: 5274

Modified:
   provenancedb/ProvSQL.g
Log:
Inclusion of built-in function compare_run, that abstract provenance correlation queries.


Modified: provenancedb/ProvSQL.g
===================================================================
--- provenancedb/ProvSQL.g	2011-11-07 19:46:44 UTC (rev 5273)
+++ provenancedb/ProvSQL.g	2011-11-09 00:10:53 UTC (rev 5274)
@@ -16,14 +16,15 @@
 	String whereClause = new String(); 
 	boolean hasWhereJoinExpression;
 	boolean hasWhereExpression = false;
+	static boolean hasCompareRunCall = false;
 	HashSet<String> relations = new HashSet<String>();
 	UndirectedGraph<String,DefaultEdge> schemaGraph;
 	HashSet<DefaultEdge> joinEdges;
+	static HashSet<String> compareRunParams = new HashSet<String>();;
 	
 	// Ideally it could receive a DB schema in SQL and build the graph automatically
 	public static UndirectedGraph<String,DefaultEdge> buildGraph() {
 		UndirectedGraph<String,DefaultEdge> schemaGraph = new Multigraph<String,DefaultEdge>(DefaultEdge.class);
-
 		schemaGraph.addVertex("a_run_t");
 		schemaGraph.addVertex("a_run_n");
 		schemaGraph.addVertex("run");
@@ -60,7 +61,6 @@
 		schemaGraph.addEdge("ds", "in_mem");
 		schemaGraph.addEdge("ds", "ds_cont");
 		schemaGraph.addEdge("ds", "ds_cont");
-		schemaGraph.addEdge("compare_run", "run");
 
 		return schemaGraph;
 	}
@@ -78,13 +78,10 @@
 			Iterator<DefaultEdge> j = (sP.getPathEdgeList()).iterator();
 			while(j.hasNext())
 				jEdges.add(j.next());
-		}
-
-		
+		}	
 		return jEdges;
 	}
 
-	
 	public static String computeFrom(UndirectedGraph<String,DefaultEdge> schemaGraph, HashSet<DefaultEdge> joinEdges, HashSet<String> qrels) {
 		HashSet<String> fromRels = new HashSet<String>();
 		String fromq = " FROM ";
@@ -116,15 +113,18 @@
 						fromRels.add(schemaGraph.getEdgeSource(aux));
 						fromRels.add(schemaGraph.getEdgeTarget(aux));				
 					}
-
 				}
 			}
 		Iterator<String> j = fromRels.iterator();
 		if(j.hasNext())
 			fromq += j.next();
 		while(j.hasNext())
-			fromq+=","+j.next();
-
+			fromq += "," + j.next();
+		if(hasCompareRunCall) {
+			if(fromRels.size() > 0)
+				fromq += ",";
+			fromq += "(" + computeCompareRunQuery(compareRunParams) + ") AS compare_run";
+		}
 		return fromq;
 	}
 
@@ -152,7 +152,6 @@
 		joinExpressions.put(schemaGraph.getEdge("ds", "in_mem"), "ds.id=in_mem.id");
 		joinExpressions.put(schemaGraph.getEdge("ds", "ds_cont"), "ds.id=ds_cont.in_id");
 		joinExpressions.put(schemaGraph.getEdge("ds", "ds_cont"), "ds.id=ds_cont.out_id");
-		joinExpressions.put(schemaGraph.getEdge("compare_run", "ds_cont"), "ds.id=ds_cont.out_id");
 
 		Iterator<DefaultEdge> i = jEdges.iterator();
 		if(i.hasNext()) {
@@ -189,9 +188,34 @@
 
 			}    		
 		}
-
 		return joinExpressionsString;
 	}
+
+	public static String computeCompareRunQuery(HashSet<String> atoms) {
+		String compareRunSelectClause = "SELECT run_id";
+		String compareRunFromClause = "FROM";
+		Iterator<String> i = atoms.iterator();
+		int nId = 0;
+		for(String arg: atoms) {
+			String[] argTokens = arg.split("=");
+			if(argTokens[0].equals("key_numeric") ||
+			   argTokens[0].equals("key_text")    ||
+			   argTokens[0].equals("parameter"))
+			{
+				String key = argTokens[1].split("'")[1];
+				nId++;
+				String sId = "j" + nId;
+				compareRunSelectClause+=", " + sId + ".value as " + key;
+				if(nId>1)
+					compareRunFromClause += " INNER JOIN";
+				compareRunFromClause += " compare_run_by_" + argTokens[0] + "(\'" + key + "\') as " + sId;
+				if(nId>1)
+					compareRunFromClause += " USING (run_id)";
+			}
+		}
+		String compareRunQuery = compareRunSelectClause + " " + compareRunFromClause;
+		return compareRunQuery;
+	}
 	
 }
 
@@ -322,7 +346,7 @@
 			selectClause+=$b.text;
 		} 
 		'(' 	{ selectClause+="("; }
-		c=entityAttribute  
+		c=entityAndAttribute  
 		{ 
 			selectClause += $c.text; 
 			relations.add($c.text.split("\\.")[0]);
@@ -331,7 +355,7 @@
 		}
 		')' 	{ selectClause+=")"; }
 		|
-		builtInAttribute
+		builtInProcedureAttribute 
 	;
 
 selectExpression
@@ -361,11 +385,16 @@
 	;
 
 whereAtom 
-	:	a=entityAndAttribute 
-		{ 
+	:	(a=entityAndAttribute 
+		{
 			relations.add($a.text.split("\\.")[0]);
 			whereClause += $a.text;
 		}
+		|
+		j=COMPARERUN { whereClause+="comapare_run"; }
+		DOT 
+		k=ID { whereClause+="."+$k.text; } 
+		)
 			(
 		NOT
 		{
@@ -560,15 +589,61 @@
 
 entityAttribute	:	ID (DOT ID)?;
 
-builtInAttribute
-	:	COMPARERUN '(' builtInAtom (COLON builtInAtom)* ')' (DOT ID)?;
-
-builtInAtom
-	:	('parameter' | 'key' ) '=' STRING ;
-	
 entityAndAttribute
 	:	ID DOT ID;
 
+
+builtInProcedureAttribute
+	:	COMPARERUN {
+			boolean hasAttribute = false;
+		}
+		{
+			hasCompareRunCall=true;
+		}
+		'(' 
+		a=builtInAtom
+		{ 
+			compareRunParams.add($a.text);
+			if(relations.size() > 0)
+				relations.add("run");
+		}
+	 	(COLON 
+	 	b=builtInAtom
+	 	{ 
+			compareRunParams.add($b.text);
+			if(relations.size() > 0)
+				relations.add("run");
+		} 		
+	 	)* ')' ( 
+	 	DOT
+	 	{
+	 		hasAttribute = true;
+	 	}
+	 	 (
+	 	c=ID
+	 	{
+	 		selectClause += "compare_run." + $c.text;
+	 	}
+	 		 | '{' 
+	 	d=ID
+	 	{
+	 		selectClause += "compare_run." + $d.text;
+	 	}
+	 	(COLON 
+	 	e=ID 
+	 	{ 
+	 		selectClause += ", compare_run." + $e.text; 
+	 	}
+	 	)* '}'))?
+	 	{
+	 		if(!hasAttribute)
+	 			selectClause += "compare_run.*";
+	 	}
+;
+
+builtInAtom 
+	:	('parameter' | 'key_numeric' | 'key_text') OP STRING;
+
 OP	:	'=' | '>' | '>=' | '<' | '<=';
 
 GROUP	:	'group';
@@ -590,39 +665,40 @@
 
 ASC	:	'asc';
 
+
 DISTINCT 
-	:	'd' 'i' 's' 't' 'i' 'n' 'c' 't';
+	:	'distinct';
 	
-WHERE	:	'w' 'h' 'e' 'r' 'e';
+WHERE	:	'where';
 
-AND	:	'a' 'n' 'd';
+AND	:	'and';
 
-OR	:	'o' 'r';
+OR	:	'or';
 
-NOT	:	'n' 'o' 't';
+NOT	:	'not';
 
-IN	:	'i' 'n';
+IN	:	'in';
 
-ANY	:	'a' 'n' 'y';
+ANY	:	'any';
 
-UNION	:	'u' 'n' 'i' 'o' 'n';
+UNION	:	'union';
 
 INTERSECT 
-	:	'i' 'n' 't' 'e' 'r' 's' 'e' 'c' 't';
+	:	'intersect';
 
-EXCEPT	:	'e' 'x' 'c' 'e' 'p' 't';
+EXCEPT	:	'except';
 
-ALL	:	'a' 'l' 'l';
+ALL	:	'all';
 
 DOT	:	'.';
 
 COLON	:	',';
 
-BETWEEN	:	'b' 'e' 't' 'w' 'e' 'e' 'n';
+BETWEEN	:	'between';
 
 HAVING	:	'having';
 
-LIKE 	:	'l' 'i' 'k' 'e';
+LIKE 	:	'like';
 
 SEMICOLON	:	';';
 



From davidk at ci.uchicago.edu  Wed Nov  9 15:05:23 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Wed,  9 Nov 2011 15:05:23 -0600 (CST)
Subject: [Swift-commit] r5278 - in wwwdev: inc main
Message-ID: <20111109210523.287BF9CC9D@svn.ci.uchicago.edu>

Author: davidk
Date: 2011-11-09 15:05:22 -0600 (Wed, 09 Nov 2011)
New Revision: 5278

Modified:
   wwwdev/inc/downloads_sidebar.php
   wwwdev/main/index.php
Log:
Swift 0.93RC4


Modified: wwwdev/inc/downloads_sidebar.php
===================================================================
--- wwwdev/inc/downloads_sidebar.php	2011-11-09 00:40:10 UTC (rev 5277)
+++ wwwdev/inc/downloads_sidebar.php	2011-11-09 21:05:22 UTC (rev 5278)
@@ -1,7 +1,7 @@
 <div id="box-prime">
 <h2><a name="stable"></a>Latest Release</h2>
-<div class="button_175"><a href="../packages/swift-0.93RC3.tar.gz">Download Now</a></div>
-<b>0.93 RC3</b> current version<br />2011/10/6
+<div class="button_175"><a href="../packages/swift-0.93RC4.tar.gz">Download Now</a></div>
+<b>0.93 RC4</b> current version<br />2011/11/9
 <br />
 </div>
 <p>&nbsp;</p>

Modified: wwwdev/main/index.php
===================================================================
--- wwwdev/main/index.php	2011-11-09 00:40:10 UTC (rev 5277)
+++ wwwdev/main/index.php	2011-11-09 21:05:22 UTC (rev 5278)
@@ -62,7 +62,7 @@
   </div>
   <!-- end .grid_9 -->
   <div class="grid_3">
-    <div id="box-prime"><div class="button_175"><a href="../downloads" onClick="javascript: pageTracker._trackPageview('../downloads');">Download Now</a></div><b>0.93 RC3</b> current version<br />2011/10/06
+    <div id="box-prime"><div class="button_175"><a href="../downloads" onClick="javascript: pageTracker._trackPageview('../downloads');">Download Now</a></div><b>0.93 RC4</b> current version<br />2011/11/09
 	<br /></div>
 	<div id="box-prime"><div class="button_175"><a href="../guides/release-0.93/quickstart/quickstart.html">Get Started</a></div>Read our Quick Start Guide
 and start using Swift today!</div>



From hategan at ci.uchicago.edu  Thu Nov 10 13:50:44 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Thu, 10 Nov 2011 13:50:44 -0600 (CST)
Subject: [Swift-commit] r5279 - branches
Message-ID: <20111110195044.540929CCA3@svn.ci.uchicago.edu>

Author: hategan
Date: 2011-11-10 13:50:44 -0600 (Thu, 10 Nov 2011)
New Revision: 5279

Added:
   branches/release-0.93.1/
Log:
branched branches/release-0.93 to branches/release-0.93.1



From jonmon at ci.uchicago.edu  Thu Nov 10 20:42:26 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Thu, 10 Nov 2011 20:42:26 -0600 (CST)
Subject: [Swift-commit] r5280 - SwiftApps/GOSwift/pykoa/tools
Message-ID: <20111111024226.C9A289CCA3@svn.ci.uchicago.edu>

Author: jonmon
Date: 2011-11-10 20:42:26 -0600 (Thu, 10 Nov 2011)
New Revision: 5280

Modified:
   SwiftApps/GOSwift/pykoa/tools/koa_goswift_run.py
Log:
o GOSwift
  -- goswift_run now outputs a more cleaner output


Modified: SwiftApps/GOSwift/pykoa/tools/koa_goswift_run.py
===================================================================
--- SwiftApps/GOSwift/pykoa/tools/koa_goswift_run.py	2011-11-10 19:50:44 UTC (rev 5279)
+++ SwiftApps/GOSwift/pykoa/tools/koa_goswift_run.py	2011-11-11 02:42:26 UTC (rev 5280)
@@ -1,4 +1,5 @@
 #!/usr/bin/python
+
 import os
 import re
 import sys
@@ -27,7 +28,11 @@
         self.lines = []
 
     def stdout_event(self, line):
-        sys.stdout.write(line)
+        line_split = line.split(" ")
+        line_split = line_split[3:]
+        for parse in line_split:
+            sys.out.write(parse+" ")
+
         self.lines.append(line)
         return
 



From lgadelha at ci.uchicago.edu  Fri Nov 11 09:29:06 2011
From: lgadelha at ci.uchicago.edu (lgadelha at ci.uchicago.edu)
Date: Fri, 11 Nov 2011 09:29:06 -0600 (CST)
Subject: [Swift-commit] r5281 - provenancedb
Message-ID: <20111111152906.096CC9CC9D@svn.ci.uchicago.edu>

Author: lgadelha
Date: 2011-11-11 09:29:05 -0600 (Fri, 11 Nov 2011)
New Revision: 5281

Modified:
   provenancedb/pql_functions.sql
Log:
Minor changes.


Modified: provenancedb/pql_functions.sql
===================================================================
--- provenancedb/pql_functions.sql	2011-11-11 02:42:26 UTC (rev 5280)
+++ provenancedb/pql_functions.sql	2011-11-11 15:29:05 UTC (rev 5281)
@@ -90,11 +90,11 @@
 
 
 
-drop type compare_run_by_param_type cascade;
-create type compare_run_by_param_type as (run_id varchar, param varchar, value varchar);
+drop type compare_run_by_parameter_type cascade;
+create type compare_run_by_parameter_type as (run_id varchar, param varchar, value varchar);
 
-create or replace function compare_run_by_param(param_name varchar)
-returns setof compare_run_by_param_type
+create or replace function compare_run_by_parameter(param_name varchar)
+returns setof compare_run_by_parameter_type
 as $$
    select run_id, param, value
    from   ds_use,proc,in_mem
@@ -102,7 +102,7 @@
 $$ language sql;
 
 -- PostgreSQL >= 9.0
--- CREATE OR REPLACE FUNCTION compare_run_by_parameter(param_name VARCHAR) 
+-- CREATE OR REPLACE FUNCTION compare_run_by_parametereter(param_name VARCHAR) 
 -- RETURNS TABLE (
 --   run_id VARCHAR, 
 --   param VARCHAR, 
@@ -123,7 +123,7 @@
 --$$ LANGUAGE SQL;
 
 
---CREATE OR REPLACE FUNCTION compare_run_by_parameter(param_name1 VARCHAR, param_name2 VARCHAR) 
+--CREATE OR REPLACE FUNCTION compare_run_by_parametereter(param_name1 VARCHAR, param_name2 VARCHAR) 
 --RETURNS TABLE (
 --  workflow_id VARCHAR, 
 --  param_name1 VARCHAR, 
@@ -133,9 +133,9 @@
 --) 
 --AS $$
 --  SELECT * 
---  FROM   compare_run_by_parameter($1) as t 
+--  FROM   compare_run_by_parametereter($1) as t 
 --         INNER JOIN 
---         compare_run_by_parameter($2) as s 
+--         compare_run_by_parametereter($2) as s 
 --         USING (workflow_id); 
 --$$ LANGUAGE SQL;
 



From ketan at ci.uchicago.edu  Fri Nov 11 14:58:51 2011
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Fri, 11 Nov 2011 14:58:51 -0600 (CST)
Subject: [Swift-commit] r5282 - branches/release-0.93/docs/userguide
Message-ID: <20111111205851.6FBC99CC9D@svn.ci.uchicago.edu>

Author: ketan
Date: 2011-11-11 14:58:51 -0600 (Fri, 11 Nov 2011)
New Revision: 5282

Modified:
   branches/release-0.93/docs/userguide/cdm
Log:
 

Modified: branches/release-0.93/docs/userguide/cdm
===================================================================
--- branches/release-0.93/docs/userguide/cdm	2011-11-11 15:29:05 UTC (rev 5281)
+++ branches/release-0.93/docs/userguide/cdm	2011-11-11 20:58:51 UTC (rev 5282)
@@ -1,12 +1,16 @@
 Collective Data Management
 --------------------------
 
+What is CDM
+~~~~~~~~~~~
+CDM is a Swift feature that broadly enables improved management of data based on its nature and location.
+
 Overview
 ~~~~~~~~ 
 . The user specifies a CDM policy in a file, customarily fs.data.  
 . fs.data is given to Swift on the command line.  
 . The Swift data module (org.globus.swift.data) is informed of the CDM policy.  
-. At job launch time, the VDL Karajan code queries the CDM policy,  
+. At job launch time, the CDM policy is queried,  
    .. altering the file staging phase, and  
    .. sending fs.data to the compute site.  
 . At job run time, the Swift wrapper script  



From davidk at ci.uchicago.edu  Sat Nov 12 01:23:11 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Sat, 12 Nov 2011 01:23:11 -0600 (CST)
Subject: [Swift-commit] r5283 - in branches/release-0.93/tests: .
	providers/ranger/local
Message-ID: <20111112072311.2C6829CCA3@svn.ci.uchicago.edu>

Author: davidk
Date: 2011-11-12 01:23:10 -0600 (Sat, 12 Nov 2011)
New Revision: 5283

Added:
   branches/release-0.93/tests/providers/ranger/local/sleep.swift
Modified:
   branches/release-0.93/tests/USAGENOTES.txt
   branches/release-0.93/tests/providers/ranger/local/tc.template.data
Log:
Regression test for SGE jobs failing with no exit code file being created


Modified: branches/release-0.93/tests/USAGENOTES.txt
===================================================================
--- branches/release-0.93/tests/USAGENOTES.txt	2011-11-11 20:58:51 UTC (rev 5282)
+++ branches/release-0.93/tests/USAGENOTES.txt	2011-11-12 07:23:10 UTC (rev 5283)
@@ -74,6 +74,7 @@
 - *.check.sh*
 - *.clean.sh*
 - *.timeout specifier.*
+- *.repeat (number of times to repeat this test)
 
 The scripts may setup and inspect files in *RUNDIR* 
 including *exec.out* which must be accessed in *stdout.txt* 

Added: branches/release-0.93/tests/providers/ranger/local/sleep.swift
===================================================================
--- branches/release-0.93/tests/providers/ranger/local/sleep.swift	                        (rev 0)
+++ branches/release-0.93/tests/providers/ranger/local/sleep.swift	2011-11-12 07:23:10 UTC (rev 5283)
@@ -0,0 +1,7 @@
+// *_THIS-SCRIPT-SHOULD-FAIL_*
+app sleep (int i)
+{
+  sleep i;
+}
+
+sleep(@toint(@arg("n","300")));

Modified: branches/release-0.93/tests/providers/ranger/local/tc.template.data
===================================================================
--- branches/release-0.93/tests/providers/ranger/local/tc.template.data	2011-11-11 20:58:51 UTC (rev 5282)
+++ branches/release-0.93/tests/providers/ranger/local/tc.template.data	2011-11-12 07:23:10 UTC (rev 5283)
@@ -5,4 +5,4 @@
 ranger 	sort 		/bin/sort	INSTALLED	INTEL32::LINUX	null
 ranger 	paste 		/bin/paste	INSTALLED	INTEL32::LINUX	null
 ranger	wc		/usr/bin/wc	INSTALLED	INTEL32::LINUX	null
-
+ranger	sleep		/bin/sleep	null		null		null



From ketan at ci.uchicago.edu  Sat Nov 12 15:54:59 2011
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Sat, 12 Nov 2011 15:54:59 -0600 (CST)
Subject: [Swift-commit] r5284 - branches/release-0.93/docs/siteguide
Message-ID: <20111112215459.3CCD49CCA3@svn.ci.uchicago.edu>

Author: ketan
Date: 2011-11-12 15:54:58 -0600 (Sat, 12 Nov 2011)
New Revision: 5284

Modified:
   branches/release-0.93/docs/siteguide/beagle
Log:
beagle docs improved

Modified: branches/release-0.93/docs/siteguide/beagle
===================================================================
--- branches/release-0.93/docs/siteguide/beagle	2011-11-12 07:23:10 UTC (rev 5283)
+++ branches/release-0.93/docs/siteguide/beagle	2011-11-12 21:54:58 UTC (rev 5284)
@@ -26,23 +26,78 @@
 stay. (say, +mkdir swift-lab+, followed by, +cd swift-lab+)
 
 *step 3.* To get started with a simple example running +/bin/cat+ to read an
-input file +data.txt+ and write to an output file +f.nnn.out+, copy the folder
-at +/home/ketan/catsn+ to the above directory. (+cp -r /home/ketan/catsn
-.+ followed by +cd catsn+).
+input file +data.txt+ and write to an output file +f.nnn.out+, start with writing a simple swift source script as follows:
 
-*step 4.*  In the sites file: +beagle-coaster.xml+, make the following two
-changes: *1)* change the path of +workdirectory+ to your preferred location
-(say to +/lustre/beagle/$USER/swift-lab/swift.workdir+) and *2)* Change the
-project name to your project (+CI-CCR000013+) . The workdirectory will contain
-execution data related to each run, e.g. wrapper scripts, system information,
-inputs and outputs.
+-----
+type file;
 
-*step 5.* Run the example using following commandline (also found in run.sh):
-+swift -config cf -tc.file tc -sites.file beagle-coaster.xml catsn.swift -n=1+
+/* App definitio */
+app (file o) cat (file i)
+{
+  cat @i stdout=@o;
+}
+
+file out[]<simple_mapper; location="outdir", prefix="f.",suffix=".out">;
+file data<"data.txt">;
+
+/* App invocation: n times */
+foreach j in [1:@toint(@arg("n","1"))] {
+  out[j] = cat(data);
+}
+-----
+
+*step 4.*  The next step is to create a sites file. An example sites file (sites.xml) is shown as follows:
+-----
+<config>
+  <pool handle="pbs">
+    <execution provider="coaster" jobmanager="local:pbs"/>
+    <!-- replace with your project -->
+    <profile namespace="globus" key="project">CI-CCR000013</profile>
+
+    <profile namespace="globus" key="providerAttributes">
+                     pbs.aprun;pbs.mpp;depth=24</profile>
+
+    <profile namespace="globus" key="jobsPerNode">24</profile>
+    <profile namespace="globus" key="maxTime">1000</profile>
+    <profile namespace="globus" key="slots">1</profile>
+    <profile namespace="globus" key="nodeGranularity">1</profile>
+    <profile namespace="globus" key="maxNodes">1</profile>
+
+    <profile namespace="karajan" key="jobThrottle">.63</profile>
+    <profile namespace="karajan" key="initialScore">10000</profile>
+
+    <filesystem provider="local"/>
+    <!-- replace this with your home on lustre -->
+    <workdirectory >/lustre/beagle/ketan/swift.workdir</workdirectory>
+  </pool>
+</config>
+-----
+
+*step 5.* In this step, we will see the config and tc files. The config file (cf) is as follows:
+-----
+wrapperlog.always.transfer=true
+sitedir.keep=true
+execution.retries=1
+lazy.errors=true
+use.provider.staging=true
+provider.staging.pin.swiftfiles=false
+foreach.max.threads=100
+provenance.log=false
+-----
+
+The tc file (tc) is as follows:
+-----
+pbs cat /bin/cat null null null
+-----
+
+More about config and tc file options can be found in the swift userguide here: http://www.ci.uchicago.edu/swift/wwwdev/guides/release-0.93/userguide/userguide.html#_swift_configuration_properties.
+
+*step 6.* Run the example using following commandline (also found in run.sh):
++swift -config cf -tc.file tc -sites.file sites.xml catsn.swift -n=1+
 . You can further change the value of +-n+ to any arbitrary number to run that
 many number of concurrent +cat+
 
-*step 6.* Check the output in the generated +outdir+ directory (+ls outdir+)
+*step 7.* Swift will show a status message as "done" after the job has completed its run in the queue. Check the output in the generated +outdir+ directory (+ls outdir+)
 
 Note: Running from sandbox node or requesting 30 minutes walltime for upto 3 nodes
 will get fast prioritized execution. Good for small tests.



From wilde at ci.uchicago.edu  Sun Nov 13 20:29:23 2011
From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu)
Date: Sun, 13 Nov 2011 20:29:23 -0600 (CST)
Subject: [Swift-commit] r5285 - branches/release-0.93/bin
Message-ID: <20111114022923.B427A9CC94@svn.ci.uchicago.edu>

Author: wilde
Date: 2011-11-13 20:29:22 -0600 (Sun, 13 Nov 2011)
New Revision: 5285

Modified:
   branches/release-0.93/bin/swift
Log:
Allow users to set an alternate value for $HOME (Java property user.home) to make Swift look for .globus in an alternate location, e.g. /lustre/beagle/$USER). Use this by \*exporting\* SWIFT_USERHOME and setting that env var to the alternate HOME location.

Modified: branches/release-0.93/bin/swift
===================================================================
--- branches/release-0.93/bin/swift	2011-11-12 21:54:58 UTC (rev 5284)
+++ branches/release-0.93/bin/swift	2011-11-14 02:29:22 UTC (rev 5285)
@@ -86,6 +86,7 @@
 updateOptions "$X509_USER_PROXY" "X509_USER_PROXY"
 updateOptions "$SWIFT_HOME" "COG_INSTALL_PATH"
 updateOptions "$SWIFT_HOME" "swift.home"
+updateOptions "$SWIFT_USERHOME" "user.home"
 #Use /dev/urandom instead of /dev/random for seeding RNGs
 #This will lower the randomness of the seed, but avoid
 #large delays if /dev/random does not have enough entropy collected



From davidk at ci.uchicago.edu  Mon Nov 14 00:56:31 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Mon, 14 Nov 2011 00:56:31 -0600 (CST)
Subject: [Swift-commit] r5286 - in branches/release-0.93: . libexec
Message-ID: <20111114065632.03D479CCC5@svn.ci.uchicago.edu>

Author: davidk
Date: 2011-11-14 00:56:30 -0600 (Mon, 14 Nov 2011)
New Revision: 5286

Added:
   branches/release-0.93/libexec/release.txt
Modified:
   branches/release-0.93/build.xml
   branches/release-0.93/libexec/svn-revision
Log:
Use release.txt for managing release name to be displayed/logged at Swift start-up


Modified: branches/release-0.93/build.xml
===================================================================
--- branches/release-0.93/build.xml	2011-11-14 02:29:22 UTC (rev 5285)
+++ branches/release-0.93/build.xml	2011-11-14 06:56:30 UTC (rev 5286)
@@ -385,7 +385,7 @@
 	</target>
 
 	<target name="generateVersion">
-		<echo file="libexec/version.txt">Swift ${version} </echo>
+		<echo file="libexec/version.txt">Swift </echo>
 		<exec os="Linux, Mac OS X" executable="libexec/svn-revision" append="true" output="libexec/version.txt"/>
 		<exec os="Linux, Mac OS X" executable="libexec/generate-buildid" append="false" output="libexec/buildid.txt"/>
 	</target>

Added: branches/release-0.93/libexec/release.txt
===================================================================
--- branches/release-0.93/libexec/release.txt	                        (rev 0)
+++ branches/release-0.93/libexec/release.txt	2011-11-14 06:56:30 UTC (rev 5286)
@@ -0,0 +1 @@
+0.93RC5

Modified: branches/release-0.93/libexec/svn-revision
===================================================================
--- branches/release-0.93/libexec/svn-revision	2011-11-14 02:29:22 UTC (rev 5285)
+++ branches/release-0.93/libexec/svn-revision	2011-11-14 06:56:30 UTC (rev 5286)
@@ -16,6 +16,7 @@
 }
 
 # Determine Swift revision
+RELEASE=$(cat libexec/release.txt)
 
 SWIFTVER=$(hereversion swift)
 
@@ -23,5 +24,5 @@
 
 COGVER=$(hereversion cog)
 
-echo $SWIFTVER $COGVER
+echo $RELEASE $SWIFTVER $COGVER
 



From davidk at ci.uchicago.edu  Mon Nov 14 01:14:38 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Mon, 14 Nov 2011 01:14:38 -0600 (CST)
Subject: [Swift-commit] r5287 - in trunk: . libexec
Message-ID: <20111114071438.BD75B9CCC5@svn.ci.uchicago.edu>

Author: davidk
Date: 2011-11-14 01:14:38 -0600 (Mon, 14 Nov 2011)
New Revision: 5287

Added:
   trunk/libexec/release.txt
Modified:
   trunk/build.xml
   trunk/libexec/svn-revision
   trunk/libexec/vdl.k
Log:
Use libexec/release.txt for managing release names


Modified: trunk/build.xml
===================================================================
--- trunk/build.xml	2011-11-14 06:56:30 UTC (rev 5286)
+++ trunk/build.xml	2011-11-14 07:14:38 UTC (rev 5287)
@@ -387,7 +387,7 @@
 	</target>
 
 	<target name="generateVersion">
-		<echo file="libexec/version.txt">Swift ${version} </echo>
+		<echo file="libexec/version.txt">Swift </echo>
 		<exec os="Linux, Mac OS X" executable="libexec/svn-revision" append="true" output="libexec/version.txt"/>
 		<exec os="Linux, Mac OS X" executable="libexec/generate-buildid" append="false" output="libexec/buildid.txt"/>
 	</target>

Added: trunk/libexec/release.txt
===================================================================
--- trunk/libexec/release.txt	                        (rev 0)
+++ trunk/libexec/release.txt	2011-11-14 07:14:38 UTC (rev 5287)
@@ -0,0 +1 @@
+trunk

Modified: trunk/libexec/svn-revision
===================================================================
--- trunk/libexec/svn-revision	2011-11-14 06:56:30 UTC (rev 5286)
+++ trunk/libexec/svn-revision	2011-11-14 07:14:38 UTC (rev 5287)
@@ -16,6 +16,7 @@
 }
 
 # Determine Swift revision
+RELEASE=$(cat libexec/release.txt)
 
 SWIFTVER=$(hereversion swift)
 
@@ -23,5 +24,5 @@
 
 COGVER=$(hereversion cog)
 
-echo $SWIFTVER $COGVER
+echo $RELEASE $SWIFTVER $COGVER
 

Modified: trunk/libexec/vdl.k
===================================================================
--- trunk/libexec/vdl.k	2011-11-14 06:56:30 UTC (rev 5286)
+++ trunk/libexec/vdl.k	2011-11-14 07:14:38 UTC (rev 5287)
@@ -19,9 +19,6 @@
 	once("vdl.k-print-version"
 		log("info",sys:file:read("{swift.home}/libexec/version.txt"))
 		echo(sys:file:read("{swift.home}/libexec/version.txt"))
-
-                log(LOG:INFO, "Swift version: TRUNK")
-
 		log("info","RUNID id=run:{VDL:RUNID}")
 		echo("RunID: {VDL:RUNID}")
 	)



From davidk at ci.uchicago.edu  Mon Nov 14 08:03:20 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Mon, 14 Nov 2011 08:03:20 -0600 (CST)
Subject: [Swift-commit] r5288 - in wwwdev: downloads inc main
Message-ID: <20111114140320.68EFF9CCC5@svn.ci.uchicago.edu>

Author: davidk
Date: 2011-11-14 08:03:20 -0600 (Mon, 14 Nov 2011)
New Revision: 5288

Modified:
   wwwdev/downloads/archive.php
   wwwdev/downloads/index.php
   wwwdev/inc/downloads_sidebar.php
   wwwdev/main/index.php
Log:
RC5


Modified: wwwdev/downloads/archive.php
===================================================================
--- wwwdev/downloads/archive.php	2011-11-14 07:14:38 UTC (rev 5287)
+++ wwwdev/downloads/archive.php	2011-11-14 14:03:20 UTC (rev 5288)
@@ -24,6 +24,14 @@
 <h2>Older releases</h2>
 <p>&nbsp;</p>
 
+<h3>Swift 0.93RC4 - 2011/11</h3>
+<p>Precompiled binary distribution
+[<a href="../packages/swift-0.93RC4.tar.gz"
+onClick="javascript: pageTracker._trackPageview('/swift/packages/swift-0.93RC4.tar.gz');"
+>swift-0.93RC3.tar.gz</a>]
+<p>&nbsp;</p>
+
+
 <h3>Swift 0.92.1 - 2011/04/13</h3>
 <p>Precompiled binary distribution
 [<a href="../packages/swift-0.92.1.tar.gz"

Modified: wwwdev/downloads/index.php
===================================================================
--- wwwdev/downloads/index.php	2011-11-14 07:14:38 UTC (rev 5287)
+++ wwwdev/downloads/index.php	2011-11-14 14:03:20 UTC (rev 5288)
@@ -25,15 +25,15 @@
 
 <h2>Latest Release</h2>
 <p>
-<p><b>Swift 0.93 RC3 - 2011/10/6</b><br><br>
+<p><b>Swift 0.93 RC5 - 2011/11/14</b><br><br>
 For the majority of users, this is the version you will want to download. This package
 contains the latest Stable release of Swift. Since Swift is written in Java, this
 package will run on all supported platforms with Java Runtime Environment 1.5 or greater.
 <br><br>
 Precompiled binary distribution
-[<a href="../packages/swift-0.93RC3.tar.gz"
-onClick="javascript: pageTracker._trackPageview('../packages/swift-0.93RC3.tar.gz');"
->swift-0.93.tar.gz</a>]
+[<a href="../packages/swift-0.93RC5.tar.gz"
+onClick="javascript: pageTracker._trackPageview('../packages/swift-0.93RC5.tar.gz');"
+>swift-0.93RC5.tar.gz</a>]
 
 
 <h2>Source Code</h2>

Modified: wwwdev/inc/downloads_sidebar.php
===================================================================
--- wwwdev/inc/downloads_sidebar.php	2011-11-14 07:14:38 UTC (rev 5287)
+++ wwwdev/inc/downloads_sidebar.php	2011-11-14 14:03:20 UTC (rev 5288)
@@ -1,7 +1,7 @@
 <div id="box-prime">
 <h2><a name="stable"></a>Latest Release</h2>
-<div class="button_175"><a href="../packages/swift-0.93RC4.tar.gz">Download Now</a></div>
-<b>0.93 RC4</b> current version<br />2011/11/9
+<div class="button_175"><a href="../packages/swift-0.93RC5.tar.gz">Download Now</a></div>
+<b>0.93 RC5</b> current version<br />2011/11/14
 <br />
 </div>
 <p>&nbsp;</p>

Modified: wwwdev/main/index.php
===================================================================
--- wwwdev/main/index.php	2011-11-14 07:14:38 UTC (rev 5287)
+++ wwwdev/main/index.php	2011-11-14 14:03:20 UTC (rev 5288)
@@ -62,7 +62,7 @@
   </div>
   <!-- end .grid_9 -->
   <div class="grid_3">
-    <div id="box-prime"><div class="button_175"><a href="../downloads" onClick="javascript: pageTracker._trackPageview('../downloads');">Download Now</a></div><b>0.93 RC4</b> current version<br />2011/11/09
+    <div id="box-prime"><div class="button_175"><a href="../downloads" onClick="javascript: pageTracker._trackPageview('../downloads');">Download Now</a></div><b>0.93 RC5</b> current version<br />2011/11/14
 	<br /></div>
 	<div id="box-prime"><div class="button_175"><a href="../guides/release-0.93/quickstart/quickstart.html">Get Started</a></div>Read our Quick Start Guide
 and start using Swift today!</div>



From ketan at ci.uchicago.edu  Mon Nov 14 14:07:14 2011
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Mon, 14 Nov 2011 14:07:14 -0600 (CST)
Subject: [Swift-commit] r5289 - in branches/release-0.93/tests/cdm: .
	absolute
Message-ID: <20111114200714.D33139CCC5@svn.ci.uchicago.edu>

Author: ketan
Date: 2011-11-14 14:07:14 -0600 (Mon, 14 Nov 2011)
New Revision: 5289

Added:
   branches/release-0.93/tests/cdm/absolute/
   branches/release-0.93/tests/cdm/absolute/README
   branches/release-0.93/tests/cdm/absolute/basic.swift
   branches/release-0.93/tests/cdm/absolute/catnap.sh
   branches/release-0.93/tests/cdm/absolute/cf.absolute
   branches/release-0.93/tests/cdm/absolute/cf.relative
   branches/release-0.93/tests/cdm/absolute/data1.txt
   branches/release-0.93/tests/cdm/absolute/data2.txt
   branches/release-0.93/tests/cdm/absolute/data3.txt
   branches/release-0.93/tests/cdm/absolute/data4.txt
   branches/release-0.93/tests/cdm/absolute/ff.kml
   branches/release-0.93/tests/cdm/absolute/ff.swift
   branches/release-0.93/tests/cdm/absolute/fr.kml
   branches/release-0.93/tests/cdm/absolute/fr.swift
   branches/release-0.93/tests/cdm/absolute/fs.data
   branches/release-0.93/tests/cdm/absolute/out1.data
   branches/release-0.93/tests/cdm/absolute/out2.data
   branches/release-0.93/tests/cdm/absolute/out3.data
   branches/release-0.93/tests/cdm/absolute/out4.data
   branches/release-0.93/tests/cdm/absolute/rf.kml
   branches/release-0.93/tests/cdm/absolute/rf.swift
   branches/release-0.93/tests/cdm/absolute/rr.kml
   branches/release-0.93/tests/cdm/absolute/rr.swift
   branches/release-0.93/tests/cdm/absolute/run.sh
   branches/release-0.93/tests/cdm/absolute/set-env.sh
   branches/release-0.93/tests/cdm/absolute/tc
Log:
Added test suite for CDM absolute path mode

Added: branches/release-0.93/tests/cdm/absolute/README
===================================================================
--- branches/release-0.93/tests/cdm/absolute/README	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/README	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1,23 @@
+This directory contains testcases to test the Swift CDM for the following cases:
+case 1. When input data is specified as relative or full path name.
+case 2. When the output data is specified as relative or full path name.
+case 3. When the config option wrapper.invocation.mode is set to 'absolute' or 'relative'
+
+The above cases lead to 8 possible combinations of input/output data specification and the wrapper.invocation.mode value.
+
+The naming of swift scripts are given based on how the data is specified:
+
+ff=both input and output data is specified as *fullpaths*
+rf=input data in relative path and output data in fullpath
+fr=input data in full path and output data in relative path
+rr=both input and output data in relative path
+
+Two different versions of config files represent the absolute and relative options for wrapper.invocation.mode:
+cf.absolute
+cf.relative
+
+A run.sh file contains the commandlines to run all the combinations.
+
+You will need to adapt the absolute pathnames in the swift sources based on where you run.
+
+The CDM policy file is fs.data.

Added: branches/release-0.93/tests/cdm/absolute/basic.swift
===================================================================
--- branches/release-0.93/tests/cdm/absolute/basic.swift	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/basic.swift	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1,18 @@
+
+type file;
+
+app (file o) catnap (string delay, file i)
+{
+ catnap delay @i stdout=@o;
+}
+
+string delay=@arg("s","1");
+
+file out[]<array_mapper; files=["/home/ketan/out1.data","/home/ketan/out2.data","out3.data","out4.data"]>;
+
+file data[]<array_mapper; files=["/home/ketan/data1.txt","data2.txt","/home/ketan/data4.txt","data4.txt"]>;
+
+foreach o, j in out {
+ out[j] = catnap(delay,data[j]);
+}
+

Added: branches/release-0.93/tests/cdm/absolute/catnap.sh
===================================================================
--- branches/release-0.93/tests/cdm/absolute/catnap.sh	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/catnap.sh	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1,4 @@
+#! /bin/sh
+sleep $1
+shift
+cat $*


Property changes on: branches/release-0.93/tests/cdm/absolute/catnap.sh
___________________________________________________________________
Added: svn:executable
   + *

Added: branches/release-0.93/tests/cdm/absolute/cf.absolute
===================================================================
--- branches/release-0.93/tests/cdm/absolute/cf.absolute	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/cf.absolute	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1,8 @@
+wrapperlog.always.transfer=true
+sitedir.keep=true
+execution.retries=0
+lazy.errors=false
+status.mode=provider
+use.provider.staging=false
+provider.staging.pin.swiftfiles=false
+wrapper.invocation.mode=absolute

Added: branches/release-0.93/tests/cdm/absolute/cf.relative
===================================================================
--- branches/release-0.93/tests/cdm/absolute/cf.relative	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/cf.relative	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1,8 @@
+wrapperlog.always.transfer=true
+sitedir.keep=true
+execution.retries=0
+lazy.errors=false
+status.mode=provider
+use.provider.staging=false
+provider.staging.pin.swiftfiles=false
+wrapper.invocation.mode=relative

Added: branches/release-0.93/tests/cdm/absolute/data1.txt
===================================================================
--- branches/release-0.93/tests/cdm/absolute/data1.txt	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/data1.txt	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1 @@
+hello from data1

Added: branches/release-0.93/tests/cdm/absolute/data2.txt
===================================================================
--- branches/release-0.93/tests/cdm/absolute/data2.txt	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/data2.txt	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1 @@
+hello from data2

Added: branches/release-0.93/tests/cdm/absolute/data3.txt
===================================================================
--- branches/release-0.93/tests/cdm/absolute/data3.txt	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/data3.txt	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1 @@
+hello from data3

Added: branches/release-0.93/tests/cdm/absolute/data4.txt
===================================================================
--- branches/release-0.93/tests/cdm/absolute/data4.txt	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/data4.txt	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1 @@
+hello from data4

Added: branches/release-0.93/tests/cdm/absolute/ff.kml
===================================================================
--- branches/release-0.93/tests/cdm/absolute/ff.kml	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/ff.kml	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1,223 @@
+<project><!-- CACHE ID 4ce39a0c-5c18-4b47-8d49-6688cfc1844b -->
+  <import file="sys.xml"/>
+  <import file="scheduler.xml"/>
+  <import file="rlog.xml"/>
+  <import file="vdl.k"/>
+  <types>
+     <xs:schema targetNamespace="http://ci.uchicago.edu/swift/2009/02/swiftscript" xmlns="http://ci.uchicago.edu/swift/2009/02/swiftscript" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xs="http://www.w3.org/2001/XMLSchema">
+      	   <xs:simpleType name="file">
+      	   <xs:restriction base="string"/>
+      	   </xs:simpleType>  
+     </xs:schema>	 
+  </types>
+  <global name="swift#string#17001">
+    <vdl:new type="string" value="1" />
+  </global>
+  <global name="swift#string#17004">
+    <vdl:new type="string" value="/home/ketan/cdm_tests/out2.data" />
+  </global>
+  <global name="swift#string#17008">
+    <vdl:new type="string" value="/home/ketan/cdm_tests/data1.txt" />
+  </global>
+  <global name="swift#string#17000">
+    <vdl:new type="string" value="s" />
+  </global>
+  <global name="swift#string#17005">
+    <vdl:new type="string" value="/home/ketan/cdm_tests/out3.data" />
+  </global>
+  <global name="swift#string#17006">
+    <vdl:new type="string" value="/home/ketan/cdm_tests/out4.data" />
+  </global>
+  <global name="swift#string#17003">
+    <vdl:new type="string" value="/home/ketan/cdm_tests/out1.data" />
+  </global>
+  <global name="swift#string#17009">
+    <vdl:new type="string" value="/home/ketan/cdm_tests/data2.txt" />
+  </global>
+  <global name="swift#string#17010">
+    <vdl:new type="string" value="/home/ketan/cdm_tests/data4.txt" />
+  </global>
+  <element name="catnap" arguments="o,_delay,i">
+    <parameterlog>
+    <string>input</string>
+    <string>_delay</string>
+    <vdl:getdatasetprovenanceid var="{_delay}" />
+    <string>{#thread}</string>
+    </parameterlog>
+    <parameterlog>
+    <string>input</string>
+    <string>i</string>
+    <vdl:getdatasetprovenanceid var="{i}" />
+    <string>{#thread}</string>
+    </parameterlog>
+    <parameterlog>
+    <string>output</string>
+    <string>o</string>
+    <vdl:getdatasetprovenanceid var="{o}" />
+    <string>{#thread}</string>
+    </parameterlog>
+    <log level="debug" message="PROCEDURE line=4 thread={#thread} name=catnap"/>
+    <vdl:execute>
+      <vdl:tr>catnap</vdl:tr>
+      <vdl:stagein var="{_delay}"/>
+      <vdl:stagein var="{i}"/>
+      <vdl:stageout var="{o}"/>
+      <vdl:arguments>
+        <variable>_delay</variable>
+        <swiftscript:filename>
+         <variable>i</variable> 
+        </swiftscript:filename>
+      </vdl:arguments>
+      <vdl:stdout>
+        <swiftscript:filename>
+         <variable>o</variable> 
+        </swiftscript:filename>
+      </vdl:stdout>
+    </vdl:execute>
+    <vdl:closedataset var="{o}"/>
+    <log level="debug" message="PROCEDURE_END line=4"/></element>
+
+  <set name="delay">
+        <vdl:new type="string" dbgname="delay" waitfor="88000" />
+  </set>
+  <parameterlog>
+  <string>intermediate</string>
+  <string>delay</string>
+  <vdl:getdatasetprovenanceid var="{delay}" />
+  <string>{#thread}</string>
+  </parameterlog>
+  <set name="swift#mapper#17002">
+        <vdl:new type="string[]" dbgname="swift#mapper#17002" waitfor="" />
+  </set>
+  <parameterlog>
+  <string>intermediate</string>
+  <string>swift#mapper#17002</string>
+  <vdl:getdatasetprovenanceid var="{swift#mapper#17002}" />
+  <string>{#thread}</string>
+  </parameterlog>
+  <set name="out">
+    <vdl:new type="file[]" dbgname="out" waitfor="88002">
+      <vdl:mapping descriptor="array_mapper">
+        <vdl:parameter name="files"><variable>swift#mapper#17002</variable></vdl:parameter>
+      </vdl:mapping>
+    </vdl:new>
+  </set>
+  <parameterlog>
+  <string>intermediate</string>
+  <string>out</string>
+  <vdl:getdatasetprovenanceid var="{out}" />
+  <string>{#thread}</string>
+  </parameterlog>
+  <set name="swift#mapper#17007">
+        <vdl:new type="string[]" dbgname="swift#mapper#17007" waitfor="" />
+  </set>
+  <parameterlog>
+  <string>intermediate</string>
+  <string>swift#mapper#17007</string>
+  <vdl:getdatasetprovenanceid var="{swift#mapper#17007}" />
+  <string>{#thread}</string>
+  </parameterlog>
+  <set name="data">
+    <vdl:new type="file[]" dbgname="data">
+      <vdl:mapping descriptor="array_mapper">
+        <vdl:parameter name="files"><variable>swift#mapper#17007</variable></vdl:parameter>
+        <vdl:parameter name="input" value="true" />
+      </vdl:mapping>
+    </vdl:new>
+  </set>
+  <parameterlog>
+  <string>intermediate</string>
+  <string>data</string>
+  <vdl:getdatasetprovenanceid var="{data}" />
+  <string>{#thread}</string>
+  </parameterlog>
+  <restartLog>
+  	<vdl:mains>
+		<vdl:startprogressticker />
+		<vdl:mainp>
+		    <parallel>
+		        <sequential>
+		             <vdl:setfieldvalue>
+		               <argument name="var">
+		                 <variable>delay</variable>
+		               </argument>
+		               <argument name="value">
+		                 <swiftscript:arg>
+		                  <variable>swift#string#17000</variable><variable>swift#string#17001</variable> 
+		                 </swiftscript:arg>
+		               </argument>
+		             </vdl:setfieldvalue>
+		            <partialCloseDataset var="{delay}" closeID="88000" />
+		        </sequential>
+		        <sequential>
+		             <vdl:setfieldvalue>
+		               <argument name="var">
+		                 <variable>swift#mapper#17002</variable>
+		               </argument>
+		               <argument name="value">
+		                 <vdl:createarray>
+		                   <list>
+		                     <variable>swift#string#17003</variable>
+		                     <variable>swift#string#17004</variable>
+		                     <variable>swift#string#17005</variable>
+		                     <variable>swift#string#17006</variable>
+		                   </list>
+		                 </vdl:createarray>
+		               </argument>
+		             </vdl:setfieldvalue>
+		        </sequential>
+		        <sequential>
+		             <vdl:setfieldvalue>
+		               <argument name="var">
+		                 <variable>swift#mapper#17007</variable>
+		               </argument>
+		               <argument name="value">
+		                 <vdl:createarray>
+		                   <list>
+		                     <variable>swift#string#17008</variable>
+		                     <variable>swift#string#17009</variable>
+		                     <variable>swift#string#17010</variable>
+		                     <variable>swift#string#17010</variable>
+		                   </list>
+		                 </vdl:createarray>
+		               </argument>
+		             </vdl:setfieldvalue>
+		        </sequential>
+		        <sequential>
+		          <vdl:tparallelFor name="$" selfClose="true">
+		            <getarrayiterator><variable>out</variable></getarrayiterator>
+		            <set names="$$, o">
+		              <each items="{$}"/>
+		            </set>
+		              <set name="j">
+		                <vdl:new type="int" value="{$$}"/>
+		              </set>	<log level="debug" message="FOREACH_IT_START line=15 thread={#thread}"/>
+		          <log level="debug"><string>SCOPE thread={#thread}</string></log>
+
+		                <sequentialWithID>
+		                    <sequential>
+		                      <catnap>
+		                        <parallel>
+		                          <vdl:getfieldsubscript>
+		                            <argument name="var"><variable>out</variable></argument>
+		                            <argument name="subscript"><variable>j</variable></argument>
+		                          </vdl:getfieldsubscript>
+		                          <variable>delay</variable>
+		                          <vdl:getfieldsubscript>
+		                            <argument name="var"><variable>data</variable></argument>
+		                            <argument name="subscript"><variable>j</variable></argument>
+		                          </vdl:getfieldsubscript>
+		                        </parallel>
+		                      </catnap>
+		                    </sequential>
+		                </sequentialWithID>
+		              <log level="debug" message="FOREACH_IT_END line=15 thread={#thread}"/>
+		          </vdl:tparallelFor>
+		            <partialCloseDataset var="{out}" closeID="88002" />
+		        </sequential>
+		    </parallel>
+		</vdl:mainp>
+		<vdl:stopprogressticker />
+	</vdl:mains>
+  </restartLog>  
+</project>

Added: branches/release-0.93/tests/cdm/absolute/ff.swift
===================================================================
--- branches/release-0.93/tests/cdm/absolute/ff.swift	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/ff.swift	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1,18 @@
+
+type file;
+
+app (file o) catnap (string _delay, file i)
+{
+ catnap _delay @i stdout=@o;
+}
+
+string delay=@arg("s","1");
+
+file out[]<array_mapper; files=["/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/out1.data","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/out2.data","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/out3.data","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/out4.data"]>;
+
+file data[]<array_mapper; files=["/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/data1.txt","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/data2.txt","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/data4.txt","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/data4.txt"]>;
+
+foreach o, j in out {
+ out[j] = catnap(delay,data[j]);
+}
+

Added: branches/release-0.93/tests/cdm/absolute/fr.kml
===================================================================
--- branches/release-0.93/tests/cdm/absolute/fr.kml	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/fr.kml	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1,223 @@
+<project><!-- CACHE ID 4ce39a0c-5c18-4b47-8d49-6688cfc1844b -->
+  <import file="sys.xml"/>
+  <import file="scheduler.xml"/>
+  <import file="rlog.xml"/>
+  <import file="vdl.k"/>
+  <types>
+     <xs:schema targetNamespace="http://ci.uchicago.edu/swift/2009/02/swiftscript" xmlns="http://ci.uchicago.edu/swift/2009/02/swiftscript" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xs="http://www.w3.org/2001/XMLSchema">
+      	   <xs:simpleType name="file">
+      	   <xs:restriction base="string"/>
+      	   </xs:simpleType>  
+     </xs:schema>	 
+  </types>
+  <global name="swift#string#17001">
+    <vdl:new type="string" value="1" />
+  </global>
+  <global name="swift#string#17008">
+    <vdl:new type="string" value="/home/ketan/cdm_tests/data1.txt" />
+  </global>
+  <global name="swift#string#17003">
+    <vdl:new type="string" value="out1.data" />
+  </global>
+  <global name="swift#string#17000">
+    <vdl:new type="string" value="s" />
+  </global>
+  <global name="swift#string#17004">
+    <vdl:new type="string" value="out2.data" />
+  </global>
+  <global name="swift#string#17009">
+    <vdl:new type="string" value="/home/ketan/cdm_tests/data2.txt" />
+  </global>
+  <global name="swift#string#17010">
+    <vdl:new type="string" value="/home/ketan/cdm_tests/data4.txt" />
+  </global>
+  <global name="swift#string#17006">
+    <vdl:new type="string" value="out4.data" />
+  </global>
+  <global name="swift#string#17005">
+    <vdl:new type="string" value="out3.data" />
+  </global>
+  <element name="catnap" arguments="o,delay,i">
+    <parameterlog>
+    <string>input</string>
+    <string>delay</string>
+    <vdl:getdatasetprovenanceid var="{delay}" />
+    <string>{#thread}</string>
+    </parameterlog>
+    <parameterlog>
+    <string>input</string>
+    <string>i</string>
+    <vdl:getdatasetprovenanceid var="{i}" />
+    <string>{#thread}</string>
+    </parameterlog>
+    <parameterlog>
+    <string>output</string>
+    <string>o</string>
+    <vdl:getdatasetprovenanceid var="{o}" />
+    <string>{#thread}</string>
+    </parameterlog>
+    <log level="debug" message="PROCEDURE line=4 thread={#thread} name=catnap"/>
+    <vdl:execute>
+      <vdl:tr>catnap</vdl:tr>
+      <vdl:stagein var="{delay}"/>
+      <vdl:stagein var="{i}"/>
+      <vdl:stageout var="{o}"/>
+      <vdl:arguments>
+        <variable>delay</variable>
+        <swiftscript:filename>
+         <variable>i</variable> 
+        </swiftscript:filename>
+      </vdl:arguments>
+      <vdl:stdout>
+        <swiftscript:filename>
+         <variable>o</variable> 
+        </swiftscript:filename>
+      </vdl:stdout>
+    </vdl:execute>
+    <vdl:closedataset var="{o}"/>
+    <log level="debug" message="PROCEDURE_END line=4"/></element>
+
+  <set name="delay">
+        <vdl:new type="string" dbgname="delay" waitfor="88000" />
+  </set>
+  <parameterlog>
+  <string>intermediate</string>
+  <string>delay</string>
+  <vdl:getdatasetprovenanceid var="{delay}" />
+  <string>{#thread}</string>
+  </parameterlog>
+  <set name="swift#mapper#17002">
+        <vdl:new type="string[]" dbgname="swift#mapper#17002" waitfor="" />
+  </set>
+  <parameterlog>
+  <string>intermediate</string>
+  <string>swift#mapper#17002</string>
+  <vdl:getdatasetprovenanceid var="{swift#mapper#17002}" />
+  <string>{#thread}</string>
+  </parameterlog>
+  <set name="out">
+    <vdl:new type="file[]" dbgname="out" waitfor="88002">
+      <vdl:mapping descriptor="array_mapper">
+        <vdl:parameter name="files"><variable>swift#mapper#17002</variable></vdl:parameter>
+      </vdl:mapping>
+    </vdl:new>
+  </set>
+  <parameterlog>
+  <string>intermediate</string>
+  <string>out</string>
+  <vdl:getdatasetprovenanceid var="{out}" />
+  <string>{#thread}</string>
+  </parameterlog>
+  <set name="swift#mapper#17007">
+        <vdl:new type="string[]" dbgname="swift#mapper#17007" waitfor="" />
+  </set>
+  <parameterlog>
+  <string>intermediate</string>
+  <string>swift#mapper#17007</string>
+  <vdl:getdatasetprovenanceid var="{swift#mapper#17007}" />
+  <string>{#thread}</string>
+  </parameterlog>
+  <set name="data">
+    <vdl:new type="file[]" dbgname="data">
+      <vdl:mapping descriptor="array_mapper">
+        <vdl:parameter name="files"><variable>swift#mapper#17007</variable></vdl:parameter>
+        <vdl:parameter name="input" value="true" />
+      </vdl:mapping>
+    </vdl:new>
+  </set>
+  <parameterlog>
+  <string>intermediate</string>
+  <string>data</string>
+  <vdl:getdatasetprovenanceid var="{data}" />
+  <string>{#thread}</string>
+  </parameterlog>
+  <restartLog>
+  	<vdl:mains>
+		<vdl:startprogressticker />
+		<vdl:mainp>
+		    <parallel>
+		        <sequential>
+		             <vdl:setfieldvalue>
+		               <argument name="var">
+		                 <variable>delay</variable>
+		               </argument>
+		               <argument name="value">
+		                 <swiftscript:arg>
+		                  <variable>swift#string#17000</variable><variable>swift#string#17001</variable> 
+		                 </swiftscript:arg>
+		               </argument>
+		             </vdl:setfieldvalue>
+		            <partialCloseDataset var="{delay}" closeID="88000" />
+		        </sequential>
+		        <sequential>
+		             <vdl:setfieldvalue>
+		               <argument name="var">
+		                 <variable>swift#mapper#17002</variable>
+		               </argument>
+		               <argument name="value">
+		                 <vdl:createarray>
+		                   <list>
+		                     <variable>swift#string#17003</variable>
+		                     <variable>swift#string#17004</variable>
+		                     <variable>swift#string#17005</variable>
+		                     <variable>swift#string#17006</variable>
+		                   </list>
+		                 </vdl:createarray>
+		               </argument>
+		             </vdl:setfieldvalue>
+		        </sequential>
+		        <sequential>
+		             <vdl:setfieldvalue>
+		               <argument name="var">
+		                 <variable>swift#mapper#17007</variable>
+		               </argument>
+		               <argument name="value">
+		                 <vdl:createarray>
+		                   <list>
+		                     <variable>swift#string#17008</variable>
+		                     <variable>swift#string#17009</variable>
+		                     <variable>swift#string#17010</variable>
+		                     <variable>swift#string#17010</variable>
+		                   </list>
+		                 </vdl:createarray>
+		               </argument>
+		             </vdl:setfieldvalue>
+		        </sequential>
+		        <sequential>
+		          <vdl:tparallelFor name="$" selfClose="true">
+		            <getarrayiterator><variable>out</variable></getarrayiterator>
+		            <set names="$$, o">
+		              <each items="{$}"/>
+		            </set>
+		              <set name="j">
+		                <vdl:new type="int" value="{$$}"/>
+		              </set>	<log level="debug" message="FOREACH_IT_START line=15 thread={#thread}"/>
+		          <log level="debug"><string>SCOPE thread={#thread}</string></log>
+
+		                <sequentialWithID>
+		                    <sequential>
+		                      <catnap>
+		                        <parallel>
+		                          <vdl:getfieldsubscript>
+		                            <argument name="var"><variable>out</variable></argument>
+		                            <argument name="subscript"><variable>j</variable></argument>
+		                          </vdl:getfieldsubscript>
+		                          <variable>delay</variable>
+		                          <vdl:getfieldsubscript>
+		                            <argument name="var"><variable>data</variable></argument>
+		                            <argument name="subscript"><variable>j</variable></argument>
+		                          </vdl:getfieldsubscript>
+		                        </parallel>
+		                      </catnap>
+		                    </sequential>
+		                </sequentialWithID>
+		              <log level="debug" message="FOREACH_IT_END line=15 thread={#thread}"/>
+		          </vdl:tparallelFor>
+		            <partialCloseDataset var="{out}" closeID="88002" />
+		        </sequential>
+		    </parallel>
+		</vdl:mainp>
+		<vdl:stopprogressticker />
+	</vdl:mains>
+  </restartLog>  
+</project>

Added: branches/release-0.93/tests/cdm/absolute/fr.swift
===================================================================
--- branches/release-0.93/tests/cdm/absolute/fr.swift	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/fr.swift	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1,18 @@
+
+type file;
+
+app (file o) catnap (string delay, file i)
+{
+ catnap delay @i stdout=@o;
+}
+
+string delay=@arg("s","1");
+
+file out[]<array_mapper; files=["out1.data","out2.data","out3.data","out4.data"]>;
+
+file data[]<array_mapper; files=["/home/ketan/cdm_tests/data1.txt","/home/ketan/cdm_tests/data2.txt","/home/ketan/cdm_tests/data4.txt","/home/ketan/cdm_tests/data4.txt"]>;
+
+foreach o, j in out {
+ out[j] = catnap(delay,data[j]);
+}
+

Added: branches/release-0.93/tests/cdm/absolute/fs.data
===================================================================
--- branches/release-0.93/tests/cdm/absolute/fs.data	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/fs.data	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1 @@
+rule \./outdir/.* DIRECT /home/ketan/cdm_test

Added: branches/release-0.93/tests/cdm/absolute/out1.data
===================================================================
--- branches/release-0.93/tests/cdm/absolute/out1.data	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/out1.data	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1 @@
+hello from data1

Added: branches/release-0.93/tests/cdm/absolute/out2.data
===================================================================
--- branches/release-0.93/tests/cdm/absolute/out2.data	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/out2.data	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1 @@
+hello from data2

Added: branches/release-0.93/tests/cdm/absolute/out3.data
===================================================================
--- branches/release-0.93/tests/cdm/absolute/out3.data	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/out3.data	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1 @@
+hello from data4

Added: branches/release-0.93/tests/cdm/absolute/out4.data
===================================================================
--- branches/release-0.93/tests/cdm/absolute/out4.data	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/out4.data	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1 @@
+hello from data4

Added: branches/release-0.93/tests/cdm/absolute/rf.kml
===================================================================
--- branches/release-0.93/tests/cdm/absolute/rf.kml	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/rf.kml	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1,223 @@
+<project><!-- CACHE ID 4ce39a0c-5c18-4b47-8d49-6688cfc1844b -->
+  <import file="sys.xml"/>
+  <import file="scheduler.xml"/>
+  <import file="rlog.xml"/>
+  <import file="vdl.k"/>
+  <types>
+     <xs:schema targetNamespace="http://ci.uchicago.edu/swift/2009/02/swiftscript" xmlns="http://ci.uchicago.edu/swift/2009/02/swiftscript" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xs="http://www.w3.org/2001/XMLSchema">
+      	   <xs:simpleType name="file">
+      	   <xs:restriction base="string"/>
+      	   </xs:simpleType>  
+     </xs:schema>	 
+  </types>
+  <global name="swift#string#17008">
+    <vdl:new type="string" value="data1.txt" />
+  </global>
+  <global name="swift#string#17001">
+    <vdl:new type="string" value="1" />
+  </global>
+  <global name="swift#string#17004">
+    <vdl:new type="string" value="/home/ketan/cdm_tests/out2.data" />
+  </global>
+  <global name="swift#string#17000">
+    <vdl:new type="string" value="s" />
+  </global>
+  <global name="swift#string#17005">
+    <vdl:new type="string" value="/home/ketan/cdm_tests/out3.data" />
+  </global>
+  <global name="swift#string#17009">
+    <vdl:new type="string" value="data2.txt" />
+  </global>
+  <global name="swift#string#17006">
+    <vdl:new type="string" value="/home/ketan/cdm_tests/out4.data" />
+  </global>
+  <global name="swift#string#17003">
+    <vdl:new type="string" value="/home/ketan/cdm_tests/out1.data" />
+  </global>
+  <global name="swift#string#17010">
+    <vdl:new type="string" value="data4.txt" />
+  </global>
+  <element name="catnap" arguments="o,delay,i">
+    <parameterlog>
+    <string>input</string>
+    <string>delay</string>
+    <vdl:getdatasetprovenanceid var="{delay}" />
+    <string>{#thread}</string>
+    </parameterlog>
+    <parameterlog>
+    <string>input</string>
+    <string>i</string>
+    <vdl:getdatasetprovenanceid var="{i}" />
+    <string>{#thread}</string>
+    </parameterlog>
+    <parameterlog>
+    <string>output</string>
+    <string>o</string>
+    <vdl:getdatasetprovenanceid var="{o}" />
+    <string>{#thread}</string>
+    </parameterlog>
+    <log level="debug" message="PROCEDURE line=4 thread={#thread} name=catnap"/>
+    <vdl:execute>
+      <vdl:tr>catnap</vdl:tr>
+      <vdl:stagein var="{delay}"/>
+      <vdl:stagein var="{i}"/>
+      <vdl:stageout var="{o}"/>
+      <vdl:arguments>
+        <variable>delay</variable>
+        <swiftscript:filename>
+         <variable>i</variable> 
+        </swiftscript:filename>
+      </vdl:arguments>
+      <vdl:stdout>
+        <swiftscript:filename>
+         <variable>o</variable> 
+        </swiftscript:filename>
+      </vdl:stdout>
+    </vdl:execute>
+    <vdl:closedataset var="{o}"/>
+    <log level="debug" message="PROCEDURE_END line=4"/></element>
+
+  <set name="delay">
+        <vdl:new type="string" dbgname="delay" waitfor="88000" />
+  </set>
+  <parameterlog>
+  <string>intermediate</string>
+  <string>delay</string>
+  <vdl:getdatasetprovenanceid var="{delay}" />
+  <string>{#thread}</string>
+  </parameterlog>
+  <set name="swift#mapper#17002">
+        <vdl:new type="string[]" dbgname="swift#mapper#17002" waitfor="" />
+  </set>
+  <parameterlog>
+  <string>intermediate</string>
+  <string>swift#mapper#17002</string>
+  <vdl:getdatasetprovenanceid var="{swift#mapper#17002}" />
+  <string>{#thread}</string>
+  </parameterlog>
+  <set name="out">
+    <vdl:new type="file[]" dbgname="out" waitfor="88002">
+      <vdl:mapping descriptor="array_mapper">
+        <vdl:parameter name="files"><variable>swift#mapper#17002</variable></vdl:parameter>
+      </vdl:mapping>
+    </vdl:new>
+  </set>
+  <parameterlog>
+  <string>intermediate</string>
+  <string>out</string>
+  <vdl:getdatasetprovenanceid var="{out}" />
+  <string>{#thread}</string>
+  </parameterlog>
+  <set name="swift#mapper#17007">
+        <vdl:new type="string[]" dbgname="swift#mapper#17007" waitfor="" />
+  </set>
+  <parameterlog>
+  <string>intermediate</string>
+  <string>swift#mapper#17007</string>
+  <vdl:getdatasetprovenanceid var="{swift#mapper#17007}" />
+  <string>{#thread}</string>
+  </parameterlog>
+  <set name="data">
+    <vdl:new type="file[]" dbgname="data">
+      <vdl:mapping descriptor="array_mapper">
+        <vdl:parameter name="files"><variable>swift#mapper#17007</variable></vdl:parameter>
+        <vdl:parameter name="input" value="true" />
+      </vdl:mapping>
+    </vdl:new>
+  </set>
+  <parameterlog>
+  <string>intermediate</string>
+  <string>data</string>
+  <vdl:getdatasetprovenanceid var="{data}" />
+  <string>{#thread}</string>
+  </parameterlog>
+  <restartLog>
+  	<vdl:mains>
+		<vdl:startprogressticker />
+		<vdl:mainp>
+		    <parallel>
+		        <sequential>
+		             <vdl:setfieldvalue>
+		               <argument name="var">
+		                 <variable>delay</variable>
+		               </argument>
+		               <argument name="value">
+		                 <swiftscript:arg>
+		                  <variable>swift#string#17000</variable><variable>swift#string#17001</variable> 
+		                 </swiftscript:arg>
+		               </argument>
+		             </vdl:setfieldvalue>
+		            <partialCloseDataset var="{delay}" closeID="88000" />
+		        </sequential>
+		        <sequential>
+		             <vdl:setfieldvalue>
+		               <argument name="var">
+		                 <variable>swift#mapper#17002</variable>
+		               </argument>
+		               <argument name="value">
+		                 <vdl:createarray>
+		                   <list>
+		                     <variable>swift#string#17003</variable>
+		                     <variable>swift#string#17004</variable>
+		                     <variable>swift#string#17005</variable>
+		                     <variable>swift#string#17006</variable>
+		                   </list>
+		                 </vdl:createarray>
+		               </argument>
+		             </vdl:setfieldvalue>
+		        </sequential>
+		        <sequential>
+		             <vdl:setfieldvalue>
+		               <argument name="var">
+		                 <variable>swift#mapper#17007</variable>
+		               </argument>
+		               <argument name="value">
+		                 <vdl:createarray>
+		                   <list>
+		                     <variable>swift#string#17008</variable>
+		                     <variable>swift#string#17009</variable>
+		                     <variable>swift#string#17010</variable>
+		                     <variable>swift#string#17010</variable>
+		                   </list>
+		                 </vdl:createarray>
+		               </argument>
+		             </vdl:setfieldvalue>
+		        </sequential>
+		        <sequential>
+		          <vdl:tparallelFor name="$" selfClose="true">
+		            <getarrayiterator><variable>out</variable></getarrayiterator>
+		            <set names="$$, o">
+		              <each items="{$}"/>
+		            </set>
+		              <set name="j">
+		                <vdl:new type="int" value="{$$}"/>
+		              </set>	<log level="debug" message="FOREACH_IT_START line=15 thread={#thread}"/>
+		          <log level="debug"><string>SCOPE thread={#thread}</string></log>
+
+		                <sequentialWithID>
+		                    <sequential>
+		                      <catnap>
+		                        <parallel>
+		                          <vdl:getfieldsubscript>
+		                            <argument name="var"><variable>out</variable></argument>
+		                            <argument name="subscript"><variable>j</variable></argument>
+		                          </vdl:getfieldsubscript>
+		                          <variable>delay</variable>
+		                          <vdl:getfieldsubscript>
+		                            <argument name="var"><variable>data</variable></argument>
+		                            <argument name="subscript"><variable>j</variable></argument>
+		                          </vdl:getfieldsubscript>
+		                        </parallel>
+		                      </catnap>
+		                    </sequential>
+		                </sequentialWithID>
+		              <log level="debug" message="FOREACH_IT_END line=15 thread={#thread}"/>
+		          </vdl:tparallelFor>
+		            <partialCloseDataset var="{out}" closeID="88002" />
+		        </sequential>
+		    </parallel>
+		</vdl:mainp>
+		<vdl:stopprogressticker />
+	</vdl:mains>
+  </restartLog>  
+</project>

Added: branches/release-0.93/tests/cdm/absolute/rf.swift
===================================================================
--- branches/release-0.93/tests/cdm/absolute/rf.swift	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/rf.swift	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1,18 @@
+
+type file;
+
+app (file o) catnap (string delay, file i)
+{
+ catnap delay @i stdout=@o;
+}
+
+string delay=@arg("s","1");
+
+file out[]<array_mapper; files=["/home/ketan/cdm_tests/out1.data","/home/ketan/cdm_tests/out2.data","/home/ketan/cdm_tests/out3.data","/home/ketan/cdm_tests/out4.data"]>;
+
+file data[]<array_mapper; files=["data1.txt","data2.txt","data4.txt","data4.txt"]>;
+
+foreach o, j in out {
+ out[j] = catnap(delay,data[j]);
+}
+

Added: branches/release-0.93/tests/cdm/absolute/rr.kml
===================================================================
--- branches/release-0.93/tests/cdm/absolute/rr.kml	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/rr.kml	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1,223 @@
+<project><!-- CACHE ID 4ce39a0c-5c18-4b47-8d49-6688cfc1844b -->
+  <import file="sys.xml"/>
+  <import file="scheduler.xml"/>
+  <import file="rlog.xml"/>
+  <import file="vdl.k"/>
+  <types>
+     <xs:schema targetNamespace="http://ci.uchicago.edu/swift/2009/02/swiftscript" xmlns="http://ci.uchicago.edu/swift/2009/02/swiftscript" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xs="http://www.w3.org/2001/XMLSchema">
+      	   <xs:simpleType name="file">
+      	   <xs:restriction base="string"/>
+      	   </xs:simpleType>  
+     </xs:schema>	 
+  </types>
+  <global name="swift#string#17008">
+    <vdl:new type="string" value="data1.txt" />
+  </global>
+  <global name="swift#string#17001">
+    <vdl:new type="string" value="1" />
+  </global>
+  <global name="swift#string#17003">
+    <vdl:new type="string" value="out1.data" />
+  </global>
+  <global name="swift#string#17000">
+    <vdl:new type="string" value="s" />
+  </global>
+  <global name="swift#string#17009">
+    <vdl:new type="string" value="data2.txt" />
+  </global>
+  <global name="swift#string#17004">
+    <vdl:new type="string" value="out2.data" />
+  </global>
+  <global name="swift#string#17006">
+    <vdl:new type="string" value="out4.data" />
+  </global>
+  <global name="swift#string#17010">
+    <vdl:new type="string" value="data4.txt" />
+  </global>
+  <global name="swift#string#17005">
+    <vdl:new type="string" value="out3.data" />
+  </global>
+  <element name="catnap" arguments="o,delay,i">
+    <parameterlog>
+    <string>input</string>
+    <string>delay</string>
+    <vdl:getdatasetprovenanceid var="{delay}" />
+    <string>{#thread}</string>
+    </parameterlog>
+    <parameterlog>
+    <string>input</string>
+    <string>i</string>
+    <vdl:getdatasetprovenanceid var="{i}" />
+    <string>{#thread}</string>
+    </parameterlog>
+    <parameterlog>
+    <string>output</string>
+    <string>o</string>
+    <vdl:getdatasetprovenanceid var="{o}" />
+    <string>{#thread}</string>
+    </parameterlog>
+    <log level="debug" message="PROCEDURE line=4 thread={#thread} name=catnap"/>
+    <vdl:execute>
+      <vdl:tr>catnap</vdl:tr>
+      <vdl:stagein var="{delay}"/>
+      <vdl:stagein var="{i}"/>
+      <vdl:stageout var="{o}"/>
+      <vdl:arguments>
+        <variable>delay</variable>
+        <swiftscript:filename>
+         <variable>i</variable> 
+        </swiftscript:filename>
+      </vdl:arguments>
+      <vdl:stdout>
+        <swiftscript:filename>
+         <variable>o</variable> 
+        </swiftscript:filename>
+      </vdl:stdout>
+    </vdl:execute>
+    <vdl:closedataset var="{o}"/>
+    <log level="debug" message="PROCEDURE_END line=4"/></element>
+
+  <set name="delay">
+        <vdl:new type="string" dbgname="delay" waitfor="88000" />
+  </set>
+  <parameterlog>
+  <string>intermediate</string>
+  <string>delay</string>
+  <vdl:getdatasetprovenanceid var="{delay}" />
+  <string>{#thread}</string>
+  </parameterlog>
+  <set name="swift#mapper#17002">
+        <vdl:new type="string[]" dbgname="swift#mapper#17002" waitfor="" />
+  </set>
+  <parameterlog>
+  <string>intermediate</string>
+  <string>swift#mapper#17002</string>
+  <vdl:getdatasetprovenanceid var="{swift#mapper#17002}" />
+  <string>{#thread}</string>
+  </parameterlog>
+  <set name="out">
+    <vdl:new type="file[]" dbgname="out" waitfor="88002">
+      <vdl:mapping descriptor="array_mapper">
+        <vdl:parameter name="files"><variable>swift#mapper#17002</variable></vdl:parameter>
+      </vdl:mapping>
+    </vdl:new>
+  </set>
+  <parameterlog>
+  <string>intermediate</string>
+  <string>out</string>
+  <vdl:getdatasetprovenanceid var="{out}" />
+  <string>{#thread}</string>
+  </parameterlog>
+  <set name="swift#mapper#17007">
+        <vdl:new type="string[]" dbgname="swift#mapper#17007" waitfor="" />
+  </set>
+  <parameterlog>
+  <string>intermediate</string>
+  <string>swift#mapper#17007</string>
+  <vdl:getdatasetprovenanceid var="{swift#mapper#17007}" />
+  <string>{#thread}</string>
+  </parameterlog>
+  <set name="data">
+    <vdl:new type="file[]" dbgname="data">
+      <vdl:mapping descriptor="array_mapper">
+        <vdl:parameter name="files"><variable>swift#mapper#17007</variable></vdl:parameter>
+        <vdl:parameter name="input" value="true" />
+      </vdl:mapping>
+    </vdl:new>
+  </set>
+  <parameterlog>
+  <string>intermediate</string>
+  <string>data</string>
+  <vdl:getdatasetprovenanceid var="{data}" />
+  <string>{#thread}</string>
+  </parameterlog>
+  <restartLog>
+  	<vdl:mains>
+		<vdl:startprogressticker />
+		<vdl:mainp>
+		    <parallel>
+		        <sequential>
+		             <vdl:setfieldvalue>
+		               <argument name="var">
+		                 <variable>delay</variable>
+		               </argument>
+		               <argument name="value">
+		                 <swiftscript:arg>
+		                  <variable>swift#string#17000</variable><variable>swift#string#17001</variable> 
+		                 </swiftscript:arg>
+		               </argument>
+		             </vdl:setfieldvalue>
+		            <partialCloseDataset var="{delay}" closeID="88000" />
+		        </sequential>
+		        <sequential>
+		             <vdl:setfieldvalue>
+		               <argument name="var">
+		                 <variable>swift#mapper#17002</variable>
+		               </argument>
+		               <argument name="value">
+		                 <vdl:createarray>
+		                   <list>
+		                     <variable>swift#string#17003</variable>
+		                     <variable>swift#string#17004</variable>
+		                     <variable>swift#string#17005</variable>
+		                     <variable>swift#string#17006</variable>
+		                   </list>
+		                 </vdl:createarray>
+		               </argument>
+		             </vdl:setfieldvalue>
+		        </sequential>
+		        <sequential>
+		             <vdl:setfieldvalue>
+		               <argument name="var">
+		                 <variable>swift#mapper#17007</variable>
+		               </argument>
+		               <argument name="value">
+		                 <vdl:createarray>
+		                   <list>
+		                     <variable>swift#string#17008</variable>
+		                     <variable>swift#string#17009</variable>
+		                     <variable>swift#string#17010</variable>
+		                     <variable>swift#string#17010</variable>
+		                   </list>
+		                 </vdl:createarray>
+		               </argument>
+		             </vdl:setfieldvalue>
+		        </sequential>
+		        <sequential>
+		          <vdl:tparallelFor name="$" selfClose="true">
+		            <getarrayiterator><variable>out</variable></getarrayiterator>
+		            <set names="$$, o">
+		              <each items="{$}"/>
+		            </set>
+		              <set name="j">
+		                <vdl:new type="int" value="{$$}"/>
+		              </set>	<log level="debug" message="FOREACH_IT_START line=15 thread={#thread}"/>
+		          <log level="debug"><string>SCOPE thread={#thread}</string></log>
+
+		                <sequentialWithID>
+		                    <sequential>
+		                      <catnap>
+		                        <parallel>
+		                          <vdl:getfieldsubscript>
+		                            <argument name="var"><variable>out</variable></argument>
+		                            <argument name="subscript"><variable>j</variable></argument>
+		                          </vdl:getfieldsubscript>
+		                          <variable>delay</variable>
+		                          <vdl:getfieldsubscript>
+		                            <argument name="var"><variable>data</variable></argument>
+		                            <argument name="subscript"><variable>j</variable></argument>
+		                          </vdl:getfieldsubscript>
+		                        </parallel>
+		                      </catnap>
+		                    </sequential>
+		                </sequentialWithID>
+		              <log level="debug" message="FOREACH_IT_END line=15 thread={#thread}"/>
+		          </vdl:tparallelFor>
+		            <partialCloseDataset var="{out}" closeID="88002" />
+		        </sequential>
+		    </parallel>
+		</vdl:mainp>
+		<vdl:stopprogressticker />
+	</vdl:mains>
+  </restartLog>  
+</project>

Added: branches/release-0.93/tests/cdm/absolute/rr.swift
===================================================================
--- branches/release-0.93/tests/cdm/absolute/rr.swift	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/rr.swift	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1,18 @@
+
+type file;
+
+app (file o) catnap (string delay, file i)
+{
+ catnap delay @i stdout=@o;
+}
+
+string delay=@arg("s","1");
+
+file out[]<array_mapper; files=["out1.data","out2.data","out3.data","out4.data"]>;
+
+file data[]<array_mapper; files=["data1.txt","data2.txt","data4.txt","data4.txt"]>;
+
+foreach o, j in out {
+ out[j] = catnap(delay,data[j]);
+}
+

Added: branches/release-0.93/tests/cdm/absolute/run.sh
===================================================================
--- branches/release-0.93/tests/cdm/absolute/run.sh	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/run.sh	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1,16 @@
+swift -config cf.absolute -sites.file sites.xml -tc.file tc -cdm.file fs.ff -runid ff_config_abs  ff.swift
+
+swift -config cf.absolute -sites.file sites.xml -tc.file tc -cdm.file fs.ff -runid fr_config_abs  fr.swift
+
+swift -config cf.absolute -sites.file sites.xml -tc.file tc -cdm.file fs.ff -runid rf_config_abs rf.swift
+
+swift -config cf.absolute -sites.file sites.xml -tc.file tc -cdm.file fs.ff -runid rr_config_abs rr.swift
+
+swift -config cf.relative -sites.file sites.xml -tc.file tc -cdm.file fs.ff -runid ff_config_rel ff.swift
+
+swift -config cf.relative -sites.file sites.xml -tc.file tc -cdm.file fs.ff -runid fr_config_rel fr.swift
+
+swift -config cf.relative -sites.file sites.xml -tc.file tc -cdm.file fs.ff -runid rf_config_rel rf.swift
+
+swift -config cf.relative -sites.file sites.xml -tc.file tc -cdm.file fs.ff -runid rr_config_rel rr.swift
+

Added: branches/release-0.93/tests/cdm/absolute/set-env.sh
===================================================================
--- branches/release-0.93/tests/cdm/absolute/set-env.sh	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/set-env.sh	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1 @@
+export PATH=/home/ketan/swift-install/trunk/cog/modules/swift/bin/grid:/home/ketan/swift-install/0.93/cog/modules/swift/dist/swift-svn/bin:$PATH

Added: branches/release-0.93/tests/cdm/absolute/tc
===================================================================
--- branches/release-0.93/tests/cdm/absolute/tc	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/tc	2011-11-14 20:07:14 UTC (rev 5289)
@@ -0,0 +1,6 @@
+localhost sh /bin/sh null null null
+localhost cat /bin/cat null null null
+pbs cat /bin/cat null null null
+mcs cat /bin/cat null null null
+localhost catnap /home/ketan/cdm_tests/catnap.sh null null GLOBUS::maxwalltime="00:01:00"
+



From ketan at ci.uchicago.edu  Mon Nov 14 14:18:29 2011
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Mon, 14 Nov 2011 14:18:29 -0600 (CST)
Subject: [Swift-commit] r5290 - in branches/release-0.93/tests/cdm/absolute:
	. indir
Message-ID: <20111114201829.5FFDC9CCC5@svn.ci.uchicago.edu>

Author: ketan
Date: 2011-11-14 14:18:29 -0600 (Mon, 14 Nov 2011)
New Revision: 5290

Added:
   branches/release-0.93/tests/cdm/absolute/indir/
   branches/release-0.93/tests/cdm/absolute/indir/data1.txt
   branches/release-0.93/tests/cdm/absolute/indir/data2.txt
   branches/release-0.93/tests/cdm/absolute/indir/data3.txt
   branches/release-0.93/tests/cdm/absolute/indir/data4.txt
   branches/release-0.93/tests/cdm/absolute/outdir/
Modified:
   branches/release-0.93/tests/cdm/absolute/fs.data
Log:
 

Modified: branches/release-0.93/tests/cdm/absolute/fs.data
===================================================================
--- branches/release-0.93/tests/cdm/absolute/fs.data	2011-11-14 20:07:14 UTC (rev 5289)
+++ branches/release-0.93/tests/cdm/absolute/fs.data	2011-11-14 20:18:29 UTC (rev 5290)
@@ -1 +1,2 @@
-rule \./outdir/.* DIRECT /home/ketan/cdm_test
+rule \./outdir/.* DIRECT /home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute
+rule \./indir/.* DIRECT /home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute

Added: branches/release-0.93/tests/cdm/absolute/indir/data1.txt
===================================================================
--- branches/release-0.93/tests/cdm/absolute/indir/data1.txt	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/indir/data1.txt	2011-11-14 20:18:29 UTC (rev 5290)
@@ -0,0 +1 @@
+hello from data1

Added: branches/release-0.93/tests/cdm/absolute/indir/data2.txt
===================================================================
--- branches/release-0.93/tests/cdm/absolute/indir/data2.txt	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/indir/data2.txt	2011-11-14 20:18:29 UTC (rev 5290)
@@ -0,0 +1 @@
+hello from data2

Added: branches/release-0.93/tests/cdm/absolute/indir/data3.txt
===================================================================
--- branches/release-0.93/tests/cdm/absolute/indir/data3.txt	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/indir/data3.txt	2011-11-14 20:18:29 UTC (rev 5290)
@@ -0,0 +1 @@
+hello from data3

Added: branches/release-0.93/tests/cdm/absolute/indir/data4.txt
===================================================================
--- branches/release-0.93/tests/cdm/absolute/indir/data4.txt	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/indir/data4.txt	2011-11-14 20:18:29 UTC (rev 5290)
@@ -0,0 +1 @@
+hello from data4



From ketan at ci.uchicago.edu  Mon Nov 14 14:25:58 2011
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Mon, 14 Nov 2011 14:25:58 -0600 (CST)
Subject: [Swift-commit] r5291 - branches/release-0.93/tests/cdm/absolute
Message-ID: <20111114202558.256B79CCC5@svn.ci.uchicago.edu>

Author: ketan
Date: 2011-11-14 14:25:58 -0600 (Mon, 14 Nov 2011)
New Revision: 5291

Modified:
   branches/release-0.93/tests/cdm/absolute/ff.swift
   branches/release-0.93/tests/cdm/absolute/fr.swift
   branches/release-0.93/tests/cdm/absolute/rf.swift
   branches/release-0.93/tests/cdm/absolute/rr.swift
Log:
improved tests

Modified: branches/release-0.93/tests/cdm/absolute/ff.swift
===================================================================
--- branches/release-0.93/tests/cdm/absolute/ff.swift	2011-11-14 20:18:29 UTC (rev 5290)
+++ branches/release-0.93/tests/cdm/absolute/ff.swift	2011-11-14 20:25:58 UTC (rev 5291)
@@ -8,9 +8,9 @@
 
 string delay=@arg("s","1");
 
-file out[]<array_mapper; files=["/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/out1.data","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/out2.data","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/out3.data","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/out4.data"]>;
+file out[]<array_mapper; files=["/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/outdir/out1.data","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/outdir/out2.data","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/outdir/out3.data","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/outdir/out4.data"]>;
 
-file data[]<array_mapper; files=["/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/data1.txt","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/data2.txt","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/data4.txt","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/data4.txt"]>;
+file data[]<array_mapper; files=["/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/indir/data1.txt","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/indir/data2.txt","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/indir/data4.txt","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/indir/data4.txt"]>;
 
 foreach o, j in out {
  out[j] = catnap(delay,data[j]);

Modified: branches/release-0.93/tests/cdm/absolute/fr.swift
===================================================================
--- branches/release-0.93/tests/cdm/absolute/fr.swift	2011-11-14 20:18:29 UTC (rev 5290)
+++ branches/release-0.93/tests/cdm/absolute/fr.swift	2011-11-14 20:25:58 UTC (rev 5291)
@@ -1,16 +1,16 @@
 
 type file;
 
-app (file o) catnap (string delay, file i)
+app (file o) catnap (string _delay, file i)
 {
- catnap delay @i stdout=@o;
+ catnap _delay @i stdout=@o;
 }
 
 string delay=@arg("s","1");
 
-file out[]<array_mapper; files=["out1.data","out2.data","out3.data","out4.data"]>;
+file out[]<array_mapper; files=["./outdir/out1.data","./outdir/out2.data","./outdir/out3.data","./outdir/out4.data"]>;
 
-file data[]<array_mapper; files=["/home/ketan/cdm_tests/data1.txt","/home/ketan/cdm_tests/data2.txt","/home/ketan/cdm_tests/data4.txt","/home/ketan/cdm_tests/data4.txt"]>;
+file data[]<array_mapper; files=["/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/indir/data1.txt","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/indir/data2.txt","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/indir/data4.txt","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/indir/data4.txt"]>;
 
 foreach o, j in out {
  out[j] = catnap(delay,data[j]);

Modified: branches/release-0.93/tests/cdm/absolute/rf.swift
===================================================================
--- branches/release-0.93/tests/cdm/absolute/rf.swift	2011-11-14 20:18:29 UTC (rev 5290)
+++ branches/release-0.93/tests/cdm/absolute/rf.swift	2011-11-14 20:25:58 UTC (rev 5291)
@@ -1,16 +1,16 @@
 
 type file;
 
-app (file o) catnap (string delay, file i)
+app (file o) catnap (string _delay, file i)
 {
- catnap delay @i stdout=@o;
+ catnap _delay @i stdout=@o;
 }
 
 string delay=@arg("s","1");
 
-file out[]<array_mapper; files=["/home/ketan/cdm_tests/out1.data","/home/ketan/cdm_tests/out2.data","/home/ketan/cdm_tests/out3.data","/home/ketan/cdm_tests/out4.data"]>;
+file out[]<array_mapper; files=["/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/outdir/out1.data","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/outdir/out2.data","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/outdir/out3.data","/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/outdir/out4.data"]>;
 
-file data[]<array_mapper; files=["data1.txt","data2.txt","data4.txt","data4.txt"]>;
+file data[]<array_mapper; files=["./indir/data1.txt","./indir/data2.txt","./indir/data4.txt","./indir/data4.txt"]>;
 
 foreach o, j in out {
  out[j] = catnap(delay,data[j]);

Modified: branches/release-0.93/tests/cdm/absolute/rr.swift
===================================================================
--- branches/release-0.93/tests/cdm/absolute/rr.swift	2011-11-14 20:18:29 UTC (rev 5290)
+++ branches/release-0.93/tests/cdm/absolute/rr.swift	2011-11-14 20:25:58 UTC (rev 5291)
@@ -1,16 +1,16 @@
 
 type file;
 
-app (file o) catnap (string delay, file i)
+app (file o) catnap (string _delay, file i)
 {
- catnap delay @i stdout=@o;
+ catnap _delay @i stdout=@o;
 }
 
 string delay=@arg("s","1");
 
-file out[]<array_mapper; files=["out1.data","out2.data","out3.data","out4.data"]>;
+file out[]<array_mapper; files=["./outdir/out1.data","./outdir/out2.data","./outdir/out3.data","./outdir/out4.data"]>;
 
-file data[]<array_mapper; files=["data1.txt","data2.txt","data4.txt","data4.txt"]>;
+file data[]<array_mapper; files=["./indir/data1.txt","./indir/data2.txt","./indir/data4.txt","./indir/data4.txt"]>;
 
 foreach o, j in out {
  out[j] = catnap(delay,data[j]);



From ketan at ci.uchicago.edu  Mon Nov 14 14:27:14 2011
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Mon, 14 Nov 2011 14:27:14 -0600 (CST)
Subject: [Swift-commit] r5292 - branches/release-0.93/tests/cdm/absolute
Message-ID: <20111114202714.9A8A49CCC5@svn.ci.uchicago.edu>

Author: ketan
Date: 2011-11-14 14:27:14 -0600 (Mon, 14 Nov 2011)
New Revision: 5292

Added:
   branches/release-0.93/tests/cdm/absolute/sites.xml
Log:
added missing sites.xml

Added: branches/release-0.93/tests/cdm/absolute/sites.xml
===================================================================
--- branches/release-0.93/tests/cdm/absolute/sites.xml	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/sites.xml	2011-11-14 20:27:14 UTC (rev 5292)
@@ -0,0 +1,7 @@
+<config>
+  <pool handle="localhost" sysinfo="INTEL32::LINUX">
+    <gridftp url="local://localhost" />
+    <execution provider="local" url="none" />
+    <workdirectory>work/</workdirectory>
+  </pool>
+</config>



From ketan at ci.uchicago.edu  Mon Nov 14 14:31:32 2011
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Mon, 14 Nov 2011 14:31:32 -0600 (CST)
Subject: [Swift-commit] r5293 - branches/release-0.93/tests/cdm/absolute
Message-ID: <20111114203132.97C089CCC5@svn.ci.uchicago.edu>

Author: ketan
Date: 2011-11-14 14:31:32 -0600 (Mon, 14 Nov 2011)
New Revision: 5293

Modified:
   branches/release-0.93/tests/cdm/absolute/ff.kml
   branches/release-0.93/tests/cdm/absolute/fr.kml
   branches/release-0.93/tests/cdm/absolute/rf.kml
   branches/release-0.93/tests/cdm/absolute/rr.kml
   branches/release-0.93/tests/cdm/absolute/run.sh
Log:
added rule for out

Modified: branches/release-0.93/tests/cdm/absolute/ff.kml
===================================================================
--- branches/release-0.93/tests/cdm/absolute/ff.kml	2011-11-14 20:27:14 UTC (rev 5292)
+++ branches/release-0.93/tests/cdm/absolute/ff.kml	2011-11-14 20:31:32 UTC (rev 5293)
@@ -14,29 +14,29 @@
     <vdl:new type="string" value="1" />
   </global>
   <global name="swift#string#17004">
-    <vdl:new type="string" value="/home/ketan/cdm_tests/out2.data" />
+    <vdl:new type="string" value="/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/outdir/out2.data" />
   </global>
-  <global name="swift#string#17008">
-    <vdl:new type="string" value="/home/ketan/cdm_tests/data1.txt" />
-  </global>
   <global name="swift#string#17000">
     <vdl:new type="string" value="s" />
   </global>
-  <global name="swift#string#17005">
-    <vdl:new type="string" value="/home/ketan/cdm_tests/out3.data" />
+  <global name="swift#string#17010">
+    <vdl:new type="string" value="/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/indir/data4.txt" />
   </global>
   <global name="swift#string#17006">
-    <vdl:new type="string" value="/home/ketan/cdm_tests/out4.data" />
+    <vdl:new type="string" value="/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/outdir/out4.data" />
   </global>
-  <global name="swift#string#17003">
-    <vdl:new type="string" value="/home/ketan/cdm_tests/out1.data" />
-  </global>
   <global name="swift#string#17009">
-    <vdl:new type="string" value="/home/ketan/cdm_tests/data2.txt" />
+    <vdl:new type="string" value="/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/indir/data2.txt" />
   </global>
-  <global name="swift#string#17010">
-    <vdl:new type="string" value="/home/ketan/cdm_tests/data4.txt" />
+  <global name="swift#string#17008">
+    <vdl:new type="string" value="/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/indir/data1.txt" />
   </global>
+  <global name="swift#string#17005">
+    <vdl:new type="string" value="/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/outdir/out3.data" />
+  </global>
+  <global name="swift#string#17003">
+    <vdl:new type="string" value="/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/outdir/out1.data" />
+  </global>
   <element name="catnap" arguments="o,_delay,i">
     <parameterlog>
     <string>input</string>

Modified: branches/release-0.93/tests/cdm/absolute/fr.kml
===================================================================
--- branches/release-0.93/tests/cdm/absolute/fr.kml	2011-11-14 20:27:14 UTC (rev 5292)
+++ branches/release-0.93/tests/cdm/absolute/fr.kml	2011-11-14 20:31:32 UTC (rev 5293)
@@ -13,35 +13,35 @@
   <global name="swift#string#17001">
     <vdl:new type="string" value="1" />
   </global>
-  <global name="swift#string#17008">
-    <vdl:new type="string" value="/home/ketan/cdm_tests/data1.txt" />
-  </global>
-  <global name="swift#string#17003">
-    <vdl:new type="string" value="out1.data" />
-  </global>
   <global name="swift#string#17000">
     <vdl:new type="string" value="s" />
   </global>
-  <global name="swift#string#17004">
-    <vdl:new type="string" value="out2.data" />
-  </global>
-  <global name="swift#string#17009">
-    <vdl:new type="string" value="/home/ketan/cdm_tests/data2.txt" />
-  </global>
   <global name="swift#string#17010">
-    <vdl:new type="string" value="/home/ketan/cdm_tests/data4.txt" />
+    <vdl:new type="string" value="/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/indir/data4.txt" />
   </global>
   <global name="swift#string#17006">
-    <vdl:new type="string" value="out4.data" />
+    <vdl:new type="string" value="./outdir/out4.data" />
   </global>
   <global name="swift#string#17005">
-    <vdl:new type="string" value="out3.data" />
+    <vdl:new type="string" value="./outdir/out3.data" />
   </global>
-  <element name="catnap" arguments="o,delay,i">
+  <global name="swift#string#17009">
+    <vdl:new type="string" value="/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/indir/data2.txt" />
+  </global>
+  <global name="swift#string#17008">
+    <vdl:new type="string" value="/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/indir/data1.txt" />
+  </global>
+  <global name="swift#string#17004">
+    <vdl:new type="string" value="./outdir/out2.data" />
+  </global>
+  <global name="swift#string#17003">
+    <vdl:new type="string" value="./outdir/out1.data" />
+  </global>
+  <element name="catnap" arguments="o,_delay,i">
     <parameterlog>
     <string>input</string>
-    <string>delay</string>
-    <vdl:getdatasetprovenanceid var="{delay}" />
+    <string>_delay</string>
+    <vdl:getdatasetprovenanceid var="{_delay}" />
     <string>{#thread}</string>
     </parameterlog>
     <parameterlog>
@@ -59,11 +59,11 @@
     <log level="debug" message="PROCEDURE line=4 thread={#thread} name=catnap"/>
     <vdl:execute>
       <vdl:tr>catnap</vdl:tr>
-      <vdl:stagein var="{delay}"/>
+      <vdl:stagein var="{_delay}"/>
       <vdl:stagein var="{i}"/>
       <vdl:stageout var="{o}"/>
       <vdl:arguments>
-        <variable>delay</variable>
+        <variable>_delay</variable>
         <swiftscript:filename>
          <variable>i</variable> 
         </swiftscript:filename>

Modified: branches/release-0.93/tests/cdm/absolute/rf.kml
===================================================================
--- branches/release-0.93/tests/cdm/absolute/rf.kml	2011-11-14 20:27:14 UTC (rev 5292)
+++ branches/release-0.93/tests/cdm/absolute/rf.kml	2011-11-14 20:31:32 UTC (rev 5293)
@@ -11,37 +11,37 @@
      </xs:schema>	 
   </types>
   <global name="swift#string#17008">
-    <vdl:new type="string" value="data1.txt" />
+    <vdl:new type="string" value="./indir/data1.txt" />
   </global>
   <global name="swift#string#17001">
     <vdl:new type="string" value="1" />
   </global>
   <global name="swift#string#17004">
-    <vdl:new type="string" value="/home/ketan/cdm_tests/out2.data" />
+    <vdl:new type="string" value="/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/outdir/out2.data" />
   </global>
   <global name="swift#string#17000">
     <vdl:new type="string" value="s" />
   </global>
-  <global name="swift#string#17005">
-    <vdl:new type="string" value="/home/ketan/cdm_tests/out3.data" />
+  <global name="swift#string#17010">
+    <vdl:new type="string" value="./indir/data4.txt" />
   </global>
   <global name="swift#string#17009">
-    <vdl:new type="string" value="data2.txt" />
+    <vdl:new type="string" value="./indir/data2.txt" />
   </global>
   <global name="swift#string#17006">
-    <vdl:new type="string" value="/home/ketan/cdm_tests/out4.data" />
+    <vdl:new type="string" value="/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/outdir/out4.data" />
   </global>
+  <global name="swift#string#17005">
+    <vdl:new type="string" value="/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/outdir/out3.data" />
+  </global>
   <global name="swift#string#17003">
-    <vdl:new type="string" value="/home/ketan/cdm_tests/out1.data" />
+    <vdl:new type="string" value="/home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute/outdir/out1.data" />
   </global>
-  <global name="swift#string#17010">
-    <vdl:new type="string" value="data4.txt" />
-  </global>
-  <element name="catnap" arguments="o,delay,i">
+  <element name="catnap" arguments="o,_delay,i">
     <parameterlog>
     <string>input</string>
-    <string>delay</string>
-    <vdl:getdatasetprovenanceid var="{delay}" />
+    <string>_delay</string>
+    <vdl:getdatasetprovenanceid var="{_delay}" />
     <string>{#thread}</string>
     </parameterlog>
     <parameterlog>
@@ -59,11 +59,11 @@
     <log level="debug" message="PROCEDURE line=4 thread={#thread} name=catnap"/>
     <vdl:execute>
       <vdl:tr>catnap</vdl:tr>
-      <vdl:stagein var="{delay}"/>
+      <vdl:stagein var="{_delay}"/>
       <vdl:stagein var="{i}"/>
       <vdl:stageout var="{o}"/>
       <vdl:arguments>
-        <variable>delay</variable>
+        <variable>_delay</variable>
         <swiftscript:filename>
          <variable>i</variable> 
         </swiftscript:filename>

Modified: branches/release-0.93/tests/cdm/absolute/rr.kml
===================================================================
--- branches/release-0.93/tests/cdm/absolute/rr.kml	2011-11-14 20:27:14 UTC (rev 5292)
+++ branches/release-0.93/tests/cdm/absolute/rr.kml	2011-11-14 20:31:32 UTC (rev 5293)
@@ -11,37 +11,37 @@
      </xs:schema>	 
   </types>
   <global name="swift#string#17008">
-    <vdl:new type="string" value="data1.txt" />
+    <vdl:new type="string" value="./indir/data1.txt" />
   </global>
   <global name="swift#string#17001">
     <vdl:new type="string" value="1" />
   </global>
-  <global name="swift#string#17003">
-    <vdl:new type="string" value="out1.data" />
-  </global>
   <global name="swift#string#17000">
     <vdl:new type="string" value="s" />
   </global>
-  <global name="swift#string#17009">
-    <vdl:new type="string" value="data2.txt" />
+  <global name="swift#string#17010">
+    <vdl:new type="string" value="./indir/data4.txt" />
   </global>
-  <global name="swift#string#17004">
-    <vdl:new type="string" value="out2.data" />
-  </global>
   <global name="swift#string#17006">
-    <vdl:new type="string" value="out4.data" />
+    <vdl:new type="string" value="./outdir/out4.data" />
   </global>
-  <global name="swift#string#17010">
-    <vdl:new type="string" value="data4.txt" />
+  <global name="swift#string#17009">
+    <vdl:new type="string" value="./indir/data2.txt" />
   </global>
   <global name="swift#string#17005">
-    <vdl:new type="string" value="out3.data" />
+    <vdl:new type="string" value="./outdir/out3.data" />
   </global>
-  <element name="catnap" arguments="o,delay,i">
+  <global name="swift#string#17004">
+    <vdl:new type="string" value="./outdir/out2.data" />
+  </global>
+  <global name="swift#string#17003">
+    <vdl:new type="string" value="./outdir/out1.data" />
+  </global>
+  <element name="catnap" arguments="o,_delay,i">
     <parameterlog>
     <string>input</string>
-    <string>delay</string>
-    <vdl:getdatasetprovenanceid var="{delay}" />
+    <string>_delay</string>
+    <vdl:getdatasetprovenanceid var="{_delay}" />
     <string>{#thread}</string>
     </parameterlog>
     <parameterlog>
@@ -59,11 +59,11 @@
     <log level="debug" message="PROCEDURE line=4 thread={#thread} name=catnap"/>
     <vdl:execute>
       <vdl:tr>catnap</vdl:tr>
-      <vdl:stagein var="{delay}"/>
+      <vdl:stagein var="{_delay}"/>
       <vdl:stagein var="{i}"/>
       <vdl:stageout var="{o}"/>
       <vdl:arguments>
-        <variable>delay</variable>
+        <variable>_delay</variable>
         <swiftscript:filename>
          <variable>i</variable> 
         </swiftscript:filename>

Modified: branches/release-0.93/tests/cdm/absolute/run.sh
===================================================================
--- branches/release-0.93/tests/cdm/absolute/run.sh	2011-11-14 20:27:14 UTC (rev 5292)
+++ branches/release-0.93/tests/cdm/absolute/run.sh	2011-11-14 20:31:32 UTC (rev 5293)
@@ -1,16 +1,16 @@
-swift -config cf.absolute -sites.file sites.xml -tc.file tc -cdm.file fs.ff -runid ff_config_abs  ff.swift
+swift -config cf.absolute -sites.file sites.xml -tc.file tc -cdm.file fs.data -runid ff_config_abs  ff.swift
 
-swift -config cf.absolute -sites.file sites.xml -tc.file tc -cdm.file fs.ff -runid fr_config_abs  fr.swift
+swift -config cf.absolute -sites.file sites.xml -tc.file tc -cdm.file fs.data -runid fr_config_abs  fr.swift
 
-swift -config cf.absolute -sites.file sites.xml -tc.file tc -cdm.file fs.ff -runid rf_config_abs rf.swift
+swift -config cf.absolute -sites.file sites.xml -tc.file tc -cdm.file fs.data -runid rf_config_abs rf.swift
 
-swift -config cf.absolute -sites.file sites.xml -tc.file tc -cdm.file fs.ff -runid rr_config_abs rr.swift
+swift -config cf.absolute -sites.file sites.xml -tc.file tc -cdm.file fs.data -runid rr_config_abs rr.swift
 
-swift -config cf.relative -sites.file sites.xml -tc.file tc -cdm.file fs.ff -runid ff_config_rel ff.swift
+swift -config cf.relative -sites.file sites.xml -tc.file tc -cdm.file fs.data -runid ff_config_rel ff.swift
 
-swift -config cf.relative -sites.file sites.xml -tc.file tc -cdm.file fs.ff -runid fr_config_rel fr.swift
+swift -config cf.relative -sites.file sites.xml -tc.file tc -cdm.file fs.data -runid fr_config_rel fr.swift
 
-swift -config cf.relative -sites.file sites.xml -tc.file tc -cdm.file fs.ff -runid rf_config_rel rf.swift
+swift -config cf.relative -sites.file sites.xml -tc.file tc -cdm.file fs.data -runid rf_config_rel rf.swift
 
-swift -config cf.relative -sites.file sites.xml -tc.file tc -cdm.file fs.ff -runid rr_config_rel rr.swift
+swift -config cf.relative -sites.file sites.xml -tc.file tc -cdm.file fs.data -runid rr_config_rel rr.swift
 



From ketan at ci.uchicago.edu  Mon Nov 14 14:53:50 2011
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Mon, 14 Nov 2011 14:53:50 -0600 (CST)
Subject: [Swift-commit] r5294 - branches/release-0.93/tests/cdm/absolute
Message-ID: <20111114205350.1A8769CCC5@svn.ci.uchicago.edu>

Author: ketan
Date: 2011-11-14 14:53:49 -0600 (Mon, 14 Nov 2011)
New Revision: 5294

Modified:
   branches/release-0.93/tests/cdm/absolute/fs.data
Log:
 

Modified: branches/release-0.93/tests/cdm/absolute/fs.data
===================================================================
--- branches/release-0.93/tests/cdm/absolute/fs.data	2011-11-14 20:31:32 UTC (rev 5293)
+++ branches/release-0.93/tests/cdm/absolute/fs.data	2011-11-14 20:53:49 UTC (rev 5294)
@@ -1,2 +1 @@
-rule \./outdir/.* DIRECT /home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute
 rule \./indir/.* DIRECT /home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute



From wozniak at ci.uchicago.edu  Mon Nov 14 16:20:55 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Mon, 14 Nov 2011 16:20:55 -0600 (CST)
Subject: [Swift-commit] r5295 - branches/release-0.93/tests
Message-ID: <20111114222055.86C699CCC5@svn.ci.uchicago.edu>

Author: wozniak
Date: 2011-11-14 16:20:55 -0600 (Mon, 14 Nov 2011)
New Revision: 5295

Modified:
   branches/release-0.93/tests/suite.sh
Log:
Use $0 to get the suite directory; whitespace


Modified: branches/release-0.93/tests/suite.sh
===================================================================
--- branches/release-0.93/tests/suite.sh	2011-11-14 20:53:49 UTC (rev 5294)
+++ branches/release-0.93/tests/suite.sh	2011-11-14 22:20:55 UTC (rev 5295)
@@ -34,9 +34,11 @@
 TOTAL_TIME=0
 INDIVIDUAL_TEST_TIME=0
 COLORIZE=0
+# The directory in which is suite.sh (this script)
+SUITE_DIR=$( dirname $0 )
 # The directory in which to start:
-TOPDIR=`readlink -f $PWD/../../../..`
-CRTDIR=`pwd`
+TOPDIR=$( readlink -f $SUITE_DIR/../../../.. )
+CRTDIR=$( /bin/pwd )
 
 # Disable usage stats in test suite
 export SWIFT_USAGE_STATS=0
@@ -357,7 +359,7 @@
 	LABEL="$2"  # Text on link to output
 	CMD=$3    # Command issued (td title)
 	RESULT=$4 # Passed or Failed
- 
+
 	if [ $TEXTREPORT == 1 ]; then
 		if [ "$TYPE" == "test" ]; then
 			if [ "$RESULT" == "Passed" ]; then
@@ -697,7 +699,7 @@
 
   process_exec $SCRIPT
   RESULT=$( result )
-   
+
   output_report test "$SYMBOL" "$LASTCMD" $RESULT
 
   check_bailout
@@ -707,7 +709,7 @@
 	GROUP=$1
 	NAME=$2
 	RESULT="None"
-	
+
         if [ -f "$GROUP/$NAME.in" ]; then
                 echo "Copying input: $NAME.in"
 		cp -v $GROUP/$NAME.in . 2>&1 >> $OUTPUT
@@ -818,7 +820,7 @@
   TEST_SHOULD_FAIL=0
   OUTPUT=$NAME.check.stdout
   if [ -x $GROUP/$CHECKSCRIPT ]; then
-    cp "$GROUP/$CHECKSCRIPT" .	
+    cp "$GROUP/$CHECKSCRIPT" .
     export TEST_LOG=$NAME.stdout
     script_exec ./$CHECKSCRIPT "&#8730;"
   else
@@ -848,7 +850,7 @@
   TIMEOUTFILE=$NAME.timeout
 
   TEST_SHOULD_FAIL=0
-  
+
   OUTPUT=$NAME.setup.stdout
   if [ -x $GROUP/$SETUPSCRIPT ]; then
     script_exec $GROUP/$SETUPSCRIPT "S"
@@ -962,7 +964,7 @@
   else
      TEMPLATE="$TESTDIR/sites/localhost.xml"
   fi
-  
+
   # Give default to _WORK_ if undefined in swift.properties
   if [ -z "$WORK" ]
   then
@@ -1072,7 +1074,7 @@
   checkfail "Could not list: $GROUP"
 
   for TEST in $SWIFTS; do
-   
+
     (( SKIP_COUNTER++ < SKIP_TESTS )) && continue
 
 
@@ -1089,7 +1091,7 @@
       ITERS_LOCAL=`cat $GROUP/$SCRIPT_BASENAME.repeat`
     else
       ITERS_LOCAL=1
-    fi    
+    fi
 
     for (( i=0; $i<$ITERS_LOCAL; i=$i+1 )); do
 
@@ -1100,7 +1102,7 @@
        mkdir -p $TESTNAMEDIR
        pushd $TESTNAMEDIR > /dev/null 2>&1
 
-       cp $TEST .    
+       cp $TEST .
        group_swift_properties
        group_sites_xml
        group_tc_data



From wozniak at ci.uchicago.edu  Mon Nov 14 17:57:28 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Mon, 14 Nov 2011 17:57:28 -0600 (CST)
Subject: [Swift-commit] r5296 - trunk/tests/cdm
Message-ID: <20111114235728.9C04B9CCC5@svn.ci.uchicago.edu>

Author: wozniak
Date: 2011-11-14 17:57:28 -0600 (Mon, 14 Nov 2011)
New Revision: 5296

Added:
   trunk/tests/cdm/211-direct-output-abs.check.sh
   trunk/tests/cdm/211-direct-output-abs.clean.sh
   trunk/tests/cdm/211-direct-output-abs.setup.sh
   trunk/tests/cdm/211-direct-output-abs.swift
Modified:
   trunk/tests/cdm/fs.template.data
Log:
New, working CDM DIRECT output absolute path test


Added: trunk/tests/cdm/211-direct-output-abs.check.sh
===================================================================
--- trunk/tests/cdm/211-direct-output-abs.check.sh	                        (rev 0)
+++ trunk/tests/cdm/211-direct-output-abs.check.sh	2011-11-14 23:57:28 UTC (rev 5296)
@@ -0,0 +1,8 @@
+#!/bin/sh
+
+set -x
+
+grep $( uname -m ) 211-input.txt  || exit 1
+grep $( uname -m ) /tmp/211-output.txt || exit 1
+
+exit 0


Property changes on: trunk/tests/cdm/211-direct-output-abs.check.sh
___________________________________________________________________
Added: svn:executable
   + *

Added: trunk/tests/cdm/211-direct-output-abs.clean.sh
===================================================================
--- trunk/tests/cdm/211-direct-output-abs.clean.sh	                        (rev 0)
+++ trunk/tests/cdm/211-direct-output-abs.clean.sh	2011-11-14 23:57:28 UTC (rev 5296)
@@ -0,0 +1,7 @@
+#!/bin/sh
+
+set -x
+
+rm -v /tmp/211-output.txt || exit 1
+
+exit 0


Property changes on: trunk/tests/cdm/211-direct-output-abs.clean.sh
___________________________________________________________________
Added: svn:executable
   + *

Added: trunk/tests/cdm/211-direct-output-abs.setup.sh
===================================================================
--- trunk/tests/cdm/211-direct-output-abs.setup.sh	                        (rev 0)
+++ trunk/tests/cdm/211-direct-output-abs.setup.sh	2011-11-14 23:57:28 UTC (rev 5296)
@@ -0,0 +1,10 @@
+#!/bin/sh
+
+set -x
+
+{
+  uname -a
+  date
+} > 211-input.txt
+
+exit 0


Property changes on: trunk/tests/cdm/211-direct-output-abs.setup.sh
___________________________________________________________________
Added: svn:executable
   + *

Added: trunk/tests/cdm/211-direct-output-abs.swift
===================================================================
--- trunk/tests/cdm/211-direct-output-abs.swift	                        (rev 0)
+++ trunk/tests/cdm/211-direct-output-abs.swift	2011-11-14 23:57:28 UTC (rev 5296)
@@ -0,0 +1,14 @@
+
+type file;
+
+app (file o) copy (file i)
+{
+  cp @i @o;
+}
+
+file f1<"211-input.txt">;
+
+// Will be transformed by CDM to /tmp/211-output.txt:
+file f2<"211-output.txt">;
+
+f2 = copy(f1);

Modified: trunk/tests/cdm/fs.template.data
===================================================================
--- trunk/tests/cdm/fs.template.data	2011-11-14 22:20:55 UTC (rev 5295)
+++ trunk/tests/cdm/fs.template.data	2011-11-14 23:57:28 UTC (rev 5296)
@@ -5,6 +5,7 @@
 
 rule .*201-output.txt DIRECT _PWD_/201
 rule .*210-input.txt EXTERNAL _PWD_/external.sh
+rule .*211-output.txt DIRECT /tmp
 rule .*301-input.txt BROADCAST _PWD_/301
 rule .*501-output.*.txt GATHER
 rule .*601-input.*.txt LOCAL cp _PWD_/601



From wozniak at ci.uchicago.edu  Mon Nov 14 18:06:08 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Mon, 14 Nov 2011 18:06:08 -0600 (CST)
Subject: [Swift-commit] r5297 - trunk/tests/cdm
Message-ID: <20111115000608.20B069CCC5@svn.ci.uchicago.edu>

Author: wozniak
Date: 2011-11-14 18:06:07 -0600 (Mon, 14 Nov 2011)
New Revision: 5297

Modified:
   trunk/tests/cdm/fs.template.data
Log:
Correction to test case


Modified: trunk/tests/cdm/fs.template.data
===================================================================
--- trunk/tests/cdm/fs.template.data	2011-11-14 23:57:28 UTC (rev 5296)
+++ trunk/tests/cdm/fs.template.data	2011-11-15 00:06:07 UTC (rev 5297)
@@ -5,7 +5,7 @@
 
 rule .*201-output.txt DIRECT _PWD_/201
 rule .*210-input.txt EXTERNAL _PWD_/external.sh
-rule .*211-output.txt DIRECT /tmp
+rule .*211-output.txt DIRECT /
 rule .*301-input.txt BROADCAST _PWD_/301
 rule .*501-output.*.txt GATHER
 rule .*601-input.*.txt LOCAL cp _PWD_/601



From wozniak at ci.uchicago.edu  Mon Nov 14 18:07:09 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Mon, 14 Nov 2011 18:07:09 -0600 (CST)
Subject: [Swift-commit] r5298 - trunk/tests/cdm
Message-ID: <20111115000709.796AE9CCC5@svn.ci.uchicago.edu>

Author: wozniak
Date: 2011-11-14 18:07:09 -0600 (Mon, 14 Nov 2011)
New Revision: 5298

Modified:
   trunk/tests/cdm/211-direct-output-abs.swift
Log:
Correction to test case SwiftScript


Modified: trunk/tests/cdm/211-direct-output-abs.swift
===================================================================
--- trunk/tests/cdm/211-direct-output-abs.swift	2011-11-15 00:06:07 UTC (rev 5297)
+++ trunk/tests/cdm/211-direct-output-abs.swift	2011-11-15 00:07:09 UTC (rev 5298)
@@ -9,6 +9,6 @@
 file f1<"211-input.txt">;
 
 // Will be transformed by CDM to /tmp/211-output.txt:
-file f2<"211-output.txt">;
+file f2<"/tmp/211-output.txt">;
 
 f2 = copy(f1);



From wozniak at ci.uchicago.edu  Mon Nov 14 18:11:46 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Mon, 14 Nov 2011 18:11:46 -0600 (CST)
Subject: [Swift-commit] r5299 - trunk/libexec
Message-ID: <20111115001146.79F3D9CCC5@svn.ci.uchicago.edu>

Author: wozniak
Date: 2011-11-14 18:11:46 -0600 (Mon, 14 Nov 2011)
New Revision: 5299

Modified:
   trunk/libexec/cdm_lib.sh
Log:
Improvement to cdm_lib.sh logging


Modified: trunk/libexec/cdm_lib.sh
===================================================================
--- trunk/libexec/cdm_lib.sh	2011-11-15 00:07:09 UTC (rev 5298)
+++ trunk/libexec/cdm_lib.sh	2011-11-15 00:11:46 UTC (rev 5299)
@@ -31,6 +31,7 @@
 	case $POLICY in
 		DIRECT)
 			DIRECT_DIR=${ARGS[0]}
+			[[ $DIRECT_DIR == "/" ]] && DIRECT_DIR=""
 			log "CDM[DIRECT]: Linking to $DIRECT_DIR/$FILE via $JOBDIR/$FILE"
 			if [ $MODE == "INPUT" ]; then
 				[ -f "$DIRECT_DIR/$FILE" ]
@@ -38,10 +39,8 @@
 				ln -s $DIRECT_DIR/$FILE $JOBDIR/$FILE
 				checkError 254 "CDM[DIRECT]: Linking to $DIRECT_DIR/$FILE failed!"
 			elif [ $MODE == "OUTPUT" ]; then
-				mkdir -p $DIRECT_DIR
-				checkError 254 "CDM[DIRECT]: mkdir -p $DIRECT_DIR failed!"
 				mkdir -p $( dirname $DIRECT_DIR/$FILE )
-				checkError 254 "CDM[DIRECT]: mkdir -p $( dirname $FILE ) failed!"
+				checkError 254 "CDM[DIRECT]: mkdir -p $( dirname $DIRECT_DIR/$FILE ) failed!"
 				touch $DIRECT_DIR/$FILE
 				checkError 254 "CDM[DIRECT]: Touching $DIRECT_DIR/$FILE failed!"
 				ln -s $DIRECT_DIR/$FILE $JOBDIR/$FILE



From wozniak at ci.uchicago.edu  Mon Nov 14 18:14:56 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Mon, 14 Nov 2011 18:14:56 -0600 (CST)
Subject: [Swift-commit] r5300 - trunk/tests/cdm
Message-ID: <20111115001456.D59AB9CCC5@svn.ci.uchicago.edu>

Author: wozniak
Date: 2011-11-14 18:14:56 -0600 (Mon, 14 Nov 2011)
New Revision: 5300

Added:
   trunk/tests/cdm/205-direct-output-abs.check.sh
   trunk/tests/cdm/205-direct-output-abs.clean.sh
   trunk/tests/cdm/205-direct-output-abs.setup.sh
   trunk/tests/cdm/205-direct-output-abs.swift
Removed:
   trunk/tests/cdm/211-direct-output-abs.check.sh
   trunk/tests/cdm/211-direct-output-abs.clean.sh
   trunk/tests/cdm/211-direct-output-abs.setup.sh
   trunk/tests/cdm/211-direct-output-abs.swift
Modified:
   trunk/tests/cdm/fs.template.data
Log:
Better numbering


Copied: trunk/tests/cdm/205-direct-output-abs.check.sh (from rev 5296, trunk/tests/cdm/211-direct-output-abs.check.sh)
===================================================================
--- trunk/tests/cdm/205-direct-output-abs.check.sh	                        (rev 0)
+++ trunk/tests/cdm/205-direct-output-abs.check.sh	2011-11-15 00:14:56 UTC (rev 5300)
@@ -0,0 +1,8 @@
+#!/bin/sh
+
+set -x
+
+grep $( uname -m ) 211-input.txt  || exit 1
+grep $( uname -m ) /tmp/211-output.txt || exit 1
+
+exit 0

Copied: trunk/tests/cdm/205-direct-output-abs.clean.sh (from rev 5296, trunk/tests/cdm/211-direct-output-abs.clean.sh)
===================================================================
--- trunk/tests/cdm/205-direct-output-abs.clean.sh	                        (rev 0)
+++ trunk/tests/cdm/205-direct-output-abs.clean.sh	2011-11-15 00:14:56 UTC (rev 5300)
@@ -0,0 +1,7 @@
+#!/bin/sh
+
+set -x
+
+rm -v /tmp/211-output.txt || exit 1
+
+exit 0

Copied: trunk/tests/cdm/205-direct-output-abs.setup.sh (from rev 5296, trunk/tests/cdm/211-direct-output-abs.setup.sh)
===================================================================
--- trunk/tests/cdm/205-direct-output-abs.setup.sh	                        (rev 0)
+++ trunk/tests/cdm/205-direct-output-abs.setup.sh	2011-11-15 00:14:56 UTC (rev 5300)
@@ -0,0 +1,10 @@
+#!/bin/sh
+
+set -x
+
+{
+  uname -a
+  date
+} > 211-input.txt
+
+exit 0

Copied: trunk/tests/cdm/205-direct-output-abs.swift (from rev 5298, trunk/tests/cdm/211-direct-output-abs.swift)
===================================================================
--- trunk/tests/cdm/205-direct-output-abs.swift	                        (rev 0)
+++ trunk/tests/cdm/205-direct-output-abs.swift	2011-11-15 00:14:56 UTC (rev 5300)
@@ -0,0 +1,14 @@
+
+type file;
+
+app (file o) copy (file i)
+{
+  cp @i @o;
+}
+
+file f1<"211-input.txt">;
+
+// Will be transformed by CDM to /tmp/211-output.txt:
+file f2<"/tmp/211-output.txt">;
+
+f2 = copy(f1);

Deleted: trunk/tests/cdm/211-direct-output-abs.check.sh
===================================================================
--- trunk/tests/cdm/211-direct-output-abs.check.sh	2011-11-15 00:11:46 UTC (rev 5299)
+++ trunk/tests/cdm/211-direct-output-abs.check.sh	2011-11-15 00:14:56 UTC (rev 5300)
@@ -1,8 +0,0 @@
-#!/bin/sh
-
-set -x
-
-grep $( uname -m ) 211-input.txt  || exit 1
-grep $( uname -m ) /tmp/211-output.txt || exit 1
-
-exit 0

Deleted: trunk/tests/cdm/211-direct-output-abs.clean.sh
===================================================================
--- trunk/tests/cdm/211-direct-output-abs.clean.sh	2011-11-15 00:11:46 UTC (rev 5299)
+++ trunk/tests/cdm/211-direct-output-abs.clean.sh	2011-11-15 00:14:56 UTC (rev 5300)
@@ -1,7 +0,0 @@
-#!/bin/sh
-
-set -x
-
-rm -v /tmp/211-output.txt || exit 1
-
-exit 0

Deleted: trunk/tests/cdm/211-direct-output-abs.setup.sh
===================================================================
--- trunk/tests/cdm/211-direct-output-abs.setup.sh	2011-11-15 00:11:46 UTC (rev 5299)
+++ trunk/tests/cdm/211-direct-output-abs.setup.sh	2011-11-15 00:14:56 UTC (rev 5300)
@@ -1,10 +0,0 @@
-#!/bin/sh
-
-set -x
-
-{
-  uname -a
-  date
-} > 211-input.txt
-
-exit 0

Deleted: trunk/tests/cdm/211-direct-output-abs.swift
===================================================================
--- trunk/tests/cdm/211-direct-output-abs.swift	2011-11-15 00:11:46 UTC (rev 5299)
+++ trunk/tests/cdm/211-direct-output-abs.swift	2011-11-15 00:14:56 UTC (rev 5300)
@@ -1,14 +0,0 @@
-
-type file;
-
-app (file o) copy (file i)
-{
-  cp @i @o;
-}
-
-file f1<"211-input.txt">;
-
-// Will be transformed by CDM to /tmp/211-output.txt:
-file f2<"/tmp/211-output.txt">;
-
-f2 = copy(f1);

Modified: trunk/tests/cdm/fs.template.data
===================================================================
--- trunk/tests/cdm/fs.template.data	2011-11-15 00:11:46 UTC (rev 5299)
+++ trunk/tests/cdm/fs.template.data	2011-11-15 00:14:56 UTC (rev 5300)
@@ -4,8 +4,9 @@
 property GATHER_TARGET _PWD_/501-output
 
 rule .*201-output.txt DIRECT _PWD_/201
-rule .*210-input.txt EXTERNAL _PWD_/external.sh
-rule .*211-output.txt DIRECT /
+rule .*205-output.txt DIRECT /
+rule .*206-input.txt  DIRECT /
+rule .*210-input.txt EXTERNAL PWD_/external.sh
 rule .*301-input.txt BROADCAST _PWD_/301
 rule .*501-output.*.txt GATHER
 rule .*601-input.*.txt LOCAL cp _PWD_/601



From wozniak at ci.uchicago.edu  Mon Nov 14 18:22:50 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Mon, 14 Nov 2011 18:22:50 -0600 (CST)
Subject: [Swift-commit] r5301 - trunk/tests/cdm
Message-ID: <20111115002250.823F29CCC5@svn.ci.uchicago.edu>

Author: wozniak
Date: 2011-11-14 18:22:50 -0600 (Mon, 14 Nov 2011)
New Revision: 5301

Added:
   trunk/tests/cdm/206-direct-input-abs.check.sh
   trunk/tests/cdm/206-direct-input-abs.clean.sh
   trunk/tests/cdm/206-direct-input-abs.setup.sh
   trunk/tests/cdm/206-direct-input-abs.swift
Log:
New CDM DIRECT input absolute path case


Added: trunk/tests/cdm/206-direct-input-abs.check.sh
===================================================================
--- trunk/tests/cdm/206-direct-input-abs.check.sh	                        (rev 0)
+++ trunk/tests/cdm/206-direct-input-abs.check.sh	2011-11-15 00:22:50 UTC (rev 5301)
@@ -0,0 +1,8 @@
+#!/bin/sh
+
+set -x
+
+grep $( uname -m ) /tmp/206-input.txt || exit 1
+grep $( uname -m ) 206-output.txt     || exit 1
+
+exit 0


Property changes on: trunk/tests/cdm/206-direct-input-abs.check.sh
___________________________________________________________________
Added: svn:executable
   + *

Added: trunk/tests/cdm/206-direct-input-abs.clean.sh
===================================================================
--- trunk/tests/cdm/206-direct-input-abs.clean.sh	                        (rev 0)
+++ trunk/tests/cdm/206-direct-input-abs.clean.sh	2011-11-15 00:22:50 UTC (rev 5301)
@@ -0,0 +1,7 @@
+#!/bin/sh
+
+set -x
+
+rm -v /tmp/206-input.txt || exit 1
+
+exit 0


Property changes on: trunk/tests/cdm/206-direct-input-abs.clean.sh
___________________________________________________________________
Added: svn:executable
   + *

Added: trunk/tests/cdm/206-direct-input-abs.setup.sh
===================================================================
--- trunk/tests/cdm/206-direct-input-abs.setup.sh	                        (rev 0)
+++ trunk/tests/cdm/206-direct-input-abs.setup.sh	2011-11-15 00:22:50 UTC (rev 5301)
@@ -0,0 +1,10 @@
+#!/bin/sh
+
+set -x
+
+{
+  uname -a
+  date
+} > /tmp/206-input.txt
+
+exit 0


Property changes on: trunk/tests/cdm/206-direct-input-abs.setup.sh
___________________________________________________________________
Added: svn:executable
   + *

Added: trunk/tests/cdm/206-direct-input-abs.swift
===================================================================
--- trunk/tests/cdm/206-direct-input-abs.swift	                        (rev 0)
+++ trunk/tests/cdm/206-direct-input-abs.swift	2011-11-15 00:22:50 UTC (rev 5301)
@@ -0,0 +1,14 @@
+
+type file;
+
+app (file o) copy (file i)
+{
+  cp @i @o;
+}
+
+// Will be transformed by CDM to /tmp/206-input.txt (no staging)
+file f1<"/tmp/206-input.txt">;
+
+file f2<"206-output.txt">;
+
+f2 = copy(f1);



From wozniak at ci.uchicago.edu  Mon Nov 14 18:25:00 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Mon, 14 Nov 2011 18:25:00 -0600 (CST)
Subject: [Swift-commit] r5302 - trunk/tests/cdm
Message-ID: <20111115002500.7B6879CCC5@svn.ci.uchicago.edu>

Author: wozniak
Date: 2011-11-14 18:25:00 -0600 (Mon, 14 Nov 2011)
New Revision: 5302

Modified:
   trunk/tests/cdm/205-direct-output-abs.check.sh
   trunk/tests/cdm/205-direct-output-abs.clean.sh
   trunk/tests/cdm/205-direct-output-abs.setup.sh
   trunk/tests/cdm/205-direct-output-abs.swift
Log:
Correct numbers


Modified: trunk/tests/cdm/205-direct-output-abs.check.sh
===================================================================
--- trunk/tests/cdm/205-direct-output-abs.check.sh	2011-11-15 00:22:50 UTC (rev 5301)
+++ trunk/tests/cdm/205-direct-output-abs.check.sh	2011-11-15 00:25:00 UTC (rev 5302)
@@ -2,7 +2,7 @@
 
 set -x
 
-grep $( uname -m ) 211-input.txt  || exit 1
-grep $( uname -m ) /tmp/211-output.txt || exit 1
+grep $( uname -m ) 205-input.txt  || exit 1
+grep $( uname -m ) /tmp/205-output.txt || exit 1
 
 exit 0

Modified: trunk/tests/cdm/205-direct-output-abs.clean.sh
===================================================================
--- trunk/tests/cdm/205-direct-output-abs.clean.sh	2011-11-15 00:22:50 UTC (rev 5301)
+++ trunk/tests/cdm/205-direct-output-abs.clean.sh	2011-11-15 00:25:00 UTC (rev 5302)
@@ -2,6 +2,6 @@
 
 set -x
 
-rm -v /tmp/211-output.txt || exit 1
+rm -v /tmp/205-output.txt || exit 1
 
 exit 0

Modified: trunk/tests/cdm/205-direct-output-abs.setup.sh
===================================================================
--- trunk/tests/cdm/205-direct-output-abs.setup.sh	2011-11-15 00:22:50 UTC (rev 5301)
+++ trunk/tests/cdm/205-direct-output-abs.setup.sh	2011-11-15 00:25:00 UTC (rev 5302)
@@ -5,6 +5,6 @@
 {
   uname -a
   date
-} > 211-input.txt
+} > 205-input.txt
 
 exit 0

Modified: trunk/tests/cdm/205-direct-output-abs.swift
===================================================================
--- trunk/tests/cdm/205-direct-output-abs.swift	2011-11-15 00:22:50 UTC (rev 5301)
+++ trunk/tests/cdm/205-direct-output-abs.swift	2011-11-15 00:25:00 UTC (rev 5302)
@@ -6,9 +6,9 @@
   cp @i @o;
 }
 
-file f1<"211-input.txt">;
+file f1<"205-input.txt">;
 
-// Will be transformed by CDM to /tmp/211-output.txt:
-file f2<"/tmp/211-output.txt">;
+// Will be transformed by CDM to /tmp/211-output.txt (no staging)
+file f2<"/tmp/205-output.txt">;
 
 f2 = copy(f1);



From wozniak at ci.uchicago.edu  Mon Nov 14 21:03:30 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Mon, 14 Nov 2011 21:03:30 -0600 (CST)
Subject: [Swift-commit] r5303 - trunk/docs/userguide
Message-ID: <20111115030330.89A009CCC5@svn.ci.uchicago.edu>

Author: wozniak
Date: 2011-11-14 21:03:30 -0600 (Mon, 14 Nov 2011)
New Revision: 5303

Modified:
   trunk/docs/userguide/cdm
Log:
Additional CDM notes


Modified: trunk/docs/userguide/cdm
===================================================================
--- trunk/docs/userguide/cdm	2011-11-15 00:25:00 UTC (rev 5302)
+++ trunk/docs/userguide/cdm	2011-11-15 03:03:30 UTC (rev 5303)
@@ -13,7 +13,7 @@
 . The user specifies a CDM policy in a file, customarily +fs.data+.
 . +fs.data+ is given to Swift on the command line.
 . The Swift data module is informed of the CDM policy.
-. At job launch time, for each file, the Swift mechanics query the CDM policy,
+. At job launch time, for each file, the Swift mechanics query the CDM file,
    .. altering the file staging phase, and
    .. sending +fs.data+ to the compute site.
 . At job run time, the Swift wrapper script
@@ -39,15 +39,15 @@
 
 The lines contain:
 
-. A directive, either rule or property
+. A directive, either +rule+ or +property+.
 . A rule has:
-   .. A regular expression to match on the file name
-   .. A policy token
-   .. Additional policy-specific arguments
-. A property has
-   .. A policy property token
-   .. The token value
-. Comments with +#+ .
+   .. A regular expression to match on the file name.
+   .. A policy token.
+   .. Additional policy-specific arguments.
+. A property has:
+   .. A policy property token.
+   .. The token value.
+. Comments with +#+.
 . Blank lines are ignored.
 
 ==== Notes
@@ -56,12 +56,13 @@
 methods.  Thus, one should only use basic features such as +.*+ .
 . Swift treats file names as URLs, while the wrapper script treats
 them as Unix file names.  Thus, one should use a wildcard in the
-beginning of each file name.
+beginning of each file name, as shown in the example.
 . For example, a lookup with the database above given the argument
-+input.txt+ would result in the DIRECT policy.
-. If the lookup does not succeed, the result is DEFAULT.
++input.txt+ would result in the +DIRECT+ policy.
+. Each rule is considered in the order given; the first match results
+in that policy line.
+. If the lookup does not succeed, the result is +DEFAULT+.
 
-
 === Policy Descriptions
 ==== DEFAULT
 
@@ -93,6 +94,26 @@
 Copies will be placed in the job directory for use by the user job.
 * Argument list: [tool] [directory] [tool arguments]*
 
+=== Specific use cases
+
+==== Matching on all file names
+
+To match all file names produced by Swift, simply use pattern +.*+
+
+A common use case is to redirect all file operations to a given
+directory, say, +/fs/dir+.  To do this, use a rule such as:
+
+----
+rule .* DIRECT /fs/dir
+----
+
+==== Use of symbolic links
+
+Swift may provide a symbolic link to the application, which the
+application may read and write normally.  The application may not
+unlink these links; if it does, the application will not be able to
+find or create Swift-compatible data.
+
 // .Gather
 // -----
 // property GATHER_LIMIT 500000000 # 500 MB



From wozniak at ci.uchicago.edu  Tue Nov 15 10:27:03 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Tue, 15 Nov 2011 10:27:03 -0600 (CST)
Subject: [Swift-commit] r5304 - trunk/docs/userguide
Message-ID: <20111115162703.BB17D9CC94@svn.ci.uchicago.edu>

Author: wozniak
Date: 2011-11-15 10:27:02 -0600 (Tue, 15 Nov 2011)
New Revision: 5304

Modified:
   trunk/docs/userguide/cdm
Log:
Note CDM absolute path use case


Modified: trunk/docs/userguide/cdm
===================================================================
--- trunk/docs/userguide/cdm	2011-11-15 03:03:30 UTC (rev 5303)
+++ trunk/docs/userguide/cdm	2011-11-15 16:27:02 UTC (rev 5304)
@@ -107,6 +107,20 @@
 rule .* DIRECT /fs/dir
 ----
 
+==== Absolute paths
+
+If your Swift script operates on files with absolute path names such as:
+
+----
+file f<"/fs/dir/f.txt">;
+----
+
+use:
+
+----
+rule .*f.txt DIRECT /
+----
+
 ==== Use of symbolic links
 
 Swift may provide a symbolic link to the application, which the



From jonmon at ci.uchicago.edu  Wed Nov 16 09:17:43 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Wed, 16 Nov 2011 09:17:43 -0600 (CST)
Subject: [Swift-commit] r5305 -
	trunk/src/org/griphyn/vdl/karajan/lib/swiftscript
Message-ID: <20111116151743.5AA219CCC5@svn.ci.uchicago.edu>

Author: jonmon
Date: 2011-11-16 09:17:42 -0600 (Wed, 16 Nov 2011)
New Revision: 5305

Modified:
   trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/Java.java
Log:
@java now selects to correct function based on parameter types and function name.



Modified: trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/Java.java
===================================================================
--- trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/Java.java	2011-11-15 16:27:02 UTC (rev 5304)
+++ trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/Java.java	2011-11-16 15:17:42 UTC (rev 5305)
@@ -1,6 +1,7 @@
 package org.griphyn.vdl.karajan.lib.swiftscript;
 
 import java.lang.reflect.Method;
+import java.util.Arrays;
 
 import org.globus.cog.karajan.arguments.Arg;
 import org.globus.cog.karajan.stack.VariableStack;
@@ -12,73 +13,93 @@
 import org.griphyn.vdl.type.Type;
 import org.griphyn.vdl.type.Types;
 
-public class Java extends VDLFunction {
+public class Java extends VDLFunction
+{
 
-    static {
+    static
+    {
         setArguments(Java.class, new Arg[] { Arg.VARGS });
     }
-    
-    protected Object function(VariableStack stack) throws ExecutionException {
+
+    protected Object function(VariableStack stack) throws ExecutionException
+    {
         AbstractDataNode[] args = waitForAllVargs(stack);
-        
-        Method method = getMethod(args); 
+
+        Method method = getMethod(args);
         Object[] p = convertInputs(method, args);
         Type type = returnType(method);
         Object value = invoke(method, p);
         DSHandle result = swiftResult(type, value);
-        
+
         return result;
     }
- 
-    /** 
+
+    /**
        Given the user args, locate the Java Method.
     */
-    Method getMethod(DSHandle[] args) {
-        Method result = null;
+
+    Method getMethod(DSHandle[] args)
+    {
+        Method result;
         Class<?> clazz;
-        
+
         String lib = "unset";
         String name = "unset";
-        
+
+        Class[] parameterTypes = new Class[args.length-2];
+
         if (args.length < 2)
             throw new RuntimeException
-            ("@java() requires at least two arguments");
-        
-        try {
+                ("@java() requires at least two arguments");
+
+        try
+        {
             lib = (String) args[0].getValue();
             name = (String) args[1].getValue();
-            clazz = Class.forName(lib); 
-            Method[] methods = clazz.getMethods();
-            result = null;
-            for (Method m : methods) { 
-                if (m.getName().equals(name)) { 
-                    result = m;
-                    break;
-                }
+            clazz = Class.forName(lib);
+
+            for (int i = 2; i < args.length; i++)
+            {
+                Class p = null;
+                Type  t = args[i].getType();
+
+                if (t.equals(Types.FLOAT))        p = double.class;
+                else if (t.equals(Types.INT))     p = int.class;
+                else if (t.equals(Types.BOOLEAN)) p = boolean.class;
+                else if (t.equals(Types.STRING))  p = String.class;
+                else                              throw new RuntimeException("Cannot use @java with non-primitive types");
+
+                parameterTypes[i-2] = p;
             }
-        } 
-        catch (Exception e) { 
-            e.printStackTrace(); 
+            result = clazz.getMethod(name, parameterTypes);
+        }
+        catch (Exception e)
+        {
+            e.printStackTrace();
             throw new RuntimeException
-            ("@java(): Error attempting to use: " + args[0]); 
+                ("@java(): Error attempting to use: " + args[0].getValue());
         }
-        
-        if (result == null) 
+
+        if (result == null)
             throw new RuntimeException
-            ("No method: " + name + " in " + lib); 
+                ("No method: " + name + " in " + lib + "with parameter types" + Arrays.toString(parameterTypes));
+
         return result;
     }
-    
-    /** 
-       Convert the user args to a Java Object array. 
+
+    /**
+       Convert the user args to a Java Object array.
     */
-    Object[] convertInputs(Method method, DSHandle[] args) {
+    Object[] convertInputs(Method method, DSHandle[] args)
+    {
         Object[] result = new Object[args.length-2];
         Object a = null;
-        try {
-            for (int i = 2; i < args.length; i++) {
+        try
+        {
+            for (int i = 2; i < args.length; i++)
+            {
                 Type   t = args[i].getType();
-                Object v = args[i].getValue(); 
+                Object v = args[i].getValue();
                 if (t.equals(Types.FLOAT))
                     a = (Double) v;
                 else if (t.equals(Types.INT))
@@ -87,25 +108,25 @@
                     a = (Boolean) v;
                 else if (t.equals(Types.STRING))
                     a = (String) v;
-                result[i-2] = a;   
+                result[i-2] = a;
             }
         }
         catch (Exception e) {
             e.printStackTrace();
             throw new RuntimeException
-            ("Error converting input arguments: \n" + 
-                " to: " + method.getDeclaringClass() + 
-                "." + method + " \n argument: " + a);
+                ("Error converting input arguments: \n" +
+                 " to: " + method.getDeclaringClass() +
+                 "." + method + " \n argument: " + a);
         }
         return result;
     }
-    
+
     Type returnType(Method method) {
         Type result = null;
-        
-        Class<?> rt = method.getReturnType(); 
+
+        Class<?> rt = method.getReturnType();
         if (rt.equals(Double.TYPE))
-            result = Types.FLOAT; 
+            result = Types.FLOAT;
         else if (rt.equals(Integer.TYPE))
             result = Types.INT;
         else if (rt.equals(Boolean.TYPE))
@@ -114,22 +135,26 @@
             result = Types.STRING;
         return result;
     }
-    
-    Object invoke(Method method, Object[] p) { 
-        Object result = null; 
-        try { 
+
+    Object invoke(Method method, Object[] p)
+    {
+        Object result = null;
+        try
+        {
             result = method.invoke(null, p);
         }
-        catch (Exception e) { 
-            e.printStackTrace(); 
+        catch (Exception e)
+        {
+            e.printStackTrace();
             throw new RuntimeException
-            ("Error attempting to invoke: " + 
-                method.getDeclaringClass() + "." + method); 
+                ("Error attempting to invoke: " +
+                 method.getDeclaringClass() + "." + method);
         }
         return result;
     }
-    
-    DSHandle swiftResult(Type type, Object value) { 
+
+    DSHandle swiftResult(Type type, Object value)
+    {
         DSHandle result = new RootDataNode(type);
         result.setValue(value);
         result.closeShallow();



From ketan at ci.uchicago.edu  Thu Nov 17 10:27:56 2011
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Thu, 17 Nov 2011 10:27:56 -0600 (CST)
Subject: [Swift-commit] r5306 - branches/release-0.93/tests/cdm/absolute
Message-ID: <20111117162756.19B149CFD6@svn.ci.uchicago.edu>

Author: ketan
Date: 2011-11-17 10:27:55 -0600 (Thu, 17 Nov 2011)
New Revision: 5306

Removed:
   branches/release-0.93/tests/cdm/absolute/outdir/
Modified:
   branches/release-0.93/tests/cdm/absolute/fs.data
Log:
improved cdm tests

Modified: branches/release-0.93/tests/cdm/absolute/fs.data
===================================================================
--- branches/release-0.93/tests/cdm/absolute/fs.data	2011-11-16 15:17:42 UTC (rev 5305)
+++ branches/release-0.93/tests/cdm/absolute/fs.data	2011-11-17 16:27:55 UTC (rev 5306)
@@ -1 +1,2 @@
+rule \./outdir/.* DIRECT /home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute
 rule \./indir/.* DIRECT /home/ketan/swift-install/0.93/cog/modules/swift/tests/cdm/absolute



From ketan at ci.uchicago.edu  Thu Nov 17 12:45:56 2011
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Thu, 17 Nov 2011 12:45:56 -0600 (CST)
Subject: [Swift-commit] r5307 - branches/release-0.93/tests/cdm/absolute
Message-ID: <20111117184556.8B8D19CFD6@svn.ci.uchicago.edu>

Author: ketan
Date: 2011-11-17 12:45:56 -0600 (Thu, 17 Nov 2011)
New Revision: 5307

Added:
   branches/release-0.93/tests/cdm/absolute/100-infullpath-outfullpath.check.sh
   branches/release-0.93/tests/cdm/absolute/100-infullpath-outfullpath.clean.sh
   branches/release-0.93/tests/cdm/absolute/100-infullpath-outfullpath.swift
   branches/release-0.93/tests/cdm/absolute/200-infullpath-outrelpath.check.sh
   branches/release-0.93/tests/cdm/absolute/200-infullpath-outrelpath.clean.sh
   branches/release-0.93/tests/cdm/absolute/200-infullpath-outrelpath.setup.sh
   branches/release-0.93/tests/cdm/absolute/200-infullpath-outrelpath.swift
   branches/release-0.93/tests/cdm/absolute/300-inrelpath-outfullpath.check.sh
   branches/release-0.93/tests/cdm/absolute/300-inrelpath-outfullpath.clean.sh
   branches/release-0.93/tests/cdm/absolute/300-inrelpath-outfullpath.setup.sh
   branches/release-0.93/tests/cdm/absolute/300-inrelpath-outfullpath.swift
   branches/release-0.93/tests/cdm/absolute/400-inrelpath-outrelpath.check.sh
   branches/release-0.93/tests/cdm/absolute/400-inrelpath-outrelpath.clean.sh
   branches/release-0.93/tests/cdm/absolute/400-inrelpath-outrelpath.setup.sh
   branches/release-0.93/tests/cdm/absolute/400-inrelpath-outrelpath.swift
   branches/release-0.93/tests/cdm/absolute/setup.sh
Modified:
   branches/release-0.93/tests/cdm/absolute/tc
Log:
improving cdm test

Added: branches/release-0.93/tests/cdm/absolute/100-infullpath-outfullpath.check.sh
===================================================================
--- branches/release-0.93/tests/cdm/absolute/100-infullpath-outfullpath.check.sh	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/100-infullpath-outfullpath.check.sh	2011-11-17 18:45:56 UTC (rev 5307)
@@ -0,0 +1,12 @@
+#!/bin/bash
+
+for aline in `ls -1 /tmp/outdir`
+do
+theline= echo $aline | cut -c 1-3
+if [ $theline == "out" ]
+then
+ ; #pass
+else
+ exit 1 #fail
+fi
+


Property changes on: branches/release-0.93/tests/cdm/absolute/100-infullpath-outfullpath.check.sh
___________________________________________________________________
Added: svn:executable
   + *

Added: branches/release-0.93/tests/cdm/absolute/100-infullpath-outfullpath.clean.sh
===================================================================
--- branches/release-0.93/tests/cdm/absolute/100-infullpath-outfullpath.clean.sh	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/100-infullpath-outfullpath.clean.sh	2011-11-17 18:45:56 UTC (rev 5307)
@@ -0,0 +1,6 @@
+#!/bin/bash
+
+rm -v outdir/* || exit 1
+
+exit 0
+


Property changes on: branches/release-0.93/tests/cdm/absolute/100-infullpath-outfullpath.clean.sh
___________________________________________________________________
Added: svn:executable
   + *

Added: branches/release-0.93/tests/cdm/absolute/100-infullpath-outfullpath.swift
===================================================================
--- branches/release-0.93/tests/cdm/absolute/100-infullpath-outfullpath.swift	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/100-infullpath-outfullpath.swift	2011-11-17 18:45:56 UTC (rev 5307)
@@ -0,0 +1,19 @@
+
+type file;
+
+app (file o) catnap (string _delay, file i)
+{
+ catnap _delay @i stdout=@o;
+}
+
+string delay=@arg("s","1");
+
+file data[]<array_mapper; files=["/tmp/indir/data1.txt","/tmp/indir/data2.txt","/tmp/indir/data4.txt","/tmp/indir/data4.txt"]>;
+
+file out[]<array_mapper; files=["/tmp/outdir/out1.data","/tmp/outdir/out2.data","/tmp/outdir/out3.data","/tmp/outdir/out4.data"]>;
+
+
+foreach o, j in out {
+ out[j] = catnap(delay,data[j]);
+}
+


Property changes on: branches/release-0.93/tests/cdm/absolute/200-infullpath-outrelpath.check.sh
___________________________________________________________________
Added: svn:executable
   + *


Property changes on: branches/release-0.93/tests/cdm/absolute/200-infullpath-outrelpath.clean.sh
___________________________________________________________________
Added: svn:executable
   + *


Property changes on: branches/release-0.93/tests/cdm/absolute/200-infullpath-outrelpath.setup.sh
___________________________________________________________________
Added: svn:executable
   + *

Added: branches/release-0.93/tests/cdm/absolute/200-infullpath-outrelpath.swift
===================================================================
--- branches/release-0.93/tests/cdm/absolute/200-infullpath-outrelpath.swift	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/200-infullpath-outrelpath.swift	2011-11-17 18:45:56 UTC (rev 5307)
@@ -0,0 +1,18 @@
+
+type file;
+
+app (file o) catnap (string _delay, file i)
+{
+ catnap _delay @i stdout=@o;
+}
+
+string delay=@arg("s","1");
+
+file data[]<array_mapper; files=["/tmp/indir/data1.txt","/tmp/indir/data2.txt","/tmp/indir/data4.txt","/tmp/indir/data4.txt"]>;
+
+file out[]<array_mapper; files=["outdir/out1.data","outdir/out2.data","outdir/out3.data","outdir/out4.data"]>;
+
+foreach o, j in out {
+ out[j] = catnap(delay,data[j]);
+}
+


Property changes on: branches/release-0.93/tests/cdm/absolute/300-inrelpath-outfullpath.check.sh
___________________________________________________________________
Added: svn:executable
   + *


Property changes on: branches/release-0.93/tests/cdm/absolute/300-inrelpath-outfullpath.clean.sh
___________________________________________________________________
Added: svn:executable
   + *


Property changes on: branches/release-0.93/tests/cdm/absolute/300-inrelpath-outfullpath.setup.sh
___________________________________________________________________
Added: svn:executable
   + *

Added: branches/release-0.93/tests/cdm/absolute/300-inrelpath-outfullpath.swift
===================================================================
--- branches/release-0.93/tests/cdm/absolute/300-inrelpath-outfullpath.swift	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/300-inrelpath-outfullpath.swift	2011-11-17 18:45:56 UTC (rev 5307)
@@ -0,0 +1,17 @@
+
+type file;
+
+app (file o) catnap (string _delay, file i)
+{
+ catnap _delay @i stdout=@o;
+}
+
+string delay=@arg("s","1");
+file data[]<array_mapper; files=["indir/data1.txt","indir/data2.txt","indir/data4.txt","indir/data4.txt"]>;
+
+file out[]<array_mapper; files=["/tmp/outdir/out1.data","/tmp/outdir/out2.data","/tmp/outdir/out3.data","/tmp/outdir/out4.data"]>;
+
+foreach o, j in out {
+ out[j] = catnap(delay,data[j]);
+}
+


Property changes on: branches/release-0.93/tests/cdm/absolute/400-inrelpath-outrelpath.check.sh
___________________________________________________________________
Added: svn:executable
   + *


Property changes on: branches/release-0.93/tests/cdm/absolute/400-inrelpath-outrelpath.clean.sh
___________________________________________________________________
Added: svn:executable
   + *


Property changes on: branches/release-0.93/tests/cdm/absolute/400-inrelpath-outrelpath.setup.sh
___________________________________________________________________
Added: svn:executable
   + *

Added: branches/release-0.93/tests/cdm/absolute/400-inrelpath-outrelpath.swift
===================================================================
--- branches/release-0.93/tests/cdm/absolute/400-inrelpath-outrelpath.swift	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/400-inrelpath-outrelpath.swift	2011-11-17 18:45:56 UTC (rev 5307)
@@ -0,0 +1,18 @@
+
+type file;
+
+app (file o) catnap (string _delay, file i)
+{
+ catnap _delay @i stdout=@o;
+}
+
+string delay=@arg("s","1");
+
+file out[]<array_mapper; files=["outdir/out1.data","outdir/out2.data","outdir/out3.data","outdir/out4.data"]>;
+
+file data[]<array_mapper; files=["indir/data1.txt","indir/data2.txt","indir/data4.txt","indir/data4.txt"]>;
+
+foreach o, j in out {
+ out[j] = catnap(delay,data[j]);
+}
+

Added: branches/release-0.93/tests/cdm/absolute/setup.sh
===================================================================
--- branches/release-0.93/tests/cdm/absolute/setup.sh	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/setup.sh	2011-11-17 18:45:56 UTC (rev 5307)
@@ -0,0 +1,6 @@
+#!/bin/bash
+
+rm -rfv /tmp/outdir
+cp -r -v indir/ /tmp/
+
+exit 0


Property changes on: branches/release-0.93/tests/cdm/absolute/setup.sh
___________________________________________________________________
Added: svn:executable
   + *

Modified: branches/release-0.93/tests/cdm/absolute/tc
===================================================================
--- branches/release-0.93/tests/cdm/absolute/tc	2011-11-17 16:27:55 UTC (rev 5306)
+++ branches/release-0.93/tests/cdm/absolute/tc	2011-11-17 18:45:56 UTC (rev 5307)
@@ -1,6 +1,3 @@
 localhost sh /bin/sh null null null
-localhost cat /bin/cat null null null
-pbs cat /bin/cat null null null
-mcs cat /bin/cat null null null
 localhost catnap /home/ketan/cdm_tests/catnap.sh null null GLOBUS::maxwalltime="00:01:00"
 



From ketan at ci.uchicago.edu  Thu Nov 17 12:56:50 2011
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Thu, 17 Nov 2011 12:56:50 -0600 (CST)
Subject: [Swift-commit] r5308 - branches/release-0.93/tests/cdm/absolute
Message-ID: <20111117185651.001649CFD6@svn.ci.uchicago.edu>

Author: ketan
Date: 2011-11-17 12:56:50 -0600 (Thu, 17 Nov 2011)
New Revision: 5308

Added:
   branches/release-0.93/tests/cdm/absolute/clean.sh
Removed:
   branches/release-0.93/tests/cdm/absolute/100-infullpath-outfullpath.clean.sh
   branches/release-0.93/tests/cdm/absolute/200-infullpath-outrelpath.clean.sh
   branches/release-0.93/tests/cdm/absolute/200-infullpath-outrelpath.setup.sh
   branches/release-0.93/tests/cdm/absolute/300-inrelpath-outfullpath.clean.sh
   branches/release-0.93/tests/cdm/absolute/300-inrelpath-outfullpath.setup.sh
   branches/release-0.93/tests/cdm/absolute/400-inrelpath-outrelpath.clean.sh
   branches/release-0.93/tests/cdm/absolute/400-inrelpath-outrelpath.setup.sh
Log:
 

Deleted: branches/release-0.93/tests/cdm/absolute/100-infullpath-outfullpath.clean.sh
===================================================================
--- branches/release-0.93/tests/cdm/absolute/100-infullpath-outfullpath.clean.sh	2011-11-17 18:45:56 UTC (rev 5307)
+++ branches/release-0.93/tests/cdm/absolute/100-infullpath-outfullpath.clean.sh	2011-11-17 18:56:50 UTC (rev 5308)
@@ -1,6 +0,0 @@
-#!/bin/bash
-
-rm -v outdir/* || exit 1
-
-exit 0
-

Deleted: branches/release-0.93/tests/cdm/absolute/200-infullpath-outrelpath.clean.sh
===================================================================
Deleted: branches/release-0.93/tests/cdm/absolute/200-infullpath-outrelpath.setup.sh
===================================================================
Deleted: branches/release-0.93/tests/cdm/absolute/300-inrelpath-outfullpath.clean.sh
===================================================================
Deleted: branches/release-0.93/tests/cdm/absolute/300-inrelpath-outfullpath.setup.sh
===================================================================
Deleted: branches/release-0.93/tests/cdm/absolute/400-inrelpath-outrelpath.clean.sh
===================================================================
Deleted: branches/release-0.93/tests/cdm/absolute/400-inrelpath-outrelpath.setup.sh
===================================================================
Copied: branches/release-0.93/tests/cdm/absolute/clean.sh (from rev 5307, branches/release-0.93/tests/cdm/absolute/100-infullpath-outfullpath.clean.sh)
===================================================================
--- branches/release-0.93/tests/cdm/absolute/clean.sh	                        (rev 0)
+++ branches/release-0.93/tests/cdm/absolute/clean.sh	2011-11-17 18:56:50 UTC (rev 5308)
@@ -0,0 +1,6 @@
+#!/bin/bash
+
+rm -v outdir/* || exit 1
+
+exit 0
+



From ketan at ci.uchicago.edu  Thu Nov 17 15:50:16 2011
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Thu, 17 Nov 2011 15:50:16 -0600 (CST)
Subject: [Swift-commit] r5309 - trunk/docs/userguide
Message-ID: <20111117215016.DEDAC9CFD6@svn.ci.uchicago.edu>

Author: ketan
Date: 2011-11-17 15:50:16 -0600 (Thu, 17 Nov 2011)
New Revision: 5309

Modified:
   trunk/docs/userguide/cdm
Log:
updated cdm doc

Modified: trunk/docs/userguide/cdm
===================================================================
--- trunk/docs/userguide/cdm	2011-11-17 18:56:50 UTC (rev 5308)
+++ trunk/docs/userguide/cdm	2011-11-17 21:50:16 UTC (rev 5309)
@@ -27,6 +27,11 @@
 -----
 
 === CDM policy file format
+A CDM policy file contains four space separated fields as follows:
+. The keyword 'rule'
+. The filename pattern expressed as a regexp
+. The rulename: DIRECT, GATHER, BROADCAST etc.
+. The path where to look for the files (optional)
 
 ==== Example
 -----
@@ -231,7 +236,7 @@
 
 // . A directive, either rule or property
 // . A rule has:
-//    .. A regular expression for site matchin
+//    .. A regular expression for site matching
 //    .. A regular expression for filename matching
 //    .. A policy token
 //    .. Additional policy-specific arguments



From wilde at ci.uchicago.edu  Thu Nov 17 23:02:09 2011
From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu)
Date: Thu, 17 Nov 2011 23:02:09 -0600 (CST)
Subject: [Swift-commit] r5310 - wwwdev/main
Message-ID: <20111118050209.DDC1B9CFD6@svn.ci.uchicago.edu>

Author: wilde
Date: 2011-11-17 23:02:09 -0600 (Thu, 17 Nov 2011)
New Revision: 5310

Modified:
   wwwdev/main/index.php
Log:
Revised text of new main page.

Modified: wwwdev/main/index.php
===================================================================
--- wwwdev/main/index.php	2011-11-17 21:50:16 UTC (rev 5309)
+++ wwwdev/main/index.php	2011-11-18 05:02:09 UTC (rev 5310)
@@ -49,7 +49,8 @@
 	<!-- end header -->
   <div class="grid_12">
     <h1 class="tagline">
-      A system for rapid and reliable specification, execution, and management of large scale workflows.
+      <!-- A system for rapid and reliable specification, execution, and management of large scale workflows. -->
+      The <b>Swift parallel scripting language</b> is an easy way to use almost any parallel computer system.
     </h1>
   </div>
   <!-- end .grid_12 -->
@@ -72,10 +73,49 @@
   <div class="clear"></div>
   <div class="grid_6">
     <p class="indented">
-      <b>Swift</b> is a system for the <b>rapid</b> and <b>reliable</b> specification, execution, and management of <b>large-scale</b> science and engineering <b>workflows</b>. It supports applications that execute many tasks coupled by disk-resident datasets?as is common, for example, when analyzing large quantities of data or performing parameter studies or ensemble simulations. </p>
-	  <p class="highlight-A">A simple scripting language to enable the concise, high-level specifications of complex parallel computations, and mappers for accessing diverse data formats in a convenient manner.</p>
-	  <p class="highlight-B">An execution engine that can manage the dispatch of many (10,000) tasks to many (100) processors, whether on parallel computers, campus grids, or multi-site grids.</p>
-	  <p class="indented">Swift users have achieved multiple-order-of-magnitude savings in program development and execution time.</p>
+
+<b>Swift</b> lets you write parallel scripts that run many copies of
+ordinary programs concurrently, using statements like this:
+
+<pre>
+foreach protein in proteinList {
+  runBLAST(protein);
+}
+</pre>
+
+
+<p class="highlight-B">
+
+<b>Swift is parallel:
+</b> it runs multiple programs concurrently as
+soon as their inputs are available, reducing the need for complex
+parallel programming.  <!-- </li> -->
+
+</p>
+
+
+<p class="highlight-A">
+
+<b>Swift is easy:</b>
+Short, simple
+scripts can do large-scale work.
+The same script runs on
+multicore computers, clusters, grids, clouds, and
+supercomputers. 
+</p>
+
+<!-- FIXME: [Swift runtime picture here, in right column] -->
+
+<p class="highlight-C">
+<b>Swift is fast:</b>
+it can run a million programs, thousands at a time, launching hundreds per second.</p>
+
+<p class="highlight-D">
+<b>Swift is flexible:</b>
+its being used in many fields of
+science, engineering, and business. Read the case studies.
+</p>
+
   </div>
   <!-- end .grid_6 -->
   <div class="grid_3">



From tga at ci.uchicago.edu  Fri Nov 18 11:43:01 2011
From: tga at ci.uchicago.edu (tga at ci.uchicago.edu)
Date: Fri, 18 Nov 2011 11:43:01 -0600 (CST)
Subject: [Swift-commit] r5311 - SwiftApps/SwiftR
Message-ID: <20111118174301.7A8E19CC94@svn.ci.uchicago.edu>

Author: tga
Date: 2011-11-18 11:43:01 -0600 (Fri, 18 Nov 2011)
New Revision: 5311

Modified:
   SwiftApps/SwiftR/checkout-swift.sh
Log:
Made build process return error when checking out swift fails



Modified: SwiftApps/SwiftR/checkout-swift.sh
===================================================================
--- SwiftApps/SwiftR/checkout-swift.sh	2011-11-18 05:02:09 UTC (rev 5310)
+++ SwiftApps/SwiftR/checkout-swift.sh	2011-11-18 17:43:01 UTC (rev 5311)
@@ -26,8 +26,12 @@
 
 svn checkout https://cogkit.svn.sourceforge.net/svnroot/cogkit/${COG_TAG} ${COG_SRC}
 echo Checking out Swift source 
-(cd ${COG_SRC}/modules &&
-    svn checkout https://svn.ci.uchicago.edu/svn/vdl2/${SWIFT_TAG} swift)
+CURR=`pwd`
+cd ${COG_SRC}/modules
+svn checkout https://svn.ci.uchicago.edu/svn/vdl2/${SWIFT_TAG} swift
+cd "$CURR"
 
 patch -d ${COG_SRC} -p0 < $SWIFT_PATCH
-(cd ${COG_SRC} && xargs -d '\n' -n 1 --arg-file=../${SWIFT_PATCH}.deleted rm -rf)
+cd ${COG_SRC} 
+xargs -d '\n' -n 1 --arg-file=../${SWIFT_PATCH}.deleted rm -rf
+cd "$CURR"



From jonmon at ci.uchicago.edu  Sat Nov 19 17:31:35 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Sat, 19 Nov 2011 17:31:35 -0600 (CST)
Subject: [Swift-commit] r5312 -
	branches/release-0.93/src/org/griphyn/vdl/karajan/lib/swiftscript
Message-ID: <20111119233135.7B61C9CCF7@svn.ci.uchicago.edu>

Author: jonmon
Date: 2011-11-19 17:31:34 -0600 (Sat, 19 Nov 2011)
New Revision: 5312

Modified:
   branches/release-0.93/src/org/griphyn/vdl/karajan/lib/swiftscript/Java.java
Log:
Fixed issue with @java not picking the correct method. Now matches name and parameter list.



Modified: branches/release-0.93/src/org/griphyn/vdl/karajan/lib/swiftscript/Java.java
===================================================================
--- branches/release-0.93/src/org/griphyn/vdl/karajan/lib/swiftscript/Java.java	2011-11-18 17:43:01 UTC (rev 5311)
+++ branches/release-0.93/src/org/griphyn/vdl/karajan/lib/swiftscript/Java.java	2011-11-19 23:31:34 UTC (rev 5312)
@@ -1,6 +1,7 @@
 package org.griphyn.vdl.karajan.lib.swiftscript;
 
 import java.lang.reflect.Method;
+import java.util.Arrays;
 
 import org.globus.cog.karajan.arguments.Arg;
 import org.globus.cog.karajan.stack.VariableStack;
@@ -8,78 +9,97 @@
 import org.griphyn.vdl.karajan.lib.VDLFunction;
 import org.griphyn.vdl.mapping.AbstractDataNode;
 import org.griphyn.vdl.mapping.DSHandle;
-import org.griphyn.vdl.mapping.HandleOpenException;
 import org.griphyn.vdl.mapping.RootDataNode;
 import org.griphyn.vdl.type.Type;
 import org.griphyn.vdl.type.Types;
 
-public class Java extends VDLFunction {
+public class Java extends VDLFunction
+{
 
-    static {
+    static
+    {
         setArguments(Java.class, new Arg[] { Arg.VARGS });
     }
-    
-    protected Object function(VariableStack stack) throws ExecutionException {
+
+    protected Object function(VariableStack stack) throws ExecutionException
+    {
         AbstractDataNode[] args = waitForAllVargs(stack);
-        
-        Method method = getMethod(args); 
+
+        Method method = getMethod(args);
         Object[] p = convertInputs(method, args);
         Type type = returnType(method);
         Object value = invoke(method, p);
         DSHandle result = swiftResult(type, value);
-        
+
         return result;
     }
- 
-    /** 
+
+    /**
        Given the user args, locate the Java Method.
     */
-    Method getMethod(DSHandle[] args) {
-        Method result = null;
-        Class clazz;
-        
+
+    Method getMethod(DSHandle[] args)
+    {
+        Method result;
+        Class<?> clazz;
+
         String lib = "unset";
         String name = "unset";
-        
+
+        Class[] parameterTypes = new Class[args.length-2];
+
         if (args.length < 2)
             throw new RuntimeException
-            ("@java() requires at least two arguments");
-        
-        try {
-            lib = (String)args[0].getValue();
-            name = (String)args[1].getValue();
-            clazz = Class.forName(lib); 
-            Method[] methods = clazz.getMethods();
-            result = null;
-            for (Method m : methods) { 
-                if (m.getName().equals(name)) { 
-                    result = m;
-                    break;
-                }
+                ("@java() requires at least two arguments");
+
+        try
+        {
+            lib = (String) args[0].getValue();
+            name = (String) args[1].getValue();
+            clazz = Class.forName(lib);
+
+            for (int i = 2; i < args.length; i++)
+            {
+                Class p = null;
+                Type  t = args[i].getType();
+
+                if (t.equals(Types.FLOAT))        p = double.class;
+                else if (t.equals(Types.INT))     p = int.class;
+                else if (t.equals(Types.BOOLEAN)) p = boolean.class;
+                else if (t.equals(Types.STRING))  p = String.class;
+                else                              throw new RuntimeException("Cannot use @java with non-primitive types");
+
+                parameterTypes[i-2] = p;
             }
-        } 
-        catch (Exception e) { 
-            e.printStackTrace(); 
+            result = clazz.getMethod(name, parameterTypes);
+        }
+        catch (Exception e)
+        {
+            e.printStackTrace();
             throw new RuntimeException
-            ("@java(): Error attempting to use: " + args[0]); 
+                ("@java(): Error attempting to use: " + args[0].getValue());
         }
-        
-        if (result == null) 
+
+        if (result == null)
             throw new RuntimeException
-            ("No method: " + name + " in " + lib); 
+                ("No method: " + name + " in " + lib + "with parameter types" + Arrays.toString(parameterTypes));
+
         return result;
     }
-    
-    /** 
-       Convert the user args to a Java Object array. 
+
+    /**
+       Convert the user args to a Java Object array.
     */
-    Object[] convertInputs(Method method, DSHandle[] args) {
+    Object[] convertInputs(Method method, DSHandle[] args)
+    {
         Object[] result = new Object[args.length-2];
         Object a = null;
-        try {
-            for (int i = 2; i < args.length; i++) {
+        try
+        {
+            for (int i = 2; i < args.length; i++)
+            {
                 Type   t = args[i].getType();
-                Object v = args[i].getValue(); 
+                Object v = args[i].getValue();
                 if (t.equals(Types.FLOAT))
                     a = (Double) v;
                 else if (t.equals(Types.INT))
@@ -88,25 +108,25 @@
                     a = (Boolean) v;
                 else if (t.equals(Types.STRING))
                     a = (String) v;
-                result[i-2] = a;   
+                result[i-2] = a;
             }
         }
         catch (Exception e) {
             e.printStackTrace();
             throw new RuntimeException
-            ("Error converting input arguments: \n" + 
-                " to: " + method.getDeclaringClass() + 
-                "." + method + " \n argument: " + a);
+                ("Error converting input arguments: \n" +
+                 " to: " + method.getDeclaringClass() +
+                 "." + method + " \n argument: " + a);
         }
         return result;
     }
-    
+
     Type returnType(Method method) {
         Type result = null;
-        
-        Class rt = method.getReturnType(); 
+
+        Class<?> rt = method.getReturnType();
         if (rt.equals(Double.TYPE))
-            result = Types.FLOAT; 
+            result = Types.FLOAT;
         else if (rt.equals(Integer.TYPE))
             result = Types.INT;
         else if (rt.equals(Boolean.TYPE))
@@ -115,22 +135,26 @@
             result = Types.STRING;
         return result;
     }
-    
-    Object invoke(Method method, Object[] p) { 
-        Object result = null; 
-        try { 
+
+    Object invoke(Method method, Object[] p)
+    {
+        Object result = null;
+        try
+        {
             result = method.invoke(null, p);
         }
-        catch (Exception e) { 
-            e.printStackTrace(); 
+        catch (Exception e)
+        {
+            e.printStackTrace();
             throw new RuntimeException
-            ("Error attempting to invoke: " + 
-                method.getDeclaringClass() + "." + method); 
+                ("Error attempting to invoke: " +
+                 method.getDeclaringClass() + "." + method);
         }
         return result;
     }
-    
-    DSHandle swiftResult(Type type, Object value) { 
+
+    DSHandle swiftResult(Type type, Object value)
+    {
         DSHandle result = new RootDataNode(type);
         result.setValue(value);
         result.closeShallow();



From jonmon at ci.uchicago.edu  Sat Nov 19 19:28:25 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Sat, 19 Nov 2011 19:28:25 -0600 (CST)
Subject: [Swift-commit] r5313 - trunk/src/org/griphyn/vdl/engine
Message-ID: <20111120012825.C32DD9CCF7@svn.ci.uchicago.edu>

Author: jonmon
Date: 2011-11-19 19:28:25 -0600 (Sat, 19 Nov 2011)
New Revision: 5313

Modified:
   trunk/src/org/griphyn/vdl/engine/Karajan.java
Log:
Fixed import compilation issue.



Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java
===================================================================
--- trunk/src/org/griphyn/vdl/engine/Karajan.java	2011-11-19 23:31:34 UTC (rev 5312)
+++ trunk/src/org/griphyn/vdl/engine/Karajan.java	2011-11-20 01:28:25 UTC (rev 5313)
@@ -196,18 +196,17 @@
 					File local = new File(swiftfilename);
 					if( !( lib_path == null || local.exists() ) )
 					{
-					    StringTokenizer st = new StringTokenizer(lib_path, ":");
-					    while(st.hasMoreTokens())
+					    String[] path = lib_path.split(":");
+					    for(String entry : path)
 					    {
-					        String path = st.nextToken();
-					        String lib_script_location = path + "/" + swiftfilename;
-					        File tmp = new File(lib_script_location);
-
-					        if(tmp.exists())
+					        String lib_script_location = entry + "/" + swiftfilename;
+					        File file = new File(lib_script_location);
+					        
+					        if(file.exists())
 					        {
-					            swiftfilename = path + "/" + swiftfilename;
-					            xmlfilename = path + "/" + xmlfilename;
-					            moduleToImport = path + "/" + moduleToImport;
+					            swiftfilename = entry + "/" + swiftfilename;
+
+					            moduleToImport = entry + "/" + moduleToImport;
 					            break;
 					        }
 					    }



From jonmon at ci.uchicago.edu  Sat Nov 19 21:23:56 2011
From: jonmon at ci.uchicago.edu (jonmon at ci.uchicago.edu)
Date: Sat, 19 Nov 2011 21:23:56 -0600 (CST)
Subject: [Swift-commit] r5314 -
	branches/release-0.93/src/org/griphyn/vdl/engine
Message-ID: <20111120032356.18A1B9CCF7@svn.ci.uchicago.edu>

Author: jonmon
Date: 2011-11-19 21:23:55 -0600 (Sat, 19 Nov 2011)
New Revision: 5314

Modified:
   branches/release-0.93/src/org/griphyn/vdl/engine/Karajan.java
Log:
commit fix to the import compilation problem



Modified: branches/release-0.93/src/org/griphyn/vdl/engine/Karajan.java
===================================================================
--- branches/release-0.93/src/org/griphyn/vdl/engine/Karajan.java	2011-11-20 01:28:25 UTC (rev 5313)
+++ branches/release-0.93/src/org/griphyn/vdl/engine/Karajan.java	2011-11-20 03:23:55 UTC (rev 5314)
@@ -157,28 +157,23 @@
 				logger.debug("Importing module "+moduleToImport);
 				if(!importedNames.contains(moduleToImport)) {
 
-					// TODO PATH/PERL5LIB-style path handling
-					//String swiftfilename = "./"+moduleToImport+".swift";
-					//String xmlfilename = "./"+moduleToImport+".xml";
 				    String lib_path = System.getenv("SWIFT_LIB");
-					String swiftfilename = moduleToImport+".swift";
-					String xmlfilename = moduleToImport+".xml";
+					String swiftfilename = "./"+moduleToImport+".swift";
+					String xmlfilename = "./"+moduleToImport+".xml";
 
 					File local = new File(swiftfilename);
 					if( !( lib_path == null || local.exists() ) )
 					{
-					    StringTokenizer st = new StringTokenizer(lib_path, ":");
-					    while(st.hasMoreTokens())
+					    String[] path = lib_path.split(":");
+					    for(String entry : path)
 					    {
-					        String path = st.nextToken();
-					        String lib_script_location = path + "/" + swiftfilename;
-					        File tmp = new File(lib_script_location);
+					        String lib_script_location = entry + "/" + swiftfilename;
+					        File file = new File(lib_script_location);
 
-					        if(tmp.exists())
+					        if(file.exists())
 					        {
-					            swiftfilename = path + "/" + swiftfilename;
-					            xmlfilename = path + "/" + xmlfilename;
-					            moduleToImport = path + "/" + moduleToImport;
+					            swiftfilename = entry + "/" + swiftfilename;
+					            moduleToImport = entry + "/" + moduleToImport;
 					            break;
 					        }
 					    }



From davidk at ci.uchicago.edu  Wed Nov 23 13:33:26 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Wed, 23 Nov 2011 13:33:26 -0600 (CST)
Subject: [Swift-commit] r5315 - in branches/release-0.93: bin etc/sites
Message-ID: <20111123193326.BDB9F9CD15@svn.ci.uchicago.edu>

Author: davidk
Date: 2011-11-23 13:33:26 -0600 (Wed, 23 Nov 2011)
New Revision: 5315

Added:
   branches/release-0.93/etc/sites/mcs
Modified:
   branches/release-0.93/bin/start-coaster-service
Log:
MCS gensite template and scs fix


Modified: branches/release-0.93/bin/start-coaster-service
===================================================================
--- branches/release-0.93/bin/start-coaster-service	2011-11-20 03:23:55 UTC (rev 5314)
+++ branches/release-0.93/bin/start-coaster-service	2011-11-23 19:33:26 UTC (rev 5315)
@@ -78,6 +78,7 @@
          echo $! >> $PID_FILE
       # Connect directly
       else  
+         ssh $WORKER_USERNAME@$MACHINE mkdir -p $WORKER_WORK > /dev/null 2>&1
          scp $SWIFT_BIN/$WORKER $WORKER_USERNAME@$MACHINE:$WORKER_WORK > /dev/null 2>&1
          echo Starting worker on $MACHINE
          ssh $WORKER_USERNAME@$MACHINE "$WORKER_WORK/$WORKER $EXECUTION_URL $MACHINE $LOG_DIR" &

Added: branches/release-0.93/etc/sites/mcs
===================================================================
--- branches/release-0.93/etc/sites/mcs	                        (rev 0)
+++ branches/release-0.93/etc/sites/mcs	2011-11-23 19:33:26 UTC (rev 5315)
@@ -0,0 +1,13 @@
+<config>
+  <pool handle="persistent-coasters">
+    <execution provider="coaster-persistent"
+               url="_EXECUTION_URL_"
+               jobmanager="local:local"/>
+    <profile namespace="globus" key="workerManager">passive</profile>
+    <profile namespace="globus" key="workersPerNode">8</profile>
+    <profile key="jobThrottle" namespace="karajan">.79</profile>
+    <profile namespace="karajan" key="initialScore">10000</profile>
+    <filesystem provider="local" url="none" />
+    <workdirectory>_WORK_</workdirectory>
+  </pool>
+</config>



From hategan at ci.uchicago.edu  Thu Nov 24 15:18:52 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Thu, 24 Nov 2011 15:18:52 -0600 (CST)
Subject: [Swift-commit] r5316 - in
	branches/release-0.93/src/org/griphyn/vdl: karajan/lib mapping
Message-ID: <20111124211852.42DAC9CCA2@svn.ci.uchicago.edu>

Author: hategan
Date: 2011-11-24 15:18:51 -0600 (Thu, 24 Nov 2011)
New Revision: 5316

Modified:
   branches/release-0.93/src/org/griphyn/vdl/karajan/lib/VDLFunction.java
   branches/release-0.93/src/org/griphyn/vdl/mapping/AbstractDataNode.java
   branches/release-0.93/src/org/griphyn/vdl/mapping/RootArrayDataNode.java
   branches/release-0.93/src/org/griphyn/vdl/mapping/RootDataNode.java
Log:
throw better exception than NPE when @filename is called with a non-mapped argument

Modified: branches/release-0.93/src/org/griphyn/vdl/karajan/lib/VDLFunction.java
===================================================================
--- branches/release-0.93/src/org/griphyn/vdl/karajan/lib/VDLFunction.java	2011-11-23 19:33:26 UTC (rev 5315)
+++ branches/release-0.93/src/org/griphyn/vdl/karajan/lib/VDLFunction.java	2011-11-24 21:18:51 UTC (rev 5316)
@@ -197,8 +197,17 @@
 
 	private static String[] leavesFileNames(DSHandle var) throws ExecutionException, HandleOpenException {
 	    Mapper mapper;
+	    	    
         synchronized (var.getRoot()) {
-            mapper = var.getMapper();
+            if (var instanceof AbstractDataNode) {
+                mapper = ((AbstractDataNode) var).getActualMapper();
+                if (mapper == null) {
+                    throw new ExecutionException(var + " is not a mapped type");
+                }
+            }
+            else {
+                mapper = var.getMapper();
+            }
         }
 		List<String> l = new ArrayList<String>();
 		try {

Modified: branches/release-0.93/src/org/griphyn/vdl/mapping/AbstractDataNode.java
===================================================================
--- branches/release-0.93/src/org/griphyn/vdl/mapping/AbstractDataNode.java	2011-11-23 19:33:26 UTC (rev 5315)
+++ branches/release-0.93/src/org/griphyn/vdl/mapping/AbstractDataNode.java	2011-11-24 21:18:51 UTC (rev 5316)
@@ -513,7 +513,7 @@
         }
     }
     
-    protected Mapper getActualMapper() {
+    public Mapper getActualMapper() {
         return null;
     }
 

Modified: branches/release-0.93/src/org/griphyn/vdl/mapping/RootArrayDataNode.java
===================================================================
--- branches/release-0.93/src/org/griphyn/vdl/mapping/RootArrayDataNode.java	2011-11-23 19:33:26 UTC (rev 5315)
+++ branches/release-0.93/src/org/griphyn/vdl/mapping/RootArrayDataNode.java	2011-11-24 21:18:51 UTC (rev 5316)
@@ -103,7 +103,7 @@
         throw new FutureNotYetAvailable(waitingMapperParam.getFutureWrapper());
 	}
 	
-	protected Mapper getActualMapper() {
+	public Mapper getActualMapper() {
         return mapper;
     }
 

Modified: branches/release-0.93/src/org/griphyn/vdl/mapping/RootDataNode.java
===================================================================
--- branches/release-0.93/src/org/griphyn/vdl/mapping/RootDataNode.java	2011-11-23 19:33:26 UTC (rev 5315)
+++ branches/release-0.93/src/org/griphyn/vdl/mapping/RootDataNode.java	2011-11-24 21:18:51 UTC (rev 5316)
@@ -213,7 +213,7 @@
         throw new FutureNotYetAvailable(waitingMapperParam.getFutureWrapper());
 	}
 	
-	protected Mapper getActualMapper() {
+	public Mapper getActualMapper() {
         return mapper;
     }
 



From hategan at ci.uchicago.edu  Fri Nov 25 17:46:42 2011
From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu)
Date: Fri, 25 Nov 2011 17:46:42 -0600 (CST)
Subject: [Swift-commit] r5317 -
	branches/release-0.93/src/org/griphyn/vdl/karajan/lib
Message-ID: <20111125234642.DAC589CD15@svn.ci.uchicago.edu>

Author: hategan
Date: 2011-11-25 17:46:41 -0600 (Fri, 25 Nov 2011)
New Revision: 5317

Modified:
   branches/release-0.93/src/org/griphyn/vdl/karajan/lib/SetFieldValue.java
Log:
avoid concurrent modification exceptions while logging setFieldValue data

Modified: branches/release-0.93/src/org/griphyn/vdl/karajan/lib/SetFieldValue.java
===================================================================
--- branches/release-0.93/src/org/griphyn/vdl/karajan/lib/SetFieldValue.java	2011-11-24 21:18:51 UTC (rev 5316)
+++ branches/release-0.93/src/org/griphyn/vdl/karajan/lib/SetFieldValue.java	2011-11-25 23:46:41 UTC (rev 5317)
@@ -81,15 +81,17 @@
 	String unpackHandles(Map<String,DSHandle> handles) { 
 	    StringBuilder sb = new StringBuilder();
 	    sb.append("{");
-	    Iterator<Map.Entry<String,DSHandle>> it = 
-	        handles.entrySet().iterator();
-	    while (it.hasNext()) { 
-	        Map.Entry<String,DSHandle> entry = it.next();
-	        sb.append(entry.getKey());
-	        sb.append('=');
-	        sb.append(entry.getValue().getValue());
-	        if (it.hasNext())
-	            sb.append(", ");
+	    synchronized(handles) {
+    	    Iterator<Map.Entry<String,DSHandle>> it = 
+    	        handles.entrySet().iterator();
+    	    while (it.hasNext()) { 
+    	        Map.Entry<String,DSHandle> entry = it.next();
+    	        sb.append(entry.getKey());
+    	        sb.append('=');
+    	        sb.append(entry.getValue().getValue());
+    	        if (it.hasNext())
+    	            sb.append(", ");
+    	    }
 	    }
 	    sb.append("}");
 	    return sb.toString();



From davidk at ci.uchicago.edu  Mon Nov 28 12:55:03 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Mon, 28 Nov 2011 12:55:03 -0600 (CST)
Subject: [Swift-commit] r5318 - in branches/release-0.93.1: . libexec
Message-ID: <20111128185503.25BDE9CCA2@svn.ci.uchicago.edu>

Author: davidk
Date: 2011-11-28 12:55:02 -0600 (Mon, 28 Nov 2011)
New Revision: 5318

Added:
   branches/release-0.93.1/libexec/release.txt
Modified:
   branches/release-0.93.1/build.xml
   branches/release-0.93.1/libexec/svn-revision
Log:
Added version information


Modified: branches/release-0.93.1/build.xml
===================================================================
--- branches/release-0.93.1/build.xml	2011-11-25 23:46:41 UTC (rev 5317)
+++ branches/release-0.93.1/build.xml	2011-11-28 18:55:02 UTC (rev 5318)
@@ -385,7 +385,7 @@
 	</target>
 
 	<target name="generateVersion">
-		<echo file="libexec/version.txt">Swift ${version} </echo>
+		<echo file="libexec/version.txt">Swift </echo>
 		<exec os="Linux, Mac OS X" executable="libexec/svn-revision" append="true" output="libexec/version.txt"/>
 		<exec os="Linux, Mac OS X" executable="libexec/generate-buildid" append="false" output="libexec/buildid.txt"/>
 	</target>

Added: branches/release-0.93.1/libexec/release.txt
===================================================================
--- branches/release-0.93.1/libexec/release.txt	                        (rev 0)
+++ branches/release-0.93.1/libexec/release.txt	2011-11-28 18:55:02 UTC (rev 5318)
@@ -0,0 +1 @@
+0.93.1 RC0

Modified: branches/release-0.93.1/libexec/svn-revision
===================================================================
--- branches/release-0.93.1/libexec/svn-revision	2011-11-25 23:46:41 UTC (rev 5317)
+++ branches/release-0.93.1/libexec/svn-revision	2011-11-28 18:55:02 UTC (rev 5318)
@@ -16,6 +16,7 @@
 }
 
 # Determine Swift revision
+RELEASE=$(cat libexec/release.txt)
 
 SWIFTVER=$(hereversion swift)
 
@@ -23,5 +24,5 @@
 
 COGVER=$(hereversion cog)
 
-echo $SWIFTVER $COGVER
+echo $RELEASE $SWIFTVER $COGVER
 



From ketan at ci.uchicago.edu  Tue Nov 29 10:12:03 2011
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Tue, 29 Nov 2011 10:12:03 -0600 (CST)
Subject: [Swift-commit] r5319 - trunk/bin/grid
Message-ID: <20111129161203.5A5579CCA8@svn.ci.uchicago.edu>

Author: ketan
Date: 2011-11-29 10:12:03 -0600 (Tue, 29 Nov 2011)
New Revision: 5319

Added:
   trunk/bin/grid/multi-service-workers
Removed:
   trunk/bin/grid/my-swift-workers
Modified:
   trunk/bin/grid/start-grid-service
   trunk/bin/grid/swift-workers
Log:
renamed multiple service workers script from my-swift-workers to multi-service-workers

Copied: trunk/bin/grid/multi-service-workers (from rev 5308, trunk/bin/grid/my-swift-workers)
===================================================================
--- trunk/bin/grid/multi-service-workers	                        (rev 0)
+++ trunk/bin/grid/multi-service-workers	2011-11-29 16:12:03 UTC (rev 5319)
@@ -0,0 +1,338 @@
+#! /usr/bin/env ruby
+
+$stdout.sync = true
+# FIXME: Is this the best way to get lib functions from bin/grid into RUBYLIB ?
+$:[$:.length] = File.dirname($0)
+
+require 'erb'
+require 'mk_catalog'
+require 'etc'
+
+=begin
+- Start the n coaster services: n is the number of sites in the whitelist
+- Build coasters.sites.xml: In order to do this get all the greensites in an array of objects and iterate through them to generate the coasters.xml
+- Start the workers where each worker connects to a servcie
+=end
+
+# **Ketan ARGV[0]=greensites, ARGV[1]=extenci (why?: it's just an appname), ARGV[2]=workercontact in the form of http://Hosturl:port
+$stdout.sync = true
+
+# FIXME: Is this the best way to get lib functions from bin/grid into RUBYLIB ?
+$:[$:.length] = File.dirname($0)
+
+
+class Site
+  attr_accessor :grid_resource, :gridftp, :data_dir, :app_dir, :name, :sport, :wport
+  attr_reader :submit_file
+
+  # **Ketan: Generate the worker job
+  def gen_submit(count = 1)
+
+    ov=$VERBOSE
+    $VERBOSE=nil
+    workerExecutable = `which worker.pl`
+    workerWrapper = `which run-worker.sh`
+    $VERBOSE=ov
+
+    #   workerContact = "http://communicado.ci.uchicago.edu:36906"
+    #   **Ketan: This will change, only read the url and supply port from the service.wports file
+    workerContact = ARGV[2]
+
+    job = %Q[
+      universe = grid
+      stream_output = False
+      stream_error = False
+      transfer_executable = true
+      periodic_remove = JobStatus == 5
+      notification = Never
+
+      globus_rsl    = (maxwalltime=240)
+      grid_resource = <%= @grid_resource %>
+      executable    = #{workerWrapper}
+      arguments     = #{workerContact}:<%= @wport.to_i %> <%= @name.gsub(/__.*/,"") %> /tmp
+      environment   = WORKER_LOGGING_LEVEL=INFO
+      Input         = #{workerExecutable}
+      Error         = condor/$(Process).err
+      Output        = condor/$(Process).out
+      log           = condor.log
+
+      queue #{count}
+    ]
+
+    # **Ketan: Does this submits the worker? If yes, this would be required to be submitted for each service.
+    ERB.new(job.gsub(/^\s+/, ""), 0, "%<>", "@submit_file").result(binding)
+  
+  end
+  
+  # **Ketan: Submit the above generated job `count` times
+  def submit_job(count)
+    puts "submit_job: Submitting #{@name} #{count} jobs"
+    count = count.to_i
+    output = ""
+    submitfile = gen_submit(count)
+    IO.popen("condor_submit", "w+") do |submit|
+      submit.puts submitfile
+      submit.close_write
+      output = submit.read
+    end
+    output
+  end
+
+  def queued
+    ov=$VERBOSE
+    $VERBOSE=nil
+    jobs = `condor_q  #{$username} -const 'GridResource == \"#{@grid_resource}\" && JobStatus == 1' -format \"%s \" GlobalJobId`
+    $VERBOSE=ov
+    jobs.split(" ").size
+  end
+
+  def running
+    ov=$VERBOSE
+    $VERBOSE=nil
+    jobs = `condor_q #{$username} -const 'GridResource == \"#{@grid_resource}\" && JobStatus == 2' -format \"%s \" GlobalJobId`
+    $VERBOSE=ov
+    jobs.split(" ").size
+  end
+end
+
+=begin
+# For reference:
+JobStatus in job ClassAds
+
+0	Unexpanded	U
+1	Idle	I
+2	Running	R
+3	Removed	X
+4	Completed	C
+5	Held	H
+6	Submission_err	E
+=end
+
+# **Ketan: Program execution starts from here
+if __FILE__ == $0
+  raise "No greenlist file" if !ARGV[0]
+
+def dump(file, template, binding)
+ file_out = File.open(file, "w")
+ file_out.puts ERB.new(template, 0, "%<>").result(binding)
+ file_out.close
+end
+
+  # **Ketan: Read the lines in greensites file and start that many services
+  numlines = %x{wc -l #{ARGV[0]}}.split.first.to_i
+
+  # **Ketan: Remove any worker or service ports file present from the previous runs
+  # removing service.sports and service.wports
+  system("rm -f service.sports service.wports")
+
+  system("start-swift-service #{numlines} &")
+  
+  service_ports=[]
+  worker_ports=[]
+  
+  puts "sleeping, waiting for all services to start and write their ports"
+  sleep 40
+  
+  sports_file=File.open("service.sports")
+  sports_file.each_line { |line|
+       service_ports.push line
+  }
+
+  sports_file.close
+  wports_file=File.open("service.wports")
+  wports_file.each_line { |line|
+      worker_ports.push line
+  }
+
+  wports_file.close
+ 
+  puts "service ports"
+  puts service_ports
+  puts "============="
+  puts "worker ports"
+  puts worker_ports
+
+  # **Ketan: This will change, should be the first line of service.
+  
+  start_port = 61100 # FIXME
+  #start_port = 
+  ctr        = 0
+  threads    = []
+  ARGV[1]    = "scec" if !ARGV[1]
+  greenlist  = IO.readlines(ARGV[0]).map { |line| line.chomp! }
+  $username = Etc.getlogin
+
+  puts "Username = #{$username}"
+
+  minSiteJobs = 2
+  paddedDemand = 0
+  swiftDemand = 0
+  totalCores = 0
+  totalRunning = 0
+
+  ress_parse(ARGV[1]) do |name, value|
+    next if not greenlist.index(name) and not greenlist.empty?
+    totalCores += (value.throttle * 100 + 2).to_i
+  end
+  
+  puts "totalCores for green sites = #{totalCores}"
+  
+  demandThread = Thread.new("monitor-demand") do |t|
+    puts "starting demand thread"
+    while true do
+      puts "in demand thread"
+      swiftDemand = IO.read("swiftDemand").to_i  # Replace this with sensor of Swift demand
+      # swiftDemand = 15
+      paddedDemand = (swiftDemand * 1.2).to_i
+      ov=$VERBOSE;$VERBOSE=nil
+      totalRunning = `condor_q #{$username} -const 'JobStatus == 2' -format \"%s \" GlobalJobId`.split(" ").size
+      $VERBOSE=ov
+      puts "*** demandThread: swiftDemand=#{swiftDemand} paddedDemand=#{paddedDemand} totalRunning=#{totalRunning}"
+      sleep 60
+    end
+  end
+
+  sites=Array.new
+      
+  ress_parse(ARGV[1]) do |name, value|
+    next if not greenlist.index(name) and not greenlist.empty?
+    site               = Site.new
+    site.name          = name
+    site.grid_resource = "gt2 #{value.url}/jobmanager-#{value.jm}"
+    #site.grid_resource = "gt2 #{value.url}/jobmanager-fork"
+    site.gridftp       = "gsiftp://#{value.url}"
+    site.app_dir       = value.app_dir
+    site.data_dir      = value.data_dir
+    #site.port          = start_port + ctr
+    site.sport          = service_ports[ctr] #**Ketan: assuming this is the worker that needs to connect back to its corresponding service
+    site.wport          = worker_ports[ctr]
+
+    sites.push site
+
+    # **Ketan: can put the coasters_osg.xml template right here.
+   coaster_sites = %q[
+      <config>
+      <% ctr = 0
+        sites.each do |asite|
+        gftp      = asite.gridftp
+        app_dir  = asite.app_dir
+        data_dir = asite.data_dir
+        sitename = asite.name
+      %>
+      <pool handle="<%=sitename%>">
+       <execution provider="coaster-persistent" url="http://localhost:<%= service_ports[ctr].chomp %>" jobmanager="local:local" />
+       <profile namespace="globus" key="workerManager">passive</profile>
+       <profile namespace="karajan" key="initialScore">10000</profile>
+       <profile namespace="karajan" key="jobThrottle">2.99</profile>
+       <profile namespace="globus" key="workerLoggingLevel">DEBUG</profile>
+       <profile namespace="globus" key="jobsPerNode">16</profile>
+       <gridftp  url="<%=gftp%>"/>
+       <workdirectory><%=data_dir%>/swift_scratch</workdirectory>
+      </pool>
+      <% ctr += 1
+         end 
+      %>
+     </config>
+    ] 
+    
+    dump("coaster_osg.xml", coaster_sites, binding) 
+    
+    # local per-site attributes:
+    cores = (value.throttle * 100 + 2).to_i
+    siteFraction = cores.to_f / totalCores.to_f
+    siteTargetRunning = [ (swiftDemand.to_f * siteFraction), minSiteJobs ].max
+    siteTargetQueued = [ (swiftDemand.to_f * siteFraction), minSiteJobs ].max
+    
+    printf "site: %5d cores %2d%% %s\n", cores, siteFraction * 100, name
+    targetQueued = 3
+    
+    site.gen_submit
+    
+    threads << Thread.new(name) do |job|
+      trip=0
+      while true do
+        if ( (swiftDemand) > totalRunning ) then
+          # demands > running: enforce N-queued algorithm
+          queued = site.queued
+          running = site.running
+          printf "trip %d site %s running %d queued %d\n", trip, name,running,queued
+          if (running+queued) == 0 then
+            newJobs = [ (paddedDemand * siteFraction).to_i, minSiteJobs ].max
+            printf "trip %d site %s empty - submitting %d (%d%% of demand %d)\n",
+              trip, name, newJobs, siteFraction * 100, paddedDemand
+            site.submit_job(newJobs)
+          elsif queued == 0 then
+            toRun = [ running * 1.2, [(paddedDemand * siteFraction).to_i, minSiteJobs ].max ].max
+            printf "trip %d site %s queued %d target %d has drained queue - submitting %d\n",
+              trip, name, queued, targetQueued, toRun
+            site.submit_job(toRun)
+          elsif queued < targetQueued
+            printf "trip %d site %s queued %d below target %d - submitting %d\n",
+              trip, name, queued, targetQueued, targetQueued-queued
+            site.submit_job(targetQueued - queued)
+          end
+          trip += 1
+          # puts "#{name}: #{total}"
+        end
+      sleep 60
+      end
+    end
+    ctr += 1
+  end
+end
+threads.each { |job| job.join }
+puts "All threads completed."
+
+# TODO:
+#
+# tag jobs for each run uniquely, and track them as a unique factory instance
+#
+
+=begin
+
+"Keep N Queued" Algorithm
+
+Goal:
+- monitor a running swift script to track its changing demand for cores
+- increase the # of running workers to meet the demand
+- let workers that are idle time out when supply is greater than demand
+
+Initially:
+- set a constant demand
+- determine #cores at each site
+
+initialPressure = 1.2  # increase demand 
+initialDemand = 50     # initial demand prior to first poll of Swift, to prime the worker pool ahead of Swift demand
+
+- set a constant target queued for each site based on ncores
+- set a target #running 
+
+THREAD 0:
+  demand = initialDemand
+  for each site
+    site.need = (site.cores/totalcores) * demand
+  sleep delay
+
+  while swiftScriptIsRunning
+    get demand
+    get #running
+
+  
+THREAD for each site
+  while swiftScriptIsRunning
+    get site.running
+    get set.queued  
+    need = demand - running
+    if need > 0
+      if running+queued = 0
+
+keep queued on each site:
+ max( expectation, 50% of observation )
+
+     toalc=1000 
+     sitec = 200 20% d=100 ex=20 q=20
+     r=50 q=25
+=end
+
+#dump("coaster_osg.xml", coaster_sites, binding)
+

Deleted: trunk/bin/grid/my-swift-workers
===================================================================
--- trunk/bin/grid/my-swift-workers	2011-11-28 18:55:02 UTC (rev 5318)
+++ trunk/bin/grid/my-swift-workers	2011-11-29 16:12:03 UTC (rev 5319)
@@ -1,337 +0,0 @@
-#! /usr/bin/env ruby
-
-$stdout.sync = true
-# FIXME: Is this the best way to get lib functions from bin/grid into RUBYLIB ?
-$:[$:.length] = File.dirname($0)
-
-require 'erb'
-require 'mk_catalog'
-require 'etc'
-
-=begin
-- Start the n coaster services: n is the number of sites in the whitelist
-- Build coasters.sites.xml: In order to do this get all the greensites in an array of objects and iterate through them to generate the coasters.xml
-- Start the workers where each worker connects to a servcie
-=end
-
-# **Ketan ARGV[0]=greensites, ARGV[1]=extenci (why?: it's just an appname), ARGV[2]=workercontact in the form of http://Hosturl:port
-$stdout.sync = true
-
-# FIXME: Is this the best way to get lib functions from bin/grid into RUBYLIB ?
-$:[$:.length] = File.dirname($0)
-
-
-class Site
-  attr_accessor :grid_resource, :gridftp, :data_dir, :app_dir, :name, :sport, :wport
-  attr_reader :submit_file
-
-  # **Ketan: Generate the worker job
-  def gen_submit(count = 1)
-
-    ov=$VERBOSE
-    $VERBOSE=nil
-    workerExecutable = `which worker.pl`
-    workerWrapper = `which run-worker.sh`
-    $VERBOSE=ov
-
-    #   workerContact = "http://communicado.ci.uchicago.edu:36906"
-    #   **Ketan: This will change, only read the url and supply port from the service.wports file
-    workerContact = ARGV[2]
-
-    job = %Q[
-      universe = grid
-      stream_output = False
-      stream_error = False
-      transfer_executable = true
-      periodic_remove = JobStatus == 5
-      notification = Never
-
-      globus_rsl    = (maxwalltime=240)
-      grid_resource = <%= @grid_resource %>
-      executable    = #{workerWrapper}
-      arguments     = #{workerContact}:<%= @wport.to_i %> <%= @name.gsub(/__.*/,"") %> /tmp
-      environment   = WORKER_LOGGING_LEVEL=INFO
-      Input         = #{workerExecutable}
-      Error         = condor/$(Process).err
-      Output        = condor/$(Process).out
-      log           = condor.log
-
-      queue #{count}
-    ]
-
-    # **Ketan: Does this submits the worker? If yes, this would be required to be submitted for each service.
-    ERB.new(job.gsub(/^\s+/, ""), 0, "%<>", "@submit_file").result(binding)
-  
-  end
-  
-  # **Ketan: Submit the above generated job `count` times
-  def submit_job(count)
-    puts "submit_job: Submitting #{@name} #{count} jobs"
-    count = count.to_i
-    output = ""
-    submitfile = gen_submit(count)
-    IO.popen("condor_submit", "w+") do |submit|
-      submit.puts submitfile
-      submit.close_write
-      output = submit.read
-    end
-    output
-  end
-
-  def queued
-    ov=$VERBOSE
-    $VERBOSE=nil
-    jobs = `condor_q  #{$username} -const 'GridResource == \"#{@grid_resource}\" && JobStatus == 1' -format \"%s \" GlobalJobId`
-    $VERBOSE=ov
-    jobs.split(" ").size
-  end
-
-  def running
-    ov=$VERBOSE
-    $VERBOSE=nil
-    jobs = `condor_q #{$username} -const 'GridResource == \"#{@grid_resource}\" && JobStatus == 2' -format \"%s \" GlobalJobId`
-    $VERBOSE=ov
-    jobs.split(" ").size
-  end
-end
-
-=begin
-# For reference:
-JobStatus in job ClassAds
-
-0	Unexpanded	U
-1	Idle	I
-2	Running	R
-3	Removed	X
-4	Completed	C
-5	Held	H
-6	Submission_err	E
-=end
-
-# **Ketan: Program execution starts from here
-if __FILE__ == $0
-  raise "No greenlist file" if !ARGV[0]
-
-def dump(file, template, binding)
- file_out = File.open(file, "w")
- file_out.puts ERB.new(template, 0, "%<>").result(binding)
- file_out.close
-end
-
-  # **Ketan: Read the lines in greensites file and start that many services
-  numlines = %x{wc -l #{ARGV[0]}}.split.first.to_i
-
-  # **Ketan: Remove any worker or service ports file present from the previous runs
-  # removing service.sports and service.wports
-  system("rm -f service.sports service.wports")
-
-  system("start-swift-service #{numlines} &")
-  
-  service_ports=[]
-  worker_ports=[]
-  
-  puts "sleeping, waiting for all services to start and write their ports"
-  sleep 40
-  
-  sports_file=File.open("service.sports")
-  sports_file.each_line { |line|
-       service_ports.push line
-  }
-
-  sports_file.close
-  wports_file=File.open("service.wports")
-  wports_file.each_line { |line|
-      worker_ports.push line
-  }
-
-  wports_file.close
- 
-  puts "service ports"
-  puts service_ports
-  puts "============="
-  puts "worker ports"
-  puts worker_ports
-
-  # **Ketan: This will change, should be the first line of service.
-  
-  start_port = 61100 # FIXME
-  #start_port = 
-  ctr        = 0
-  threads    = []
-  ARGV[1]    = "scec" if !ARGV[1]
-  greenlist  = IO.readlines(ARGV[0]).map { |line| line.chomp! }
-  $username = Etc.getlogin
-
-  puts "Username = #{$username}"
-
-  minSiteJobs = 2
-  paddedDemand = 0
-  swiftDemand = 0
-  totalCores = 0
-  totalRunning = 0
-
-  ress_parse(ARGV[1]) do |name, value|
-    next if not greenlist.index(name) and not greenlist.empty?
-    totalCores += (value.throttle * 100 + 2).to_i
-  end
-  
-  puts "totalCores for green sites = #{totalCores}"
-  
-  demandThread = Thread.new("monitor-demand") do |t|
-    puts "starting demand thread"
-    while true do
-      puts "in demand thread"
-      swiftDemand = IO.read("swiftDemand").to_i  # Replace this with sensor of Swift demand
-      # swiftDemand = 15
-      paddedDemand = (swiftDemand * 1.2).to_i
-      ov=$VERBOSE;$VERBOSE=nil
-      totalRunning = `condor_q #{$username} -const 'JobStatus == 2' -format \"%s \" GlobalJobId`.split(" ").size
-      $VERBOSE=ov
-      puts "*** demandThread: swiftDemand=#{swiftDemand} paddedDemand=#{paddedDemand} totalRunning=#{totalRunning}"
-      sleep 60
-    end
-  end
-
-  sites=Array.new
-      
-  ress_parse(ARGV[1]) do |name, value|
-    next if not greenlist.index(name) and not greenlist.empty?
-    site               = Site.new
-    site.name          = name
-    site.grid_resource = "gt2 #{value.url}/jobmanager-#{value.jm}"
-    #site.grid_resource = "gt2 #{value.url}/jobmanager-fork"
-    site.gridftp       = "gsiftp://#{value.url}"
-    site.app_dir       = value.app_dir
-    site.data_dir      = value.data_dir
-    #site.port          = start_port + ctr
-    site.sport          = service_ports[ctr] #**Ketan: assuming this is the worker that needs to connect back to its corresponding service
-    site.wport          = worker_ports[ctr]
-
-    sites.push site
-
-    # **Ketan: can put the coasters_osg.xml template right here.
-   coaster_sites = %q[
-      <config>
-      <% ctr = 0
-        sites.each do |asite|
-        gftp      = asite.gridftp
-        app_dir  = asite.app_dir
-        data_dir = asite.data_dir
-        sitename = asite.name
-      %>
-      <pool handle="<%=sitename%>">
-       <execution provider="coaster-persistent" url="http://localhost:<%= service_ports[ctr].chomp %>" jobmanager="local:local" />
-       <profile namespace="globus" key="workerManager">passive</profile>
-       <profile namespace="karajan" key="initialScore">10000</profile>
-       <profile namespace="karajan" key="jobThrottle">2.99</profile>
-       <profile namespace="globus" key="jobsPerNode">16</profile>
-       <gridftp  url="<%=gftp%>"/>
-       <workdirectory><%=data_dir%>/swift_scratch</workdirectory>
-      </pool>
-      <% ctr += 1
-         end 
-      %>
-     </config>
-    ] 
-    
-    dump("coaster_osg.xml", coaster_sites, binding) 
-    
-    # local per-site attributes:
-    cores = (value.throttle * 100 + 2).to_i
-    siteFraction = cores.to_f / totalCores.to_f
-    siteTargetRunning = [ (swiftDemand.to_f * siteFraction), minSiteJobs ].max
-    siteTargetQueued = [ (swiftDemand.to_f * siteFraction), minSiteJobs ].max
-    
-    printf "site: %5d cores %2d%% %s\n", cores, siteFraction * 100, name
-    targetQueued = 3
-    
-    site.gen_submit
-    
-    threads << Thread.new(name) do |job|
-      trip=0
-      while true do
-        if ( (swiftDemand) > totalRunning ) then
-          # demands > running: enforce N-queued algorithm
-          queued = site.queued
-          running = site.running
-          printf "trip %d site %s running %d queued %d\n", trip, name,running,queued
-          if (running+queued) == 0 then
-            newJobs = [ (paddedDemand * siteFraction).to_i, minSiteJobs ].max
-            printf "trip %d site %s empty - submitting %d (%d%% of demand %d)\n",
-              trip, name, newJobs, siteFraction * 100, paddedDemand
-            site.submit_job(newJobs)
-          elsif queued == 0 then
-            toRun = [ running * 1.2, [(paddedDemand * siteFraction).to_i, minSiteJobs ].max ].max
-            printf "trip %d site %s queued %d target %d has drained queue - submitting %d\n",
-              trip, name, queued, targetQueued, toRun
-            site.submit_job(toRun)
-          elsif queued < targetQueued
-            printf "trip %d site %s queued %d below target %d - submitting %d\n",
-              trip, name, queued, targetQueued, targetQueued-queued
-            site.submit_job(targetQueued - queued)
-          end
-          trip += 1
-          # puts "#{name}: #{total}"
-        end
-      sleep 60
-      end
-    end
-    ctr += 1
-  end
-end
-threads.each { |job| job.join }
-puts "All threads completed."
-
-# TODO:
-#
-# tag jobs for each run uniquely, and track them as a unique factory instance
-#
-
-=begin
-
-"Keep N Queued" Algorithm
-
-Goal:
-- monitor a running swift script to track its changing demand for cores
-- increase the # of running workers to meet the demand
-- let workers that are idle time out when supply is greater than demand
-
-Initially:
-- set a constant demand
-- determine #cores at each site
-
-initialPressure = 1.2  # increase demand 
-initialDemand = 50     # initial demand prior to first poll of Swift, to prime the worker pool ahead of Swift demand
-
-- set a constant target queued for each site based on ncores
-- set a target #running 
-
-THREAD 0:
-  demand = initialDemand
-  for each site
-    site.need = (site.cores/totalcores) * demand
-  sleep delay
-
-  while swiftScriptIsRunning
-    get demand
-    get #running
-
-  
-THREAD for each site
-  while swiftScriptIsRunning
-    get site.running
-    get set.queued  
-    need = demand - running
-    if need > 0
-      if running+queued = 0
-
-keep queued on each site:
- max( expectation, 50% of observation )
-
-     toalc=1000 
-     sitec = 200 20% d=100 ex=20 q=20
-     r=50 q=25
-=end
-
-#dump("coaster_osg.xml", coaster_sites, binding)
-

Modified: trunk/bin/grid/start-grid-service
===================================================================
--- trunk/bin/grid/start-grid-service	2011-11-28 18:55:02 UTC (rev 5318)
+++ trunk/bin/grid/start-grid-service	2011-11-29 16:12:03 UTC (rev 5319)
@@ -62,6 +62,7 @@
       <profile namespace="karajan" key="initialScore">10000</profile>
       <!-- <filesystem provider="local" url="none" /> -->
       <profile namespace="swift" key="stagingMethod">proxy</profile>
+      <profile namespace="globus" key="workerLoggingLevel">DEBUG</profile>
       <workdirectory>/tmp/$USER</workdirectory>
     </pool>
     
@@ -72,6 +73,7 @@
       <profile key="jobThrottle" namespace="karajan">$THROTTLE</profile>
       <profile namespace="karajan" key="initialScore">10000</profile>
       <filesystem provider="local" url="none" />
+      <profile namespace="globus" key="workerLoggingLevel">DEBUG</profile>
       <profile namespace="swift" key="stagingMethod">proxy</profile>
       <workdirectory>/tmp/$USER</workdirectory>
     </pool>

Modified: trunk/bin/grid/swift-workers
===================================================================
--- trunk/bin/grid/swift-workers	2011-11-28 18:55:02 UTC (rev 5318)
+++ trunk/bin/grid/swift-workers	2011-11-29 16:12:03 UTC (rev 5319)
@@ -43,13 +43,13 @@
       stream_error = False
       transfer_executable = true
       periodic_remove = JobStatus == 5
-      notification = Never
+      notification = NEVER
 
       globus_rsl = (maxwalltime=240)
       grid_resource = <%= @grid_resource %>
       executable = #{workerWrapper}
       arguments = #{workerContact} <%= @name.gsub(/__.*/,"") %> /tmp
-      environment = WORKER_LOGGING_LEVEL=INFO
+      environment = WORKER_LOGGING_LEVEL=DEBUG
       Input   = #{workerExecutable}
       Error   = condor/$(Process).err
       Output  = condor/$(Process).out



From ketan at ci.uchicago.edu  Tue Nov 29 10:23:35 2011
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Tue, 29 Nov 2011 10:23:35 -0600 (CST)
Subject: [Swift-commit] r5320 - trunk/bin/grid
Message-ID: <20111129162335.6FB6F9CCA8@svn.ci.uchicago.edu>

Author: ketan
Date: 2011-11-29 10:23:35 -0600 (Tue, 29 Nov 2011)
New Revision: 5320

Added:
   trunk/bin/grid/gwms-swift-workers
Log:
added gwms swift workers script

Added: trunk/bin/grid/gwms-swift-workers
===================================================================
--- trunk/bin/grid/gwms-swift-workers	                        (rev 0)
+++ trunk/bin/grid/gwms-swift-workers	2011-11-29 16:23:35 UTC (rev 5320)
@@ -0,0 +1,256 @@
+#! /usr/bin/env ruby
+
+$stdout.sync = true
+
+# FIXME: Is this the best way to get lib functions from bin/grid into RUBYLIB ?
+
+$:[$:.length] = File.dirname($0)
+
+require 'mk_catalog'
+require 'etc'
+
+class Site
+  attr_accessor :grid_resource, :gridftp, :data_dir, :app_dir, :name, :port
+  attr_reader :submit_file
+
+#      executable = <%= @app_dir %>/worker.pl  # FIXME (below)
+
+#       transfer_executable = True
+
+#      executable = /home/wilde/swift/src/0.92/cog/modules/swift/dist/swift-svn/bin/worker.pl
+#      arguments = http://128.135.125.17:<%= port %> <%= name %> /tmp 14400
+
+# WORKER_LOGGING_LEVEL=$LOGLEVEL $HOME/swift_gridtools/worker.pl $SERVICEURL swork${worker} $LOGDIR >& /dev/null &
+
+  def gen_submit(count = 1)
+
+    ov=$VERBOSE
+    $VERBOSE=nil
+    workerExecutable = `which worker.pl`
+    workerWrapper = `which run-worker.sh`
+    $VERBOSE=ov
+#   workerContact = "http://communicado.ci.uchicago.edu:36906"
+    workerContact = ARGV[2]
+
+# for submit file:      log = condor.log
+#       <% count.times { %>queue
+#       <% } %>
+#      log     = condor/$(Process).log
+
+    job = %Q[
+      universe = grid
+      stream_output = False
+      stream_error = False
+      transfer_executable = true
+      periodic_remove = JobStatus == 5
+      notification = NEVER
+
+      globus_rsl = (maxwalltime=240)
+      grid_resource = <%= @grid_resource %>
+      executable = #{workerWrapper}
+      arguments = #{workerContact} <%= @name.gsub(/__.*/,"") %> /tmp
+      environment = WORKER_LOGGING_LEVEL=DEBUG
+      Input   = #{workerExecutable}
+      Error   = condor/$(Process).err
+      Output  = condor/$(Process).out
+      log     = condor.log
+
+      queue #{count}
+    ]
+    ERB.new(job.gsub(/^\s+/, ""), 0, "%<>", "@submit_file").result(binding)
+  end
+
+  def submit_job(count)
+    puts "submit_job: Submitting #{@name} #{count} jobs"
+    count = count.to_i
+    output = ""
+    submitfile = gen_submit(count)
+    IO.popen("condor_submit", "w+") do |submit|
+      submit.puts submitfile
+      submit.close_write
+      output = submit.read
+    end
+    output
+  end
+
+  def queued
+    ov=$VERBOSE
+    $VERBOSE=nil
+    jobs = `condor_q  #{$username} -const 'GridResource == \"#{@grid_resource}\" && JobStatus == 1' -format \"%s \" GlobalJobId`
+    $VERBOSE=ov
+    jobs.split(" ").size
+  end
+
+  def running
+    ov=$VERBOSE
+    $VERBOSE=nil
+    jobs = `condor_q #{$username} -const 'GridResource == \"#{@grid_resource}\" && JobStatus == 2' -format \"%s \" GlobalJobId`
+    $VERBOSE=ov
+    jobs.split(" ").size
+  end
+
+end
+
+=begin
+# For reference:
+JobStatus in job ClassAds
+
+0	Unexpanded	U
+1	Idle	I
+2	Running	R
+3	Removed	X
+4	Completed	C
+5	Held	H
+6	Submission_err	E
+=end
+
+if __FILE__ == $0
+  raise "No greenlist file" if !ARGV[0]
+
+  start_port = 61100 # FIXME
+  ctr        = 0
+  threads    = []
+  ARGV[1]    = "scec" if !ARGV[1]
+  greenlist  = IO.readlines(ARGV[0]).map { |line| line.chomp! }
+  $username = Etc.getlogin
+
+  puts "Username = #{$username}"
+
+  minSiteJobs = 2
+  paddedDemand = 0
+  swiftDemand = 0
+  totalCores = 0
+  totalRunning = 0
+
+  ress_parse(ARGV[1]) do |name, value|
+    next if not greenlist.index(name) and not greenlist.empty?
+    totalCores += (value.throttle * 100 + 2).to_i
+  end
+  puts "totalCores for green sites = #{totalCores}"
+
+  demandThread = Thread.new("monitor-demand") do |t|
+    puts "starting demand thread"
+    while true do
+      puts "in demand thread"
+      swiftDemand = IO.read("swiftDemand").to_i  # Replace this with sensor of Swift demand
+      # swiftDemand = 15
+      paddedDemand = (swiftDemand * 1.2).to_i
+      ov=$VERBOSE;$VERBOSE=nil
+      totalRunning = `condor_q #{$username} -const 'JobStatus == 2' -format \"%s \" GlobalJobId`.split(" ").size
+      $VERBOSE=ov
+      puts "*** demandThread: swiftDemand=#{swiftDemand} paddedDemand=#{paddedDemand} totalRunning=#{totalRunning}"
+      sleep 60
+    end
+  end
+
+  ress_parse(ARGV[1]) do |name, value|
+    next if not greenlist.index(name) and not greenlist.empty?
+    site               = Site.new
+    site.name          = name
+    site.grid_resource = "gt2 #{value.url}/jobmanager-#{value.jm}"
+    #site.grid_resource = "fork"
+    site.gridftp       = "gsiftp://#{value.url}"
+    site.app_dir       = value.app_dir
+    site.data_dir      = value.data_dir
+    site.port          = start_port + ctr
+
+    # local per-site attributes:
+
+    cores = (value.throttle * 100 + 2).to_i
+    siteFraction = cores.to_f / totalCores.to_f
+    siteTargetRunning = [ (swiftDemand.to_f * siteFraction), minSiteJobs ].max
+    siteTargetQueued = [ (swiftDemand.to_f * siteFraction), minSiteJobs ].max
+
+    printf "site: %5d cores %2d%% %s\n", cores, siteFraction * 100, name
+    targetQueued = 3
+
+    site.gen_submit
+
+    threads << Thread.new(name) do |job|
+      trip=0
+      while true do
+        if ( (swiftDemand) > totalRunning ) then
+          # demands > running: enforce N-queued algorithm
+          queued = site.queued
+          running = site.running
+          printf "trip %d site %s running %d queued %d\n", trip, name,running,queued
+          if (running+queued) == 0 then
+            newJobs = [ (paddedDemand * siteFraction).to_i, minSiteJobs ].max
+            printf "trip %d site %s empty - submitting %d (%d%% of demand %d)\n",
+              trip, name, newJobs, siteFraction * 100, paddedDemand
+            site.submit_job(newJobs)
+          elsif queued == 0 then
+            toRun = [ running * 1.2, [(paddedDemand * siteFraction).to_i, minSiteJobs ].max ].max
+            printf "trip %d site %s queued %d target %d has drained queue - submitting %d\n",
+              trip, name, queued, targetQueued, toRun
+            site.submit_job(toRun)
+          elsif queued < targetQueued
+            printf "trip %d site %s queued %d below target %d - submitting %d\n",
+              trip, name, queued, targetQueued, targetQueued-queued
+            site.submit_job(targetQueued - queued)
+          end
+          trip += 1
+          # puts "#{name}: #{total}"
+        end
+      sleep 60
+      end
+    end
+
+    ctr += 1
+  end
+end
+threads.each { |job| job.join }
+puts "All threads completed."
+
+# TODO:
+#
+# tag jobs for each run uniquely, and track them as a unique factory instance
+#
+
+=begin
+
+"Keep N Queued" Algorithm
+
+Goal:
+- monitor a running swift script to track its changing demand for cores
+- increase the # of running workers to meet the demand
+- let workers that are idle time out when supply is greater than demand
+
+Initially:
+- set a constant demand
+- determine #cores at each site
+
+initialPressure = 1.2  # increase demand 
+initialDemand = 50     # initial demand prior to first poll of Swift, to prime the worker pool ahead of Swift demand
+
+- set a constant target queued for each site based on ncores
+- set a target #running 
+
+THREAD 0:
+  demand = initialDemand
+  for each site
+    site.need = (site.cores/totalcores) * demand
+  sleep delay
+
+
+  while swiftScriptIsRunning
+    get demand
+    get #running
+
+
+  
+THREAD for each site
+  while swiftScriptIsRunning
+    get site.running
+    get set.queued  
+    need = demand - running
+    if need > 0
+      if running+queued = 0
+
+keep queued on each site:
+ max( expectation, 50% of observation )
+
+     toalc=1000 
+     sitec = 200 20% d=100 ex=20 q=20
+     r=50 q=25
+=end


Property changes on: trunk/bin/grid/gwms-swift-workers
___________________________________________________________________
Added: svn:executable
   + *



From ketan at ci.uchicago.edu  Tue Nov 29 10:31:59 2011
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Tue, 29 Nov 2011 10:31:59 -0600 (CST)
Subject: [Swift-commit] r5321 - trunk/bin/grid
Message-ID: <20111129163159.11FC79CCA8@svn.ci.uchicago.edu>

Author: ketan
Date: 2011-11-29 10:31:58 -0600 (Tue, 29 Nov 2011)
New Revision: 5321

Modified:
   trunk/bin/grid/gwms-swift-workers
Log:
adjusting parameters in gwms worker

Modified: trunk/bin/grid/gwms-swift-workers
===================================================================
--- trunk/bin/grid/gwms-swift-workers	2011-11-29 16:23:35 UTC (rev 5320)
+++ trunk/bin/grid/gwms-swift-workers	2011-11-29 16:31:58 UTC (rev 5321)
@@ -2,92 +2,75 @@
 
 $stdout.sync = true
 
-# FIXME: Is this the best way to get lib functions from bin/grid into RUBYLIB ?
-
 $:[$:.length] = File.dirname($0)
 
 require 'mk_catalog'
 require 'etc'
 
 class Site
-  attr_accessor :grid_resource, :gridftp, :data_dir, :app_dir, :name, :port
-  attr_reader :submit_file
+    attr_accessor :grid_resource, :gridftp, :data_dir, :app_dir, :name, :port
+    attr_reader :submit_file
 
-#      executable = <%= @app_dir %>/worker.pl  # FIXME (below)
+    def gen_submit(count = 1)
 
-#       transfer_executable = True
+        ov=$VERBOSE
+        $VERBOSE=nil
+        workerExecutable = `which worker.pl`
+        workerWrapper = `which run-worker.sh`
+        $VERBOSE=ov
+        workerContact = ARGV[2]
 
-#      executable = /home/wilde/swift/src/0.92/cog/modules/swift/dist/swift-svn/bin/worker.pl
-#      arguments = http://128.135.125.17:<%= port %> <%= name %> /tmp 14400
+        job = %Q[
+          universe = vanilla
+          stream_output = False
+          stream_error = False
+          transfer_executable = true
+          periodic_remove = JobStatus == 5
+          notification = NEVER
 
-# WORKER_LOGGING_LEVEL=$LOGLEVEL $HOME/swift_gridtools/worker.pl $SERVICEURL swork${worker} $LOGDIR >& /dev/null &
+          #globus_rsl = (maxwalltime=240)
+          #grid_resource = <%= @grid_resource %>
+          executable = #{workerWrapper}
+          arguments = #{workerContact} <%= @name.gsub(/__.*/,"") %> /tmp
+          environment = WORKER_LOGGING_LEVEL=DEBUG
+          Input   = #{workerExecutable}
+          Error   = condor/$(Process).err
+          Output  = condor/$(Process).out
+          log     = condor.log
 
-  def gen_submit(count = 1)
-
-    ov=$VERBOSE
-    $VERBOSE=nil
-    workerExecutable = `which worker.pl`
-    workerWrapper = `which run-worker.sh`
-    $VERBOSE=ov
-#   workerContact = "http://communicado.ci.uchicago.edu:36906"
-    workerContact = ARGV[2]
-
-# for submit file:      log = condor.log
-#       <% count.times { %>queue
-#       <% } %>
-#      log     = condor/$(Process).log
-
-    job = %Q[
-      universe = grid
-      stream_output = False
-      stream_error = False
-      transfer_executable = true
-      periodic_remove = JobStatus == 5
-      notification = NEVER
-
-      globus_rsl = (maxwalltime=240)
-      grid_resource = <%= @grid_resource %>
-      executable = #{workerWrapper}
-      arguments = #{workerContact} <%= @name.gsub(/__.*/,"") %> /tmp
-      environment = WORKER_LOGGING_LEVEL=DEBUG
-      Input   = #{workerExecutable}
-      Error   = condor/$(Process).err
-      Output  = condor/$(Process).out
-      log     = condor.log
-
-      queue #{count}
+          queue #{count}
     ]
-    ERB.new(job.gsub(/^\s+/, ""), 0, "%<>", "@submit_file").result(binding)
-  end
 
-  def submit_job(count)
-    puts "submit_job: Submitting #{@name} #{count} jobs"
-    count = count.to_i
-    output = ""
-    submitfile = gen_submit(count)
-    IO.popen("condor_submit", "w+") do |submit|
-      submit.puts submitfile
-      submit.close_write
-      output = submit.read
+        ERB.new(job.gsub(/^\s+/, ""), 0, "%<>", "@submit_file").result(binding)
     end
-    output
-  end
+    def submit_job(count)
+        puts "submit_job: Submitting #{@name} #{count} jobs"
+        count = count.to_i
+        output = ""
+        submitfile = gen_submit(count)
+        IO.popen("condor_submit", "w+") do |submit|
+            submit.puts submitfile
+            submit.close_write
+            output = submit.read
+        end
+        output
+    end
 
-  def queued
-    ov=$VERBOSE
-    $VERBOSE=nil
-    jobs = `condor_q  #{$username} -const 'GridResource == \"#{@grid_resource}\" && JobStatus == 1' -format \"%s \" GlobalJobId`
-    $VERBOSE=ov
-    jobs.split(" ").size
-  end
+    def queued
+        ov=$VERBOSE
+        $VERBOSE=nil
+        jobs = `condor_q  #{$username} -const 'GridResource == \"#{@grid_resource}\" && JobStatus == 1' -format \"%s \" GlobalJobId`
+        $VERBOSE=ov
+        jobs.split(" ").size
+    end
 
-  def running
-    ov=$VERBOSE
-    $VERBOSE=nil
-    jobs = `condor_q #{$username} -const 'GridResource == \"#{@grid_resource}\" && JobStatus == 2' -format \"%s \" GlobalJobId`
-    $VERBOSE=ov
-    jobs.split(" ").size
-  end
+    def running
+        ov=$VERBOSE
+        $VERBOSE=nil
+        jobs = `condor_q #{$username} -const 'GridResource == \"#{@grid_resource}\" && JobStatus == 2' -format \"%s \" GlobalJobId`
+        $VERBOSE=ov
+        jobs.split(" ").size
+    end
 
 end
 
@@ -105,99 +88,99 @@
 =end
 
 if __FILE__ == $0
-  raise "No greenlist file" if !ARGV[0]
+    raise "No greenlist file" if !ARGV[0]
 
-  start_port = 61100 # FIXME
-  ctr        = 0
-  threads    = []
-  ARGV[1]    = "scec" if !ARGV[1]
-  greenlist  = IO.readlines(ARGV[0]).map { |line| line.chomp! }
-  $username = Etc.getlogin
+    start_port = 61100 # FIXME
+    ctr        = 0
+    threads    = []
+    ARGV[1]    = "scec" if !ARGV[1]
+    greenlist  = IO.readlines(ARGV[0]).map { |line| line.chomp! }
+    $username = Etc.getlogin
 
-  puts "Username = #{$username}"
+    puts "Username = #{$username}"
 
-  minSiteJobs = 2
-  paddedDemand = 0
-  swiftDemand = 0
-  totalCores = 0
-  totalRunning = 0
+    minSiteJobs = 2
+    paddedDemand = 0
+    swiftDemand = 0
+    totalCores = 0
+    totalRunning = 0
 
-  ress_parse(ARGV[1]) do |name, value|
-    next if not greenlist.index(name) and not greenlist.empty?
-    totalCores += (value.throttle * 100 + 2).to_i
-  end
-  puts "totalCores for green sites = #{totalCores}"
+    ress_parse(ARGV[1]) do |name, value|
+        next if not greenlist.index(name) and not greenlist.empty?
+        totalCores += (value.throttle * 100 + 2).to_i
+    end
+    puts "totalCores for green sites = #{totalCores}"
 
-  demandThread = Thread.new("monitor-demand") do |t|
-    puts "starting demand thread"
-    while true do
-      puts "in demand thread"
-      swiftDemand = IO.read("swiftDemand").to_i  # Replace this with sensor of Swift demand
-      # swiftDemand = 15
-      paddedDemand = (swiftDemand * 1.2).to_i
-      ov=$VERBOSE;$VERBOSE=nil
-      totalRunning = `condor_q #{$username} -const 'JobStatus == 2' -format \"%s \" GlobalJobId`.split(" ").size
-      $VERBOSE=ov
-      puts "*** demandThread: swiftDemand=#{swiftDemand} paddedDemand=#{paddedDemand} totalRunning=#{totalRunning}"
-      sleep 60
+    demandThread = Thread.new("monitor-demand") do |t|
+        puts "starting demand thread"
+        while true do
+            puts "in demand thread"
+            swiftDemand = IO.read("swiftDemand").to_i  # Replace this with sensor of Swift demand
+            # swiftDemand = 15
+            paddedDemand = (swiftDemand * 1.2).to_i
+            ov=$VERBOSE;$VERBOSE=nil
+            totalRunning = `condor_q #{$username} -const 'JobStatus == 2' -format \"%s \" GlobalJobId`.split(" ").size
+            $VERBOSE=ov
+            puts "*** demandThread: swiftDemand=#{swiftDemand} paddedDemand=#{paddedDemand} totalRunning=#{totalRunning}"
+            sleep 60
+        end
     end
-  end
 
-  ress_parse(ARGV[1]) do |name, value|
-    next if not greenlist.index(name) and not greenlist.empty?
-    site               = Site.new
-    site.name          = name
-    site.grid_resource = "gt2 #{value.url}/jobmanager-#{value.jm}"
-    #site.grid_resource = "fork"
-    site.gridftp       = "gsiftp://#{value.url}"
-    site.app_dir       = value.app_dir
-    site.data_dir      = value.data_dir
-    site.port          = start_port + ctr
+    ress_parse(ARGV[1]) do |name, value|
+        next if not greenlist.index(name) and not greenlist.empty?
+        site               = Site.new
+        site.name          = name
+        site.grid_resource = "gt2 #{value.url}/jobmanager-#{value.jm}"
+        #site.grid_resource = "fork"
+        site.gridftp       = "gsiftp://#{value.url}"
+        site.app_dir       = value.app_dir
+        site.data_dir      = value.data_dir
+        site.port          = start_port + ctr
 
-    # local per-site attributes:
+        # local per-site attributes:
 
-    cores = (value.throttle * 100 + 2).to_i
-    siteFraction = cores.to_f / totalCores.to_f
-    siteTargetRunning = [ (swiftDemand.to_f * siteFraction), minSiteJobs ].max
-    siteTargetQueued = [ (swiftDemand.to_f * siteFraction), minSiteJobs ].max
+        cores = (value.throttle * 100 + 2).to_i
+        siteFraction = cores.to_f / totalCores.to_f
+        siteTargetRunning = [ (swiftDemand.to_f * siteFraction), minSiteJobs ].max
+        siteTargetQueued = [ (swiftDemand.to_f * siteFraction), minSiteJobs ].max
 
-    printf "site: %5d cores %2d%% %s\n", cores, siteFraction * 100, name
-    targetQueued = 3
+        printf "site: %5d cores %2d%% %s\n", cores, siteFraction * 100, name
+        targetQueued = 3
 
-    site.gen_submit
+        site.gen_submit
 
-    threads << Thread.new(name) do |job|
-      trip=0
-      while true do
-        if ( (swiftDemand) > totalRunning ) then
-          # demands > running: enforce N-queued algorithm
-          queued = site.queued
-          running = site.running
-          printf "trip %d site %s running %d queued %d\n", trip, name,running,queued
-          if (running+queued) == 0 then
-            newJobs = [ (paddedDemand * siteFraction).to_i, minSiteJobs ].max
-            printf "trip %d site %s empty - submitting %d (%d%% of demand %d)\n",
-              trip, name, newJobs, siteFraction * 100, paddedDemand
-            site.submit_job(newJobs)
-          elsif queued == 0 then
-            toRun = [ running * 1.2, [(paddedDemand * siteFraction).to_i, minSiteJobs ].max ].max
-            printf "trip %d site %s queued %d target %d has drained queue - submitting %d\n",
-              trip, name, queued, targetQueued, toRun
-            site.submit_job(toRun)
-          elsif queued < targetQueued
-            printf "trip %d site %s queued %d below target %d - submitting %d\n",
-              trip, name, queued, targetQueued, targetQueued-queued
-            site.submit_job(targetQueued - queued)
-          end
-          trip += 1
-          # puts "#{name}: #{total}"
+        threads << Thread.new(name) do |job|
+            trip=0
+            while true do
+                if ( (swiftDemand) > totalRunning ) then
+                    # demands > running: enforce N-queued algorithm
+                    queued = site.queued
+                    running = site.running
+                    printf "trip %d site %s running %d queued %d\n", trip, name,running,queued
+                    if (running+queued) == 0 then
+                        newJobs = [ (paddedDemand * siteFraction).to_i, minSiteJobs ].max
+                        printf "trip %d site %s empty - submitting %d (%d%% of demand %d)\n",
+                            trip, name, newJobs, siteFraction * 100, paddedDemand
+                        site.submit_job(newJobs)
+                    elsif queued == 0 then
+                        toRun = [ running * 1.2, [(paddedDemand * siteFraction).to_i, minSiteJobs ].max ].max
+                        printf "trip %d site %s queued %d target %d has drained queue - submitting %d\n",
+                            trip, name, queued, targetQueued, toRun
+                        site.submit_job(toRun)
+                    elsif queued < targetQueued
+                        printf "trip %d site %s queued %d below target %d - submitting %d\n",
+                            trip, name, queued, targetQueued, targetQueued-queued
+                        site.submit_job(targetQueued - queued)
+                    end
+                    trip += 1
+                    # puts "#{name}: #{total}"
+                end
+                sleep 60
+            end
         end
-      sleep 60
-      end
-    end
 
-    ctr += 1
-  end
+        ctr += 1
+    end
 end
 threads.each { |job| job.join }
 puts "All threads completed."
@@ -238,7 +221,7 @@
     get #running
 
 
-  
+
 THREAD for each site
   while swiftScriptIsRunning
     get site.running



From ketan at ci.uchicago.edu  Tue Nov 29 10:42:02 2011
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Tue, 29 Nov 2011 10:42:02 -0600 (CST)
Subject: [Swift-commit] r5322 - trunk/docs/siteguide
Message-ID: <20111129164202.A22599CCA8@svn.ci.uchicago.edu>

Author: ketan
Date: 2011-11-29 10:42:02 -0600 (Tue, 29 Nov 2011)
New Revision: 5322

Modified:
   trunk/docs/siteguide/grid
Log:
minor update on siteguide

Modified: trunk/docs/siteguide/grid
===================================================================
--- trunk/docs/siteguide/grid	2011-11-29 16:31:58 UTC (rev 5321)
+++ trunk/docs/siteguide/grid	2011-11-29 16:42:02 UTC (rev 5322)
@@ -177,7 +177,7 @@
 
 -----
 $ ./foreachsite -help
-./foreachsite foreachsite [-resource fork|worker ] [-sites alt-sites-file] scriptname
+./foreachsite [-resource fork|worker ] [-sites alt-sites-file] scriptname
 $ 
 -----
 
@@ -215,8 +215,10 @@
 
 This single coaster service will service all grid sites:
 
+-----
 start-grid-service --loglevel INFO --throttle 3.99 --jobspernode 1 \
                    >& start-grid-service.out
+-----
 
 Starting workers on OSG sites
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~



From ketan at ci.uchicago.edu  Tue Nov 29 11:26:03 2011
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Tue, 29 Nov 2011 11:26:03 -0600 (CST)
Subject: [Swift-commit] r5323 - trunk/bin/grid
Message-ID: <20111129172603.DA8AB9CCA8@svn.ci.uchicago.edu>

Author: ketan
Date: 2011-11-29 11:26:03 -0600 (Tue, 29 Nov 2011)
New Revision: 5323

Modified:
   trunk/bin/grid/gwms-swift-workers
Log:
 

Modified: trunk/bin/grid/gwms-swift-workers
===================================================================
--- trunk/bin/grid/gwms-swift-workers	2011-11-29 16:42:02 UTC (rev 5322)
+++ trunk/bin/grid/gwms-swift-workers	2011-11-29 17:26:03 UTC (rev 5323)
@@ -24,7 +24,11 @@
           universe = vanilla
           stream_output = False
           stream_error = False
+
           transfer_executable = true
+          should_transfer_files = YES
+          WhenToTransferOutput = ON_EXIT
+
           periodic_remove = JobStatus == 5
           notification = NEVER
 



From wozniak at ci.uchicago.edu  Wed Nov 30 14:32:25 2011
From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu)
Date: Wed, 30 Nov 2011 14:32:25 -0600 (CST)
Subject: [Swift-commit] r5325 - in branches/release-0.93: bin etc
Message-ID: <20111130203225.8F4849CCA2@svn.ci.uchicago.edu>

Author: wozniak
Date: 2011-11-30 14:32:25 -0600 (Wed, 30 Nov 2011)
New Revision: 5325

Modified:
   branches/release-0.93/bin/start-coaster-service
   branches/release-0.93/etc/coaster-service.conf
Log:
Multiple minor fixes to start-coaster-service for EC2


Modified: branches/release-0.93/bin/start-coaster-service
===================================================================
--- branches/release-0.93/bin/start-coaster-service	2011-11-29 21:20:57 UTC (rev 5324)
+++ branches/release-0.93/bin/start-coaster-service	2011-11-30 20:32:25 UTC (rev 5325)
@@ -41,16 +41,16 @@
       fi
 
       # Copy and start worker script
-      scp $SWIFT_BIN/$WORKER $WORKER_USERNAME@$MACHINE:$WORKER_WORK > /dev/null 2>&1
+      scp $SWIFT_BIN/$WORKER $WORKER_USERNAME@$MACHINE:$WORKER_LOCATION > /dev/null 2>&1
       echo "Starting worker on $MACHINE"
-      ssh $WORKER_USERNAME@$MACHINE "$WORKER_WORK/$WORKER $EXECUTION_URL $MACHINE $LOG_DIR" &
-      echo $! >> $PID_FILE 
+      ssh $WORKER_USERNAME@$MACHINE "$WORKER_LOCATION/$WORKER $EXECUTION_URL $MACHINE $LOG_DIR" &
+      echo $! >> $PID_FILE
    done
 }
 
 # Start SSH workers
 start-workers-ssh()
-{ 
+{
    PORT=$1
    EXECUTION_URL=http://$IPADDR:$PORT
    if [ -z "$PORT" ]; then
@@ -71,17 +71,17 @@
 
       # Use a relay host
       if [ -n "$WORKER_RELAY_HOST" ]; then
-         ssh $WORKER_USERNAME@$WORKER_RELAY_HOST ssh $MACHINE mkdir -p $WORKER_WORK > /dev/null 2>&1
-         ssh $WORKER_USERNAME@$WORKER_RELAY_HOST "scp /tmp/$WORKER $WORKER_USERNAME@$MACHINE:$WORKER_WORK" > /dev/null 2>&1
+         ssh $WORKER_USERNAME@$WORKER_RELAY_HOST ssh $MACHINE mkdir -p $WORKER_LOCATION > /dev/null 2>&1
+         ssh $WORKER_USERNAME@$WORKER_RELAY_HOST "scp /tmp/$WORKER $WORKER_USERNAME@$MACHINE:$WORKER_LOCATION" > /dev/null 2>&1
          echo Starting worker on $MACHINE
-         ssh $WORKER_USERNAME@$WORKER_RELAY_HOST ssh $WORKER_USERNAME@$MACHINE "WORKER_LOGGING_LEVEL=$WORKER_LOGGING_LEVEL $WORKER_WORK/$WORKER $EXECUTION_URL $MACHINE $WORKER_LOG_DIR" &
+         ssh $WORKER_USERNAME@$WORKER_RELAY_HOST ssh $WORKER_USERNAME@$MACHINE "WORKER_LOGGING_LEVEL=$WORKER_LOGGING_LEVEL $WORKER_LOCATION/$WORKER $EXECUTION_URL $MACHINE $WORKER_LOG_DIR" &
          echo $! >> $PID_FILE
       # Connect directly
-      else  
-         ssh $WORKER_USERNAME@$MACHINE mkdir -p $WORKER_WORK > /dev/null 2>&1
-         scp $SWIFT_BIN/$WORKER $WORKER_USERNAME@$MACHINE:$WORKER_WORK > /dev/null 2>&1
+      else
+         ssh $WORKER_USERNAME@$MACHINE mkdir -p $WORKER_LOCATION > /dev/null 2>&1
+         scp $SWIFT_BIN/$WORKER $WORKER_USERNAME@$MACHINE:$WORKER_LOCATION > /dev/null 2>&1
          echo Starting worker on $MACHINE
-         ssh $WORKER_USERNAME@$MACHINE "$WORKER_WORK/$WORKER $EXECUTION_URL $MACHINE $LOG_DIR" &
+         ssh $WORKER_USERNAME@$MACHINE "$WORKER_LOCATION/$WORKER $EXECUTION_URL $MACHINE $LOG_DIR" &
          echo $! >> $PID_FILE
       fi
    done
@@ -89,7 +89,7 @@
 }
 
 # Start local workers
-start-workers-local() 
+start-workers-local()
 {
    PORT=$1
    EXECUTION_URL=http://$IPADDR:$PORT
@@ -145,16 +145,22 @@
    crash "Cannot find coaster-service.conf!"
 fi
 
+echo "start-coaster-service..."
+echo "configuration: $CONFIG_FILE"
+
 source $CONFIG_FILE
 
-# Determine information needed about this machine
+# Determine IP address to which workers should connect
 if [ -z "$IPADDR" ]; then
-   if [ -x "/sbin/ifconfig" ]; then
+   if [ "$SSH_TUNNELING" == "yes" ]; then
+      IPADDR=localhost
+   elif [ -x "/sbin/ifconfig" ]; then
       IPADDR=$( /sbin/ifconfig | grep 'inet addr' | grep -v 127.0.0.1 | cut -d ':' -f 2 | awk '{print $1}' |head -1)
    else
       crash "Unable to determine IP address of system. Please add to coaster-service.conf"
    fi
 fi
+echo Service address: $IPADDR
 
 # Find swift
 if [ ! -x "$SWIFT" ]; then
@@ -208,7 +214,7 @@
    $SWIFT_BIN/coaster-service -nosec -portfile $SERVICE_PORT_FILE --localport $LOCAL_PORT -passive > $COASTER_LOG 2>&1 &
 elif [ -n  "$SERVICE_PORT" ] && [ -n "$LOCAL_PORT" ]; then
    $SWIFT_BIN/coaster-service -nosec -port $SERVICE_PORT -localport $LOCAL_PORT -passive > $COASTER_LOG 2>&1 &
-else 
+else
    crash "Unknown SERVICE_PORT type specified!"
 fi
 
@@ -250,7 +256,7 @@
    futuregrid)
       start-workers-futuregrid $LOCAL_PORT
       ;;
-   *) 
+   *)
       crash "Unknown WORKER_MODE. Please modify coaster-service.conf"
       ;;
 esac
@@ -261,7 +267,7 @@
 if [ -f "gensites.template" ]; then
    gensites `cat gensites.template` -p $CONFIG_FILE > $RUN_DIR/sites.xml
 else
-   gensites persistent-coasters -p $CONFIG_FILE > $RUN_DIR/sites.xml 
+   gensites persistent-coasters -p $CONFIG_FILE > $RUN_DIR/sites.xml
 fi
 
 # Generate config file
@@ -272,6 +278,11 @@
 wrapperlog.always.transfer=false
 execution.retries=0
 provider.staging.pin.swiftfiles=false
+sitedir.keep=true
 EOF
 fi
 
+# Local Variables:
+# tab-width: 3
+# sh-basic-offset: 3
+# End:

Modified: branches/release-0.93/etc/coaster-service.conf
===================================================================
--- branches/release-0.93/etc/coaster-service.conf	2011-11-29 21:20:57 UTC (rev 5324)
+++ branches/release-0.93/etc/coaster-service.conf	2011-11-30 20:32:25 UTC (rev 5325)
@@ -5,7 +5,7 @@
 export SWIFT=
 
 # Where to copy worker.pl on the remote machine for sites.xml
-export WORKER_WORK=$HOME/swiftwork
+export WORKER_LOCATION=$HOME/swiftwork
 
 # How to launch workers: local, ssh, cobalt, or futuregrid
 export WORKER_MODE=ssh
@@ -29,7 +29,10 @@
 export LOCAL_PORT=
 export SERVICE_PORT=
 
-# start-coaster-service tries to automatically detect the IP address of this system. Specify here if you have multiple network interfaces
+# This is the IP address to which the workers will connect
+# If not given, start-coaster-service tries to automatically detect
+#   the IP address of this system via ifconfig
+# Specify this if you have multiple network interfaces
 export IPADDR=
 
 # Location of the swift-vm-boot scripts



From davidk at ci.uchicago.edu  Wed Nov 30 15:10:02 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Wed, 30 Nov 2011 15:10:02 -0600 (CST)
Subject: [Swift-commit] r5326 - branches/release-0.93/bin
Message-ID: <20111130211002.061DD9CCA2@svn.ci.uchicago.edu>

Author: davidk
Date: 2011-11-30 15:10:01 -0600 (Wed, 30 Nov 2011)
New Revision: 5326

Modified:
   branches/release-0.93/bin/start-coaster-service
   branches/release-0.93/bin/stop-coaster-service
Log:
Better handling for futuregrid VM termination and cleanup


Modified: branches/release-0.93/bin/start-coaster-service
===================================================================
--- branches/release-0.93/bin/start-coaster-service	2011-11-30 20:32:25 UTC (rev 5325)
+++ branches/release-0.93/bin/start-coaster-service	2011-11-30 21:10:01 UTC (rev 5326)
@@ -19,15 +19,15 @@
    fi
 
    echo Starting virtual machines.. please wait
-   $SWIFTVMBOOT_DIR/bin/bootit.sh
+   $SWIFTVMBOOT_DIR/bin/bootit.sh | tee -a bootit.log
    SWIFTVMBOOT_OUTPUT=$SWIFTVMBOOT_DIR/output.json
 
    if [ ! -f "$SWIFTVMBOOT_OUTPUT" ]; then
       crash "Error: Swift VM output file $SWIFTVMBOOT_OUTPUT does not exist!"
    fi
 
-   SWIFTVM_INSTANCES=`grep instance_id $SWIFTVMBOOT_OUTPUT |awk '{print $2}'|sed 's/\"//g;s/,//g;s/null//g'`
-   echo $SWIFTVM_INSTANCES > .swiftvm_instances
+   SWIFTVM_INSTANCE=`grep "Starting up run" bootit.log |awk '{print $4}'`
+   echo $SWIFTVM_INSTANCE >> $HOME/.swift/.swiftvm_instance
 
    WORKER_HOSTS=`grep hostname $SWIFTVMBOOT_OUTPUT |awk '{print $2}'|sed 's/\"//g;s/,//g;s/null//g'`
 
@@ -218,7 +218,7 @@
    crash "Unknown SERVICE_PORT type specified!"
 fi
 
-echo $! > $PID_FILE
+echo $! >> $PID_FILE
 sleep 5
 
 # Determine SERVICE_PORT

Modified: branches/release-0.93/bin/stop-coaster-service
===================================================================
--- branches/release-0.93/bin/stop-coaster-service	2011-11-30 20:32:25 UTC (rev 5325)
+++ branches/release-0.93/bin/stop-coaster-service	2011-11-30 21:10:01 UTC (rev 5326)
@@ -10,7 +10,7 @@
 
 # Location of required files
 PID_FILE="$HOME/.swift/.coaster-service-pids"
-SWIFTVM_INSTANCES="$HOME/.swift/.swiftvm_instances"
+SWIFTVM_INSTANCES="$HOME/.swift/.swiftvm_instance"
 
 # Import settings
 if [ -f "./coaster-service.conf" ]; then



From vytas at ci.uchicago.edu  Tue Nov  8 18:12:06 2011
From: vytas at ci.uchicago.edu (vytas at ci.uchicago.edu)
Date: Wed, 09 Nov 2011 00:12:06 -0000
Subject: [Swift-commit] r5275 - in wwwdev: case_studies case_studies/images
	inc
Message-ID: <20111109001113.373F79CCA3@svn.ci.uchicago.edu>

Author: vytas
Date: 2011-11-08 18:11:13 -0600 (Tue, 08 Nov 2011)
New Revision: 5275

Added:
   wwwdev/case_studies/CIM-Earth.php
   wwwdev/case_studies/images/
   wwwdev/case_studies/images/CIM-Earth.jpg
   wwwdev/case_studies/index.php
   wwwdev/inc/case_study_sidebar.php
Removed:
   wwwdev/case_studies/index.php
Log:
New case study files.

Added: wwwdev/case_studies/CIM-Earth.php
===================================================================
--- wwwdev/case_studies/CIM-Earth.php	                        (rev 0)
+++ wwwdev/case_studies/CIM-Earth.php	2011-11-09 00:11:13 UTC (rev 5275)
@@ -0,0 +1,56 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml">
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
+<title>Swift - Case Studies</title>
+<link href="../css/960_12_col.css" rel="stylesheet" type="text/css" />
+<link href="../css/reset.css" rel="stylesheet" type="text/css" />
+<link href="../css/style3.css" rel="stylesheet" type="text/css" />
+<script type="text/javascript" src="http://use.typekit.com/dbf2lqy.js"></script>
+<script type="text/javascript">try{Typekit.load();}catch(e){}</script>
+</head>
+
+<body>
+<div class="container_12">
+	<!-- header -->
+	<?php require('../inc/header2.php') ?>
+	<!-- end header -->
+<div class="grid_12 page-header">				
+	<h1>Case Studies</h1>
+</div>
+<div class="clear"></div>
+<!-- end .grid_12 -->
+
+ <div class="grid_9">
+	<h3>Energy/Climate/Economics Modelling</h3>
+	<p>The CIM-EARTH project develops a large-scale integrated modeling framework for decision makers in climate and energy policy.  (Foster, Elliott)</p>
+	<img src="images/CIM-Earth.jpg" style="padding:0px 0px 15px 20px;" />
+	<p><b>Approach.</b> Beagle is being used to study land use, land cover, and the impacts of climate change on agriculture and the global food supply.  Using a DSSAT 4.0 (?decision support system for agrotechnology transfer?) crop systems model ported from Windows, a parallel simulation framework was implemented using Swift. Benchmarks of this framework have been performed on a prototype simulation campaign, measuring yield and climate impact for a single crop (maize) across the conterminous USA with daily weather data and climate model output spanning 120 years (1981-2100) and 16 different configurations of local management (fertilizer and irrigation) and cultivar choice.</p>
+	<p><b>Results.</b> Preliminary results of parallel DSSAT on Beagle have been presented in an NSF/advisory board meeting of the CIM-EARTH project. At right, top 2 maps: Preliminary results of parallel DSSAT: maize yields across the USA with intensive nitrogen application and full irrigation; bottom 2 maps show results with no irrigation. Each model run is ~120,000 DSSAT invocations.</p>
+	<p>&nbsp;</p>
+	</div>
+  <!-- end .grid_9 -->
+  <div class="grid_3">
+  <?php require('../inc/case_study_sidebar.php') ?>
+    <p>&nbsp;</p>
+  </div>
+  <!-- end .grid_3 -->
+  
+  <div class="clear"></div>
+				
+  </div>
+<!-- end .container_12 -->
+<!-- footer -->
+<?php require('../inc/footer2.php') ?> 
+<!-- end footer -->
+<script type="text/javascript">
+var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
+document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
+</script>
+<script type="text/javascript">
+try {
+var pageTracker = _gat._getTracker("UA-106257-5");
+pageTracker._trackPageview();
+} catch(err) {}</script>
+</body>
+</html>

Added: wwwdev/case_studies/images/CIM-Earth.jpg
===================================================================
(Binary files differ)


Property changes on: wwwdev/case_studies/images/CIM-Earth.jpg
___________________________________________________________________
Added: svn:mime-type
   + application/octet-stream

Deleted: wwwdev/case_studies/index.php
===================================================================
--- wwwdev/case_studies/index.php	2011-11-09 00:10:53 UTC (rev 5274)
+++ wwwdev/case_studies/index.php	2011-11-09 00:11:13 UTC (rev 5275)
@@ -1,55 +0,0 @@
-<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<html xmlns="http://www.w3.org/1999/xhtml">
-<head>
-<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
-<meta name="robots" content="noindex">
-<title>Swift - Case Studies</title>
-<link href="../css/960_12_col.css" rel="stylesheet" type="text/css" />
-<link href="../css/reset.css" rel="stylesheet" type="text/css" />
-<link href="../css/style3.css" rel="stylesheet" type="text/css" />
-<script type="text/javascript" src="http://use.typekit.com/dbf2lqy.js"></script>
-<script type="text/javascript">try{Typekit.load();}catch(e){}</script>
-</head>
-
-<body>
-<div class="container_12">
-	<!-- header -->
-	<?php require('../inc/header2.php') ?>
-	<!-- end header -->
-<div class="grid_12 page-header">				
-	<h1>Case Studies</h1>
-</div>
-<div class="clear"></div>
-<!-- end .grid_12 -->
-
- <div class="grid_9">
-	<h3>Main Header</h3>
-	<p>Main column content.</p>
-	<p>&nbsp;</p>
-	</div>
-  <!-- end .grid_9 -->
-  <div class="grid_3">
-  	<h3>Right Header</h3>
-    <p>Right column content.</p>
-    <p>&nbsp;</p>
-  </div>
-  <!-- end .grid_3 -->
-  
-  <div class="clear"></div>
-				
-  </div>
-<!-- end .container_12 -->
-<!-- footer -->
-<?php require('../inc/footer2.php') ?> 
-<!-- end footer -->
-<script type="text/javascript">
-var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
-document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
-</script>
-<script type="text/javascript">
-try {
-var pageTracker = _gat._getTracker("UA-106257-5");
-pageTracker._trackPageview();
-} catch(err) {}</script>
-</body>
-</html>

Added: wwwdev/case_studies/index.php
===================================================================
--- wwwdev/case_studies/index.php	                        (rev 0)
+++ wwwdev/case_studies/index.php	2011-11-09 00:11:13 UTC (rev 5275)
@@ -0,0 +1,89 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml">
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
+<title>Swift - Case Studies</title>
+<link href="../css/960_12_col.css" rel="stylesheet" type="text/css" />
+<link href="../css/reset.css" rel="stylesheet" type="text/css" />
+<link href="../css/style3.css" rel="stylesheet" type="text/css" />
+<script type="text/javascript" src="http://use.typekit.com/dbf2lqy.js"></script>
+<script type="text/javascript">try{Typekit.load();}catch(e){}</script>
+</head>
+
+<body>
+<div class="container_12">
+	<!-- header -->
+	<?php require('../inc/header2.php') ?>
+	<!-- end header -->
+<div class="grid_12 page-header">				
+	<h1>Case Studies</h1>
+</div>
+<div class="clear"></div>
+<!-- end .grid_12 -->
+
+ <div class="grid_12">
+	<h3>Energy/Climate/Economics Modelling</h3>
+	<p>The CIM-EARTH project develops a large-scale integrated modeling framework for decision makers in climate and energy policy.  (Foster, Elliott)</p>
+	<a href="../case_studies/CIM-Earth.php" class="trailingLink">learn more</a>
+	<p>&nbsp;</p>
+	 <div class="clear"></div>
+	 
+	<h3>Protein Structure Prediction</h3>
+	<p>The laboratories of Karl Freed and Tobin Sosnick use Beagle to develop and validate methods to predict protein structure using homology-free approaches.</p>
+	<a href="#" class="trailingLink">learn more</a>
+	<p>&nbsp;</p>
+	 <div class="clear"></div>
+	
+	<h3>Protein-RNA interaction modeling</h3>
+	<p>M. Parisien (with T. Sosnick, T. Pan, and K. Freed) used Beagle to develop a first-generation algorithm for the prediction of the RNA-protein interactome.</p>
+	<a href="#" class="trailingLink">learn more</a>
+	<p>&nbsp;</p>
+	 <div class="clear"></div>
+	
+	<h3>Glass Structure Modeling</h3>
+	<p>This project models of aspects of glass structure at a theoretical chemistry level. (Hocky/Reichman)</p>
+	<a href="#" class="trailingLink">learn more</a>
+	<p>&nbsp;</p>
+	 <div class="clear"></div>
+	
+	<h3>Modeling climate impact on hydrology</h3>
+	<p>Projecting biofuel production impact on hydrology (E. Yan)</p>
+	<a href="#" class="trailingLink">learn more</a>
+	<p>&nbsp;</p>
+	 <div class="clear"></div>
+	
+	<h3>Computational neuroscience application research infrastructure</h3>
+	<p>needs text here</p>
+	<a href="#" class="trailingLink">learn more</a>
+	<p>&nbsp;</p>
+	 <div class="clear"></div>
+	
+	
+	
+	</div>
+  <!-- end .grid_9 -->
+  <!--<div class="grid_3">
+  	<h3>Right Header</h3>
+    <p>Right column content.</p>
+    <p>&nbsp;</p>
+  </div>-->
+  <!-- end .grid_3 -->
+  
+  <div class="clear"></div>
+				
+  </div>
+<!-- end .container_12 -->
+<!-- footer -->
+<?php require('../inc/footer2.php') ?> 
+<!-- end footer -->
+<script type="text/javascript">
+var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
+document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
+</script>
+<script type="text/javascript">
+try {
+var pageTracker = _gat._getTracker("UA-106257-5");
+pageTracker._trackPageview();
+} catch(err) {}</script>
+</body>
+</html>

Added: wwwdev/inc/case_study_sidebar.php
===================================================================
--- wwwdev/inc/case_study_sidebar.php	                        (rev 0)
+++ wwwdev/inc/case_study_sidebar.php	2011-11-09 00:11:13 UTC (rev 5275)
@@ -0,0 +1,9 @@
+<h3>More Case Studies</h3>
+<ul class="swift-bullet">
+	<li><a href="#">study 1</a></li>
+	<li><a href="#">study 2</a></li>
+	<li><a href="#">study 3</a></li>
+	<li><a href="#">study 4</a></li>
+	<li><a href="#">study 5</a></li>
+	<li><a href="#">study 6</a></li>
+</ul>
\ No newline at end of file



From vytas at ci.uchicago.edu  Tue Nov  8 18:36:13 2011
From: vytas at ci.uchicago.edu (vytas at ci.uchicago.edu)
Date: Wed, 09 Nov 2011 00:36:13 -0000
Subject: [Swift-commit] r5276 - wwwdev/images
Message-ID: <20111109003520.60D3D9CCA3@svn.ci.uchicago.edu>

Author: vytas
Date: 2011-11-08 18:35:20 -0600 (Tue, 08 Nov 2011)
New Revision: 5276

Added:
   wwwdev/images/icon_special_C.png
   wwwdev/images/icon_special_D.png
Log:
New home page icons for "fast" and "flexible".

Added: wwwdev/images/icon_special_C.png
===================================================================
(Binary files differ)


Property changes on: wwwdev/images/icon_special_C.png
___________________________________________________________________
Added: svn:mime-type
   + application/octet-stream

Added: wwwdev/images/icon_special_D.png
===================================================================
(Binary files differ)


Property changes on: wwwdev/images/icon_special_D.png
___________________________________________________________________
Added: svn:mime-type
   + application/octet-stream



From vytas at ci.uchicago.edu  Tue Nov  8 18:41:03 2011
From: vytas at ci.uchicago.edu (vytas at ci.uchicago.edu)
Date: Wed, 09 Nov 2011 00:41:03 -0000
Subject: [Swift-commit] r5277 - wwwdev/css
Message-ID: <20111109004010.69E239CCA3@svn.ci.uchicago.edu>

Author: vytas
Date: 2011-11-08 18:40:10 -0600 (Tue, 08 Nov 2011)
New Revision: 5277

Modified:
   wwwdev/css/style3.css
Log:
Added "fast" and "flexible" special bullet styles for use on the home page.

Modified: wwwdev/css/style3.css
===================================================================
--- wwwdev/css/style3.css	2011-11-09 00:35:20 UTC (rev 5276)
+++ wwwdev/css/style3.css	2011-11-09 00:40:10 UTC (rev 5277)
@@ -544,6 +544,14 @@
 	padding: 10px 40px 25px 120px;
 	background: url(../images/icon_special_B.png) 60px 13px no-repeat;
 }
+.highlight-C {
+	padding: 10px 40px 25px 120px;
+	background: url(../images/icon_special_C.png) 60px 13px no-repeat;
+}
+.highlight-D {
+	padding: 10px 40px 25px 120px;
+	background: url(../images/icon_special_D.png) 60px 13px no-repeat;
+}
 #nav .current {
 	color: #fff;
 }



From davidk at ci.uchicago.edu  Tue Nov 29 15:20:57 2011
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Tue, 29 Nov 2011 15:20:57 -0600 (CST)
Subject: [Swift-commit] r5324 - in usertools/swift-vm-boot: . bin plan
Message-ID: <20111129212057.AC0DE9CCA8@svn.ci.uchicago.edu>

Author: davidk
Date: 2011-11-29 15:20:57 -0600 (Tue, 29 Nov 2011)
New Revision: 5324

Added:
   usertools/swift-vm-boot/bin/
   usertools/swift-vm-boot/bin/bootit.sh
   usertools/swift-vm-boot/bin/handleoutput.py
   usertools/swift-vm-boot/bin/nimbus-register-key.py
   usertools/swift-vm-boot/bin/register_key.py
   usertools/swift-vm-boot/bin/virtualenv.py
   usertools/swift-vm-boot/env.sh
   usertools/swift-vm-boot/hosts.txt
   usertools/swift-vm-boot/install.sh
   usertools/swift-vm-boot/plan/
   usertools/swift-vm-boot/plan/test-level1.conf
   usertools/swift-vm-boot/plan/top.conf
Removed:
   usertools/swift-vm-boot/bin/
   usertools/swift-vm-boot/env.sh
   usertools/swift-vm-boot/hosts.txt
   usertools/swift-vm-boot/install.sh
   usertools/swift-vm-boot/plan/
Log:
Fixes for registering futuregrid keys


Added: usertools/swift-vm-boot/bin/bootit.sh
===================================================================
--- usertools/swift-vm-boot/bin/bootit.sh	                        (rev 0)
+++ usertools/swift-vm-boot/bin/bootit.sh	2011-11-29 21:20:57 UTC (rev 5324)
@@ -0,0 +1,22 @@
+#!/bin/bash
+
+dir=`dirname $0`
+cd $dir
+cd ..
+
+source env.sh
+source ve/bin/activate
+
+output="output.json"
+cloudinitd -l debug -v -v -v boot plan/top.conf -o $output
+if [ $? -ne 0 ]; then
+    echo "The boot failed.  Check the logs"
+    exit 1
+fi
+
+echo "getting the hostnames..."
+python bin/handleoutput.py $output
+if [ $? -ne 0 ]; then
+    echo "The output parse failed"
+    exit 1
+fi


Property changes on: usertools/swift-vm-boot/bin/bootit.sh
___________________________________________________________________
Added: svn:executable
   + *

Added: usertools/swift-vm-boot/bin/handleoutput.py
===================================================================
--- usertools/swift-vm-boot/bin/handleoutput.py	                        (rev 0)
+++ usertools/swift-vm-boot/bin/handleoutput.py	2011-11-29 21:20:57 UTC (rev 5324)
@@ -0,0 +1,16 @@
+#!/usr/bin/env python
+
+import os
+import sys
+import simplejson as json
+
+x = json.load(open(sys.argv[1], "r"))
+
+levels = x['levels']
+lvl1 = levels[0]
+services = lvl1['services']
+
+for svc in services:
+    print svc['hostname']
+
+


Property changes on: usertools/swift-vm-boot/bin/handleoutput.py
___________________________________________________________________
Added: svn:executable
   + *

Added: usertools/swift-vm-boot/bin/nimbus-register-key.py
===================================================================
--- usertools/swift-vm-boot/bin/nimbus-register-key.py	                        (rev 0)
+++ usertools/swift-vm-boot/bin/nimbus-register-key.py	2011-11-29 21:20:57 UTC (rev 5324)
@@ -0,0 +1,39 @@
+#!/usr/bin/env python
+
+import os
+import sys
+import base64
+import boto
+from boto.ec2.regioninfo import RegionInfo
+import urllib2
+
+
+def add_key(ec2conn, keyname, keytext):
+    pairs = ec2conn.get_all_key_pairs([keyname,])
+    if not pairs:
+        print "Adding a new key pair named %s" % (keyname)
+        ec2conn.import_key_pair(keyname, keytext)
+    else:
+        print "Keyname %s already exists" % (keyname)
+
+try:
+    access_id = os.environ['FUTUREGRID_IAAS_ACCESS_KEY']
+    access_secret = os.environ['FUTUREGRID_IAAS_SECRET_KEY']
+
+    iaas_url = os.environ['FUTUREGRID_IAAS_URL']
+except Exception, ex:
+    print "Make sure the envs CLOUDINITD_* are set %s" % (str(ex))
+    sys.exit(1)
+
+url_parts = urllib2.urlparse.urlparse(iaas_url)
+region = RegionInfo(name="nimbus", endpoint=url_parts.hostname)
+ec2conn = boto.connect_ec2(access_id, access_secret, region=region, port=url_parts.port)
+
+keyname = sys.argv[1]
+keylocation = sys.argv[2]
+keytext = open(keylocation).read()
+keytext = base64.b64encode(keytext)
+add_key(ec2conn, keyname, keytext)
+
+sys.exit(0)
+

Added: usertools/swift-vm-boot/bin/register_key.py
===================================================================
--- usertools/swift-vm-boot/bin/register_key.py	                        (rev 0)
+++ usertools/swift-vm-boot/bin/register_key.py	2011-11-29 21:20:57 UTC (rev 5324)
@@ -0,0 +1,37 @@
+#!/usr/bin/env python
+
+import os
+import sys
+import base64
+import boto
+from boto.ec2.regioninfo import RegionInfo
+
+
+try:
+    access_id = os.environ['FUTUREGRID_IAAS_ACCESS_KEY']
+    access_secret = os.environ['FUTUREGRID_IAAS_SECRET_KEY']
+except:
+    print "Make sure the envs FUTUREGRID_IAAS_ACCESS_KEY and FUTUREGRID_IAAS_SECRET_KEY are set"
+    sys.exit(1)
+
+hostfile = sys.argv[1]
+keyname = "swiftkey"
+if len(sys.argv) > 2:
+    keylocation = sys.argv[2]
+else:
+    keylocation = os.path.expanduser("~/.ssh/id_rsa.pub")
+
+keytext = open(keylocation).read()
+
+f = open(hostfile, "r")
+for line in f:
+    host = line.strip()
+    print "creating key %s on %s" % (keyname, host)
+
+    region = RegionInfo(name="nimbus", endpoint=host)
+    ec2conn = boto.connect_ec2(access_id, access_secret, region=region, port=8444)
+
+#    keytext = base64.b64encode(keytext)
+    ec2conn.import_key_pair(keyname, keytext)
+
+sys.exit(0)


Property changes on: usertools/swift-vm-boot/bin/register_key.py
___________________________________________________________________
Added: svn:executable
   + *

Added: usertools/swift-vm-boot/bin/virtualenv.py
===================================================================
--- usertools/swift-vm-boot/bin/virtualenv.py	                        (rev 0)
+++ usertools/swift-vm-boot/bin/virtualenv.py	2011-11-29 21:20:57 UTC (rev 5324)
@@ -0,0 +1,1651 @@
+#!/usr/bin/env python
+"""Create a "virtual" Python installation
+"""
+
+virtualenv_version = "1.5.1"
+
+import sys
+import os
+import optparse
+import re
+import shutil
+import logging
+import tempfile
+import distutils.sysconfig
+try:
+    import subprocess
+except ImportError, e:
+    if sys.version_info <= (2, 3):
+        print 'ERROR: %s' % e
+        print 'ERROR: this script requires Python 2.4 or greater; or at least the subprocess module.'
+        print 'If you copy subprocess.py from a newer version of Python this script will probably work'
+        sys.exit(101)
+    else:
+        raise
+try:
+    set
+except NameError:
+    from sets import Set as set
+
+join = os.path.join
+py_version = 'python%s.%s' % (sys.version_info[0], sys.version_info[1])
+
+is_jython = sys.platform.startswith('java')
+is_pypy = hasattr(sys, 'pypy_version_info')
+
+if is_pypy:
+    expected_exe = 'pypy-c'
+elif is_jython:
+    expected_exe = 'jython'
+else:
+    expected_exe = 'python'
+
+
+REQUIRED_MODULES = ['os', 'posix', 'posixpath', 'nt', 'ntpath', 'genericpath',
+                    'fnmatch', 'locale', 'encodings', 'codecs',
+                    'stat', 'UserDict', 'readline', 'copy_reg', 'types',
+                    're', 'sre', 'sre_parse', 'sre_constants', 'sre_compile',
+                    'zlib']
+
+REQUIRED_FILES = ['lib-dynload', 'config']
+
+if sys.version_info[:2] >= (2, 6):
+    REQUIRED_MODULES.extend(['warnings', 'linecache', '_abcoll', 'abc'])
+if sys.version_info[:2] >= (2, 7):
+    REQUIRED_MODULES.extend(['_weakrefset'])
+if sys.version_info[:2] <= (2, 3):
+    REQUIRED_MODULES.extend(['sets', '__future__'])
+if is_pypy:
+    # these are needed to correctly display the exceptions that may happen
+    # during the bootstrap
+    REQUIRED_MODULES.extend(['traceback', 'linecache'])
+
+class Logger(object):
+
+    """
+    Logging object for use in command-line script.  Allows ranges of
+    levels, to avoid some redundancy of displayed information.
+    """
+
+    DEBUG = logging.DEBUG
+    INFO = logging.INFO
+    NOTIFY = (logging.INFO+logging.WARN)/2
+    WARN = WARNING = logging.WARN
+    ERROR = logging.ERROR
+    FATAL = logging.FATAL
+
+    LEVELS = [DEBUG, INFO, NOTIFY, WARN, ERROR, FATAL]
+
+    def __init__(self, consumers):
+        self.consumers = consumers
+        self.indent = 0
+        self.in_progress = None
+        self.in_progress_hanging = False
+
+    def debug(self, msg, *args, **kw):
+        self.log(self.DEBUG, msg, *args, **kw)
+    def info(self, msg, *args, **kw):
+        self.log(self.INFO, msg, *args, **kw)
+    def notify(self, msg, *args, **kw):
+        self.log(self.NOTIFY, msg, *args, **kw)
+    def warn(self, msg, *args, **kw):
+        self.log(self.WARN, msg, *args, **kw)
+    def error(self, msg, *args, **kw):
+        self.log(self.WARN, msg, *args, **kw)
+    def fatal(self, msg, *args, **kw):
+        self.log(self.FATAL, msg, *args, **kw)
+    def log(self, level, msg, *args, **kw):
+        if args:
+            if kw:
+                raise TypeError(
+                    "You may give positional or keyword arguments, not both")
+        args = args or kw
+        rendered = None
+        for consumer_level, consumer in self.consumers:
+            if self.level_matches(level, consumer_level):
+                if (self.in_progress_hanging
+                    and consumer in (sys.stdout, sys.stderr)):
+                    self.in_progress_hanging = False
+                    sys.stdout.write('\n')
+                    sys.stdout.flush()
+                if rendered is None:
+                    if args:
+                        rendered = msg % args
+                    else:
+                        rendered = msg
+                    rendered = ' '*self.indent + rendered
+                if hasattr(consumer, 'write'):
+                    consumer.write(rendered+'\n')
+                else:
+                    consumer(rendered)
+
+    def start_progress(self, msg):
+        assert not self.in_progress, (
+            "Tried to start_progress(%r) while in_progress %r"
+            % (msg, self.in_progress))
+        if self.level_matches(self.NOTIFY, self._stdout_level()):
+            sys.stdout.write(msg)
+            sys.stdout.flush()
+            self.in_progress_hanging = True
+        else:
+            self.in_progress_hanging = False
+        self.in_progress = msg
+
+    def end_progress(self, msg='done.'):
+        assert self.in_progress, (
+            "Tried to end_progress without start_progress")
+        if self.stdout_level_matches(self.NOTIFY):
+            if not self.in_progress_hanging:
+                # Some message has been printed out since start_progress
+                sys.stdout.write('...' + self.in_progress + msg + '\n')
+                sys.stdout.flush()
+            else:
+                sys.stdout.write(msg + '\n')
+                sys.stdout.flush()
+        self.in_progress = None
+        self.in_progress_hanging = False
+
+    def show_progress(self):
+        """If we are in a progress scope, and no log messages have been
+        shown, write out another '.'"""
+        if self.in_progress_hanging:
+            sys.stdout.write('.')
+            sys.stdout.flush()
+
+    def stdout_level_matches(self, level):
+        """Returns true if a message at this level will go to stdout"""
+        return self.level_matches(level, self._stdout_level())
+
+    def _stdout_level(self):
+        """Returns the level that stdout runs at"""
+        for level, consumer in self.consumers:
+            if consumer is sys.stdout:
+                return level
+        return self.FATAL
+
+    def level_matches(self, level, consumer_level):
+        """
+        >>> l = Logger()
+        >>> l.level_matches(3, 4)
+        False
+        >>> l.level_matches(3, 2)
+        True
+        >>> l.level_matches(slice(None, 3), 3)
+        False
+        >>> l.level_matches(slice(None, 3), 2)
+        True
+        >>> l.level_matches(slice(1, 3), 1)
+        True
+        >>> l.level_matches(slice(2, 3), 1)
+        False
+        """
+        if isinstance(level, slice):
+            start, stop = level.start, level.stop
+            if start is not None and start > consumer_level:
+                return False
+            if stop is not None or stop <= consumer_level:
+                return False
+            return True
+        else:
+            return level >= consumer_level
+
+    #@classmethod
+    def level_for_integer(cls, level):
+        levels = cls.LEVELS
+        if level < 0:
+            return levels[0]
+        if level >= len(levels):
+            return levels[-1]
+        return levels[level]
+
+    level_for_integer = classmethod(level_for_integer)
+
+def mkdir(path):
+    if not os.path.exists(path):
+        logger.info('Creating %s', path)
+        os.makedirs(path)
+    else:
+        logger.info('Directory %s already exists', path)
+
+def copyfile(src, dest, symlink=True):
+    if not os.path.exists(src):
+        # Some bad symlink in the src
+        logger.warn('Cannot find file %s (bad symlink)', src)
+        return
+    if os.path.exists(dest):
+        logger.debug('File %s already exists', dest)
+        return
+    if not os.path.exists(os.path.dirname(dest)):
+        logger.info('Creating parent directories for %s' % os.path.dirname(dest))
+        os.makedirs(os.path.dirname(dest))
+    if symlink and hasattr(os, 'symlink'):
+        logger.info('Symlinking %s', dest)
+        os.symlink(os.path.abspath(src), dest)
+    else:
+        logger.info('Copying to %s', dest)
+        if os.path.isdir(src):
+            shutil.copytree(src, dest, True)
+        else:
+            shutil.copy2(src, dest)
+
+def writefile(dest, content, overwrite=True):
+    if not os.path.exists(dest):
+        logger.info('Writing %s', dest)
+        f = open(dest, 'wb')
+        f.write(content)
+        f.close()
+        return
+    else:
+        f = open(dest, 'rb')
+        c = f.read()
+        f.close()
+        if c != content:
+            if not overwrite:
+                logger.notify('File %s exists with different content; not overwriting', dest)
+                return
+            logger.notify('Overwriting %s with new content', dest)
+            f = open(dest, 'wb')
+            f.write(content)
+            f.close()
+        else:
+            logger.info('Content %s already in place', dest)
+
+def rmtree(dir):
+    if os.path.exists(dir):
+        logger.notify('Deleting tree %s', dir)
+        shutil.rmtree(dir)
+    else:
+        logger.info('Do not need to delete %s; already gone', dir)
+
+def make_exe(fn):
+    if hasattr(os, 'chmod'):
+        oldmode = os.stat(fn).st_mode & 07777
+        newmode = (oldmode | 0555) & 07777
+        os.chmod(fn, newmode)
+        logger.info('Changed mode of %s to %s', fn, oct(newmode))
+
+def _find_file(filename, dirs):
+    for dir in dirs:
+        if os.path.exists(join(dir, filename)):
+            return join(dir, filename)
+    return filename
+
+def _install_req(py_executable, unzip=False, distribute=False):
+    if not distribute:
+        setup_fn = 'setuptools-0.6c11-py%s.egg' % sys.version[:3]
+        project_name = 'setuptools'
+        bootstrap_script = EZ_SETUP_PY
+        source = None
+    else:
+        setup_fn = None
+        source = 'distribute-0.6.14.tar.gz'
+        project_name = 'distribute'
+        bootstrap_script = DISTRIBUTE_SETUP_PY
+        try:
+            # check if the global Python has distribute installed or plain
+            # setuptools
+            import pkg_resources
+            if not hasattr(pkg_resources, '_distribute'):
+                location = os.path.dirname(pkg_resources.__file__)
+                logger.notify("A globally installed setuptools was found (in %s)" % location)
+                logger.notify("Use the --no-site-packages option to use distribute in "
+                              "the virtualenv.")
+        except ImportError:
+            pass
+
+    search_dirs = file_search_dirs()
+
+    if setup_fn is not None:
+        setup_fn = _find_file(setup_fn, search_dirs)
+
+    if source is not None:
+        source = _find_file(source, search_dirs)
+
+    if is_jython and os._name == 'nt':
+        # Jython's .bat sys.executable can't handle a command line
+        # argument with newlines
+        fd, ez_setup = tempfile.mkstemp('.py')
+        os.write(fd, bootstrap_script)
+        os.close(fd)
+        cmd = [py_executable, ez_setup]
+    else:
+        cmd = [py_executable, '-c', bootstrap_script]
+    if unzip:
+        cmd.append('--always-unzip')
+    env = {}
+    remove_from_env = []
+    if logger.stdout_level_matches(logger.DEBUG):
+        cmd.append('-v')
+
+    old_chdir = os.getcwd()
+    if setup_fn is not None and os.path.exists(setup_fn):
+        logger.info('Using existing %s egg: %s' % (project_name, setup_fn))
+        cmd.append(setup_fn)
+        if os.environ.get('PYTHONPATH'):
+            env['PYTHONPATH'] = setup_fn + os.path.pathsep + os.environ['PYTHONPATH']
+        else:
+            env['PYTHONPATH'] = setup_fn
+    else:
+        # the source is found, let's chdir
+        if source is not None and os.path.exists(source):
+            os.chdir(os.path.dirname(source))
+            # in this case, we want to be sure that PYTHONPATH is unset (not
+            # just empty, really unset), else CPython tries to import the
+            # site.py that it's in virtualenv_support
+            remove_from_env.append('PYTHONPATH')
+        else:
+            logger.info('No %s egg found; downloading' % project_name)
+        cmd.extend(['--always-copy', '-U', project_name])
+    logger.start_progress('Installing %s...' % project_name)
+    logger.indent += 2
+    cwd = None
+    if project_name == 'distribute':
+        env['DONT_PATCH_SETUPTOOLS'] = 'true'
+
+    def _filter_ez_setup(line):
+        return filter_ez_setup(line, project_name)
+
+    if not os.access(os.getcwd(), os.W_OK):
+        cwd = tempfile.mkdtemp()
+        if source is not None and os.path.exists(source):
+            # the current working dir is hostile, let's copy the
+            # tarball to a temp dir
+            target = os.path.join(cwd, os.path.split(source)[-1])
+            shutil.copy(source, target)
+    try:
+        call_subprocess(cmd, show_stdout=False,
+                        filter_stdout=_filter_ez_setup,
+                        extra_env=env,
+                        remove_from_env=remove_from_env,
+                        cwd=cwd)
+    finally:
+        logger.indent -= 2
+        logger.end_progress()
+        if os.getcwd() != old_chdir:
+            os.chdir(old_chdir)
+        if is_jython and os._name == 'nt':
+            os.remove(ez_setup)
+
+def file_search_dirs():
+    here = os.path.dirname(os.path.abspath(__file__))
+    dirs = ['.', here,
+            join(here, 'virtualenv_support')]
+    if os.path.splitext(os.path.dirname(__file__))[0] != 'virtualenv':
+        # Probably some boot script; just in case virtualenv is installed...
+        try:
+            import virtualenv
+        except ImportError:
+            pass
+        else:
+            dirs.append(os.path.join(os.path.dirname(virtualenv.__file__), 'virtualenv_support'))
+    return [d for d in dirs if os.path.isdir(d)]
+
+def install_setuptools(py_executable, unzip=False):
+    _install_req(py_executable, unzip)
+
+def install_distribute(py_executable, unzip=False):
+    _install_req(py_executable, unzip, distribute=True)
+
+_pip_re = re.compile(r'^pip-.*(zip|tar.gz|tar.bz2|tgz|tbz)$', re.I)
+def install_pip(py_executable):
+    filenames = []
+    for dir in file_search_dirs():
+        filenames.extend([join(dir, fn) for fn in os.listdir(dir)
+                          if _pip_re.search(fn)])
+    filenames = [(os.path.basename(filename).lower(), i, filename) for i, filename in enumerate(filenames)]
+    filenames.sort()
+    filenames = [filename for basename, i, filename in filenames]
+    if not filenames:
+        filename = 'pip'
+    else:
+        filename = filenames[-1]
+    easy_install_script = 'easy_install'
+    if sys.platform == 'win32':
+        easy_install_script = 'easy_install-script.py'
+    cmd = [py_executable, join(os.path.dirname(py_executable), easy_install_script), filename]
+    if filename == 'pip':
+        logger.info('Installing pip from network...')
+    else:
+        logger.info('Installing %s' % os.path.basename(filename))
+    logger.indent += 2
+    def _filter_setup(line):
+        return filter_ez_setup(line, 'pip')
+    try:
+        call_subprocess(cmd, show_stdout=False,
+                        filter_stdout=_filter_setup)
+    finally:
+        logger.indent -= 2
+
+def filter_ez_setup(line, project_name='setuptools'):
+    if not line.strip():
+        return Logger.DEBUG
+    if project_name == 'distribute':
+        for prefix in ('Extracting', 'Now working', 'Installing', 'Before',
+                       'Scanning', 'Setuptools', 'Egg', 'Already',
+                       'running', 'writing', 'reading', 'installing',
+                       'creating', 'copying', 'byte-compiling', 'removing',
+                       'Processing'):
+            if line.startswith(prefix):
+                return Logger.DEBUG
+        return Logger.DEBUG
+    for prefix in ['Reading ', 'Best match', 'Processing setuptools',
+                   'Copying setuptools', 'Adding setuptools',
+                   'Installing ', 'Installed ']:
+        if line.startswith(prefix):
+            return Logger.DEBUG
+    return Logger.INFO
+
+def main():
+    parser = optparse.OptionParser(
+        version=virtualenv_version,
+        usage="%prog [OPTIONS] DEST_DIR")
+
+    parser.add_option(
+        '-v', '--verbose',
+        action='count',
+        dest='verbose',
+        default=0,
+        help="Increase verbosity")
+
+    parser.add_option(
+        '-q', '--quiet',
+        action='count',
+        dest='quiet',
+        default=0,
+        help='Decrease verbosity')
+
+    parser.add_option(
+        '-p', '--python',
+        dest='python',
+        metavar='PYTHON_EXE',
+        help='The Python interpreter to use, e.g., --python=python2.5 will use the python2.5 '
+        'interpreter to create the new environment.  The default is the interpreter that '
+        'virtualenv was installed with (%s)' % sys.executable)
+
+    parser.add_option(
+        '--clear',
+        dest='clear',
+        action='store_true',
+        help="Clear out the non-root install and start from scratch")
+
+    parser.add_option(
+        '--no-site-packages',
+        dest='no_site_packages',
+        action='store_true',
+        help="Don't give access to the global site-packages dir to the "
+             "virtual environment")
+
+    parser.add_option(
+        '--unzip-setuptools',
+        dest='unzip_setuptools',
+        action='store_true',
+        help="Unzip Setuptools or Distribute when installing it")
+
+    parser.add_option(
+        '--relocatable',
+        dest='relocatable',
+        action='store_true',
+        help='Make an EXISTING virtualenv environment relocatable.  '
+        'This fixes up scripts and makes all .pth files relative')
+
+    parser.add_option(
+        '--distribute',
+        dest='use_distribute',
+        action='store_true',
+        help='Use Distribute instead of Setuptools. Set environ variable '
+        'VIRTUALENV_USE_DISTRIBUTE to make it the default ')
+
+    parser.add_option(
+        '--prompt=',
+        dest='prompt',
+        help='Provides an alternative prompt prefix for this environment')
+
+    if 'extend_parser' in globals():
+        extend_parser(parser)
+
+    options, args = parser.parse_args()
+
+    global logger
+
+    if 'adjust_options' in globals():
+        adjust_options(options, args)
+
+    verbosity = options.verbose - options.quiet
+    logger = Logger([(Logger.level_for_integer(2-verbosity), sys.stdout)])
+
+    if options.python and not os.environ.get('VIRTUALENV_INTERPRETER_RUNNING'):
+        env = os.environ.copy()
+        interpreter = resolve_interpreter(options.python)
+        if interpreter == sys.executable:
+            logger.warn('Already using interpreter %s' % interpreter)
+        else:
+            logger.notify('Running virtualenv with interpreter %s' % interpreter)
+            env['VIRTUALENV_INTERPRETER_RUNNING'] = 'true'
+            file = __file__
+            if file.endswith('.pyc'):
+                file = file[:-1]
+            popen = subprocess.Popen([interpreter, file] + sys.argv[1:], env=env)
+            raise SystemExit(popen.wait())
+
+    if not args:
+        print 'You must provide a DEST_DIR'
+        parser.print_help()
+        sys.exit(2)
+    if len(args) > 1:
+        print 'There must be only one argument: DEST_DIR (you gave %s)' % (
+            ' '.join(args))
+        parser.print_help()
+        sys.exit(2)
+
+    home_dir = args[0]
+
+    if os.environ.get('WORKING_ENV'):
+        logger.fatal('ERROR: you cannot run virtualenv while in a workingenv')
+        logger.fatal('Please deactivate your workingenv, then re-run this script')
+        sys.exit(3)
+
+    if 'PYTHONHOME' in os.environ:
+        logger.warn('PYTHONHOME is set.  You *must* activate the virtualenv before using it')
+        del os.environ['PYTHONHOME']
+
+    if options.relocatable:
+        make_environment_relocatable(home_dir)
+        return
+
+    create_environment(home_dir, site_packages=not options.no_site_packages, clear=options.clear,
+                       unzip_setuptools=options.unzip_setuptools,
+                       use_distribute=options.use_distribute,
+                       prompt=options.prompt)
+    if 'after_install' in globals():
+        after_install(options, home_dir)
+
+def call_subprocess(cmd, show_stdout=True,
+                    filter_stdout=None, cwd=None,
+                    raise_on_returncode=True, extra_env=None,
+                    remove_from_env=None):
+    cmd_parts = []
+    for part in cmd:
+        if len(part) > 40:
+            part = part[:30]+"..."+part[-5:]
+        if ' ' in part or '\n' in part or '"' in part or "'" in part:
+            part = '"%s"' % part.replace('"', '\\"')
+        cmd_parts.append(part)
+    cmd_desc = ' '.join(cmd_parts)
+    if show_stdout:
+        stdout = None
+    else:
+        stdout = subprocess.PIPE
+    logger.debug("Running command %s" % cmd_desc)
+    if extra_env or remove_from_env:
+        env = os.environ.copy()
+        if extra_env:
+            env.update(extra_env)
+        if remove_from_env:
+            for varname in remove_from_env:
+                env.pop(varname, None)
+    else:
+        env = None
+    try:
+        proc = subprocess.Popen(
+            cmd, stderr=subprocess.STDOUT, stdin=None, stdout=stdout,
+            cwd=cwd, env=env)
+    except Exception, e:
+        logger.fatal(
+            "Error %s while executing command %s" % (e, cmd_desc))
+        raise
+    all_output = []
+    if stdout is not None:
+        stdout = proc.stdout
+        while 1:
+            line = stdout.readline()
+            if not line:
+                break
+            line = line.rstrip()
+            all_output.append(line)
+            if filter_stdout:
+                level = filter_stdout(line)
+                if isinstance(level, tuple):
+                    level, line = level
+                logger.log(level, line)
+                if not logger.stdout_level_matches(level):
+                    logger.show_progress()
+            else:
+                logger.info(line)
+    else:
+        proc.communicate()
+    proc.wait()
+    if proc.returncode:
+        if raise_on_returncode:
+            if all_output:
+                logger.notify('Complete output from command %s:' % cmd_desc)
+                logger.notify('\n'.join(all_output) + '\n----------------------------------------')
+            raise OSError(
+                "Command %s failed with error code %s"
+                % (cmd_desc, proc.returncode))
+        else:
+            logger.warn(
+                "Command %s had error code %s"
+                % (cmd_desc, proc.returncode))
+
+
+def create_environment(home_dir, site_packages=True, clear=False,
+                       unzip_setuptools=False, use_distribute=False,
+                       prompt=None):
+    """
+    Creates a new environment in ``home_dir``.
+
+    If ``site_packages`` is true (the default) then the global
+    ``site-packages/`` directory will be on the path.
+
+    If ``clear`` is true (default False) then the environment will
+    first be cleared.
+    """
+    home_dir, lib_dir, inc_dir, bin_dir = path_locations(home_dir)
+
+    py_executable = os.path.abspath(install_python(
+        home_dir, lib_dir, inc_dir, bin_dir,
+        site_packages=site_packages, clear=clear))
+
+    install_distutils(home_dir)
+
+    if use_distribute or os.environ.get('VIRTUALENV_USE_DISTRIBUTE'):
+        install_distribute(py_executable, unzip=unzip_setuptools)
+    else:
+        install_setuptools(py_executable, unzip=unzip_setuptools)
+
+    install_pip(py_executable)
+
+    install_activate(home_dir, bin_dir, prompt)
+
+def path_locations(home_dir):
+    """Return the path locations for the environment (where libraries are,
+    where scripts go, etc)"""
+    # XXX: We'd use distutils.sysconfig.get_python_inc/lib but its
+    # prefix arg is broken: http://bugs.python.org/issue3386
+    if sys.platform == 'win32':
+        # Windows has lots of problems with executables with spaces in
+        # the name; this function will remove them (using the ~1
+        # format):
+        mkdir(home_dir)
+        if ' ' in home_dir:
+            try:
+                import win32api
+            except ImportError:
+                print 'Error: the path "%s" has a space in it' % home_dir
+                print 'To handle these kinds of paths, the win32api module must be installed:'
+                print '  http://sourceforge.net/projects/pywin32/'
+                sys.exit(3)
+            home_dir = win32api.GetShortPathName(home_dir)
+        lib_dir = join(home_dir, 'Lib')
+        inc_dir = join(home_dir, 'Include')
+        bin_dir = join(home_dir, 'Scripts')
+    elif is_jython:
+        lib_dir = join(home_dir, 'Lib')
+        inc_dir = join(home_dir, 'Include')
+        bin_dir = join(home_dir, 'bin')
+    elif is_pypy:
+        lib_dir = home_dir
+        inc_dir = join(home_dir, 'include')
+        bin_dir = join(home_dir, 'bin')
+    else:
+        lib_dir = join(home_dir, 'lib', py_version)
+        inc_dir = join(home_dir, 'include', py_version)
+        bin_dir = join(home_dir, 'bin')
+    return home_dir, lib_dir, inc_dir, bin_dir
+
+
+def change_prefix(filename, dst_prefix):
+    prefixes = [sys.prefix]
+    if hasattr(sys, 'real_prefix'):
+        prefixes.append(sys.real_prefix)
+    prefixes = map(os.path.abspath, prefixes)
+    filename = os.path.abspath(filename)
+    for src_prefix in prefixes:
+        if filename.startswith(src_prefix):
+            _, relpath = filename.split(src_prefix, 1)
+            assert relpath[0] == os.sep
+            relpath = relpath[1:]
+            return join(dst_prefix, relpath)
+    assert False, "Filename %s does not start with any of these prefixes: %s" % \
+        (filename, prefixes)
+
+def copy_required_modules(dst_prefix):
+    import imp
+    for modname in REQUIRED_MODULES:
+        if modname in sys.builtin_module_names:
+            logger.info("Ignoring built-in bootstrap module: %s" % modname)
+            continue
+        try:
+            f, filename, _ = imp.find_module(modname)
+        except ImportError:
+            logger.info("Cannot import bootstrap module: %s" % modname)
+        else:
+            if f is not None:
+                f.close()
+            dst_filename = change_prefix(filename, dst_prefix)
+            copyfile(filename, dst_filename)
+            if filename.endswith('.pyc'):
+                pyfile = filename[:-1]
+                if os.path.exists(pyfile):
+                    copyfile(pyfile, dst_filename[:-1])
+
+
+def install_python(home_dir, lib_dir, inc_dir, bin_dir, site_packages, clear):
+    """Install just the base environment, no distutils patches etc"""
+    if sys.executable.startswith(bin_dir):
+        print 'Please use the *system* python to run this script'
+        return
+
+    if clear:
+        rmtree(lib_dir)
+        ## FIXME: why not delete it?
+        ## Maybe it should delete everything with #!/path/to/venv/python in it
+        logger.notify('Not deleting %s', bin_dir)
+
+    if hasattr(sys, 'real_prefix'):
+        logger.notify('Using real prefix %r' % sys.real_prefix)
+        prefix = sys.real_prefix
+    else:
+        prefix = sys.prefix
+    mkdir(lib_dir)
+    fix_lib64(lib_dir)
+    stdlib_dirs = [os.path.dirname(os.__file__)]
+    if sys.platform == 'win32':
+        stdlib_dirs.append(join(os.path.dirname(stdlib_dirs[0]), 'DLLs'))
+    elif sys.platform == 'darwin':
+        stdlib_dirs.append(join(stdlib_dirs[0], 'site-packages'))
+    if hasattr(os, 'symlink'):
+        logger.info('Symlinking Python bootstrap modules')
+    else:
+        logger.info('Copying Python bootstrap modules')
+    logger.indent += 2
+    try:
+        # copy required files...
+        for stdlib_dir in stdlib_dirs:
+            if not os.path.isdir(stdlib_dir):
+                continue
+            for fn in os.listdir(stdlib_dir):
+                if fn != 'site-packages' and os.path.splitext(fn)[0] in REQUIRED_FILES:
+                    copyfile(join(stdlib_dir, fn), join(lib_dir, fn))
+        # ...and modules
+        copy_required_modules(home_dir)
+    finally:
+        logger.indent -= 2
+    mkdir(join(lib_dir, 'site-packages'))
+    import site
+    site_filename = site.__file__
+    if site_filename.endswith('.pyc'):
+        site_filename = site_filename[:-1]
+    elif site_filename.endswith('$py.class'):
+        site_filename = site_filename.replace('$py.class', '.py')
+    site_filename_dst = change_prefix(site_filename, home_dir)
+    site_dir = os.path.dirname(site_filename_dst)
+    writefile(site_filename_dst, SITE_PY)
+    writefile(join(site_dir, 'orig-prefix.txt'), prefix)
+    site_packages_filename = join(site_dir, 'no-global-site-packages.txt')
+    if not site_packages:
+        writefile(site_packages_filename, '')
+    else:
+        if os.path.exists(site_packages_filename):
+            logger.info('Deleting %s' % site_packages_filename)
+            os.unlink(site_packages_filename)
+
+    if is_pypy:
+        stdinc_dir = join(prefix, 'include')
+    else:
+        stdinc_dir = join(prefix, 'include', py_version)
+    if os.path.exists(stdinc_dir):
+        copyfile(stdinc_dir, inc_dir)
+    else:
+        logger.debug('No include dir %s' % stdinc_dir)
+
+    if sys.exec_prefix != prefix:
+        if sys.platform == 'win32':
+            exec_dir = join(sys.exec_prefix, 'lib')
+        elif is_jython:
+            exec_dir = join(sys.exec_prefix, 'Lib')
+        else:
+            exec_dir = join(sys.exec_prefix, 'lib', py_version)
+        for fn in os.listdir(exec_dir):
+            copyfile(join(exec_dir, fn), join(lib_dir, fn))
+
+    if is_jython:
+        # Jython has either jython-dev.jar and javalib/ dir, or just
+        # jython.jar
+        for name in 'jython-dev.jar', 'javalib', 'jython.jar':
+            src = join(prefix, name)
+            if os.path.exists(src):
+                copyfile(src, join(home_dir, name))
+        # XXX: registry should always exist after Jython 2.5rc1
+        src = join(prefix, 'registry')
+        if os.path.exists(src):
+            copyfile(src, join(home_dir, 'registry'), symlink=False)
+        copyfile(join(prefix, 'cachedir'), join(home_dir, 'cachedir'),
+                 symlink=False)
+
+    mkdir(bin_dir)
+    py_executable = join(bin_dir, os.path.basename(sys.executable))
+    if 'Python.framework' in prefix:
+        if re.search(r'/Python(?:-32|-64)*$', py_executable):
+            # The name of the python executable is not quite what
+            # we want, rename it.
+            py_executable = os.path.join(
+                    os.path.dirname(py_executable), 'python')
+
+    logger.notify('New %s executable in %s', expected_exe, py_executable)
+    if sys.executable != py_executable:
+        ## FIXME: could I just hard link?
+        executable = sys.executable
+        if sys.platform == 'cygwin' and os.path.exists(executable + '.exe'):
+            # Cygwin misreports sys.executable sometimes
+            executable += '.exe'
+            py_executable += '.exe'
+            logger.info('Executable actually exists in %s' % executable)
+        shutil.copyfile(executable, py_executable)
+        make_exe(py_executable)
+        if sys.platform == 'win32' or sys.platform == 'cygwin':
+            pythonw = os.path.join(os.path.dirname(sys.executable), 'pythonw.exe')
+            if os.path.exists(pythonw):
+                logger.info('Also created pythonw.exe')
+                shutil.copyfile(pythonw, os.path.join(os.path.dirname(py_executable), 'pythonw.exe'))
+        if is_pypy:
+            # make a symlink python --> pypy-c
+            python_executable = os.path.join(os.path.dirname(py_executable), 'python')
+            logger.info('Also created executable %s' % python_executable)
+            copyfile(py_executable, python_executable)
+
+    if os.path.splitext(os.path.basename(py_executable))[0] != expected_exe:
+        secondary_exe = os.path.join(os.path.dirname(py_executable),
+                                     expected_exe)
+        py_executable_ext = os.path.splitext(py_executable)[1]
+        if py_executable_ext == '.exe':
+            # python2.4 gives an extension of '.4' :P
+            secondary_exe += py_executable_ext
+        if os.path.exists(secondary_exe):
+            logger.warn('Not overwriting existing %s script %s (you must use %s)'
+                        % (expected_exe, secondary_exe, py_executable))
+        else:
+            logger.notify('Also creating executable in %s' % secondary_exe)
+            shutil.copyfile(sys.executable, secondary_exe)
+            make_exe(secondary_exe)
+
+    if 'Python.framework' in prefix:
+        logger.debug('MacOSX Python framework detected')
+
+        # Make sure we use the the embedded interpreter inside
+        # the framework, even if sys.executable points to
+        # the stub executable in ${sys.prefix}/bin
+        # See http://groups.google.com/group/python-virtualenv/
+        #                              browse_thread/thread/17cab2f85da75951
+        original_python = os.path.join(
+            prefix, 'Resources/Python.app/Contents/MacOS/Python')
+        shutil.copy(original_python, py_executable)
+
+        # Copy the framework's dylib into the virtual
+        # environment
+        virtual_lib = os.path.join(home_dir, '.Python')
+
+        if os.path.exists(virtual_lib):
+            os.unlink(virtual_lib)
+        copyfile(
+            os.path.join(prefix, 'Python'),
+            virtual_lib)
+
+        # And then change the install_name of the copied python executable
+        try:
+            call_subprocess(
+                ["install_name_tool", "-change",
+                 os.path.join(prefix, 'Python'),
+                 '@executable_path/../.Python',
+                 py_executable])
+        except:
+            logger.fatal(
+                "Could not call install_name_tool -- you must have Apple's development tools installed")
+            raise
+
+        # Some tools depend on pythonX.Y being present
+        py_executable_version = '%s.%s' % (
+            sys.version_info[0], sys.version_info[1])
+        if not py_executable.endswith(py_executable_version):
+            # symlinking pythonX.Y > python
+            pth = py_executable + '%s.%s' % (
+                    sys.version_info[0], sys.version_info[1])
+            if os.path.exists(pth):
+                os.unlink(pth)
+            os.symlink('python', pth)
+        else:
+            # reverse symlinking python -> pythonX.Y (with --python)
+            pth = join(bin_dir, 'python')
+            if os.path.exists(pth):
+                os.unlink(pth)
+            os.symlink(os.path.basename(py_executable), pth)
+
+    if sys.platform == 'win32' and ' ' in py_executable:
+        # There's a bug with subprocess on Windows when using a first
+        # argument that has a space in it.  Instead we have to quote
+        # the value:
+        py_executable = '"%s"' % py_executable
+    cmd = [py_executable, '-c', 'import sys; print sys.prefix']
+    logger.info('Testing executable with %s %s "%s"' % tuple(cmd))
+    proc = subprocess.Popen(cmd,
+                            stdout=subprocess.PIPE)
+    proc_stdout, proc_stderr = proc.communicate()
+    proc_stdout = os.path.normcase(os.path.abspath(proc_stdout.strip()))
+    if proc_stdout != os.path.normcase(os.path.abspath(home_dir)):
+        logger.fatal(
+            'ERROR: The executable %s is not functioning' % py_executable)
+        logger.fatal(
+            'ERROR: It thinks sys.prefix is %r (should be %r)'
+            % (proc_stdout, os.path.normcase(os.path.abspath(home_dir))))
+        logger.fatal(
+            'ERROR: virtualenv is not compatible with this system or executable')
+        if sys.platform == 'win32':
+            logger.fatal(
+                'Note: some Windows users have reported this error when they installed Python for "Only this user".  The problem may be resolvable if you install Python "For all users".  (See https://bugs.launchpad.net/virtualenv/+bug/352844)')
+        sys.exit(100)
+    else:
+        logger.info('Got sys.prefix result: %r' % proc_stdout)
+
+    pydistutils = os.path.expanduser('~/.pydistutils.cfg')
+    if os.path.exists(pydistutils):
+        logger.notify('Please make sure you remove any previous custom paths from '
+                      'your %s file.' % pydistutils)
+    ## FIXME: really this should be calculated earlier
+    return py_executable
+
+def install_activate(home_dir, bin_dir, prompt=None):
+    if sys.platform == 'win32' or is_jython and os._name == 'nt':
+        files = {'activate.bat': ACTIVATE_BAT,
+                 'deactivate.bat': DEACTIVATE_BAT}
+        if os.environ.get('OS') == 'Windows_NT' and os.environ.get('OSTYPE') == 'cygwin':
+            files['activate'] = ACTIVATE_SH
+    else:
+        files = {'activate': ACTIVATE_SH}
+
+        # suppling activate.fish in addition to, not instead of, the
+        # bash script support.
+        files['activate.fish'] = ACTIVATE_FISH
+
+        # same for csh/tcsh support...
+        files['activate.csh'] = ACTIVATE_CSH
+
+
+
+    files['activate_this.py'] = ACTIVATE_THIS
+    vname = os.path.basename(os.path.abspath(home_dir))
+    for name, content in files.items():
+        content = content.replace('__VIRTUAL_PROMPT__', prompt or '')
+        content = content.replace('__VIRTUAL_WINPROMPT__', prompt or '(%s)' % vname)
+        content = content.replace('__VIRTUAL_ENV__', os.path.abspath(home_dir))
+        content = content.replace('__VIRTUAL_NAME__', vname)
+        content = content.replace('__BIN_NAME__', os.path.basename(bin_dir))
+        writefile(os.path.join(bin_dir, name), content)
+
+def install_distutils(home_dir):
+    distutils_path = change_prefix(distutils.__path__[0], home_dir)
+    mkdir(distutils_path)
+    ## FIXME: maybe this prefix setting should only be put in place if
+    ## there's a local distutils.cfg with a prefix setting?
+    home_dir = os.path.abspath(home_dir)
+    ## FIXME: this is breaking things, removing for now:
+    #distutils_cfg = DISTUTILS_CFG + "\n[install]\nprefix=%s\n" % home_dir
+    writefile(os.path.join(distutils_path, '__init__.py'), DISTUTILS_INIT)
+    writefile(os.path.join(distutils_path, 'distutils.cfg'), DISTUTILS_CFG, overwrite=False)
+
+def fix_lib64(lib_dir):
+    """
+    Some platforms (particularly Gentoo on x64) put things in lib64/pythonX.Y
+    instead of lib/pythonX.Y.  If this is such a platform we'll just create a
+    symlink so lib64 points to lib
+    """
+    if [p for p in distutils.sysconfig.get_config_vars().values()
+        if isinstance(p, basestring) and 'lib64' in p]:
+        logger.debug('This system uses lib64; symlinking lib64 to lib')
+        assert os.path.basename(lib_dir) == 'python%s' % sys.version[:3], (
+            "Unexpected python lib dir: %r" % lib_dir)
+        lib_parent = os.path.dirname(lib_dir)
+        assert os.path.basename(lib_parent) == 'lib', (
+            "Unexpected parent dir: %r" % lib_parent)
+        copyfile(lib_parent, os.path.join(os.path.dirname(lib_parent), 'lib64'))
+
+def resolve_interpreter(exe):
+    """
+    If the executable given isn't an absolute path, search $PATH for the interpreter
+    """
+    if os.path.abspath(exe) != exe:
+        paths = os.environ.get('PATH', '').split(os.pathsep)
+        for path in paths:
+            if os.path.exists(os.path.join(path, exe)):
+                exe = os.path.join(path, exe)
+                break
+    if not os.path.exists(exe):
+        logger.fatal('The executable %s (from --python=%s) does not exist' % (exe, exe))
+        sys.exit(3)
+    return exe
+
+############################################################
+## Relocating the environment:
+
+def make_environment_relocatable(home_dir):
+    """
+    Makes the already-existing environment use relative paths, and takes out
+    the #!-based environment selection in scripts.
+    """
+    home_dir, lib_dir, inc_dir, bin_dir = path_locations(home_dir)
+    activate_this = os.path.join(bin_dir, 'activate_this.py')
+    if not os.path.exists(activate_this):
+        logger.fatal(
+            'The environment doesn\'t have a file %s -- please re-run virtualenv '
+            'on this environment to update it' % activate_this)
+    fixup_scripts(home_dir)
+    fixup_pth_and_egg_link(home_dir)
+    ## FIXME: need to fix up distutils.cfg
+
+OK_ABS_SCRIPTS = ['python', 'python%s' % sys.version[:3],
+                  'activate', 'activate.bat', 'activate_this.py']
+
+def fixup_scripts(home_dir):
+    # This is what we expect at the top of scripts:
+    shebang = '#!%s/bin/python' % os.path.normcase(os.path.abspath(home_dir))
+    # This is what we'll put:
+    new_shebang = '#!/usr/bin/env python%s' % sys.version[:3]
+    activate = "import os; activate_this=os.path.join(os.path.dirname(__file__), 'activate_this.py'); execfile(activate_this, dict(__file__=activate_this)); del os, activate_this"
+    if sys.platform == 'win32':
+        bin_suffix = 'Scripts'
+    else:
+        bin_suffix = 'bin'
+    bin_dir = os.path.join(home_dir, bin_suffix)
+    home_dir, lib_dir, inc_dir, bin_dir = path_locations(home_dir)
+    for filename in os.listdir(bin_dir):
+        filename = os.path.join(bin_dir, filename)
+        if not os.path.isfile(filename):
+            # ignore subdirs, e.g. .svn ones.
+            continue
+        f = open(filename, 'rb')
+        lines = f.readlines()
+        f.close()
+        if not lines:
+            logger.warn('Script %s is an empty file' % filename)
+            continue
+        if not lines[0].strip().startswith(shebang):
+            if os.path.basename(filename) in OK_ABS_SCRIPTS:
+                logger.debug('Cannot make script %s relative' % filename)
+            elif lines[0].strip() == new_shebang:
+                logger.info('Script %s has already been made relative' % filename)
+            else:
+                logger.warn('Script %s cannot be made relative (it\'s not a normal script that starts with %s)'
+                            % (filename, shebang))
+            continue
+        logger.notify('Making script %s relative' % filename)
+        lines = [new_shebang+'\n', activate+'\n'] + lines[1:]
+        f = open(filename, 'wb')
+        f.writelines(lines)
+        f.close()
+
+def fixup_pth_and_egg_link(home_dir, sys_path=None):
+    """Makes .pth and .egg-link files use relative paths"""
+    home_dir = os.path.normcase(os.path.abspath(home_dir))
+    if sys_path is None:
+        sys_path = sys.path
+    for path in sys_path:
+        if not path:
+            path = '.'
+        if not os.path.isdir(path):
+            continue
+        path = os.path.normcase(os.path.abspath(path))
+        if not path.startswith(home_dir):
+            logger.debug('Skipping system (non-environment) directory %s' % path)
+            continue
+        for filename in os.listdir(path):
+            filename = os.path.join(path, filename)
+            if filename.endswith('.pth'):
+                if not os.access(filename, os.W_OK):
+                    logger.warn('Cannot write .pth file %s, skipping' % filename)
+                else:
+                    fixup_pth_file(filename)
+            if filename.endswith('.egg-link'):
+                if not os.access(filename, os.W_OK):
+                    logger.warn('Cannot write .egg-link file %s, skipping' % filename)
+                else:
+                    fixup_egg_link(filename)
+
+def fixup_pth_file(filename):
+    lines = []
+    prev_lines = []
+    f = open(filename)
+    prev_lines = f.readlines()
+    f.close()
+    for line in prev_lines:
+        line = line.strip()
+        if (not line or line.startswith('#') or line.startswith('import ')
+            or os.path.abspath(line) != line):
+            lines.append(line)
+        else:
+            new_value = make_relative_path(filename, line)
+            if line != new_value:
+                logger.debug('Rewriting path %s as %s (in %s)' % (line, new_value, filename))
+            lines.append(new_value)
+    if lines == prev_lines:
+        logger.info('No changes to .pth file %s' % filename)
+        return
+    logger.notify('Making paths in .pth file %s relative' % filename)
+    f = open(filename, 'w')
+    f.write('\n'.join(lines) + '\n')
+    f.close()
+
+def fixup_egg_link(filename):
+    f = open(filename)
+    link = f.read().strip()
+    f.close()
+    if os.path.abspath(link) != link:
+        logger.debug('Link in %s already relative' % filename)
+        return
+    new_link = make_relative_path(filename, link)
+    logger.notify('Rewriting link %s in %s as %s' % (link, filename, new_link))
+    f = open(filename, 'w')
+    f.write(new_link)
+    f.close()
+
+def make_relative_path(source, dest, dest_is_directory=True):
+    """
+    Make a filename relative, where the filename is dest, and it is
+    being referred to from the filename source.
+
+        >>> make_relative_path('/usr/share/something/a-file.pth',
+        ...                    '/usr/share/another-place/src/Directory')
+        '../another-place/src/Directory'
+        >>> make_relative_path('/usr/share/something/a-file.pth',
+        ...                    '/home/user/src/Directory')
+        '../../../home/user/src/Directory'
+        >>> make_relative_path('/usr/share/a-file.pth', '/usr/share/')
+        './'
+    """
+    source = os.path.dirname(source)
+    if not dest_is_directory:
+        dest_filename = os.path.basename(dest)
+        dest = os.path.dirname(dest)
+    dest = os.path.normpath(os.path.abspath(dest))
+    source = os.path.normpath(os.path.abspath(source))
+    dest_parts = dest.strip(os.path.sep).split(os.path.sep)
+    source_parts = source.strip(os.path.sep).split(os.path.sep)
+    while dest_parts and source_parts and dest_parts[0] == source_parts[0]:
+        dest_parts.pop(0)
+        source_parts.pop(0)
+    full_parts = ['..']*len(source_parts) + dest_parts
+    if not dest_is_directory:
+        full_parts.append(dest_filename)
+    if not full_parts:
+        # Special case for the current directory (otherwise it'd be '')
+        return './'
+    return os.path.sep.join(full_parts)
+
+
+
+############################################################
+## Bootstrap script creation:
+
+def create_bootstrap_script(extra_text, python_version=''):
+    """
+    Creates a bootstrap script, which is like this script but with
+    extend_parser, adjust_options, and after_install hooks.
+
+    This returns a string that (written to disk of course) can be used
+    as a bootstrap script with your own customizations.  The script
+    will be the standard virtualenv.py script, with your extra text
+    added (your extra text should be Python code).
+
+    If you include these functions, they will be called:
+
+    ``extend_parser(optparse_parser)``:
+        You can add or remove options from the parser here.
+
+    ``adjust_options(options, args)``:
+        You can change options here, or change the args (if you accept
+        different kinds of arguments, be sure you modify ``args`` so it is
+        only ``[DEST_DIR]``).
+
+    ``after_install(options, home_dir)``:
+
+        After everything is installed, this function is called.  This
+        is probably the function you are most likely to use.  An
+        example would be::
+
+            def after_install(options, home_dir):
+                subprocess.call([join(home_dir, 'bin', 'easy_install'),
+                                 'MyPackage'])
+                subprocess.call([join(home_dir, 'bin', 'my-package-script'),
+                                 'setup', home_dir])
+
+        This example immediately installs a package, and runs a setup
+        script from that package.
+
+    If you provide something like ``python_version='2.4'`` then the
+    script will start with ``#!/usr/bin/env python2.4`` instead of
+    ``#!/usr/bin/env python``.  You can use this when the script must
+    be run with a particular Python version.
+    """
+    filename = __file__
+    if filename.endswith('.pyc'):
+        filename = filename[:-1]
+    f = open(filename, 'rb')
+    content = f.read()
+    f.close()
+    py_exe = 'python%s' % python_version
+    content = (('#!/usr/bin/env %s\n' % py_exe)
+               + '## WARNING: This file is generated\n'
+               + content)
+    return content.replace('##EXT' 'END##', extra_text)
+
+##EXTEND##
+
+##file site.py
+SITE_PY = """
+eJzVPP1z2zaWv/OvQOXJUEplOh/dzo5T98ZJnNZ7buJt0mluXY+WkiCJNUWyBGlZe3P3t9/7AECA
+pGS77f5wmkwskcDDw8P7xgMGg8FpUchsLtb5vE6lUDIuZytRxNVKiUVeimqVlPPDIi6rLTyd3cRL
+qUSVC7VVEbaKguDpH/wET8WnVaIMCvAtrqt8HVfJLE7TrUjWRV5Wci7mdZlkS5FkSZXEafIvaJFn
+kXj6xzEIzjMBM08TWYpbWSqAq0S+EJfbapVnYlgXOOfn0V/il6OxULMyKSpoUGqcgSKruAoyKeeA
+JrSsFZAyqeShKuQsWSQz23CT1+lcFGk8k+Kf/+SpUdMwDFS+lpuVLKXIABmAKQFWgXjA16QUs3wu
+IyFey1mMA/DzhlgBQxvjmikkY5aLNM+WMKdMzqRScbkVw2ldESBCWcxzwCkBDKokTYNNXt6oESwp
+rccGHomY2cOfDLMHzBPH73IO4PghC37KkrsxwwbuQXDVitmmlIvkTsQIFn7KOzmb6GfDZCHmyWIB
+NMiqETYJGAEl0mR6VNByfKNX6NsjwspyZQxjSESZG/NL6hEF55WIUwVsWxdII0WYv5XTJM6AGtkt
+DAcQgaRB3zjzRFV2HJqdyAFAietYgZSslRiu4yQDZv0hnhHaPyfZPN+oEVEAVkuJX2tVufMf9hAA
+WjsEGAe4WGY16yxNbmS6HQECnwD7Uqo6rVAg5kkpZ1VeJlIRAEBtK+QdID0WcSk1CZkzjdyOif5E
+kyTDhUUBQ4HHl0iSRbKsS5IwsUiAc4Er3n34Ubw9e31++l7zmAHGMrtcA84AhRbawQkGEEe1Ko/S
+HAQ6Ci7wj4jncxSyJY4PeDUNju5d6WAIcy+idh9nwYHsenH1MDDHCpQJjRVQv/+GLmO1Avr8zz3r
+HQSnu6hCE+dvm1UOMpnFaylWMfMXckbwjYbzbVRUq1fADQrhVEAqhYuDCCYID0ji0myYZ1IUwGJp
+kslRABSaUlt/FYEV3ufZIa11ixMAQhlk8NJ5NqIRMwkT7cJ6hfrCNN7SzHSTwK7zOi9JcQD/ZzPS
+RWmc3RCOihiKv03lMskyRAh5IQgPQhpY3STAifNIXFAr0gumkQhZe3FLFIkaeAmZDnhS3sXrIpVj
+Fl/UrfvVCA0mK2HWOmWOg5YVqVdatWaqvbz3Ivrc4jpCs1qVEoDXU0/oFnk+FlPQ2YRNEa9ZvKpN
+TpwT9MgTdUKeoJbQF78DRU+VqtfSvkReAc1CDBUs8jTNN0Cy4yAQ4gAbGaPsMye8hXfwP8DF/1NZ
+zVZB4IxkAWtQiPwuUAgETILMNFdrJDxu06zcVjJJxpoiL+eypKEeRuwjRvyBjXGuwfu80kaNp4ur
+nK+TClXSVJvMhC1eFlasH1/xvGEaYLkV0cw0bei0xumlxSqeSuOSTOUCJUEv0iu77DBm0DMm2eJK
+rNnKwDsgi0zYgvQrFlQ6i0qSEwAwWPjiLCnqlBopZDARw0DrguCvYzTpuXaWgL3ZLAeokNh8z8D+
+AG7/AjHarBKgzwwggIZBLQXLN02qEh2ERh8FvtE3/Xl84NTzhbZNPOQiTlJt5eMsOKeHZ2VJ4juT
+BfYaa2IomGFWoWu3zICOKOaDwSAIjDu0VeZrbr9NJtM6QXs3mQRVuT0G7hAo5AFDF+9hojQcv1mU
++RpfW/Q+gj4AvYw9ggNxSYpCso/rMdMrpICrlQvTFM2vw5ECVUlw+ePZu/PPZx/FibhqtNK4rZKu
+YcyzLAbOJKUOfNEatlFH0BJ1V4LqS7wDC03rCiaJepMEyriqgf0A9U9lTa9hGjPvZXD2/vT1xdnk
+p49nP04+nn86AwTBVMjggKaMFq4Gn09FwN/AWHMVaRMZdHrQg9enH+2DYJKoSbEttvAAbB1wYTmE
++Y5FiA8n2oxOkmyRhyNq/Cv70SesGbTTdHX81bU4ORHhr/FtHAbguDRNeRF/IB7+tC0kdK3gzzBX
+oyCYywXw+41EqRg+JWd0xB2AiNAy18bx1zzJzHt67Q1BQjukHoDDZDJLY6Ww8WQSAmmpQ88HOkTs
+0SKrD6FjsXW7jjQq+CklLEWGXcb4Xw+K8ZT6IRqMotvFNAIZWc9iJbkVTR/6TSaoKCaToR4QJIh4
+HLwclv1QmCaoKMoEnEniFVQcU5Wn+BPho+iRyGA8g6oJF0nHK9FtnNZSDZ1JARGHwxYZUbslijgI
+/IIhmL9m6UajNjUNz0AzIF+ag+oqW5TDzwE4GaAjTOSE0RUHPEwzxPRv7N4TDuDnhahjlWpBYZUk
+Ls8uxctnLw7Rh4BAb26p4zVHs5hktbQPF7BaS1k5CHOvcEzCMHLpskDlhk+P98NcR3Zluqyw0Etc
+ynV+K+eALTKws8riR3oD4TDMYxbDKoIyJSPMSs84azEGfzx7kBY02EC9NUEx62+W/oAjcJkpUB0c
+zRKpdajN9qco89sELfx0q1+CgQL1hmbKeBOBs3Aek6EdAg0BrmeGlNrIEBRYWbOXSHgjSFTx80YV
+RgTuAnXrNX29yfJNNuHw8wTV5HBkWRcFSzMvNmiW4EC8A8MBSOYQTTVEYyjgZwuUrUNAHqYP0wXK
+kkMPgMC6KoqRHFgmvqIpcqiGwyKM0StBwltKNNK3ZgiKbwwxHEj0NrIPjJZASDA5q+CsatBMhrJm
+msHADkl8rruIOO7zAbSoGIGhG2po3MjQ7+oYlLO4cJWS0w9t6OfPn5lt1IqSGojYFCeNdntB5i0q
+tmAKE9AJxg3iFAmxwQY8SgBTK82a4vCjyAt2gWA9L7Vsg+WGkKqqiuOjo81mE+mQPi+XR2px9Je/
+fv31X5+xTpzPiX9gOo606PxWdETv0I2MvjEW6Fuzci1+TDKfGwnWUJIrRP4f4vddncxzcXw4svoT
+ubgxrPi/cT5AgUzMoExloO2gweiJOnwSvVQD8UQM3bbDEXsS2qRaK+ZbXehR5WC7wdOY5XVWhY4i
+VeJLsG4QFs/ltF6GdnDPRpofMFWU06HlgcPn14iBzxmGr4wpnqCWILZAi++Q/kdmm5j8Ga0hkLxo
+ojoh67Zfixnizh8u79Y7dITGzDBRyB0oEX6TBwugbdyVHPxoZxTtnuOMmo9nCIylDwzzaldwiIJD
+uOBajF2pc7gafVSQpg2rZlAwrmoEBQ1u3ZSprcGRjQwRJHo3JsLmhdUtgE6tdJ0Jys0qQAt3nI61
+a7OC4wkhD5yI5/REglN73Hn3jJe2TlPKorR41KMKA/YWGu10Dnw5NADGYlD+NOCWelnOP7QWhdeg
+B1jOiRdksEWHmfCN6wMODgY97NSx+rt6M437QOAiUfuHASeMT3iAUoEwFUOfcXdxuKUtJ5taCO82
+OMRTZpVIotUO2Wrrjl6Z2muXFkmGqtdZo2iW5uAUW6VIfNS8930FClzwcZ8t0wKoydCQw2l0Qs6e
+J3+hbocpq2WNwb2b+0CM1oki44ZkWsF/4FVQToESQEBLgmbBPFTI/In9CSJn56u/7GAPS2hkCLfp
+Li+kYzA0HPP+QCAZdQYEhCADEnZlkTxH1gYpcJizQJ5sw2u5U7gJRqRAzBwDQloGcKeXXnyDTyLc
+dSABRch3lZKF+FIMYPnakvow1f2ncqnJGgydBuQp6HTDiZuKcNIQJ620hM/QfkKC9ieKHDh4Ch6P
+m1x32dwwrc2SgK/u622LFChkSpwMRi6q14YwbgL3ixOnRUMsM4hhKG8gbxvFjDQK7HJr0LDgBoy3
+5u2x9GM3YYF9h2GuXsj1HYR/YZmoWa5CjG87qQv3o7miSxuL7UUyHcAfbwEGo2sPkkx1+gKTLL9j
+kNCDHvZB9yaLWZF5XG6SLCQFpul34i9NBw9LSs/GHX2kaOoIJopZxqN3JQgIbTcegTihJoCgXIZK
+e/1dsHunOLBwufvA85qvjl9ed4k73pXgsZ/+pTq7q8pY4WqlvGgsFLhaXfuNShcmF2dbvWGoN5Qx
+SihzBUGk+PDxs0BCcC51E28fN/WG4RGbe+fkfQzqoNfuJVdrdsQugAhqRWSUo/DxHPlwZB87uT0T
+ewSQRzHMnkUxkDSf/B44+xYKxjicbzNMo7VVBn7g9ddfTXoSoy6SX381uGeUFjH6xH7Y8gTtyLSR
+L3qnbbqUMk7J13A6UVIxa3jHtilGrNAp/NNMdt3jdOLHvDcmo4Hfad6JG83ngOgBUXY+/RViVaXT
+W7dxklJOHtA4PEQ9Z8Jszhz04+NB2o8ypqTAY3k27o2E1NUzWJiQ4/pRdzraLzo1qd+eeNR8ilh1
+UTnQW+jNDpC3Le7u/u2W/V5L/W/SWY8E5M1m0EPAB87B7E7+/58JKyuGppXVqKX1ldyv5w2wB6jD
+HW7OHjekOzRvZi2MM8Fyp8RTFNCnYkNb0pTKw40JgDJnP6MHDi6j3th8U5clb0+SnBeyPMT9urHA
+ahzjaVCRTxfM0XtZISa22YxSo07tRt6nOkOd7LQzCRs/tV9kV7lJkcjsNimhL2iVYfj9hx/Owi4D
+6GGwUz84dx0NlzzcTiHcRzBtqIkTPqYPU+gxXX6/VLVdZZ+gZsvYJCA12bqE7eQdTdzavwb3ZCC8
+/UHeh8WIcLaSs5uJpL1lZFPs6uRg3+BrxMRuOfs1PipeUKESzGSW1kgrdvSwwmxRZzNKx1cS7Lku
+B8XyENox5nTTIo2XYkid55jq0NxI2ZDbuNTeTlHmWIAo6mR+tEzmQv5WxymGkXKxAFxwr0S/inh4
+yniIt7zpzYVpSs7qMqm2QIJY5XqrifbHnYbTLU906CHJuwpMQNwxPxYfcdr4ngk3N+QywaifYMdJ
+YpyHHcxeIHIXPYf3WT7BUSdUxzlmpLrbwPQ4aI+QA4ABAIX5D0Y6U+S/kfTK3c+iNXeJilrSI6Ub
+2ebkcSCU4Qgja/5NP31GdHlrB5bL3Vgu92O5bGO57MVy6WO53I+lKxK4sDZJYiShL1HSzqL3FmS4
+OQ4e5iyerbgd1vdhHR9AFIUJ6IxMcZmrl0nh7SQCQmrb2d+kh02BRcKFg2XOKVcNErkf90x08GgK
+lJ3OVK6hO/NUjM+2q8jE73sURVQONKXuLG/zuIojTy6WaT4FsbXojhsAY9GuN+HcXHY7mXI2sWWp
+Bpf/9en7D++xOYIamN106oaLiIYFpzJ8GpdL1ZWmJtgogB2ppV/3Qd00wIMHZnJ4lAP+7y0VFCDj
+iA1tiOeiAA+Ayn5sM7c4Jgxbz3UVjX7OTM57GydikFWDZlI7iHR6efn29NPpgFJMg/8duAJjaOtL
+h4uPaWEbdP03t7mlOPYBoda5lMb4uXPyaN1wxP021oBtub3PrlsPXjzEYPeGpf4s/62UgiUBQkU6
+2fgYQj04+PlDYUKHPoYRO9Vh7k4OOyv2nSN7joviiH5fmrs9gL+3hjHGBAigXaihiQyaYKql9K15
+3UNRB+gDfb0/HIK1Q692JONT1E6ixwF0KGub7Xb/vH0BNnpKVq/Pvjt/f3H++vL00/eOC4iu3IeP
+Ry/E2Q+fBZUjoAFjnyjGnfgKC1/AsLiHWcQ8h381pjfmdcVJSej19uJC7wys8TgD1reizYngOVfN
+WGico+Gsp32oy10Qo1QHSM65EaoOoXMlGC+t+cyCynUNLB1HmaKzWuvQS58HMueGaBs1AumDxi4p
+GARXNMErqlSuTFRY8o6TPkvTg5S20bYOIaUcVGd32tlvMdl8LzFHneFJ01kr+qvQxTW8jlSRJhDJ
+vQqtLOluWI3RMI5+aDdUGa8+Deh0h5F1Q571TizQar0KeW66/6hhtN9qwLBhsLcw70xSNQLV6GIt
+lQixEe8chPIOvtql12ugYMFwY6nCRTRMl8DsYwiuxSqBAAJ4cgXWF+MEgNBaCT8BfexkB2SOxQDh
+m/X88O+hJojf+pdfeppXZXr4D1FAFCS4ciXsIabb+C0EPpGMxNmHd6OQkaNKUPH3GkvAwSGhLJ8j
+7VQuwzu2k6GS6UKXM/j6AF9oP4Fet7qXsih1937XOEQJeKKG5DU8UYZ+IVYXWdhjnMqoBRqr2y1m
+eErM3fY2nwPxcSXTVBdEn7+9OAPfEQvuUYJ4n+cMhuN8CW7Z6lovPsXWAoUbuvC6RDYu0YWlTf15
+5DXrzcyiyFFvrw7ArhNlP7u9OqnOMk6Ui/YQp82wnJLzCLkZlsOsLHN3txnS2W1GdEfJYcaYXJZU
+NelzBnA0PY05MIKICYv6TbKZ9y6TrDJlcmkyA20KihfU6hhEBUmMJ9eI//KM0715qcyBF3hYbMtk
+uaowpQ6dIyq2x+Y/nH6+OH9P1esvXja+dw+LjikeGHPpwgnWpWHOA764tWbIW5NJH+fqVwgDdRD8
+ab/imogTHqDTj9OL+Kf9ik8cnTjxIM8A1FRdtIUEwwCnW5/0NBLBuNpoGD9u3VmDmQ+GMpJ4wEGX
+F7jz6/KjbdkyKJT9MS8fsVexKDQNh6azWwfV/ug5LgrcXJkP+xvB2z4JM58pdL3pvNlVceV+OrKI
+hx8Bo25rfwxTk9RpqqfjMNsubqHgVlvaXzInY+q0m2UoykDEodt55DJZvyrWzZkDvdrdDjDxjUbX
+SGKvQh/8kg20n+FhYondiVZMRzo7QaYA8xlSHxGpwZNCuwAKhEpOh47kjkdPX3hzdGzC/XPUugss
+5PegCHUBKB0syEvgRPjyG7uP/IrQQlV6LELHX8lkltvqJPxsVuhbPvfn2CsDlMpEsSvjbCmHDGts
+YH7pE3tHIpa0rccxV0mrWkJzN3iodzsYvCsW/bsnBrMWH3Ta3chtWxv51MEGvccPfAhlvAHtXtTV
+kNdq52YBNtdbsMMQkyS/hTvodQ96Ghb6Xb/17OHgh4ll3Etrr1pHW0L7QvuVsxICpkrRZoljhY2H
+6BrmxgaeNFZ4YJ/qihH7u+e8kFPl6sJlFFyo3gwHukEr1B/wyRU+uZdQZXRzsEK/m8tbmebgFkHE
+hYXvv9rC91FkUx29NUF/BoKX28ttP3r0pkHu2BTno+OkCljIKJPVEWLUm5C5B7kGH1z2X3TQEGc3
+5Me++fl8LN68/xH+fy0/QOSD59fG4h+AiXiTlxAB8hlKOtyOpf0Vh3Z5rfCQG0GjzQS+BwBdqkuP
+2rhxoc8c+IcNrBYTWGdZrvnyCUCR50jnihsbbirp4bc56tN1Fo0j17c0A/0SybD7AAQeGjjSLaNV
+tU5RnTupjGZNrwYX52/O3n88i6o75Hbzc+CkOvwqHZyR3sgtcdNqLOyTWY1Prh2/9nuZFj1urY4M
+zWEKjAxFCMFDYaNBvtsgthFAXGJ4L4rtPJ9F2BJ4n89vVRvwc0dOEHivHfaMIMIajvRWV+Ns42Og
+hvilrZcG0JD66DlRT0IonuJBIn4cDfot5VhQ/hn+PL3ZzN30tT4RQhNsY9rMeuh3t6pxxXTW8Fxm
+ItRO7EqYc4JpEqv1dOaeH/uQCX07BSg92o+Qi7hOKyEzEGEKxumaAND97pEvlhPmFrY4dA6K0inp
+Jt4qpyImVmKAow7opDNunFBmD2LlH+IbthB4Fk3UfKgVoBOiFOHkTldVz1Ysxxy0EAF7CgQ2Sfby
+RdghMg/KkeyscTVhnujYMUZLWen584Ph6Op5Y+wpezzzDnzOCrCDLqccgA4tnj59OhD/cb9/wqhE
+aZ7fgOMEsPvCVnFBr3d4FnpydrW6vrd5EwFLzlbyCh5cU5bbPq8zSiHu6UoLIu1fAyPEtQktP5r2
+LUvNybWSN4S5BW8saRPyU5bQHTSYApKocvVVPpgeMgJFLAm6IYzVLElCTifAemzzGs9qYTpQ84u8
+A45PEMwY3+JOFgfDK/QBqbDSco9F50QMCPCACp14NDrsSqeVAM/J5VajOTnPkqo5Z/DM3eTUh7or
+e7WM5isRb1AyzDxaxHCO/Xms2vjA+V4W9WKKfHblJgZbs+TX9+EOrA2Sli8WBlN4aBZplstyZowq
+rlgySyoHjGmHcLgz3ahDBigKelAagIYnwzC3Em3ffmHXxcX0A+33HpqRdJlPZW8p4iROnLWq3aKo
+GZ/SRZaQlm/NlxGM8p7Sz9of8MYSX+jkJxaZe5cpuMfd6kxfksB1Fs3NCQCHLuaxCtKyo6cjnNug
+LHxmWh1uNHcqODXxGEQTbrdJWdVxOtEH+SfouU3sBrjG0x6T2nsA0Pos4Pbn4BAf6pJu8B1MNQzS
+EysyTcn+iVjoJELkHj3yT+kUOfp6Lzw9jqnpZ3wRgKPBseWX5vDKQ1S+OULROX3gYjmm2qNw1K6o
+7LTCfQ5TIm+d7HYc8KghW7B8h31WbPFOHpjWk3lE/0LfkaPLFHBj6tGDp8mUBgv7Co/v76srATH+
+W4OgLBI5P3yiEDvG+Y9C1VAMddxA4REzDOnuCQL5ZWsnzykv5NrfXds3HaBff7UPrKuCewufac/E
+V8v6aJtbidxs2uDnwHrEK3C6UW/MzWFkrZb43CbqEDaI9qy5qVdpH5mB1w+f8p4JP2BHNMTBNHe4
+8rqPVha/faRqGgW/i0q6Vz+t0AnGUtFVzG9QmdXFsQ0V+TBfRmn2oVtAhJ/qpre0Psa7j4jRq5tw
+3/S5/7656xaBnbnZP+vM3T9C49JA993NL300YAddE+JBVbkWo8mfI7pjvbXbn6LSn4W9hZEzVcSD
+GrWxZsl1PHO/Y4HBIV/i6B6HClyQZtVbc+qcD2uzc5eTu9zMm6n43J6QpB3yuWYvNud0pc+Ea64m
+crlUkxhvhJqQD0j1AR3jbryKd3QbkIzV1jgDeOcCgDCsoiu53GJNWHXwM/lmSt5edw7XCxqaitCc
+qjaVzDm2154HgIs4pqf+JnPEZWmDVGI2RtVlUYKzNtD3F/K+b1+pXAPUxJfrWN0Y1E2Psb7ODofg
+YgNzhIozCewAetQBQvDJCudmF67znEzsO+CXZ81R0WRsGUJm9VqWcdXckuDvLyXiW2cEOjiHC+xE
+kI3YtTjFRSyx/OEghTGc/f6ldo4832/P+dCRVWkPZyvqoZMTjzl66ki54ebkzt6S5N7OMadrMSle
+5Ns1hG3WcJ+9GQKWwlz5Q4pQh3T8Vl9DwvfTcc4Jq+ocPgK5d4+t+NWNVmexw2DRcJ65iqF77wSe
+fCRD23edVIcLuhdH+czQjO/rDcssnd2EHY0tFU+4Ra/iaUYbNYEOFiLdE+j4xaaPDHQ8+A8MdPTl
+X2BNND5aH/SWn94TEbGacG/SahgB+kyASLhh0rqHydjDoVvMCeFKcjewl1GyznROiBgzgRzZvWKF
+QPCNWcqtfPNutDHj9kUivnTR4+8uPrw+vSBaTC5P3/zn6Xe0zY9ZvZbNenAkmOWHTO1Dr6zQjQr1
+1mzf4A22PVfTcW28htB539nW6oHQfw6ib0Hbisx9vatDp5682wkQ3z/tFtRdKrsXcsf50rXL7oZs
+q/4v0E+5WMv8cvbWzCOTU2ZxaBLG5n2T49My2kmB7Fo4p2yqq060U6ovM9uRnhnZ4j1aAUztIX/Z
+zJ6pxLb5I3ZU2leEU8UhnmIxNwGAFM6kcyEV3UXFoCr/LvISlF2MOxTsMI7tvZ7UjrOYyl5Yi7sU
+MxkZgnjHSAbd+bnCPpfpDioEASs8fd0SI2L0n877272yJ0pcHdKBtUNUNtf2F66ZdnJ/TnBHrLL3
+liiz5Y27AdB4UafuLpft0+lAzh8lTfOFUyENmu8I6NyIpwL2Rp+JFeJ0K0KIEvVWDhZdER31nUMO
+8mg3HewNrZ6Jw13HmdzjPEI8391w3joxpHu84B7qnh6qNodGHAuMdT+7zimJbwkyZ90FXVTiOR+4
+26Ovx4Svt1fPj23KFvkdX7vXYCDtB45hv2pOBuy9GsvpTbxSjqn+A4uNRm3w1wOHNRdid4DTqXPe
+EQSZ7TiGNPDe99dGmB7enb2DNqKW745hQmL4RI1oUk5luMbdPhl1JtuorC4MLnK/H0ZH+wEohNLv
+m+CHb2MB9fxMx4PTmu4TtA4nHg115IEKHXxe4B7G62uwa3eno2kP6k4l//agADdo855ebxBr9hq4
+lZfo2G0L2jNveGCH7edDfv39nz+gf7ckxnZ/sc+htq1e9h4sYScWi6hw87pFIfM4AusCCnNIahrr
+b42E4+H9howONzVTQ65Ah4/qsvCuUAosyImdaMtvjUHwf71Zz9M=
+""".decode("base64").decode("zlib")
+
+##file ez_setup.py
+EZ_SETUP_PY = """
+eJzNWmuP28YV/a5fwShYSIJlLt8PGXKRJi5gIEiDPAoU9lY7zxVrilRJyhu1yH/vmeFDJLVU2iIf
+ysDZXXJ45z7PuXekL784nqt9ns3m8/kf87wqq4IcjVJUp2OV52lpJFlZkTQlVYJFs/fSOOcn45lk
+lVHlxqkUw7XqaWEcCftEnsSirB+ax/Pa+PuprLCApScujGqflDOZpEK9Uu0hhByEwZNCsCovzsZz
+Uu2NpFobJOMG4Vy/oDZUa6v8aOSy3qmVv9nMZgYuWeQHQ/xzp+8byeGYF5XScnfRUq8b3lquriwr
+xD9OUMcgRnkULJEJMz6LooQT1N6XV9fqd6zi+XOW5oTPDklR5MXayAvtHZIZJK1EkZFKdIsulq71
+pgyreG6UuUHPRnk6HtNzkj3NlLHkeCzyY5Go1/OjCoL2w+Pj2ILHR3M2+0m5SfuV6Y2VRGEUJ/xe
+KlNYkRy1eU1UtZbHp4LwfhxNlQyzxnnluZx98+5PX/387U+7v7z74cf3f/7O2BpzywyYbc+7Rz//
+8K3yq3q0r6rj5v7+eD4mZp1cZl483TdJUd7flff4r9vtfm7cqV3Mxr8fNu7DbHbg/o6TikDgv3TE
+Fpc3XmNzar8+nh3TNcXT02JjLKLIcRiRsWU7vsUjL6JxHNBQOj4LRMDIYn1DitdKoWFMIuJZrvB8
+y5GURr4QrrRjzw5dn9EJKc5QFz/ww9CPeUQCHknmeVZokZhboRM6PI5vS+l08WAAibgdxNyhIghs
+SVyHBMJ3hCcjZ8oid6gLpa7NLMlCN45J4PphHIc+IzyWPrECO7oppdPFjUjEcJcHgnHHcbxQ2mEs
+Q06CIJaETUjxhroEjuX5xPEE94QtKAtDKSw3JsQTgQyFf1PKxS+MOsSOfOgRccKkpA63oY/lUpfa
+zHtZChvlC3WlQ33fjXmAuIYy9AgPY9uBIBJb0YRFbJwvsIcLDk8GIXe4I6WwPcuK3cCTDvEmIs1s
+a6gMgzscQn3uEsvxA88PEB9mu5FlkdCKrdtiOm38kONFxCimkRWGDvNj4rsk8lyX+JxPeqYW47di
+uPACwiL4Mg5ZFPt+6AhfRD7SUdCIhbfFBJ02kUAlESGtAA5ymAg824M0B0bC4RPRBqgMfeNQIghq
+2HY53kcZOZEIKfGpT6ARF7fFXCLFAzeWMbUgzGOe48Wh5XpcMEcwizmTkbKHvgk8FnvSpTIkIbLQ
+FSxyhUUdhDv0YurcFtP5hkoSO7ZlUY4wcdQEJAnOXQQ+8KwomBAzwhlpWYFHZUCIQ0NuQS141kNi
+W5EdMmcqUCOcCezAjh0hmOtLLxSImh0wHhDbgVQnnJIywhlpRwAogC+XSBXi+DGLIUXaPKRhJCfQ
+io1wRliCh14QOSyOIyppCE9HFrLXQsxDeyrY7jBIhAppB5JzGOb7vu1Fns1C4BePozjwp6SM0Ipa
+NLZdmzBCXceCM4BzofQ85gMoQlvelNJZhCSR2DPgnqTSRUVRGXsBs+AqoJ6YShhvaFGk0BrA7zqM
+05iFDmXSA3w5gXQiIqfQyh9aJEQseWRBHRQkMla6ApjuhwAMHtnBVKT9oUVEAqu4BKvYoWULAeeG
+ICefMhAeCaZQxh/FKOKuDAAIHmOERKHtIXG4G1LGuMt9PiElGFqEgonA8pFtB2CiKPJCByLAmL4X
+o7SngDMYsRvzAyL9kMK/6B5QDYEFQzzPRYH5ZAobgqFF1JERCX0HZA/YpS5I2kKoufAlWgnfnZAS
+juDOQoxkTDhzSWD7wrdtH2WIliICBE7mSzhiAhLJ2PfAAhxYbkkahEza0kEY8MiZqoBwaJEHjiXA
+W4mWAQXouZ5t25KLyLXxL5zSJRp1Q5bqhZwYHok5+EOlIAA8ci3VWFm3pXQWMUrcCNiAnsOLXGap
+nEW2wdkMzDJJA9HQIjt07BAgh0DHnNm+5ccW8SPqCtR57E9FOh5aBN2ZZ6GZsZWHqRcHwmOSCiuC
+rcyainQ8QgYkGRo7cKsbRTwAOhEhrADgxQLXm+rvGimdRVIgtK7wiR1S22EIE/M9m4bgXjC/mGKS
+eMhHjKBsbKlQkziCA5js2AWzhdSPHfQ4kPLrrDcRYLwpZ1Vx3tQD156U+zSh7byF3n0mfmECo8Z7
+feedGomatXjYXzfjQhq7zyRN0O2LHW4todMuwzy4NtQAsNpoAxJptPfVzNiOB/VDdfEEs0WFcUGJ
+0C+ae/FLfRfzXbsMcpqVX2w7KR9a0Q8XeerC3IVp8O1bNZ2UFRcF5rrlYIW65sqkxoJmPrzDFEYw
+hvEvDGP5fV6WCU174x9GOvx9+MNqfiXsrjNz8Gg1+EvpI35JqqVT3y8Q3CLT7qodOhoO9aJmvNqO
+hrl1p9aOklJsewPdGpPiDqPqNi9NdirwW51M3QtcpOS8tf1ZEySMjV+dqvwAPzBMl2eMohm/78zu
+nRSouf5APiGWGJ4/w1VEOQjOU6YdSbWvx/nHRulHo9znp5SraZbUvu5Layfz7HSgojCqPakMDMKd
+YC1LTcCZ8q4hMfV2Sp0yrl8RxuPAEY+GGmmXz/uE7dvdBbRWRxO1PGNxv1iZULL20qPaUsnpHWPs
+RTE4IHlOMHPTSyYIvkZG1gmuVc5y+CMtBOHni/rY473sqafdrrdrzia0mKrRUkujQqvSOESfWLA8
+42Xtm1aNI0GiKKfCI6qskipB6LKn3nlGHfHG/jwT+jyhPhvhtV5wap4qH754PqK0bA4bRCNMn+UU
++Qk7iVqVus6IcRBlSZ5EfcBxKbrHR50vBUlKYfx4LitxePeL8ldWByIzSIV79ckGoQpalPEqBZUx
+9amH2Wao/vlMyl2NQrB/ayyOn552hSjzU8FEuVAIo7Y/5PyUilKdkvQAdPy4rglUHUceNG5bri5I
+olJueymaXl02HhuVYFt261GhXTCgLRITnhVFtbTWapMeyDVA3e30pn+6Q9tjvl0TmJ0G5q2SUQcI
+wD6WNXCQfvgCwncvtYDUd0jz6HqHgWizSa7l/KLx2+38VeOq1ZtGdl+FoYC/1Cu/zjOZJqyCazZ9
+9O9H/r9F+/lP+0v2T+T78u32rlx1tdzWsD7K/JgNAX/OSLaoVEl1JQLMUMd3ukaa4zpVLacsQyqb
+xvepQIa0y6/kqRpSpQwAErCl1VAmRQlHnEpVDgtIOLehN17/3FN+YY7kfcw+ZsuvT0UBaYDzWsBd
+MeKtFVjrksvCJMVT+cF6uM1ZOn5pKYYxQKIPw7nuV9qHUZ0+qFe+hLUayfNPA1Ev5eB01nyToCQS
+elIM/l1e/SkHL9zO55ppXyrr35tuVfGjPAc8+80LpKrLmFxIwUhzVrckGj5rG5KqPiHWLcb/KcnW
+EK0+A2hJ9rc4Vt1Tu14TbI37jxfOnODFvGbDlgwVqbDqRNKLEQ3JDImk/YihANdQB9m6RwqldZ61
+/erW6IHZ67sSvfddqVrveb9wRkfgda5Cbp87lM+MV8MWsSSfBbTfoiWvSeHveZItWwppl9biyoIp
+cbpP/g5s3rbWCqra11GkZVUua7GrjSqwrz7niUqgoyCKL1t1yq4+BniuLp2KHIKUN8rWS2n+NFil
+mnEVl+G76sJK85kU2VL5+fXvd9WfkDTA2iB5+VKW3+mUUJ+cLMVnkak/YM4Rys72Ij2qvu99nW29
+3qNLFTQnKv/VZztL5YoZKGFtAF1m6tYB5ZwJOBKvoA5V5wuEFs8KjwnG2bLUb/c5QCO4OWu2BHQ3
+Pc5lR6jM22w2Z7MlQExslIe1mANhe9Vu8VzUxLRHeKFE9ZwXn5pN18axZpecVqT5XE4hhUaJu3I2
+UygCDzDdtesFkHypxKZyCtGwVd8Ac/V7RhFJsb5KmR7oXjVUOsvWqpquXkNHoZO1StRk2TROqRDH
+N/WP5aj3GmZnC8OaF8u53mLEe7rkGnww8TM/imx5texL4wc0/ffPRVIBfBBj+Fe328DwT2v10eCz
+ip5qF1ihyhDQyPKiOOnkSMVImI57Pz1UF14Jvb7FxPZqPmabGsJhgKkGkuVqqHGNItqaGivW82c6
+hzvxwNR21GN49xKGQTUUbsYQgA02eheW5qVYrq4goqw2Wmj/ecNmLWhBwVT90sLW7D+5FH8fkOlL
+NCyf11OMfeHc97c+NNUc+w6tVbOqJYiXmunRh9G3Oul6eOiw+kriZc3tAUNP6tZ1SzYcIwZThI6Z
+Ko3e7MDywwGGmoMesj3OIc1A1l5NjLSLU3CB9vPqlTpteVjpNH0Wi0KntTAUjf9mqihLlZ9HXKXU
+vuYQLDplmAA/LTuzhg1n0m/czd2u8dZuZ2wxElqmZdqL/3pE+CsAXoOrmotpmacCtToxGrdNP8ik
+buyvGvpCHPLPGm91JOrvPOgJGMxRAXrT38DdUac+2ZI3RfWPYbPSm7z63c71MPgfDHT4eaP/Hk1t
+m+ls/59T8laZdYJ/U8pVNr9Ud225PQxndu1sa4XEh1WK/RE4pjNFPXk5Q9Uuv5MDOvW15jemsDrN
+5z9etUXzdYsoc4DgkyaiQh3/IgnRJF0Sev6CvMXyB7RT8/bbOebxPJw+5/X3bq6/mmKuFs2x5rHj
+p3aEKS/w/LN+aqgSoackrV7X58QQ+aSGu7NC5H4WF838o3qt9ly5E3txiO65L921+lOtWF66ai2k
+5UJNmouCLi7PumNm9e5Dc0QtW1J98ZhadmRXj4A1RX+Yqz/uig3+rYEVGB+aTrNuyNqNTJDvoVyu
+HrqXzRIWd9R5VEPFfF5PCjVJ9x2DCGCErNqJQX+faNveNZ9EVRetur/sT+c73THsdk3Wdy5pZKwN
+7ZY3TUvUOuDN2NgDqTANbqGnWQpSsP1y/jHrfx/oY7b88LdfH16tfp3r9mTVH2P02z0segGxQeT6
+G1mpIRQKfDG/LtIWEWtV8f8PGy3Y1K330l49YAzTjnyln9YPMbri0ebhZfMXz01OyKY96lTvOWAG
+M1o/breL3U4V7G636D4FSZVEqKlr+K2j6bD9+4P9gHdev4az6lLp0VevdrrlzubhJV7UGHGRqRbV
+178BYnMUkw==
+""".decode("base64").decode("zlib")
+
+##file distribute_setup.py
+DISTRIBUTE_SETUP_PY = """
+eJztG2tz2zbyu34FTh4PqYSi7TT3GM+pM2nj9DzNJZnYaT8kHhoiIYk1X+XDsvrrb3cBkCAJyc61
+dzM3c7qrIxGLxWLfuwCP/lTs6k2eTabT6Xd5Xld1yQsWxfBvvGxqweKsqnmS8DoGoMnliu3yhm15
+VrM6Z00lWCXqpqjzPKkAFkdLVvDwjq+FU8lBv9h57JemqgEgTJpIsHoTV5NVnCB6+AFIeCpg1VKE
+dV7u2DauNyyuPcaziPEoogm4IMLWecHylVxJ4z8/n0wYfFZlnhrUBzTO4rTIyxqpDTpqCb7/yJ2N
+dliKXxsgi3FWFSKMV3HI7kVZATOQhm6qh98BKsq3WZLzaJLGZZmXHstL4hLPGE9qUWYceKqBuh17
+tGgIUFHOqpwtd6xqiiLZxdl6gpvmRVHmRRnj9LxAYRA/bm+HO7i99SeTa2QX8TekhRGjYGUD3yvc
+SljGBW1PSZeoLNYlj0x5+qgUE8W8vNLfql37tY5Tob+vspTX4aYdEmmBFLS/eUk/Wwk1dYwqI0eT
+fD2Z1OXuvJNiFaP2yeFPVxcfg6vL64uJeAgFkH5Jzy+QxXJKC8EW7F2eCQObJrtZAgtDUVVSVSKx
+YoFU/iBMI/cZL9fVTE7BD/4EZC5s1xcPImxqvkyEN2PPaaiFK4FfZWag90PgqEvY2GLBTid7iT4C
+RQfmg2hAihFbgRQkQeyF/80fSuQR+7XJa1AmfNykIquB9StYPgNd7MDgEWIqwNyBmBTJdwDmmxdO
+t6QmCxEK3OasP6bwOPA/MG4YHw8bbHOmx9XUYccIOIJTMMMhtenPHQXEOviiVqxuhtLJK78qOFid
+C98+BD+/urz22IBp7Jkps9cXb159ensd/HTx8ery/TtYb3rq/8U/ezlthz59fIuPN3VdnJ+cFLsi
+9qWo/LxcnygnWJ1U4KhCcRKddH7pZDq5urj+9OH6/fu3V8GbVz9evB4sFJ6dTScm0Icffwgu3715
+j+PT6ZfJP0XNI17z+U/SHZ2zM/908g786LlhwpN29LiaXDVpysEq2AN8Jv/IUzEvgEL6PXnVAOWl
++X0uUh4n8snbOBRZpUBfC+lACC8+AIJAgvt2NJlMSI2Vr3HBEyzh35m2AfEAMSck5ST3LodpsE4L
+cJGwZe1N/PQuwu/gqXEc3Ia/5WXmOhcdEtCB48rx1GQJmCdRsI0AEYh/LepwGykMrZcgKLDdDcxx
+zakExYkI6cL8vBBZu4sWJlD7UFvsTfbDJK8EhpfOINe5IhY33QaCFgD8idw6EFXweuP/AvCKMA8f
+JqBNBq2fT29m441ILN1Ax7B3+ZZt8/LO5JiGNqhUQsMwNMZx2Q6y161uOzPTnWR53XNgjo7YsJyj
+kDsDD9ItcAU6CqEf8G/BZbFtmcPXqCm1rpjJiW8sPMAiBEEL9LwsBRcNWs/4Mr8XetIqzgCPTRWk
+5sy0Ei+bGB6I9dqF/zytrPAlD5B1/9fp/wGdJhlSLMwYSNGC6LsWwlBshO0EIeXdcWqfjs9/xb9L
+9P2oNvRojr/gT2kgeqIayh3IqKa1qxRVk9R95YGlJLCyQc1x8QBLVzTcrVLyGFLUy/eUmrjO93mT
+RDSLOCVtZ71GW1FWEAHRKod1VTrstVltsOSV0BszHkci4Tu1KrJyqAYK3unC5Py4mhe748iH/yPv
+rIkEfI5ZRwUGdfUDIs4qBx2yPDy7mT2dPcosgOB2L0bGvWf/+2gdfPZwqdOrRxwOAVLOhuSDPxRl
+7Z56rJO/yn77dY+R5C911acDdEDp94JMQ8p7UGOoHS8GKdKAAwsjTbJyQ+5ggSrelBYmLM7+7IFw
+ghW/E4vrshGtd005mXjVQGG2peSZdJQvqzxBQ0VeTLolDE0DEPzXNbm35VUguSTQmzrF3ToAk6Ks
+raIkFvmb5lGTiAorpS/tbpyOK0PAsSfu/TBE01uvDyCVc8MrXtel2wMEQwkiI+hak3CcrThoz8Jp
+qF8BD0GUc+hqlxZiX1nTzpS59+/xFvuZ12OGr8p0d9qx5NvF9LlabWYha7iLPj6VNn+fZ6skDuv+
+0gK0RNYOIXkTdwb+ZCg4U6vGvMfpEOogI/G3JRS67ghiek2enbYVmT0Hozfjfrs4hoIFan0UNL+H
+dJ0qmS/ZdIwPWykhz5wa601l6oB5u8E2AfVXVFsAvpVNhtHFZx8SAeKx4tOtA87SvERSQ0zRNKGr
+uKxqD0wT0FinO4B4p10Om38y9uX4Fvgv2ZfM/b4pS1gl2UnE7LicAfKe/xc+VnGYOYxVWQotrt0X
+/TGRVBb7AA1kA5Mz7PvzwE/c4BSMzNTYye/2FbNfYw1PiiH7LMaq1202A6u+y+s3eZNFv9toHyXT
+RuIo1TnkroKwFLwWQ28V4ObIAtssCsPVgSj9e2MWfSyBS8Ur5YWhHn7dtfhac6W42jYSwfaSPKTS
+hdqcivFxLTt3GVTyMim8VbTfsmpDmdkS25H3PIl72LXlZU26FCVYNCdTbr0C4cL2HyW91DFp+5Cg
+BTRFsNseP24Z9jhc8BHhRq8uskiGTezRcuacODOf3Uqe3OKKvdwf/IsohU4h236XXkVEvtwjcbCd
+rvZAHdYwzyLqdRYcA/1SrNDdYFszrBuedB1X2l+NlVTtazH8RxKGXiwioTYlVMFLikIC29yq31wm
+WFZNDGu0xkoDxQvb3Hr9W4DqgK2fXnLsYxm2/g0doJK+bGqXvVwVBcmet1hk/sfvBbB0TwquQVV2
+WYaIDvalWquGtQ7yZol2do48f3Wfx6jVBVpu1JLTZTijkN4WL631kI+vph5uqe+yJVGKS+5o+Ih9
+FDw6odjKMMBAcgaksyWY3J2HHfYtKiFGQ+laQJPDvCzBXZD1DZDBbkmrtb3EeNZRC4LXKqw/2JTD
+BKEMQR94NMioJBuJaMksj023y+kISKUFiKwbG/lMJQlYy5JiAAG6RB/AA35LuINFTfiuc0oShr0k
+ZAlKxqoSBHddgfda5g/uqslC9GbKCdKwOU7tVY89e3a3nR3IimXzv6tP1HRtGK+1Z7mSzw8lzENY
+zJmhkLYly0jtfZzLVtKozW5+Cl5Vo4HhSj6uA4IeP28XeQKOFhYw7Z9X4LELlS5YJD0hsekmvOEA
+8OR8fjhvvwyV7miN6In+UW1Wy4zpPswgqwisSZ0d0lR6U2+VohNVAfoGF83AA3cBHiCru5D/M8U2
+Ht41BXmLlUysRSZ3BJFdByTyluDbAoVDewREPDO9BnBjDLvQS3ccOgIfh9N2mnmWntarPoTZLlW7
+7rShm/UBobEU8PUEyCYxNgTkDIhimc+ZmwBD2zq2YKncmuadPRNc2fwQ6fbEEAOsZ3oXY0T7JjxU
+1myzCk27uCHvDR4rVKM9SwSZ2OrIjE8hyjr++7ev/eMKj7TwdNTHP6PO7kdEJ4MbBpJc9hQliRqn
+avJibYs/Xduo2oB+2BKb5veQLINpBGaH3C0SHooNKLvQnepBGI8r7DWOwfrUf8ruIBD2mu+QeKk9
+GHP369cK646e/8F0VF8IMBrBdlKAanXa7Kt/XZzrmf2YZ9gxnGNxMHT3evGRt1yC9O9Mtqz65VHH
+ga5DSim8eWhurjtgwGSkBSAn1AKRCHkkmzc1Jr3oPbZ819mcrnOGCZvBHo9J1VfkDySq5huc6Jy5
+shwgO+jBSlfViyCjSdIfqhkes5xXqs624ujIt3fcAFPgQxflsT41VmU6AsxblojaqRgqfut8h/xs
+FU3xG3XNNVt43qD5p1r4eBMBvxrc0xgOyUPB9I7Dhn1mBTKodk1vM8Iyjuk2vQSnKhv3wFZNrOLE
+nja6c9Vd5ImMNoEz2EnfH+/zNUPvvA9O+2q+gnS6PSLG9RVTjACGIO2NlbZt3dpIx3ssVwADnoqB
+/09TICLIl7+43YGjr3vdBZSEUHfJyPZYl6Hn3CTdXzOl53JNckElLcXUY27YImzNHN1YGLsg4tTu
+nngEJqcilfvkUxNZEXYbVZHYsCJ1aFN1fhAW+NLTOXffVQFP0vYVTm9Aysj/aV6OHaDV80jwA35n
+6MO/R/nLSD6a1aVErYM8nBZZ3ScB7E+RJKvqNifazypDRj5McIZJyWAr9cbgaLcV9fixrfTIMDpl
+Q3k9vr/HTGzoaR4Bn/Xy+TbodTndkQolEIHCO1SlGH/Z8uu9Cioz4IsffpijCDGEgDjl969Q0HiU
+wh6Ms/tiwlPjquHbu9i6J9kH4tO7lm/9RwdZMXvEtB/l3H/FpgxW9MoOpS32ykMNav2Sfco2oo2i
+2Xeyj7k3nFlO5hRmatYGRSlW8YOrPX0XXNogR6FBHUpC/X1vnPcbe8Pf6kKdBvysv0CUjMSDETaf
+n53ftFkUDXr62p3ImlSUXF7IM3snCCpvrMp8az4vYa/yHoTcxDBBh00ADh/WLOsK28yoxAsMIxKP
+pTFT54WSDM0skrh2HVxn4cw+zwencwYLNPvMxRSu4RGRpApLQ0mF9cA1Ac2Utwi/lfyx95B65Faf
+CfK5hcqvpbSjEZjbVKJ06GihuxyrjgqxjWvt2NhWaWdbDENq5EhVh8p+FXI6UDTOHfX1SJvt7j0Y
+P9ShOmJb4YBFhUCCJcgb2S0opHGrJ8qFZEolRIrnDObx6LhLQj+3aC79UkHdO0I2jDdkxCFMTGHy
+tvIxa+uf6fsf5XkvJtvgFUtwRr3yxJ64D7SFYj5iWJAbVx5Xce56V4gR37BVaRwkvfpw+QcTPuuK
+wCFCUMi+Mpq3ucx3C8ySRBbmdtEcsUjUQt2aw+CNJ/FtBERNjYY5bHsMtxiS5+uhoT6b7zwYRY9c
+GrRbt0Msqyhe0KGC9IWokOQL4wcitijz+zgSkXz9IV4pePNFi8poPkTqwl3qdYcauuNoVhz9wGGj
+zC4FhQ0Y6g0JBkTyLMR2D3SsrfJGONCygfpjf43SS8PAKqUcK/O6ntqSZRO+yCIVNOjO2J5NZXN5
+m68TXo8OtO/9fTSrVPVkRRrgsHlYS1PFuPC5n6R9GZOFlMMJlCLR3Zd/os71uxFfkYPuTUIPNJ8H
+vOnPG7efTd1oj+7QrOl8Wbo/Ous1/H0mhqLtZ/+/V54Deum0MxNGwzzhTRZuuhSuezKMlB/VSG/P
+GNrYhmNrC99IkhBU8Os3WiRUERcs5eUdnuXnjNMBLO8mLJvWeNpU7/ybG0wXPjvz0LyRTdkZXrFJ
+xFy1AObigd5fgpx5nvIMYnfk3BghTmM8vWn7Adg0MxPMz/03Lm7Y83baROOg+znWl2la7hmXkiuR
+rGTjfDH1px5LBV4cqBYYU7qTGXWRmg6CFYQ8ZqRLACVwW7IWf4byipG+R6z3111oQJ+M73rl2wyr
+6jSP8K0w6f+x2U8AhSjTuKroNa3uyE4jiUEJqeEFMo8qn93iBpz2Ygi+ogVIV4IIGV2jBkIVB+Ar
+TFY7ctATy9SUJ0REiq/c0WUR4CeRTA1AjQd77EqLQWOXO7YWtcLlzvo3KFRCFubFzvwNhRhk/OpG
+oGSovE6uARTju2uDJgdAH27avECLZZQP6AGMzclq0lYfsBL5Q4goCqRXOath1f8e+KUjTViPHnWh
+peIrgVIVg2P9DtLnBVSgkavW6LsyTdeCuOXjn4OAeJ8M+zYvX/6NcpcwTkF8VDQBfad/PT01krFk
+5SvRa5xS+duc4qNAaxWsQu6bJJuGb/b02N+Z+8JjLw0OoY3hfFG6gOHMQzwvZtZyIUwLgvGxSSAB
+/e50asg2ROpKzHaAUlLv2o4eRojuxG6hFdDH435QX6TZQQKcmccUNnl1WDMIMje66AG4WgturRZV
+l8SBqdyQeQOlM8Z7RNI5oLWtoQXeZ9Do7JykHG6AuE7GCu9sDNjQ+eITAMMN7OwAoCoQTIv9N269
+ShXFyQlwP4Eq+GxcAdON4kF1bbunQMiCaLl2QQmnyrXgm2x44UnocJDymGrue4/tueTXBYLLQ6+7
+kgpc8GqnoLTzO3z9X8X44cttQFxM918weQqoIg8CJDUI1LuURHcbNc/Ob2aTfwH3muVf
+""".decode("base64").decode("zlib")
+
+##file activate.sh
+ACTIVATE_SH = """
+eJytVU1v4jAQPW9+xTT0ANVS1GsrDlRFAqmFqmG72m0rY5IJsRRslDiktNr/vuMQ8tFQpNU2B4I9
+H36eeW/SglkgYvBFiLBKYg0LhCRGD1KhA7BjlUQuwkLIHne12HCNNpz5kVrBgsfBmdWCrUrA5VIq
+DVEiQWjwRISuDreW5eE+CtodeLeAnhZEGKMGFXqAciMiJVcoNWx4JPgixDjzEj48QVeCfcqmtzfs
+cfww+zG4ZfeD2ciGF7gCHaDMPM1jtvuHXAsPfF2rSGeOxV4iDY5GUGb3xVEYv2aj6WQ0vRseAlMY
+G5DKsAawwnQUXt2LQOYlzZoYByqhonqoqfxZf4BLD97i4DukgXADCPgGgdOLTK5arYxZB1xnrc9T
+EQFcHoZEAa1gSQioo/TPV5FZrDlxJA+NzwF+Ek1UonOzFnKZp6k5mgLBqSkuuAGXS4whJb5xz/xs
+wXCHjiVerAk5eh9Kfz1wqOldtVv9dkbscfjgjKeTA8XPrtaNauX5rInOxaHuOReNtpFjo1/OxdFG
+5eY9hJ3L3jqcPJbATggXAemDLZX0MNZRYjSDH7C1wMHQh73DyYfTu8a0F9v+6D8W6XNnF1GEIXW/
+JrSKPOtnW1YFat9mrLJkzLbyIlTvYzV0RGXcaTBfVLx7jF2PJ2wyuBsydpm7VSVa4C4Zb6pFO2TR
+huypCEPwuQjNftUrNl6GsYZzuFrrLdC9iJjQ3omAPBbcI2lsU77tUD43kw1NPZhTrnZWzuQKLomx
+Rd4OXM1ByExVVkmoTwfBJ7Lt10Iq1Kgo23Bmd8Ib1KrGbsbO4Pp2yO4fpnf3s6MnZiwuiJuls1/L
+Pu4yUCvhpA+vZaJvWWDTr0yFYYyVnHMqCEq+QniuYX225xmnzRENjbXACF3wkCYNVZ1mBwxoR9Iw
+WAo3/36oSOTfgjwEEQKt15e9Xpqm52+oaXxszmnE9GLl65RH2OMmS6+u5acKxDmlPgj2eT5/gQOX
+LLK0j1y0Uwbmn438VZkVpqlfNKa/YET/53j+99G8H8tUhr9ZSXs2
+""".decode("base64").decode("zlib")
+
+##file activate.fish
+ACTIVATE_FISH = """
+eJydVm1v4jgQ/s6vmA1wBxUE7X2stJVYlVWR2lK13d6d9laRk0yIr8HmbIe0++tvnIQQB9pbXT5A
+Ys/LM55nZtyHx5RrSHiGsMm1gRAh1xhDwU0Kng8hFzMWGb5jBv2E69SDs0TJDdj3MxilxmzPZzP7
+pVPMMl+q9bjXh1eZQ8SEkAZULoAbiLnCyGSvvV6SC7IoBcS4Nw0wjcFbvJDcjiuTswzFDpiIQaHJ
+lQAjQUi1YRmUboC2uZJig8J4PaCnT5IaDcgsbm/CjinOwgx1KcUTMEhhTgV4g2B1fRk8Le8fv86v
+g7v545UHpZB9rKnp+gXsMhxLunIIpwVQxP/l9c/Hq9Xt1epm4R27bva6AJqN92G4YhbMG2i+LB+u
+grv71c3dY7B6WtzfLy9bePbp0taDTXSwJQJszUnnp0y57mvpPcrF7ZODyhswtd59+/jdgw+fwBNS
+xLSscksUPIDqwwNmCez3PpxGeyBYg6HE0YdcWBxcKczYzuVJi5Wu915vn5oWePCCoPUZBN5B7IgV
+MCi54ZDLG7TUZ0HweXkb3M5vFmSpFm/gthhBx0UrveoPpv9AJ9unIbQYdUoe21bKg2q48sPFGVwu
+H+afrxd1qvclaNlRFyh1EQ2sSccEuNAGWQwysfVpz1tPajUqbqJUnEcIJkWo6OXDaodK8ZiLdbmM
+L1wb+9H0D+pcyPSrX5u5kgWSygRYXCnJUi/KKcuU4cqsAyTKZBiissLc7NFwizvjxtieKBVCIdWz
+fzilzPaYyljZN0cGN1v7NnaIPNCGmVy3GKuJaQ6iVjE1Qfm+36hglErwmnAD8hu0dDy4uICBA8ZV
+pQr/q/+O0KFW2kjelu9Dgb9SDBsWV4F4x5CswgS0zBVlk5tDMP5bVtUGpslbm81Lu2sdKq7uNMGh
+MVQ4fy9xhogC1lS5guhISa0DlBWv0O8odT6/LP+4WZzDV6FzIkEqC0uolGZSZoMnlpxplmD2euaT
+O4hkTpPnbztDccey0bhjDaBIqaWQa0uwEtQEwtyU56i4fq54F9IE3ORR6mKriODM4XOYZwaVYLYz
+7SPbKkz4i7VkB6/Ot1upDE3znNqYKpM8raa0Bx8vfvntJ32UENsM4aI6gJL+jJwhxhh3jVIDOcpi
+m0r2hmEtS8XXXNBk71QCDXTBNhhPiHX2LtHkrVIlhoEshH/EZgdq53Eirqs5iFKMnkOmqZTtr3Xq
+djvPTWZT4S3NT5aVLgurMPUWI07BRVYqkQrmtCKohNY8qu9EdACoT6ki0a66XxVF4f9AQ3W38yO5
+mWmZmIIpnDFrbXakvKWeZhLwhvrbUH8fahhqD0YUcBDJjEBMQwiznE4y5QbHrbhHBOnUAYzb2tVN
+jJa65e+eE2Ya30E2GurxUP8ssA6e/wOnvo3V78d3vTcvMB3n7l3iX1JXWqk=
+""".decode("base64").decode("zlib")
+
+##file activate.csh
+ACTIVATE_CSH = """
+eJx9U11vmzAUffevOCVRu+UB9pws29Kl0iq1aVWllaZlcgxciiViItsQdb9+xiQp+dh4QOB7Pu49
+XHqY59IgkwVhVRmLmFAZSrGRNkdgykonhFiqSCRW1sJSmJg8wCDT5QrucRCyHn6WFRKhVGmhKwVp
+kUpNiS3emup3TY6XIn7DVNQyJUwlrgthJD6n/iCNv72uhCzCpFx9CRkThRQGKe08cWXJ9db/yh/u
+pvzl9mn+PLnjj5P5D1yM8QmXlzBkSdXwZ0H/BBc0mEo5FE5qI2jKhclHOOvy9HD/OO/6YO1mX9vx
+sY0H/tPIV0dtqel0V7iZvWyNg8XFcBA0ToEqVeqOdNUEQFvN41SumAv32VtJrakQNSmLWmgp4oJM
+yDoBHgoydtoEAs47r5wHHnUal5vbJ8oOI+9wI86vb2d8Nrm/4Xy4RZ8R85E4uTZPB5EZPnTaaAGu
+E59J8BE2J8XgrkbLeXMlVoQxznEYFYY8uFFdxsKQRx90Giwx9vSueHP1YNaUSFG4vTaErNSYuBOF
+lXiVyXa9Sy3JdClEyK1dD6Nos9mEf8iKlOpmqSNTZnYjNEWiUYn2pKNB3ttcLJ3HmYYXy6Un76f7
+r8rRsC1TpTJj7f19m5sUf/V3Ir+x/yjtLu8KjLX/CmN/AcVGUUo=
+""".decode("base64").decode("zlib")
+
+##file activate.bat
+ACTIVATE_BAT = """
+eJyFUkEKgzAQvAfyhz0YaL9QEWpRqlSjWGspFPZQTevFHOr/adQaU1GaUzI7Mzu7ZF89XhKkEJS8
+qxaKMMsvboQ+LxxE44VICSW1gEa2UFaibqoS0iyJ0xw2lIA6nX5AHCu1jpRsv5KRjknkac9VLVug
+sX9mtzxIeJDE/mg4OGp47qoLo3NHX2jsMB3AiDht5hryAUOEifoTdCXbSh7V0My2NMq/Xbh5MEjU
+ZT63gpgNT9lKOJ/CtHsvT99re3pX303kydn4HeyOeAg5cjf2EW1D6HOPkg9NGKhu
+""".decode("base64").decode("zlib")
+
+##file deactivate.bat
+DEACTIVATE_BAT = """
+eJxzSE3OyFfIT0vj4spMU0hJTcvMS01RiPf3cYkP8wwKCXX0iQ8I8vcNCFHQ4FIAguLUEgWIgK0q
+FlWqXJpcICVYpGzx2BAZ4uHv5+Hv6wq1BWINXBTdKriEKkI1DhW2QAfhttcxxANiFZCBbglQSJUL
+i2dASrm4rFz9XLgAwJNbyQ==
+""".decode("base64").decode("zlib")
+
+##file distutils-init.py
+DISTUTILS_INIT = """
+eJytV92L4zYQf9dfMU0ottuse/TeFkKh3MvC0Ydy0IdlMVpbTtR1JCMpm+T++s5Y/pBs53oPZ1hQ
+pPnSb34zo5WnVhsH2jLpV/Y2Li/cKKkOFoYN3Za6ErAdFtKC0g44vEvjzrwR6h1Oujo3YgdWw0VA
+yRWcLUo6cBpqqSpwRwHWVY18ZRB9W3jq3HDlfoIvqK7NG2gF7a297VANvZ3O1sGrQI/eDe5yB0ZY
+WQkLUpHxhVX09NDe3FGr31BL1lJUD9f8ln+FShpROm1ujOFS8ZOAPUKRt9wd836Hjqw7O9nYgvYD
+iX+1VOlMPPXQ5EVRy0YURbaDZDSQZEzWo7rS5kSLNHaQwX4RRLrQGe1nj92Fh1zltEhHDDZfEO0g
+O6MraHn5xg8IpYOfLfC2FdxYShLC64EES4A0uuROYhq49Zs368RpMvTHJmOiscKHUXRXKIpcKiuM
+Sz/sYHa7TkxcRYkkEhN8HZaxKCJXFFJJh+baW5JluRG8SjM20JHEA9qWWtXywBjbbvF2rjzC61k2
+VSGuDibTUGlhVeLgTekLHPEP73wQrrscUsUGrPCGjkTCC1JXXyw8EJWP3FSUZY8IiSCCRp97dnfO
+RUUx5a0RtbxSzLX/3XBXYxIpyQka/fh74pGrjQ5QzUt9OnFV5dMV+otOG5gQjctxozNTNtzaSSiN
+JHqu0FeJmsqRN/KrKHRLGbaQWtHUgRB9FDfu5giN4eZWIDqWCv8vrcTjrNZgRXQPzy+RmGjQpLRI
+EKz0UqQLlR28ciusM8jn7PtcLPZy2zbSDeyyos0iO+ybBgPyRvSk/CEFm8IndQebz8iXTRbbjhDP
+5xh7iJfBrKd/Nenjj6Jvgp2B+W7AnP102BXH5IZWPV3tI2MUOvXowpdS12IIXhLLP0lKyeuZrpEv
+pFhPqHg3JFTd1cceVp0EsPgGU0wFO2u4iyYRoFYfEm9kG/RZcUUBm87t9mFtx9iCtC9kx4Rt4R8a
+OdgzSt40vtyFecAZZ8BfCOhCrC8djMGPFaz2Vlt5TSZCk053+37wbLDLRXfZ+F45NtdVpVWdudSC
+xgODI8EsiLoTl5aO0lhoigX7GHZDHAY4LxoMIu1gXPYPksmFquxF4uRKZhEnKzXu82HESb+LlNQz
+Fh/RvFJVuhK+Ee5slBdj30FcRGdJ5rhKxtkyKxWcGoV/WOCYKqkNDYJ5fNQVx3g400tpJBS2FSU+
+Tco9ss8nZ08dtscGQfSby87b73fOw+4UgrEMNnY6uMzYvSDxPVPpsij6+l0/ZPfuH0Iz010giY34
+HpL0ZLyLJB4ukaQRU+GwptO7yIZCQE33B0K9iCqO6X+AR4n7wAeH68DPkJzpTsD3x+/cj9LIVHC2
+An1wmv7CzWHoqR02vb0VL73siP+3nkX0YbQ0l9f6WDyOm24cj3rxO2MMip6kpcu6VCefn/789PR3
+0v0fg21sFIp70rj9PCi8YDRDXFucym/43qN+iENh1Jy/dIIIqF3OIkDvBMsdx+huWv8Kz73vl8g5
+WQ3JOGqwu3lb4dfKKbvLigXDQsb8B/xt39Q=
+""".decode("base64").decode("zlib")
+
+##file distutils.cfg
+DISTUTILS_CFG = """
+eJxNj00KwkAMhfc9xYNuxe4Ft57AjYiUtDO1wXSmNJnK3N5pdSEEAu8nH6lxHVlRhtDHMPATA4uH
+xJ4EFmGbvfJiicSHFRzUSISMY6hq3GLCRLnIvSTnEefN0FIjw5tF0Hkk9Q5dRunBsVoyFi24aaLg
+9FDOlL0FPGluf4QjcInLlxd6f6rqkgPu/5nHLg0cXCscXoozRrP51DRT3j9QNl99AP53T2Q=
+""".decode("base64").decode("zlib")
+
+##file activate_this.py
+ACTIVATE_THIS = """
+eJyNUlGL2zAMfvevEBlHEujSsXsL9GGDvW1jD3sZpQQ3Ua7aJXawnbT595Ocpe0dO5ghseVP+vRJ
+VpIkn2cYPZknwAvWLXWYhRP5Sk4baKgOWRWNqtpdgTyH2Y5wpq5Tug406YAgKEzkwqg7NBPwR86a
+Hk0olPopaK0NHJHzYQPnE5rI0o8+yBUwiBfyQcT8mMPJGiAT0A0O+b8BY4MKJ7zPcSSzHaKrSpJE
+qeDmUgGvVbPCS41DgO+6xy/OWbfAThMn/OQ9ukDWRCSLiKzk1yrLjWapq6NnvHUoHXQ4bYPdrsVX
+4lQMc/q6ZW975nmSK+oH6wL42a9H65U6aha342Mh0UVDzrD87C1bH73s16R5zsStkBZDp0NrXQ+7
+HaRnMo8f06UBnljKoOtn/YT+LtdvSyaT/BtIv9KR60nF9f3qmuYKO4//T9ItJMsjPfgUHqKwCZ3n
+xu/Lx8M/UvCLTxW7VULHxB1PRRbrYfvWNY5S8it008jOjcleaMqVBDnUXcWULV2YK9JEQ92OfC96
+1Tv4ZicZZZ7GpuEpZbbeQ7DxquVx5hdqoyFSSmXwfC90f1Dc7hjFs/tK99I0fpkI8zSLy4tSy+sI
+3vMWehjQNJmE5VePlZbL61nzX3S93ZcfDqznnkb9AZ3GWJU=
+""".decode("base64").decode("zlib")
+
+if __name__ == '__main__':
+    main()
+
+## TODO:
+## Copy python.exe.manifest
+## Monkeypatch distutils.sysconfig


Property changes on: usertools/swift-vm-boot/bin/virtualenv.py
___________________________________________________________________
Added: svn:executable
   + *

Deleted: usertools/swift-vm-boot/env.sh
===================================================================
--- usertools/swift-vm-boot/env.sh	2011-11-29 17:26:03 UTC (rev 5323)
+++ usertools/swift-vm-boot/env.sh	2011-11-29 21:20:57 UTC (rev 5324)
@@ -1,5 +0,0 @@
-export FUTUREGRID_IAAS_ACCESS_KEY=<replace this>
-export FUTUREGRID_IAAS_SECRET_KEY=<and this>
-
-export FUTUREGRID_HOTEL_NODES=2
-export FUTUREGRID_SIERRA_NODES=2

Added: usertools/swift-vm-boot/env.sh
===================================================================
--- usertools/swift-vm-boot/env.sh	                        (rev 0)
+++ usertools/swift-vm-boot/env.sh	2011-11-29 21:20:57 UTC (rev 5324)
@@ -0,0 +1,9 @@
+export FUTUREGRID_IAAS_ACCESS_KEY=XXXX
+export FUTUREGRID_IAAS_SECRET_KEY=XXXXXXXXX
+
+export EC2_ACCESS_KEY=$FUTUREGRID_IAAS_ACCESS_KEY
+export EC2_SECRET_KEY=$FUTUREGRID_IAAS_SECRET_KEY
+
+export EC2_URL=https://s83r.idp.sdsc.futuregrid.org:8444
+
+export FUTUREGRID_SIERRA_NODES=2

Deleted: usertools/swift-vm-boot/hosts.txt
===================================================================
--- usertools/swift-vm-boot/hosts.txt	2011-11-29 17:26:03 UTC (rev 5323)
+++ usertools/swift-vm-boot/hosts.txt	2011-11-29 21:20:57 UTC (rev 5324)
@@ -1,2 +0,0 @@
-svc.uc.futuregrid.org
-s83r.idp.sdsc.futuregrid.org

Added: usertools/swift-vm-boot/hosts.txt
===================================================================
--- usertools/swift-vm-boot/hosts.txt	                        (rev 0)
+++ usertools/swift-vm-boot/hosts.txt	2011-11-29 21:20:57 UTC (rev 5324)
@@ -0,0 +1 @@
+s83r.idp.sdsc.futuregrid.org

Deleted: usertools/swift-vm-boot/install.sh
===================================================================
--- usertools/swift-vm-boot/install.sh	2011-11-29 17:26:03 UTC (rev 5323)
+++ usertools/swift-vm-boot/install.sh	2011-11-29 21:20:57 UTC (rev 5324)
@@ -1,28 +0,0 @@
-#!/bin/bash
-
-cd `dirname $0`
-
-source env.sh
-python bin/virtualenv.py ve
-if [ $? -ne 0 ]; then
-    echo "Failed to created the needed python virtual environment"
-    exit 1
-fi
-
-source ve/bin/activate
-easy_install cloudinitd
-if [ $? -ne 0 ]; then
-    echo "Failed to install cloudinitd"
-    exit 1
-fi
-
-echo "Registering the key names in all the clouds"
-python bin/register_key.py hosts.txt
-if [ $? -ne 0 ]; then
-    echo "Failed to register the key names"
-    exit 1
-fi
-
-echo "Success!"
-echo ""
-echo 0

Added: usertools/swift-vm-boot/install.sh
===================================================================
--- usertools/swift-vm-boot/install.sh	                        (rev 0)
+++ usertools/swift-vm-boot/install.sh	2011-11-29 21:20:57 UTC (rev 5324)
@@ -0,0 +1,28 @@
+#!/bin/bash
+
+cd `dirname $0`
+
+source env.sh
+python bin/virtualenv.py ve
+if [ $? -ne 0 ]; then
+    echo "Failed to created the needed python virtual environment"
+    exit 1
+fi
+
+source ve/bin/activate
+easy_install cloudinitd
+if [ $? -ne 0 ]; then
+    echo "Failed to install cloudinitd"
+    exit 1
+fi
+
+echo "Registering the key names in all the clouds"
+python bin/register_key.py hosts.txt
+if [ $? -ne 0 ]; then
+    echo "Failed to register the key names"
+    exit 1
+fi
+
+echo "Success!"
+echo ""
+echo 0


Property changes on: usertools/swift-vm-boot/install.sh
___________________________________________________________________
Added: svn:executable
   + *

Added: usertools/swift-vm-boot/plan/test-level1.conf
===================================================================
--- usertools/swift-vm-boot/plan/test-level1.conf	                        (rev 0)
+++ usertools/swift-vm-boot/plan/test-level1.conf	2011-11-29 21:20:57 UTC (rev 5324)
@@ -0,0 +1,12 @@
+[svc-sierra]
+
+replica_count: env.FUTUREGRID_SIERRA_NODES
+iaas_key: env.FUTUREGRID_IAAS_ACCESS_KEY
+iaas_secret: env.FUTUREGRID_IAAS_SECRET_KEY
+sshkeyname: swiftkey
+localsshkeypath: ~/.ssh/id_rsa.pub
+iaas_url: https://s83r.idp.sdsc.futuregrid.org:8444
+image: debian-lenny.gz
+iaas: nimbus
+ssh_username: root
+

Added: usertools/swift-vm-boot/plan/top.conf
===================================================================
--- usertools/swift-vm-boot/plan/top.conf	                        (rev 0)
+++ usertools/swift-vm-boot/plan/top.conf	2011-11-29 21:20:57 UTC (rev 5324)
@@ -0,0 +1,10 @@
+# This is a sample top level configuration file.  Each entry under runlevels
+# is a file with a single runlevel description.  All of the services in that
+# file are run at the same time but the next level is not begun until 
+# all of these services in the previous successfully complete.
+
+[defaults]
+
+[runlevels]
+level1: test-level1.conf
+