From noreply at svn.ci.uchicago.edu Tue Feb 3 03:53:11 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 3 Feb 2009 03:53:11 -0600 (CST) Subject: [Swift-commit] r2491 - trunk/docs Message-ID: <20090203095311.C9E4C228187@www.ci.uchicago.edu> Author: benc Date: 2009-02-03 03:53:10 -0600 (Tue, 03 Feb 2009) New Revision: 2491 Modified: trunk/docs/userguide.xml Log: replace "Invoking an application from swift" section with more detailed notes previous sent to swift-devel mailing list Modified: trunk/docs/userguide.xml =================================================================== --- trunk/docs/userguide.xml 2009-01-30 19:22:33 UTC (rev 2490) +++ trunk/docs/userguide.xml 2009-02-03 09:53:10 UTC (rev 2491) @@ -1465,47 +1465,306 @@ -
Invoking an application from Swift +
Executing <literal>app</literal> procedures -There are certain requirements on the behaviour of application programs -used in SwiftScript programs. These requirements are primarily to ensure -that the Swift can run your application in different places. +This section describes how Swift executes app procedures, +and requirements on the behaviour of application programs used in +app procedures. +These requirements are primarily to ensure +that the Swift can run your application in different places and with the +various fault tolerance mechanisms in place. + +
Mapping of <literal>app</literal> semantics into unix +process execution semantics + +This section describes how an app procedure +invocation is translated into a (remote) unix process execution. It does not +describe the mechanisms by which Swift performs that translation; that +is described in the next section. + +In this section, this example SwiftScript program is used +for reference: + + + type file; + + app (file o) count(file i) { + wc @i stdout=@o; + } + + file q <"input.txt">; + file r <"output.txt">; + + -Swift must know about all of your data files - when Swift has decided where -to run your application, it will transfer the necessary input files there -before execution and transfer the output files back to the submitting -system afterwards. If Swift does not know about your files, then it cannot -do this. The way to tell Swift about files is by mapping them to variables -and using those variables as parameters to your application. +The executable for wc will be looked up in tc.data. + -Applications should take the name of input and output files on the -command line - Sometimes Swift will decide on the name of your input -and output files automatically (for example, if you do not specify a mapping -explicitly for an input or output variable). Swift must be able to -tell your application which filename it has chosen, and the commandline -is the way it does that. Use the -@filename function to determine -the filename of a variable. +This unix executable will then be executed in some application +procedure workspace. This means: + -Applications should not assume that they are running in a particular -location or on a particular host - Swift will decide which site to run -a job on automatically (based on the sites that it knows have the -application installed, by looking at the transformation catalog). On that -site, it will create a unique working directory every time that it runs -your jobs. Your job should expect to be run in an arbitrary working directory -on any of the available hosts. +Each application procedure workspace will have an application workspace +directory. (TODO: can collapse terms //application procedure workspace// +and //application workspace directory// ? + -Running your application on the same input files multiple times should -always give equivalent output files. Swift expects to be able to run a job -multiple times, perhaps on the same site, perhaps on different sites, in -order to deal with error conditions. For example, applications should not -make modifications to external databases that causes their output to -differ if they are run more than once. +This application workspace directory will not be shared with any other +application procedure execution attempt; all +application procedure +execution attempts will run with distinct application procedure +workspaces. (for the avoidance of doubt: + If a SwiftScript procedure invocation is subject +to multiple application procedure execution attempts (due to Swift-level +restarts, retries or replication) then each of those application procedure +execution attempts will be made in a different application procedure workspace. +) + + +The application workspace directory will be a directory on a POSIX +filesystem accessible throughout the application execution by the +application executable. + + +Before the application executable is executed: + + + + + +The application workspace directory will exist. + + + +The input files will exist inside the application workspace +directory (but not necessarily as direct children; there may be +subdirectories within the application workspace directory). + + + +The input files will be those files mapped +to input parameters of the application procedure +invocation. (In the example, this means that the file +input.txt will exist in the application workspace +directory) + + + +For each input file dataset, it will be the case that +@filename or +@filenames invoked with that dataset as a parameter +will return the path +relative to the application workspace directory for the file(s) that are +associated with that dataset. (In the example, that means that @i will +evaluate to the path input.txt) + + + +For each file-bound parameter of the Swift procedure invocation, the +associated files (determined by data type?) will always exist. + + + +The input files must be treated as read only files. This may or may not +be enforced by unix file system permissions. They may or may not be copies +of the source file (conversely, they may be links to the actual source file). + + + + + +During/after the application executable execution, +the following must be true: + + + + +If the application executable execution was successful (in the opinion +of the application executable), then the application executable should +exit with unix return code 0; +if the application executable execution +was unsuccessful (in the opinion of the application executable), then the +application executable should exit with unix return code not equal to +0. + + + +Each file mapped from an output parameter of the SwiftScript procedure +call must exist. Files will be mapped in the same way as for input files. + + +(? Is it defined that output subdirectories will be precreated before +execution or should app executables expect to make them? That's probably +determined by the present behaviour of wrapper.sh) + + + +Output produced by running the application executable on some inputs should +be the same no matter how many times, when or where that application +executable is run. 'The same' can vary depending on application (for example, +in an application it might be acceptable for a PNG->JPEG conversion to +produce different, similar looking, output jpegs depending on the +environment) + + + + + +Things to not assume: + + + + + +anything about the path of the application workspace directory + + + +that either the application workspace directory will be deleted or will +continue to exist or will remain unmodified after execution has finished + + + +that files can be passed(?def) between application procedure invocations +through any mechanism except through files known to Swift through the +mapping mechanism (there is some exception here for extern +datasets - there are a separate set of assertions that hold for +extern datasets) + + + +that application executables will run on any particular site of those +available, or than any combination of applications will run on the same or +different sites. + + + + +
+ +
+notes on how swift implements file input and output + + + +This section describes the implementation of the semantics described +in the previous section. + + + +Swift executes application procedures on one or more sites. + + + +Each site consists of: + + + + +worker nodes. There is some execution mechanism +through which the Swift client side executable can execute its +wrapper script on those +worker nodes. This is commonly GRAM or Falkon or coasters. + + + +a site-shared file system. This site shared filesystem is accessible +through some file transfer mechanism from the +Swift client side +executable. This is commonly GridFTP or coasters. This site shared +filesystem is also accessible through the posix file system on all worker +nodes, mounted at the same location as seen through the file transfer +mechanism. Swift is configured with the location of some site working +directory on that site-shared file system. + + + + +There is no assumption that the site shared file system for one site is +accessible from another site. + + + +For each workflow run, on each site that is used by that run, a run +directory is created in the site working directory, by the Swift client +side. + + + +In that run directory are placed several subdirectories: + + + + +shared/ - site shared files cache + + + +kickstart/ - when kickstart is used, kickstart record files +for each job that has generated a kickstart record. + + + + +info/ - wrapper script log files + + + +status/ - job status files + + + +jobs/ //application workspace directories// (optionally placed here - +see below) + + + + +Application execution looks like this: + + + +For each application procedure call: + + + +The Swift client side selects a site; copies the input files for that +procedure call to the site shared file cache if they are not already in +the cache, using the file transfer mechanism; and then invokes the wrapper +script on that site using the execution mechanism. + + + +The wrapper script creates the application workspace directory; places the +input files for that job into the application workspace directory using +either cp or ln -s (depending on a configuration option); executes the +application unix executable; copies output files from the application +workspace directory to the site shared directory using cp; creates a +status file under the status/ directory; and exits, returning control to +the Swift client side. Logs created during the execution of the wrapper +script are stored under the info/ directory. + + + +The Swift client side then checks for the presence of and deletes a status +file indicating success; copies files from the site shared directory to +the appropriate client side location. + + + +The job directory is created (in the default mode) under the jobs/ +directory. However, it can be created under an arbitrary other path, which +allows it to be created on a different file system (such as a worker node +local file system in the case that the worker node has a local file +system). + + +
From noreply at svn.ci.uchicago.edu Tue Feb 3 06:25:39 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 3 Feb 2009 06:25:39 -0600 (CST) Subject: [Swift-commit] r2492 - trunk/docs Message-ID: <20090203122539.6E5432281DA@www.ci.uchicago.edu> Author: benc Date: 2009-02-03 06:25:38 -0600 (Tue, 03 Feb 2009) New Revision: 2492 Modified: trunk/docs/userguide.xml Log: reformat mappers section to be more concisely formatted Modified: trunk/docs/userguide.xml =================================================================== --- trunk/docs/userguide.xml 2009-02-03 09:53:10 UTC (rev 2491) +++ trunk/docs/userguide.xml 2009-02-03 12:25:38 UTC (rev 2492) @@ -815,12 +815,11 @@
The single file mapper - - - Name: single_file_mapper - Description: -A single file mapper maps a single physical file to a dataset. + +The single_file_mapper maps a single physical file to a dataset. + + Swift variable -------------------> Filename @@ -832,59 +831,65 @@ f.bar INVALID - - Parameter: - - file: - The location of the physical file including path and file name. - - - + - Example: + + + + + parameter + meaning + + + + fileThe location of the physical file including path and file name. + + +
+ +Example: - file f <single_file_mapper;file="plot_outfile_param">; - + file f <single_file_mapper;file="plot_outfile_param">; There is a simplified syntax for this mapper: - file f <"plot_outfile_param">; - - - - + file f <"plot_outfile_param">; +
The simple mapper - - Name: simple_mapper - Description: -A mapper that maps a file or a list of files into an array -by prefix, suffix, and pattern. -If more than one file is matched, each of the file names will -be mapped as a subelement of the dataset. - - Parameters: - - location: - A directory that the files are located. - - prefix: - The prefix of the files - - suffix: - The suffix of the files, for instance: ".txt" - - pattern: -A UNIX glob style pattern, for instance: "*foo*" would match -all file names that contain foo. When this mapper is used to specify -output filenames, pattern is ignored. - - - - Examples: +The simple_mapper maps a file or a list of files +into an array by prefix, suffix, and pattern. If more than one file is +matched, each of the file names will be mapped as a subelement of the dataset. + + + + + + + parameter + meaning + + + + locationA directory that the files are located. + prefixThe prefix of the files + suffixThe suffix of the files, for instance: ".txt" + patternA UNIX glob style pattern, for instance: +"*foo*" would match all file names that +contain foo. When this mapper is used to specify output +filenames, pattern is ignored. + + +
+ + + +Examples: + + type file; file f <simple_mapper;prefix="foo", suffix=".txt">; @@ -899,9 +904,9 @@ f foo.txt - + - + type messagefile; @@ -917,10 +922,10 @@ This will output the string 'hi' to the file foo.txt. - + - -The simple_mapper can be used to map arrays. It will map the array index + +The simple_mapper can be used to map arrays. It will map the array index into the filename between the prefix and suffix. @@ -949,10 +954,10 @@ - + - -simple_mapper can be used to map structures. It will map the name of the + +simple_mapper can be used to map structures. It will map the name of the structure member into the filename, between the prefix and the suffix. @@ -988,90 +993,93 @@ out.right quxright.txt - + -
concurrent mapper - - Name: concurrent_mapper - Description: -Concurrent mapper is almost the same as the simple mapper, + +concurrent_mapper is almost the same as the simple mapper, except that it is used to map an output file, and the filename generated will contain an extract sequence that is unique. This mapper is the default mapper for variables when no mapper is specified. - - Parameters: - - location: - A directory that the files are located. - - prefix: - The prefix of the files - - suffix: - The suffix of the files, for instance: ".txt" - - pattern: -A UNIX glob style pattern, for instance: "*foo*" would -match all file names that contain foo. When this mapper is used -to specify output filenames, pattern is ignored. - - - - Example: + + + + + + + parameter + meaning + + + + locationA directory that the files are located. + prefixThe prefix of the files + suffixThe suffix of the files, for instance: ".txt" + patternA UNIX glob style pattern, for instance: +"*foo*" would match all file names that +contain foo. When this mapper is used to specify output +filenames, pattern is ignored. + + +
+ + + Example: file f1; file f2 <concurrent_mapper;prefix="foo", suffix=".txt">; -The above example would use concurrent mapper for f1 and f2, and -generate f2 filename with prefix "foo" and extension ".txt" - - +The above example would use concurrent mapper for f1 and +f2, and +generate f2 filename with prefix "foo" and extension ".txt" +
file system mapper + +filesys_mapper is similar to the simple mapper, +but maps a file or +a list of files to an array. Each of the filename is +mapped as an element in the array. The order of files in the resulting +array is not defined. + + TODO: note on difference between location as a relative vs absolute path wrt staging to remote location - as mihael said: It's because you specify that location in the mapper. Try location="." instead of location="/sandbox/..." - - Name: filesys_mapper - Description: -This mapper is similar to the simple mapper, but maps a file or -a list of files to an array. Each of the filename is -mapped as an element in the array. The order of files in the resulting -array is not defined. - - Parameters: - - location: - The directory where the files are located. - - prefix: - The prefix of the files - - suffix: - The suffix of the files, for instance: ".txt" - - pattern: -A UNIX glob style pattern, for instance: "*foo*" would -match all file names that contain foo. - - - - - - Example: + + + + + + parameter + meaning + + + + locationThe directory where the files are located. + prefixThe prefix of the files + suffixThe suffix of the files, for instance: ".txt" + patternA UNIX glob style pattern, for instance: +"*foo*" would match all file names that +contain foo. + + + +
+ + Example: file texts[] <filesys_mapper;prefix="foo", suffix=".txt">; -The above example would map all filenames that start with "foo" -and have an extension ".txt" into the array texts. -For example, if the specified directory contains files: foo1.txt, footest.txt, -foo__1.txt, then the mapping might be: +The above example would map all filenames that start with "foo" +and have an extension ".txt" into the array texts. +For example, if the specified directory contains files: foo1.txt, footest.txt, +foo__1.txt, then the mapping might be: Swift variable -------------------> Filename @@ -1081,26 +1089,28 @@ texts[2] foo__1.txt - - +
fixed array mapper - - Name: fixed_array_mapper - Description: -This mapper maps from a string that contains a list of filenames -into a file array. - - Parameter: - - files: -A string that contains a list of filenames, separated by space, comma or colon - - - +The fixed_array_mapper maps from a string that +contains a list of filenames into a file array. - Example: + + + + + parameter + meaning + + + + filesA string that contains a list of filenames, separated by space, comma or colon + + +
+ + Example: file texts[] <fixed_array_mapper;files="file1.txt, fileB.txt, file3.txt">; @@ -1114,22 +1124,28 @@ texts[2] file3.txt - - +
array mapper - - Name: array_mapper - Description: This mapper froms an array of strings into a file -array. - Parameter: - - files: -An array of strings containing one filename per element - - - Example: + The array_mapper maps from an array of strings +into a file + + + + + + parameter + meaning + + + + filesAn array of strings containing one filename per element + + +
+ + Example: string s[] = [ "a.txt", "b.txt", "c.txt" ]; @@ -1146,101 +1162,103 @@ - - +
regular expression mapper - - Name: regexp_mapper - Description: -This mapper transforms one file name to another using -regular expression matching. - - Parameters: - - source: - The source file name - - match: -Regular expression pattern to match, use ( ) to match whatever regular -expression is inside the parentheses, and indicate the start and end -of a group; the contents of a group can be retrieved with the -\number special sequence - - transform: -The pattern of the file name to transform to, use -\number to reference the group matched. - - - +The regexp_mapper transforms one file name to +another using regular expression matching. - Example: - - string s = "picture.gif"; - file f <regexp_mapper;source=s,match="(.*)gif",transform="\1jpg">; - -This example transforms a string "gif" into one -ending with "jpg" and maps that to a file. + + + + + + parameter + meaning + + + + sourceThe source file name + matchRegular expression pattern to match, use +() to match whatever regular expression is inside the +parentheses, and indicate the start and end of a group; the contents of a +group can be retrieved with the \number special +sequence + transformThe pattern of the file name to +transform to, use \number to reference the +group matched. + + +
+ +Example: + + string s = "picture.gif"; + file f <regexp_mapper; + source=s, + match="(.*)gif", + transform="\1jpg">; + +This example transforms a string ending gif into one +ending jpg and maps that to a file. + - Swift variable -------------------> Filename - f picture.jpg - + f picture.jpg - - -
-
csv mapper - - Name: csv_mapper - Description: -This mapper maps the content of a CSV (comma-separated value) file into -an array of structures. The dataset type needs to be correctly defined to -conform to the column names in the file. For instance, if the file -contains columns: - name age GPA - then the type needs to have the same member elements, say: - - type student { - File name; - File age; - File GPA; - } - -If the file does not contain a header with column info, then the -column names are assumed as "column1", "column2", etc. - - Parameters: - - file: -The name of the CSV file to read mappings from. - - header: -Whether the file has a line describing header info; default is true - - skip: -The number of lines to skip at the beginning (after header line); default is 0. - - hdelim: -Header field delimiter; default is the value of the "delim" parameter - - delim: -Content field delimiters; defaults are space, tab and comma. - - - + - Example: +
+ +
csv mapper + + +The csv_mapper maps the content of a CSV (comma-separated +value) file into an array of structures. The dataset type needs to be +correctly defined to conform to the column names in the +file. For instance, if the file contains columns: +name age GPA then the type needs to have member elements +like this: + + type student { + file name; + file age; + file GPA; + } + + +If the file does not contain a header with column info, then the column +names are assumed as column1, column2, +etc. + + + + + + + parameter + meaning + + + + fileThe name of the CSV file to read mappings from. + headerWhether the file has a line describing header info; default is true + skipThe number of lines to skip at the beginning (after header line); default is 0. + hdelimHeader field delimiter; default is the value of the delim parameter + delimContent field delimiters; defaults are space, tab and comma + + +
+ + Example: student stus[] <csv_mapper;file="stu_list.txt">; The above example would read a list of student info from file -"stu_list.txt" and map them into a student array. By default, -the file should contain a header line specifying the names of the columns. - +"stu_list.txt" and map them into a student array. By default, the file should contain a header line specifying the names of the columns. If stu_list.txt contains the following: name,age,gpa @@ -1264,36 +1282,41 @@ stus[2].gpa s - - - +
external mapper - - Name: ext - Description: -This mapper maps based on the output of a supplied Unix executable. - - Parameters: - - exec: -The name of the executable (relative to the current directory, if an -absolute path is not specified) - - Other parameters are passed to the executable prefixed -by a - symbol. - - - - + +The external mapper, ext maps based on the output of a +supplied Unix executable. + + + + + + + parameter + meaning + + + + execThe name of the executable +(relative to the current directory, if an absolute path is not +specified) + *Other parameters are passed to the +executable prefixed with a - symbol + + +
+ + The output of the executable should consist of two columns of data, separated by a space. The first column should be the path of the mapped variable, in SwiftScript syntax (for example [2] means the 2nd element of an array) or the symbol $ to represent the root of the mapped variable. - + - Example: + Example: With the following in mapper.sh, #!/bin/bash @@ -1320,8 +1343,7 @@ - - +
From noreply at svn.ci.uchicago.edu Tue Feb 3 14:35:41 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 3 Feb 2009 14:35:41 -0600 (CST) Subject: [Swift-commit] r2493 - www/inc Message-ID: <20090203203541.7289D2281E6@www.ci.uchicago.edu> Author: benc Date: 2009-02-03 14:35:38 -0600 (Tue, 03 Feb 2009) New Revision: 2493 Modified: www/inc/home_sidebar.php Log: update front page to refer to 0.8 release (it was still talking about 0.6) Modified: www/inc/home_sidebar.php =================================================================== --- www/inc/home_sidebar.php 2009-02-03 12:25:38 UTC (rev 2492) +++ www/inc/home_sidebar.php 2009-02-03 20:35:38 UTC (rev 2493) @@ -1,7 +1,7 @@

What's New?

-

SWIFT 0.6 RELEASE - 25 AUG 2008

+

SWIFT 0.8 RELEASE - 03 FEB 2009

-The latest release of Swift, v0.6, is available from the +The latest release of Swift, v0.8, is available from the downloads page.


From noreply at svn.ci.uchicago.edu Tue Feb 3 15:54:46 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 3 Feb 2009 15:54:46 -0600 (CST) Subject: [Swift-commit] r2494 - trunk/src/org/griphyn/vdl/mapping Message-ID: <20090203215446.B74302281E9@www.ci.uchicago.edu> Author: benc Date: 2009-02-03 15:54:45 -0600 (Tue, 03 Feb 2009) New Revision: 2494 Modified: trunk/src/org/griphyn/vdl/mapping/AbstractDataNode.java Log: incorrect form of EMPTY_PATH returned sometimes in fringe paths. (roll on java 1.5 generics) Modified: trunk/src/org/griphyn/vdl/mapping/AbstractDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/AbstractDataNode.java 2009-02-03 20:35:38 UTC (rev 2493) +++ trunk/src/org/griphyn/vdl/mapping/AbstractDataNode.java 2009-02-03 21:54:45 UTC (rev 2494) @@ -331,7 +331,7 @@ public void getFringePaths(List list, Path parentPath) throws HandleOpenException { checkMappingException(); if (getField().getType().getBaseType() != null) { - list.add(Path.EMPTY_PATH.toString()); + list.add(Path.EMPTY_PATH); } else { Iterator i = getField().getType().getFields().iterator(); From noreply at svn.ci.uchicago.edu Thu Feb 5 07:17:38 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Thu, 5 Feb 2009 07:17:38 -0600 (CST) Subject: [Swift-commit] r2495 - trunk/docs Message-ID: <20090205131738.575D62281E2@www.ci.uchicago.edu> Author: benc Date: 2009-02-05 07:17:37 -0600 (Thu, 05 Feb 2009) New Revision: 2495 Modified: trunk/docs/userguide.xml Log: typo Modified: trunk/docs/userguide.xml =================================================================== --- trunk/docs/userguide.xml 2009-02-03 21:54:45 UTC (rev 2494) +++ trunk/docs/userguide.xml 2009-02-05 13:17:37 UTC (rev 2495) @@ -1929,7 +1929,7 @@ string v = @regexp("abcdefghi", "c(def)g","monkey"); -will assing the value "abmonkeyhi" to the variable v. +will assign the value "abmonkeyhi" to the variable v.
@strcat From noreply at svn.ci.uchicago.edu Thu Feb 5 08:48:16 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Thu, 5 Feb 2009 08:48:16 -0600 (CST) Subject: [Swift-commit] r2496 - trunk/tests/language Message-ID: <20090205144816.8B333228187@www.ci.uchicago.edu> Author: benc Date: 2009-02-05 08:48:15 -0600 (Thu, 05 Feb 2009) New Revision: 2496 Modified: trunk/tests/language/run Log: add some commented out code to the language tests that is useful when changing the XML form of the language to bulk update all tests Modified: trunk/tests/language/run =================================================================== --- trunk/tests/language/run 2009-02-05 13:17:37 UTC (rev 2495) +++ trunk/tests/language/run 2009-02-05 14:48:15 UTC (rev 2496) @@ -37,7 +37,12 @@ if [ "$?" -ne "0" ]; then echo "OUTPUT DIFFERS" diff -u ../working-base/$b $b - exit 1 +# echo press enter to accept this as valid or press ctrl-c to die +# read +# pushd .. +# quilt add working-base/$b +# cp working/$b working-base/$b +# popd fi echo PASS done From noreply at svn.ci.uchicago.edu Thu Feb 5 08:52:04 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Thu, 5 Feb 2009 08:52:04 -0600 (CST) Subject: [Swift-commit] r2497 - trunk/tests/language-behaviour Message-ID: <20090205145204.A01562281AD@www.ci.uchicago.edu> Author: benc Date: 2009-02-05 08:52:03 -0600 (Thu, 05 Feb 2009) New Revision: 2497 Added: trunk/tests/language-behaviour/07514-fixed-array-mapper-filenames.a.in trunk/tests/language-behaviour/07514-fixed-array-mapper-filenames.b.in trunk/tests/language-behaviour/07514-fixed-array-mapper-filenames.real.out.expected trunk/tests/language-behaviour/07514-fixed-array-mapper-filenames.swift Log: test for @filenames Added: trunk/tests/language-behaviour/07514-fixed-array-mapper-filenames.a.in =================================================================== --- trunk/tests/language-behaviour/07514-fixed-array-mapper-filenames.a.in (rev 0) +++ trunk/tests/language-behaviour/07514-fixed-array-mapper-filenames.a.in 2009-02-05 14:52:03 UTC (rev 2497) @@ -0,0 +1 @@ +foo Added: trunk/tests/language-behaviour/07514-fixed-array-mapper-filenames.b.in =================================================================== --- trunk/tests/language-behaviour/07514-fixed-array-mapper-filenames.b.in (rev 0) +++ trunk/tests/language-behaviour/07514-fixed-array-mapper-filenames.b.in 2009-02-05 14:52:03 UTC (rev 2497) @@ -0,0 +1 @@ +foo Added: trunk/tests/language-behaviour/07514-fixed-array-mapper-filenames.real.out.expected =================================================================== --- trunk/tests/language-behaviour/07514-fixed-array-mapper-filenames.real.out.expected (rev 0) +++ trunk/tests/language-behaviour/07514-fixed-array-mapper-filenames.real.out.expected 2009-02-05 14:52:03 UTC (rev 2497) @@ -0,0 +1 @@ +07514-fixed-array-mapper-filenames.a.in 07514-fixed-array-mapper-filenames.b.in Added: trunk/tests/language-behaviour/07514-fixed-array-mapper-filenames.swift =================================================================== --- trunk/tests/language-behaviour/07514-fixed-array-mapper-filenames.swift (rev 0) +++ trunk/tests/language-behaviour/07514-fixed-array-mapper-filenames.swift 2009-02-05 14:52:03 UTC (rev 2497) @@ -0,0 +1,17 @@ +type messagefile; + +(messagefile t) write(messagefile s[]) { + app { + echo @filenames(s) stdout=@filename(t); + } +} + +string fns="07514-fixed-array-mapper-filenames.a.in 07514-fixed-array-mapper-filenames.b.in"; + +messagefile outfile[] ; +messagefile realoutput <"07514-fixed-array-mapper-filenames.real.out">; + +string fn; + +realoutput = write(outfile); + From noreply at svn.ci.uchicago.edu Fri Feb 6 05:31:09 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 6 Feb 2009 05:31:09 -0600 (CST) Subject: [Swift-commit] r2499 - trunk/docs Message-ID: <20090206113109.B56FF22819E@www.ci.uchicago.edu> Author: benc Date: 2009-02-06 05:31:08 -0600 (Fri, 06 Feb 2009) New Revision: 2499 Modified: trunk/docs/userguide.xml Log: formatting of function and procedure reference Modified: trunk/docs/userguide.xml =================================================================== --- trunk/docs/userguide.xml 2009-02-05 15:16:59 UTC (rev 2498) +++ trunk/docs/userguide.xml 2009-02-06 11:31:08 UTC (rev 2499) @@ -1878,10 +1878,10 @@ default value and returns the value of that string parameter from the command line. If no default value is specified and the command line parameter is missing, an error is generated. If a default value is specified and the -command line parameter is missing, @arg will return the default value. +command line parameter is missing, @arg will return the default value. -Command line parameters recognized by @arg begin with exactly one hyphen +Command line parameters recognized by @arg begin with exactly one hyphen and need to be positioned after the script name. @@ -1903,48 +1903,48 @@
@extractint - at extractint(file) will read the specified file, parse an integer from the +@extractint(file) will read the specified file, parse an integer from the file contents and return that integer.
@filename - at filename(v) will return a string containing the filename(s) for the file(s) -mapped to the variable v. When more than one filename is returned, the +@filename(v) will return a string containing the filename(s) for the file(s) +mapped to the variable v. When more than one filename is returned, the filenames will be space separated inside a single string return value.
@filenames - at filenames(v) will return multiple values (!) containing the filename(s) for -the file(s) mapped to the variable v. (compare to +@filenames(v) will return multiple values (!) containing the filename(s) for +the file(s) mapped to the variable v. (compare to @filename)
@regexp - at regexp(input,pattern,replacement) will apply regular expression -substitution using the Java java.util.regexp API. For example: +@regexp(input,pattern,replacement) will apply regular expression +substitution using the Java java.util.regexp API. For example: string v = @regexp("abcdefghi", "c(def)g","monkey"); -will assign the value "abmonkeyhi" to the variable v. +will assign the value "abmonkeyhi" to the variable v.
@strcat - at strcat(a,b,c,d,...) will return a string containing all of the strings +@strcat(a,b,c,d,...) will return a string containing all of the strings passed as parameters joined into a single string. There may be any number of parameters. -The + operator concatenates two strings: @strcat(a,b) is the same as a + b +The + operator concatenates two strings: @strcat(a,b) is the same as a + b
@strcut - at strcut(input,pattern) will match the regular expression in the pattern +@strcut(input,pattern) will match the regular expression in the pattern parameter against the supplied input string and return the section that matches the first matching parenthesised group. @@ -1960,13 +1960,13 @@ -will output the message 'Your name is John'. +will output the message: Your name is John.
@strsplit - at strsplit(input,pattern) will split the input string based on separators +@strsplit(input,pattern) will split the input string based on separators that match the given pattern and return a string array. (since Swift 0.9) @@ -1991,8 +1991,8 @@
@toint - at toint(input) will parse its input string into an integer. This can be -used with @arg to pass input parameters to a SwiftScript program as +@toint(input) will parse its input string into an integer. This can be +used with @arg to pass input parameters to a SwiftScript program as integers.
@@ -2006,7 +2006,7 @@
readData -readData will read data from a specified file. +readData will read data from a specified file. The format of the input file is controlled by the type of the return @@ -2040,7 +2040,7 @@
readdata2 -readdata2 will read data from a specified file, like readdata, but using +readdata2 will read data from a specified file, like readdata, but using a different file format more closely related to that used by the ext mapper. @@ -2077,7 +2077,7 @@
print -Deprecated - use trace instead. +Deprecated - use trace instead. print will print its parameters to stdout; but will do this at a point @@ -2087,7 +2087,7 @@
trace -Trace will log its parameters. By default these will appear on both stdout +trace will log its parameters. By default these will appear on both stdout and in the run log file. Some formatting occurs to produce the log message. The particular output format should not be relied upon. (new in 0.4) From noreply at svn.ci.uchicago.edu Fri Feb 6 12:47:48 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 6 Feb 2009 12:47:48 -0600 (CST) Subject: [Swift-commit] r2500 - SwiftApps/SIDGrid Message-ID: <20090206184749.1868C228187@www.ci.uchicago.edu> Author: skenny Date: 2009-02-06 12:47:48 -0600 (Fri, 06 Feb 2009) New Revision: 2500 Removed: SwiftApps/SIDGrid/dot_swift/ Log: moving clean scripts to subdir of swift From noreply at svn.ci.uchicago.edu Fri Feb 6 12:50:33 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 6 Feb 2009 12:50:33 -0600 (CST) Subject: [Swift-commit] r2501 - in SwiftApps/SIDGrid/swift: . swift_scripts swift_scripts/R swift_scripts/db_query swift_scripts/freesurfer Message-ID: <20090206185033.BFF412281DA@www.ci.uchicago.edu> Author: skenny Date: 2009-02-06 12:50:33 -0600 (Fri, 06 Feb 2009) New Revision: 2501 Added: SwiftApps/SIDGrid/swift/swift_scripts/ SwiftApps/SIDGrid/swift/swift_scripts/R/ SwiftApps/SIDGrid/swift/swift_scripts/R/ccf-perm-wf.swift SwiftApps/SIDGrid/swift/swift_scripts/R/permutations.swift SwiftApps/SIDGrid/swift/swift_scripts/db_query/ SwiftApps/SIDGrid/swift/swift_scripts/db_query/ccf.swift SwiftApps/SIDGrid/swift/swift_scripts/db_query/query.swift SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/ SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/README SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/autorecon1.swift SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/reconall.swift Log: all working swift scripts Added: SwiftApps/SIDGrid/swift/swift_scripts/R/ccf-perm-wf.swift =================================================================== --- SwiftApps/SIDGrid/swift/swift_scripts/R/ccf-perm-wf.swift (rev 0) +++ SwiftApps/SIDGrid/swift/swift_scripts/R/ccf-perm-wf.swift 2009-02-06 18:50:33 UTC (rev 2501) @@ -0,0 +1,30 @@ +type file{} + +(file pdfFile, file resA, file resB) permScript (file scriptFile, file tsfile, file preprocR, file peakfitR){ + app{ + RPermInvoke @filename(scriptFile) @filename(tsfile) @filename(pdfFile) @filename(resA) @filename(resB); + } +} + + +genPerms(){ + file script<"scripts/Shellpeak.R">; + file preprocR<"inputs/preprocess.Enewsmooth.R">; + file peakfitR<"inputs/peakfit_2.1.R">; + + int inputFileNo[] = [57:70]; + foreach file in inputFileNo { + string inputName=@strcat("inputs/ts.",file); + string outputPDFName=@strcat("result.",file,".pdf"); + string outputAName=@strcat("result.",file,".a"); + string outputBName=@strcat("result.",file,".b"); + + file inFile; + file outPDFFile; + file outAFile; + file outBFile; + (outPDFFile,outAFile,outBFile)=permScript(script,inFile, preprocR, peakfitR); + } +} + +genPerms(); Added: SwiftApps/SIDGrid/swift/swift_scripts/R/permutations.swift =================================================================== --- SwiftApps/SIDGrid/swift/swift_scripts/R/permutations.swift (rev 0) +++ SwiftApps/SIDGrid/swift/swift_scripts/R/permutations.swift 2009-02-06 18:50:33 UTC (rev 2501) @@ -0,0 +1,35 @@ +# for running the script permutations.R +# swift will give R the permutations number +# and the matrices for running the t-test +# and will produce a file .dat + +type file{} + +(file rout) perm_r (file infilename, file pmatrix, int pnum, string outfilename, file scriptname) + { + app + { + Rcmd @filename(infilename) @filename(pmatrix) pnum outfilename @filename(scriptname); + } + } + +string user = @arg("user"); + +## the .R script is hardcoded here to use the version which +## will take the commandline arg for the permutation number + +file r_script; + + +file perm_matrix; +file ip; + + +foreach i in [1:10] + { + file r_out ; + r_out = perm_r(ip, perm_matrix, i, @strcat(i,".dat"), r_script); + } + + + Added: SwiftApps/SIDGrid/swift/swift_scripts/db_query/ccf.swift =================================================================== --- SwiftApps/SIDGrid/swift/swift_scripts/db_query/ccf.swift (rev 0) +++ SwiftApps/SIDGrid/swift/swift_scripts/db_query/ccf.swift 2009-02-06 18:50:33 UTC (rev 2501) @@ -0,0 +1,44 @@ +type file{} + +(file qout, file rout) run_query (string allcatargs, file config, file dot_r){ + app{ + Mediator allcatargs stdout=@filename(qout) @filename(dot_r); + } +} + +string user = @arg("user"); +string db = "EMBLEM1"; +string host = "tp-neurodb.ci.uchicago.edu"; + +string query_outline = "SELECT subject, voxel, speech0B, speech1B, speech2B, speech3B, speech4B, speech5B, speech6B, speech7B, speech8B, gesture0B, gesture1B, gesture2B, gesture3B, gesture4B, gesture5B, gesture6B, gesture7B, gesture8B, gestspeech0B, gestspeech1B, gestspeech2B, gestspeech3B, gestspeech4B, gestspeech5B, gestspeech6B, gestspeech7B, gestspeech8B, grasp0B, grasp1B, grasp2B, grasp3B, grasp4B, grasp5B, grasp6B, grasp7B, grasp8B FROM emblemfemlh WHERE voxel between BEGIN_BATCH and END_BATCH"; + +file r_script; +file config; + +string id = "backinCHI"; + +loop_query(int bvox, string user, string db, string host, string query_outline, file r_script, file config, string id){ + int evox = bvox+4999; + string r_swift_args = @strcat("ccf"); + string med_args = @strcat("--user ","andric", + " --conf ", "user.config", + " --db ", db, + " --host ", host, + " --query ", query_outline, + " --r_script ", @filename(r_script), + " --begin_vox ", bvox, + " --end_vox ", evox, + " --subject ", id, + " --batchstep ", "1000", + " --r_swift_args ", r_swift_args, + " --subject ", id); + file q_result ; + file r_result ; + (q_result, r_result) = run_query(med_args, r_script, config); +} + +int mybatches = [0:10000:5000]; + +foreach batch in mybatches { + loop_query(batch, user, db, host, query_outline, r_script, config, id); +} Added: SwiftApps/SIDGrid/swift/swift_scripts/db_query/query.swift =================================================================== --- SwiftApps/SIDGrid/swift/swift_scripts/db_query/query.swift (rev 0) +++ SwiftApps/SIDGrid/swift/swift_scripts/db_query/query.swift 2009-02-06 18:50:33 UTC (rev 2501) @@ -0,0 +1,53 @@ +## this is a script for running queries and passing the result to R +## for analysis (via Mediator.py). +## all of the swift variables should be given your own values +## for a query. in the 'query_outline', BEGIN_BATCH, END_BATCH +## and SUBJECT will be filled-in on the remote site by Mediator.py +## which will loop through and fill them in for each run based on the +## voxel range and step size you provide. + +type file{} + +(file qout, file rout) run_query (string allcatargs, file config, file dot_r) + { + app + { + Mediator allcatargs stdout=@filename(qout) @filename(dot_r); + } + } + +string user = @arg("user"); +string db = "EMBLEM1"; +string host = "tp-neurodb.ci.uchicago.edu"; + +string query_outline = "select gesture4B, gesture5B, gesture6B from emblemfemlh where voxel between BEGIN_BATCH and END_BATCH and subject=SUBJECT"; + +file r_script; +file config; + +string group_begin_vox = "1"; +string group_end_vox = "5000"; +string group_batchstep = "1000"; + +string subj_ids = ["3","5","6","7","8","10","11","12","13","14","15"]; +foreach id,i in subj_ids { + string r_swift_args = @strcat("subj",id); + string med_args = @strcat("--user ", user, + " --conf ", @filename(config), + " --db ", db, + " --host ", host, + " --query ", query_outline, + " --r_script ", @filename(r_script), + " --begin_vox ", group_begin_vox, + " --end_vox ", group_end_vox, + " --subject ", id, + " --batchstep ", "1000", + " --r_swift_args ", r_swift_args, + " --subject ", id); + file q_result ; + file r_result ; + (q_result, r_result) = run_query(med_args, r_script, config); +} + + + Added: SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/README =================================================================== --- SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/README (rev 0) +++ SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/README 2009-02-06 18:50:33 UTC (rev 2501) @@ -0,0 +1,7 @@ + +######## RECONALL + +this version runs on uc/anl teragrid site using freesurfer 3. currently runs a single subject. see example +execution below which points to the correct tc.data and sites.xml files. + +swift reconall.swift -sites.file /disks/gpfs/fmri/swifft/config/swift_sites_ucanl32.xml -tc.file /disks/gpfs/fmri/swift/config/tc.data -user=skenny -subj=dude -user=skenny -inimage=001.mgz Added: SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/autorecon1.swift =================================================================== --- SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/autorecon1.swift (rev 0) +++ SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/autorecon1.swift 2009-02-06 18:50:33 UTC (rev 2501) @@ -0,0 +1,51 @@ +type file {} +type mgzfile; + +(file job_00) new_subject (string subj, mgzfile inimage) { + app { + create_subject @arg("user") subj @inimage stdout=@filename(job_00); + } +} + +(file job_01) autorecon1_4 (string subj, file job_00) { + app { + reconall @arg("user") @strcat("-s ", subj, " -nuintensitycor") stdout=@filename(job_01); + } +} + +(file job_02) autorecon1_5 (string subj, file job_01) { + app { + reconall @arg("user") @strcat("-s ", subj, " -talairach") stdout=@filename(job_02); + } +} + +(file job_03) autorecon1_6 (string subj, file job_02) { + app { + reconall @arg("user") @strcat("-s ", subj, " -normalization") stdout=@filename(job_03); + } +} + +(file job_04) autorecon1_7 (string subj, file job_03) { + app { + reconall @arg("user") @strcat("-s ", subj, " -skullstrip") stdout=@filename(job_04); + } +} + +### tars the subject dir on remote site for tranfer back to host + +(file subject_data) get_subject (string subj, file job_04) { + app { + get_subject @arg("subject_path") subj @subject_data; + } +} + +mgzfile inimage ; +string subj = @arg("subj"); + +file sigstring00 = new_subject(subj, inimage); +file sigstring01 = autorecon1_4(subj, sigstring00); +file sigstring02 = autorecon1_5(subj, sigstring01); +file sigstring03 = autorecon1_6(subj, sigstring02); +file sigstring04 = autorecon1_7(subj, sigstring03); +file subject_data ; +subject_data = get_subject(subj, sigstring04); Added: SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/reconall.swift =================================================================== --- SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/reconall.swift (rev 0) +++ SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/reconall.swift 2009-02-06 18:50:33 UTC (rev 2501) @@ -0,0 +1,43 @@ + type file {} +type mgzfile; + +(file subj_dir) recon1 (string subj, mgzfile inimage) { + app { + recon1 @arg("user") subj @inimage; + } +} + +(file subj_dirL) recon2L (string subj, file subj_dir) { + app { + recon2L @arg("user") subj @subj_dir; + } +} + + +(file subj_dirR) recon2R (string subj, file subj_dir) { + app { + recon2R @arg("user") subj @subj_dir; + } +} + +(file subj_dirCOMPLETE) reconFINAL (string subj, file subj_dirL, file subj_dirR) { + app { + reconFINAL @arg("user") subj @subj_dirL @subj_dirR; + } +} + + +mgzfile inimage ; +string subj = @arg("subj"); +file subject_data ; +file subject_data2L ; +file subject_data2R ; +file subject_dataCOMPLETE ; + +subject_data = recon1(subj, inimage); +subject_data2L = recon2L (subj, subject_data); +subject_data2R = recon2R (subj, subject_data); +subject_dataCOMPLETE = reconFINAL (subj, subject_data2L, subject_data2R); + + + From noreply at svn.ci.uchicago.edu Fri Feb 6 13:01:51 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 6 Feb 2009 13:01:51 -0600 (CST) Subject: [Swift-commit] r2502 - SwiftApps/SIDGrid/swift/swift_scripts/freesurfer Message-ID: <20090206190151.D9F532281E6@www.ci.uchicago.edu> Author: skenny Date: 2009-02-06 13:01:51 -0600 (Fri, 06 Feb 2009) New Revision: 2502 Added: SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/autorecon2_3.swift Log: for running autorecon 2 and 3 Added: SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/autorecon2_3.swift =================================================================== --- SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/autorecon2_3.swift (rev 0) +++ SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/autorecon2_3.swift 2009-02-06 19:01:51 UTC (rev 2502) @@ -0,0 +1,45 @@ +type file {} +type mgzfile; + +(file subj_dir) autorecon2 (string subj file subj_dir_r1) { + app { + autorecon2_BOTH @arg("user") subj @subj_dir; + } +} + +(file subj_dirL) recon2L (string subj, file subj_dir) { + app { + recon2L @arg("user") subj @subj_dir; + } +} + + +(file subj_dirR) recon2R (string subj, file subj_dir) { + app { + recon2R @arg("user") subj @subj_dir; + } +} + +(file subj_dirCOMPLETE) reconFINAL (string subj, file subj_dirL, file subj_dirR) { + app { + reconFINAL @arg("user") subj @subj_dirL @subj_dirR; + } +} + + +string subj = @arg("subj"); +# initial input subject dir should already be tar'ed +file subject_data_r1 ; + +file subject_data ; +file subject_data2L ; +file subject_data2R ; +file subject_dataCOMPLETE ; + +subject_data = autorecon2(subj, subject_data_r1); +subject_data2L = recon2L (subj, subject_data); +subject_data2R = recon2R (subj, subject_data); +subject_dataCOMPLETE = reconFINAL (subj, subject_data2L, subject_data2R); + + + From noreply at svn.ci.uchicago.edu Fri Feb 6 13:13:25 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 6 Feb 2009 13:13:25 -0600 (CST) Subject: [Swift-commit] r2503 - in SwiftApps/SIDGrid/swift/swift_scripts: . afni Message-ID: <20090206191325.885E12281AD@www.ci.uchicago.edu> Author: skenny Date: 2009-02-06 13:13:24 -0600 (Fri, 06 Feb 2009) New Revision: 2503 Added: SwiftApps/SIDGrid/swift/swift_scripts/afni/ SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIPreProcs.swift SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIdeconvolution.swift Log: afni preprocessing scripts Added: SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIPreProcs.swift =================================================================== --- SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIPreProcs.swift (rev 0) +++ SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIPreProcs.swift 2009-02-06 19:13:24 UTC (rev 2503) @@ -0,0 +1,64 @@ +#---- doing volreg, despike, mean, percent sig change for PK1 +type file{} + +type AFNI_obj{ + file HEAD; + file BRIK; +}; + + +app (AFNI_obj volregResult, file resultVolregMotion) AFNI_volreg (string baseName, AFNI_obj inputTS, AFNI_obj referenceTS, string volregBaseString){ + AFNI_3dvolreg "-twopass" "-twodup" "-dfile" @strcat("mot_",baseName) "-base" @strcat(volregBaseString,"+orig[199]") "-prefix" @strcat("volreg.",baseName) @inputTS.BRIK; +} + +app (AFNI_obj despikeResult) AFNI_despike (string baseName, AFNI_obj volregdInputTS){ + AFNI_3dDespike "-prefix" @strcat("despiked.",baseName) @volregdInputTS.BRIK; +} + +app (AFNI_obj meanResult) AFNI_mean (AFNI_obj despikeResult, string baseName){ + AFNI_3dTstat "-mean" "-prefix" @strcat("mean.",baseName) @despikeResult.BRIK; +} + +app (AFNI_obj nrmlzPerChngResult) AFNI_normalizePerChng (AFNI_obj despikeResult, AFNI_obj meanResult, string baseName){ + AFNI_3dcalc "-a" @despikeResult.BRIK "-b" @meanResult.BRIK "-expr" "((a-b)/b)*100" "-fscale" "-prefix" @strcat("nrmlzdPerChng.",baseName); +} + +app (AFNI_obj TScatResult) AFNI_TScat (string subject, AFNI_obj PerCh2, AFNI_obj PerCh3, AFNI_obj PerCh4, AFNI_obj PerCh5, AFNI_obj PerCh6, AFNI_obj PerCh7, AFNI_obj PerCh8, AFNI_obj PerCh9){ + AFNI_3dTcat "-prefix" @strcat("nrmlzdPerChng_TScat.",subject) @PerCh2.BRIK @PerCh3.BRIK @PerCh4.BRIK @PerCh5.BRIK @PerCh6.BRIK @PerCh7.BRIK @PerCh8.BRIK @PerCh9.BRIK; +} + +(AFNI_obj despikeResult, AFNI_obj meanResult, AFNI_obj nrmlzPerChngResult) AFNI_PerChngTS (string baseName, AFNI_obj volregdInput){ + (despikeResult) = AFNI_despike(baseName, volregdInput); + (meanResult) = AFNI_mean(despikeResult, baseName); + (nrmlzPerChngResult) = AFNI_normalizePerChng(despikeResult, meanResult, baseName); +} + + +string declarelist[] = ["PK1"]; +foreach subject in declarelist{ + int runs[] = [2:9]; + foreach run in runs{ + string baseName = @strcat(subject,".runLOOP",run); + AFNI_obj inputTS; + AFNI_obj volregResult; + file resultVolregMotion; + AFNI_obj volregRefTS; + string volregBaseString = @strcat("ts.5"); + AFNI_obj despikeResult; + AFNI_obj meanResult; + AFNI_obj nrmlzPerChngResult; + (volregResult, resultVolregMotion) = AFNI_volreg(baseName, inputTS, volregRefTS, volregBaseString); + (despikeResult, meanResult, nrmlzPerChngResult) = AFNI_PerChngTS(baseName, volregResult); + } + string subj = @strcat(subject); + AFNI_obj PerCh2; + AFNI_obj PerCh3; + AFNI_obj PerCh4; + AFNI_obj PerCh5; + AFNI_obj PerCh6; + AFNI_obj PerCh7; + AFNI_obj PerCh8; + AFNI_obj PerCh9; + AFNI_obj TScatResult; + (TScatResult) = AFNI_TScat(subj, PerCh2, PerCh3, PerCh4, PerCh5, PerCh6, PerCh7, PerCh8, PerCh9); +} Added: SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIdeconvolution.swift =================================================================== --- SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIdeconvolution.swift (rev 0) +++ SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIdeconvolution.swift 2009-02-06 19:13:24 UTC (rev 2503) @@ -0,0 +1,49 @@ +#---- pinche deconvolution +type file{} + +type AFNI_obj{ + file HEAD; + file BRIK; +} + +type stims{ + string labels[]; + string fnames[]; +} + +app (AFNI_obj deconvBucketResult, file deconvXsave) AFNI_pincheDeconvolution (AFNI_obj normalized, stims stimuli, file concat, file motionfile, file designmatrix){ + AFNI_3dDeconvolve "-input" @filename(normalized.BRIK) "-tshift" "-num_stimts" 8 + "-stim_file" 1 stimuli.fnames[0] "-stim_label" 1 stimuli.labels[0] + "-stim_file" 2 stimuli.fnames[1] "-stim_label" 2 stimuli.labels[1] + "-stim_file" 3 stimuli.fnames[2] "-stim_base" 3 "-stim_label" 3 stimuli.labels[2] + "-stim_file" 4 stimuli.fnames[3] "-stim_base" 4 "-stim_label" 4 stimuli.labels[3] + "-stim_file" 5 stimuli.fnames[4] "-stim_base" 5 "-stim_label" 5 stimuli.labels[4] + "-stim_file" 6 stimuli.fnames[5] "-stim_base" 6 "-stim_label" 6 stimuli.labels[5] + "-stim_file" 7 stimuli.fnames[6] "-stim_base" 7 "-stim_label" 7 stimuli.labels[6] + "-stim_file" 8 stimuli.fnames[7] "-stim_base" 8 "-stim_label" 8 stimuli.labels[7] + "-full_first" "-tout" "-fout" "-nodmbase" "-xsave" "-nfirst" "0" "-concat" @filename(concat) "-bucket" + "TESTdeconvolveResults_PK1"; +} + +string declarelist[] = ["PK1"]; +foreach subject in declarelist +{ + ## map inputs + file designmatrix; + file motionfile; + file concat; + AFNI_obj normalized; + + ## map outputs + file xsave; + AFNI_obj bucket; + + stims stimuli; + stimuli.labels = ["pakata","whitenoise","roll","pitch","yaw","dx","dy","dz"]; + stimuli.fnames = [@strcat(@filename(designmatrix),"[0]"), at strcat(@filename(designmatrix),"[1]"), + @strcat(@filename(motionfile),"[1]"), at strcat(@filename(motionfile),"[2]"), + @strcat(@filename(motionfile),"[3]"), at strcat(@filename(motionfile),"[4]"), + @strcat(@filename(motionfile),"[5]"), at strcat(@filename(motionfile),"[6]")]; + + (bucket, xsave) = AFNI_pincheDeconvolution(normalized, stimuli, concat, motionfile, designmatrix); +} From noreply at svn.ci.uchicago.edu Fri Feb 6 13:15:49 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 6 Feb 2009 13:15:49 -0600 (CST) Subject: [Swift-commit] r2504 - SwiftApps/SIDGrid/config Message-ID: <20090206191549.C386D228187@www.ci.uchicago.edu> Author: skenny Date: 2009-02-06 13:15:49 -0600 (Fri, 06 Feb 2009) New Revision: 2504 Modified: SwiftApps/SIDGrid/config/local_sites.xml Log: path adjustment Modified: SwiftApps/SIDGrid/config/local_sites.xml =================================================================== --- SwiftApps/SIDGrid/config/local_sites.xml 2009-02-06 19:13:24 UTC (rev 2503) +++ SwiftApps/SIDGrid/config/local_sites.xml 2009-02-06 19:15:49 UTC (rev 2504) @@ -8,7 +8,7 @@ - /disks/gpfs/fmri/swift/work + /disks/ci-gpfs/fmri/swift/work From noreply at svn.ci.uchicago.edu Fri Feb 6 13:23:44 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 6 Feb 2009 13:23:44 -0600 (CST) Subject: [Swift-commit] r2505 - SwiftApps/SIDGrid/swift/swift_scripts/R Message-ID: <20090206192344.362B922819E@www.ci.uchicago.edu> Author: skenny Date: 2009-02-06 13:23:43 -0600 (Fri, 06 Feb 2009) New Revision: 2505 Added: SwiftApps/SIDGrid/swift/swift_scripts/R/modgenproc.swift Log: generates and runs mx model objects Added: SwiftApps/SIDGrid/swift/swift_scripts/R/modgenproc.swift =================================================================== --- SwiftApps/SIDGrid/swift/swift_scripts/R/modgenproc.swift (rev 0) +++ SwiftApps/SIDGrid/swift/swift_scripts/R/modgenproc.swift 2009-02-06 19:23:43 UTC (rev 2505) @@ -0,0 +1,28 @@ +### generate openmx models in parallel +### by number of connections + +type file{} + +(file min) mxmodel_processor (file cov_matrix, file dot_r, int numcol, int modnum){ + app{ + RInvoke @filename(dot_r) @filename(cov_matrix) numcol modnum; + } +} + +file cov_script; +file covmx; + +float initweight = .75; +int numcol = 4; +int totalperms[] = [0:65534]; + +foreach perm in totalperms{ + file modmin; + (modmin) = mxmodel_processor(covmx, cov_script, numcol, perm); + } + + + + + + From noreply at svn.ci.uchicago.edu Fri Feb 6 16:22:01 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 6 Feb 2009 16:22:01 -0600 (CST) Subject: [Swift-commit] r2506 - trunk/src/org/griphyn/vdl/karajan/lib Message-ID: <20090206222201.6F995228187@www.ci.uchicago.edu> Author: benc Date: 2009-02-06 16:22:00 -0600 (Fri, 06 Feb 2009) New Revision: 2506 Added: trunk/src/org/griphyn/vdl/karajan/lib/ExpandArguments.java Log: r2498 forgot to add this file Added: trunk/src/org/griphyn/vdl/karajan/lib/ExpandArguments.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/ExpandArguments.java (rev 0) +++ trunk/src/org/griphyn/vdl/karajan/lib/ExpandArguments.java 2009-02-06 22:22:00 UTC (rev 2506) @@ -0,0 +1,72 @@ +package org.griphyn.vdl.karajan.lib; + +import java.util.ArrayList; +import java.util.Map; +import java.util.Collections; +import java.util.Comparator; +import java.util.TreeSet; +import java.util.Set; +import java.util.Iterator; + +import org.apache.log4j.Logger; +import org.globus.cog.karajan.arguments.Arg; +import org.globus.cog.karajan.stack.VariableStack; +import org.globus.cog.karajan.workflow.ExecutionException; +import org.globus.cog.karajan.workflow.futures.FutureNotYetAvailable; + +import org.griphyn.vdl.mapping.ArrayDataNode; +import org.griphyn.vdl.mapping.DSHandle; +import org.griphyn.vdl.mapping.HandleOpenException; +import org.griphyn.vdl.mapping.InvalidPathException; +import org.griphyn.vdl.mapping.Path; + +public class ExpandArguments extends VDLFunction { + public static final Logger logger = Logger.getLogger(ExpandArguments.class); + + static { + setArguments(ExpandArguments.class, new Arg[] { Arg.VARGS }); + } + + public Object function(VariableStack stack) throws ExecutionException { + ArrayList l = new ArrayList(); + Object[] items = Arg.VARGS.asArray(stack); + for (int i = 0; i < items.length; i++) { + Object item = items[i]; + if(!(item instanceof DSHandle)) { + throw new RuntimeException("Cannot handle argument implemented by "+item.getClass()); + } + + if(item instanceof ArrayDataNode) { + ArrayDataNode array = (ArrayDataNode) item; + Map m=array.getArrayValue(); + Set keySet = m.keySet(); + TreeSet sortedKeySet = new TreeSet(new StringsAsIntegersComparator()); + sortedKeySet.addAll(keySet); + Iterator it = sortedKeySet.iterator(); + while(it.hasNext()) { + Object key = it.next(); + l.add(m.get(key)); + } + } else { + l.add(item); + } + // TODO this does not correctly handle structs or + // externals - at the moment, probably neither of + // those should be usable as a string. It also + // does not handle nested arrays. However, none of + // those should get here in normal operation due + // to static type-checking + } + return l; + } + + class StringsAsIntegersComparator implements Comparator { + public int compare(Object l, Object r) { + Integer lnum = new Integer((String)l); + Integer rnum = new Integer((String)r); + return lnum.compareTo(rnum); + } + } + +} + From noreply at svn.ci.uchicago.edu Fri Feb 6 20:13:50 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 6 Feb 2009 20:13:50 -0600 (CST) Subject: [Swift-commit] r2507 - in provider-deef: . src/org/globus/cog/abstraction/impl/execution/deef Message-ID: <20090207021350.85AE92281AD@www.ci.uchicago.edu> Author: hategan Date: 2009-02-06 20:13:49 -0600 (Fri, 06 Feb 2009) New Revision: 2507 Modified: provider-deef/project.properties provider-deef/src/org/globus/cog/abstraction/impl/execution/deef/JobSubmissionTaskHandler.java Log: cancel message; removed dependency on karajan; updated version Modified: provider-deef/project.properties =================================================================== --- provider-deef/project.properties 2009-02-06 22:22:00 UTC (rev 2506) +++ provider-deef/project.properties 2009-02-07 02:13:49 UTC (rev 2507) @@ -1,6 +1,6 @@ module.name = provider-deef long.name = DeeF provider for abstractions -version = 1.0 +version = 1.1 project = Java CoG Kit lib.deps = FalkonStubs.jar, concurrent.jar debug = true Modified: provider-deef/src/org/globus/cog/abstraction/impl/execution/deef/JobSubmissionTaskHandler.java =================================================================== --- provider-deef/src/org/globus/cog/abstraction/impl/execution/deef/JobSubmissionTaskHandler.java 2009-02-06 22:22:00 UTC (rev 2506) +++ provider-deef/src/org/globus/cog/abstraction/impl/execution/deef/JobSubmissionTaskHandler.java 2009-02-07 02:13:49 UTC (rev 2507) @@ -6,18 +6,17 @@ package org.globus.cog.abstraction.impl.execution.deef; -import java.net.MalformedURLException; -import java.net.URL; -import java.util.Vector; import java.util.Collection; import java.util.Iterator; +import java.util.Vector; -import org.apache.axis.components.uuid.UUIDGenFactory; -import org.apache.axis.message.addressing.Address; import org.apache.axis.message.addressing.EndpointReferenceType; -import org.apache.axis.types.NonNegativeInteger; -import org.apache.axis.types.PositiveInteger; import org.apache.log4j.Logger; +import org.globus.GenericPortal.stubs.GPService_instance.DataFiles; +import org.globus.GenericPortal.stubs.GPService_instance.DeInit; +import org.globus.GenericPortal.stubs.GPService_instance.DeInitResponse; +import org.globus.GenericPortal.stubs.GPService_instance.Executable; +import org.globus.GenericPortal.stubs.GPService_instance.GPPortType; import org.globus.cog.abstraction.impl.common.StatusImpl; import org.globus.cog.abstraction.impl.common.task.IllegalSpecException; import org.globus.cog.abstraction.impl.common.task.InvalidSecurityContextException; @@ -28,30 +27,10 @@ import org.globus.cog.abstraction.interfaces.ServiceContact; import org.globus.cog.abstraction.interfaces.Status; import org.globus.cog.abstraction.interfaces.Task; - -import org.xml.sax.InputSource; -import org.oasis.wsrf.lifetime.WSResourceLifetimeServiceAddressingLocator; import org.oasis.wsrf.lifetime.ImmediateResourceTermination; -import org.oasis.wsrf.properties.GetResourcePropertyResponse; -import org.globus.wsrf.NotificationConsumerManager; -import org.globus.wsrf.utils.AddressingUtils; -import org.globus.wsrf.ResourceKey; -import org.globus.wsrf.encoding.ObjectDeserializer; +import org.oasis.wsrf.lifetime.WSResourceLifetimeServiceAddressingLocator; -import org.globus.GenericPortal.stubs.GPService_instance.*; -import org.globus.GenericPortal.stubs.GPService_instance.GPPortType; -import org.globus.GenericPortal.stubs.GPService_instance.service.GPServiceAddressingLocator; -import org.globus.GenericPortal.stubs.Factory.CreateResource; -import org.globus.GenericPortal.stubs.Factory.CreateResourceResponse; -import org.globus.GenericPortal.stubs.Factory.FactoryPortType; -import org.globus.GenericPortal.stubs.Factory.service.FactoryServiceAddressingLocator; - -import org.globus.GenericPortal.common.Notification; - -import org.globus.cog.karajan.util.TypeUtil; - - //import org.globus.GenericPortal.clients.GPService_instance.*; public class JobSubmissionTaskHandler implements DelegatedTaskHandler @@ -206,12 +185,17 @@ TaskSubmissionException { // not implemented yet } - + public void cancel() throws InvalidSecurityContextException, TaskSubmissionException { + cancel(null); + } + + public void cancel(String message) throws InvalidSecurityContextException, + TaskSubmissionException { try { - this.task.setStatus(Status.CANCELED); + this.task.setStatus(new StatusImpl(Status.CANCELED, message, null)); //cleanup(); } catch (Exception e) { @@ -226,7 +210,7 @@ String s = (String)spec.getAttribute("maxwalltime"); if (s != null) { - return timeToSeconds(TypeUtil.toString(s)); + return timeToSeconds(s); } else return 0; From noreply at svn.ci.uchicago.edu Fri Feb 6 20:54:28 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 6 Feb 2009 20:54:28 -0600 (CST) Subject: [Swift-commit] r2508 - in provider-wonky: . src/org/globus/cog/abstraction/impl/execution/wonky Message-ID: <20090207025428.9B8062281DA@www.ci.uchicago.edu> Author: hategan Date: 2009-02-06 20:54:27 -0600 (Fri, 06 Feb 2009) New Revision: 2508 Modified: provider-wonky/project.properties provider-wonky/src/org/globus/cog/abstraction/impl/execution/wonky/JobSubmissionTaskHandler.java Log: cancel message; cleanup; updated version; option to lose completion notification Modified: provider-wonky/project.properties =================================================================== --- provider-wonky/project.properties 2009-02-07 02:13:49 UTC (rev 2507) +++ provider-wonky/project.properties 2009-02-07 02:54:27 UTC (rev 2508) @@ -1,6 +1,6 @@ module.name = provider-wonky long.name = Local provider, but wonky -version = 0.1 +version = 0.2 project = Java CoG Kit lib.deps = - debug = true Modified: provider-wonky/src/org/globus/cog/abstraction/impl/execution/wonky/JobSubmissionTaskHandler.java =================================================================== --- provider-wonky/src/org/globus/cog/abstraction/impl/execution/wonky/JobSubmissionTaskHandler.java 2009-02-07 02:13:49 UTC (rev 2507) +++ provider-wonky/src/org/globus/cog/abstraction/impl/execution/wonky/JobSubmissionTaskHandler.java 2009-02-07 02:54:27 UTC (rev 2508) @@ -24,6 +24,7 @@ import java.util.Random; import org.apache.log4j.Logger; +import org.globus.cog.abstraction.impl.common.AbstractDelegatedTaskHandler; import org.globus.cog.abstraction.impl.common.StatusImpl; import org.globus.cog.abstraction.impl.common.execution.JobException; import org.globus.cog.abstraction.impl.common.task.IllegalSpecException; @@ -32,14 +33,13 @@ import org.globus.cog.abstraction.impl.common.task.TaskSubmissionException; import org.globus.cog.abstraction.impl.common.util.NullOutputStream; import org.globus.cog.abstraction.impl.common.util.OutputStreamMultiplexer; -import org.globus.cog.abstraction.interfaces.DelegatedTaskHandler; import org.globus.cog.abstraction.interfaces.FileLocation; import org.globus.cog.abstraction.interfaces.JobSpecification; import org.globus.cog.abstraction.interfaces.ServiceContact; import org.globus.cog.abstraction.interfaces.Status; import org.globus.cog.abstraction.interfaces.Task; -public class JobSubmissionTaskHandler implements DelegatedTaskHandler, +public class JobSubmissionTaskHandler extends AbstractDelegatedTaskHandler implements Runnable { private static Logger logger = Logger .getLogger(JobSubmissionTaskHandler.class); @@ -58,7 +58,6 @@ public static final int BUFFER_SIZE = 1024; - private Task task = null; private Thread thread = null; private Process process; private volatile boolean killed; @@ -66,81 +65,72 @@ public void submit(Task task) throws IllegalSpecException, InvalidSecurityContextException, InvalidServiceContactException, TaskSubmissionException { - if (this.task != null) { - throw new TaskSubmissionException( - "JobSubmissionTaskHandler cannot handle two active jobs simultaneously"); - } - else if (task.getStatus().getStatusCode() != Status.UNSUBMITTED) { - throw new TaskSubmissionException("Task is not in unsubmitted state"); - } - else { - logger.info("Submitting wonky job "+jobNumber); - this.task = task; + checkAndSetTask(task); + logger.info("Submitting wonky job "+jobNumber); - ServiceContact serviceContact = this.task.getService(0) - .getServiceContact(); - String server = serviceContact.getContact(); + ServiceContact serviceContact = task.getService(0) + .getServiceContact(); + String server = serviceContact.getContact(); - logger.debug("Contact is: "+server); - String[] splitServer = server.split("/"); - siteName = splitServer[0]; - siteOptions = Arrays.asList(splitServer); + logger.debug("Contact is: "+server); + String[] splitServer = server.split("/"); + siteName = splitServer[0]; + siteOptions = Arrays.asList(splitServer); - if(!failDelay("submitting")) { - this.task.setStatus(Status.FAILED); - return; - } - task.setStatus(Status.SUBMITTING); - JobSpecification spec; - try { - spec = (JobSpecification) this.task.getSpecification(); - } catch (Exception e) { - throw new IllegalSpecException( - "Exception while retrieving Job Specification", e); - } - if (logger.isDebugEnabled()) { - logger.debug(spec.toString()); - } + if(!failDelay("submitting")) { + task.setStatus(Status.FAILED); + return; + } + task.setStatus(Status.SUBMITTING); + JobSpecification spec; + try { + spec = (JobSpecification) task.getSpecification(); + } catch (Exception e) { + throw new IllegalSpecException( + "Exception while retrieving Job Specification", e); + } + if (logger.isDebugEnabled()) { + logger.debug(spec.toString()); + } - Collection attributeNames = spec.getAttributeNames(); - Iterator attributeIterator = attributeNames.iterator(); + Collection attributeNames = spec.getAttributeNames(); + Iterator attributeIterator = attributeNames.iterator(); - if(siteOptions.contains("strictattr")) { - logger.debug("Strict attribute checking is enabled"); - while(attributeIterator.hasNext()) { - String attrName = (String)attributeIterator.next(); - if(!siteOptions.contains("permitattr=" + attrName)) { - logger.error("Job specification attribute was passed that should not have been: "+attrName); - this.task.setStatus(Status.FAILED); - return; - } + if(siteOptions.contains("strictattr")) { + logger.debug("Strict attribute checking is enabled"); + while(attributeIterator.hasNext()) { + String attrName = (String)attributeIterator.next(); + if(!siteOptions.contains("permitattr=" + attrName)) { + logger.error("Job specification attribute was passed that should not have been: "+attrName); + task.setStatus(Status.FAILED); + return; } } + } - try { - if (logger.isInfoEnabled()) { - logger.info("Submitting task " + task); - } - synchronized(this) { - this.thread = new Thread(this); - if (this.task.getStatus().getStatusCode() != Status.CANCELED) { - if(!failDelay("submitted")) { - this.task.setStatus(Status.FAILED); - return; - } - this.task.setStatus(Status.SUBMITTED); - this.thread.start(); - if (spec.isBatchJob()) { - this.task.setStatus(Status.COMPLETED); - } + try { + if (logger.isInfoEnabled()) { + logger.info("Submitting task " + task); + } + synchronized(this) { + this.thread = new Thread(this); + if (task.getStatus().getStatusCode() != Status.CANCELED) { + if(!failDelay("submitted")) { + task.setStatus(Status.FAILED); + return; } + task.setStatus(Status.SUBMITTED); + this.thread.start(); + if (spec.isBatchJob()) { + task.setStatus(Status.COMPLETED); + } } - } catch (Exception e) { - throw new TaskSubmissionException("Cannot submit job", e); } + } catch (Exception e) { + throw new TaskSubmissionException("Cannot submit job", e); } } @@ -154,13 +144,13 @@ logger.info("Resume called on wonky job "+jobNumber); } - public void cancel() throws InvalidSecurityContextException, + public void cancel(String message) throws InvalidSecurityContextException, TaskSubmissionException { logger.info("Cancel called on wonky job "+jobNumber); synchronized(this) { killed = true; process.destroy(); - this.task.setStatus(Status.CANCELED); + getTask().setStatus(new StatusImpl(Status.CANCELED, message, null)); } } @@ -169,7 +159,7 @@ public void run() { try { - JobSpecification spec = (JobSpecification) this.task + JobSpecification spec = (JobSpecification) getTask() .getSpecification(); File dir = null; @@ -210,10 +200,10 @@ logger.info("Wonky job running now"); process = Runtime.getRuntime().exec(cmdarray, buildEnvp(spec), dir); if(!failDelay("active")) { - this.task.setStatus(Status.FAILED); + getTask().setStatus(Status.FAILED); return; } - this.task.setStatus(Status.ACTIVE); + getTask().setStatus(Status.ACTIVE); // reusable byte buffer byte[] buf = new byte[BUFFER_SIZE]; @@ -232,7 +222,7 @@ if (logger.isDebugEnabled()) { logger.debug("STDOUT from job: " + out); } - this.task.setStdOutput(out); + getTask().setStdOutput(out); } } @@ -244,7 +234,7 @@ if (logger.isDebugEnabled()) { logger.debug("STDERR from job: " + err); } - this.task.setStdError(err); + getTask().setStdError(err); } } @@ -262,13 +252,18 @@ if(siteOptions.contains("nofailonexit")) { // suppress failures caused by exit code - this.task.setStatus(Status.COMPLETED); + getTask().setStatus(Status.COMPLETED); } else { // normal fail behaviour if (exitCode == 0) { if(failDelay("completed")) { - this.task.setStatus(Status.COMPLETED); + if (siteOptions.contains("runawayjob") && jobNumber == 0) { + System.out.println("Wonky job completion notification gets lost"); + } + else { + getTask().setStatus(Status.COMPLETED); + } } else { - this.task.setStatus(Status.FAILED); + getTask().setStatus(Status.FAILED); } } else { throw new JobException(exitCode); @@ -281,12 +276,7 @@ if (logger.isDebugEnabled()) { logger.debug("Exception while running local executable", e); } - Status newStatus = new StatusImpl(); - Status oldStatus = this.task.getStatus(); - newStatus.setPrevStatusCode(oldStatus.getStatusCode()); - newStatus.setStatusCode(Status.FAILED); - newStatus.setException(e); - this.task.setStatus(newStatus); + failTask(null, e); } } From noreply at svn.ci.uchicago.edu Fri Feb 6 20:57:41 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 6 Feb 2009 20:57:41 -0600 (CST) Subject: [Swift-commit] r2509 - trunk/src/org/griphyn/vdl/karajan/lib Message-ID: <20090207025741.681582281DA@www.ci.uchicago.edu> Author: hategan Date: 2009-02-06 20:57:40 -0600 (Fri, 06 Feb 2009) New Revision: 2509 Modified: trunk/src/org/griphyn/vdl/karajan/lib/TCProfile.java Log: add tr name as an attribute Modified: trunk/src/org/griphyn/vdl/karajan/lib/TCProfile.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/TCProfile.java 2009-02-07 02:54:27 UTC (rev 2508) +++ trunk/src/org/griphyn/vdl/karajan/lib/TCProfile.java 2009-02-07 02:57:40 UTC (rev 2509) @@ -67,6 +67,7 @@ attrs = attributesFromTC(tce, attrs); } + attrs = addAttribute(attrs, "tr", tr); addAttributes(named, attrs); return null; } @@ -116,6 +117,14 @@ } named.add(GridExec.A_ATTRIBUTES, attrs); } + + private Map addAttribute(Map attrs, String key, Object value) { + if (attrs == null) { + attrs = new HashMap(); + } + attrs.put(key, value); + return attrs; + } private Map attributesFromTC(TransformationCatalogEntry tce, Map attrs) { List l = tce.getProfiles(Profile.GLOBUS); From noreply at svn.ci.uchicago.edu Fri Feb 6 20:59:27 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 6 Feb 2009 20:59:27 -0600 (CST) Subject: [Swift-commit] r2510 - trunk/src/org/griphyn/vdl/karajan/lib Message-ID: <20090207025927.2AFA8228187@www.ci.uchicago.edu> Author: hategan Date: 2009-02-06 20:59:26 -0600 (Fri, 06 Feb 2009) New Revision: 2510 Modified: trunk/src/org/griphyn/vdl/karajan/lib/RuntimeStats.java Log: option to disable runtime stats Modified: trunk/src/org/griphyn/vdl/karajan/lib/RuntimeStats.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/RuntimeStats.java 2009-02-07 02:57:40 UTC (rev 2509) +++ trunk/src/org/griphyn/vdl/karajan/lib/RuntimeStats.java 2009-02-07 02:59:26 UTC (rev 2510) @@ -1,17 +1,19 @@ package org.griphyn.vdl.karajan.lib; +import java.io.IOException; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.Iterator; +import java.util.List; +import java.util.Map; + import org.globus.cog.karajan.arguments.Arg; import org.globus.cog.karajan.stack.VariableStack; +import org.globus.cog.karajan.util.TypeUtil; import org.globus.cog.karajan.workflow.ExecutionException; import org.globus.cog.karajan.workflow.nodes.functions.FunctionsCollection; -import org.globus.cog.karajan.util.TypeUtil; +import org.griphyn.vdl.util.VDL2Config; -import java.util.Map; -import java.util.HashMap; -import java.util.Iterator; -import java.util.List; -import java.util.ArrayList; - /** this is an icky class that does too much with globals, but is for proof of concept. */ @@ -87,13 +89,26 @@ List states = new ArrayList(); long lastDumpTime = 0; + private boolean disabled; public ProgressTicker() { super("Progress ticker"); + try { + if ("true".equalsIgnoreCase(VDL2Config.getConfig().getProperty("ticker.disable"))) { + logger.info("Ticker disabled in configuration file"); + disabled = true; + } + } + catch (IOException e) { + logger.debug("Could not read swift properties", e); + } } public void run() { - while(true) { + if (disabled) { + return; + } + while(true) { dumpState(); try { @@ -106,6 +121,9 @@ } void dumpState() { + if (disabled) { + return; + } long now = System.currentTimeMillis(); if(lastDumpTime + MIN_PERIOD_MS > now) return; lastDumpTime = now; @@ -113,6 +131,9 @@ } void finalDumpState() { + if (disabled) { + return; + } printStates("Final status: "); } From noreply at svn.ci.uchicago.edu Fri Feb 6 21:05:50 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 6 Feb 2009 21:05:50 -0600 (CST) Subject: [Swift-commit] r2511 - trunk/src/org/griphyn/vdl/karajan/lib/replication Message-ID: <20090207030550.96E82228187@www.ci.uchicago.edu> Author: hategan Date: 2009-02-06 21:05:49 -0600 (Fri, 06 Feb 2009) New Revision: 2511 Modified: trunk/src/org/griphyn/vdl/karajan/lib/replication/ReplicationManager.java Log: kill jobs that have been running for twice their walltime Modified: trunk/src/org/griphyn/vdl/karajan/lib/replication/ReplicationManager.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/replication/ReplicationManager.java 2009-02-07 02:59:26 UTC (rev 2510) +++ trunk/src/org/griphyn/vdl/karajan/lib/replication/ReplicationManager.java 2009-02-07 03:05:49 UTC (rev 2511) @@ -5,131 +5,201 @@ import java.util.Date; import java.util.HashMap; +import java.util.HashSet; import java.util.Iterator; import java.util.Map; +import java.util.Set; import org.apache.log4j.Logger; +import org.globus.cog.abstraction.impl.common.execution.WallTime; +import org.globus.cog.abstraction.interfaces.JobSpecification; import org.globus.cog.abstraction.interfaces.Task; import org.globus.cog.karajan.scheduler.Scheduler; import org.griphyn.vdl.util.VDL2Config; public class ReplicationManager { - public static final Logger logger = Logger.getLogger(ReplicationManager.class); + public static final Logger logger = Logger + .getLogger(ReplicationManager.class); - public static final int STATUS_NEEDS_REPLICATION = 100; - - public static final int INITIAL_QUEUE_TIME_ESTIMATE = 30; //seconds + public static final int STATUS_NEEDS_REPLICATION = 100; - private int n; - private long s; - private long s2; - private Map queued; - private int minQueueTime, limit; - private boolean enabled; - private ReplicationGroups replicationGroups; - private Scheduler scheduler; + public static final int INITIAL_QUEUE_TIME_ESTIMATE = 30; // seconds + + public static final int WALLTIME_DEADLINE_MULTIPLIER = 2; - public ReplicationManager(Scheduler scheduler) { - this.replicationGroups = new ReplicationGroups(scheduler); - queued = new HashMap(); - try { - minQueueTime = Integer.parseInt(VDL2Config.getConfig().getProperty( - "replication.min.queue.time")); - enabled = Boolean.valueOf(VDL2Config.getConfig().getProperty("replication.enabled")).booleanValue(); - limit = Integer.parseInt(VDL2Config.getConfig().getProperty("replication.limit")); - } - catch (Exception e) { - logger.warn("Failed to get value of replication.min.queue.time property " - + "from Swift configuration. Using default (60s).", e); - minQueueTime = 60; - } - if (enabled) { - Sweeper.getSweeper().register(this); - } - } + private int n; + private long s; + private long s2; + private Map queued, running; + private int minQueueTime, limit; + private boolean enabled; + private ReplicationGroups replicationGroups; + private Scheduler scheduler; - public void register(String rg, Task task) throws CanceledReplicaException { - if (enabled) { - replicationGroups.add(rg, task); - } - } + public ReplicationManager(Scheduler scheduler) { + this.replicationGroups = new ReplicationGroups(scheduler); + this.scheduler = scheduler; + queued = new HashMap(); + running = new HashMap(); + try { + minQueueTime = Integer.parseInt(VDL2Config.getConfig().getProperty( + "replication.min.queue.time")); + enabled = Boolean.valueOf( + VDL2Config.getConfig().getProperty("replication.enabled")) + .booleanValue(); + limit = Integer.parseInt(VDL2Config.getConfig().getProperty( + "replication.limit")); + } + catch (Exception e) { + logger.warn( + "Failed to get value of replication.min.queue.time property " + + "from Swift configuration. Using default (60s).", + e); + minQueueTime = 60; + } + if (enabled) { + Sweeper.getSweeper().register(this); + } + } - public void submitted(Task task, Date time) { - if (enabled) { - synchronized (queued) { - queued.put(task, time); - } - } - } + public void register(String rg, Task task) throws CanceledReplicaException { + if (enabled) { + replicationGroups.add(rg, task); + } + } - public void active(Task task, Date time) { - if (enabled) { - Date submitted; - synchronized (queued) { - submitted = (Date) queued.remove(task); - } - if (submitted != null) { - long delta = (time.getTime() - submitted.getTime()) / 1000; - synchronized (this) { - n++; - s += delta; - s2 += delta * delta; - } - } - replicationGroups.active(task); - } - } + public void submitted(Task task, Date time) { + if (enabled) { + synchronized (this) { + queued.put(task, time); + } + } + } - public synchronized int getN() { - return n; - } + public void active(Task task, Date time) { + if (enabled) { + Date submitted; + synchronized (this) { + submitted = (Date) queued.remove(task); + registerRunning(task, time); + if (submitted != null) { + long delta = (time.getTime() - submitted.getTime()) / 1000; + n++; + s += delta; + s2 += delta * delta; + } + } + replicationGroups.active(task); + } + } - public synchronized double getMean() { - if (n == 0) { - return INITIAL_QUEUE_TIME_ESTIMATE; - } - else { - return s / n; - } - } + private static final Set warnedAboutWalltime = new HashSet(); - public synchronized double getStandardDeviation() { - if (n == 0) { - return 0; - } - else { - return Math.sqrt((s2 - s * s / n) / n); - } - } + protected void registerRunning(Task task, Date time) { + JobSpecification spec = (JobSpecification) task.getSpecification(); + Object walltime = spec.getAttribute("maxwalltime"); + String tr = (String) spec.getAttribute("tr"); + if (walltime == null) { + warn(tr, "Warning: missing walltime specification for \"" + tr + + "\". Assuming 10 minutes."); + walltime = "10"; + } + int seconds; + try { + seconds = WallTime.timeToSeconds(walltime.toString()); + } + catch (IllegalArgumentException e) { + warn(tr, "Warning: invalid walltime specification for \"" + tr + + "\" (" + walltime + "). Assuming 10 minutes."); + seconds = 10 * 60; + } + Date deadline = new Date(time.getTime() + WALLTIME_DEADLINE_MULTIPLIER * seconds * 1000); + running.put(task, deadline); + } - public void checkTasks() { - Map m; - synchronized (queued) { - m = new HashMap(queued); - } - Iterator i = m.entrySet().iterator(); - while (i.hasNext()) { - Map.Entry e = (Map.Entry) i.next(); - Task t = (Task) e.getKey(); - Date d = (Date) e.getValue(); - if (shouldBeReplicated(t, d)) { - replicationGroups.requestReplica(t); - } - } - } + private void warn(String tr, String message) { + synchronized (warnedAboutWalltime) { + if (warnedAboutWalltime.add(tr)) { + System.out.println(message); + } + } + } - private boolean shouldBeReplicated(Task t, Date d) { - if (t.getStatus().getStatusCode() == STATUS_NEEDS_REPLICATION) { - // don't keep replicating the same job - return false; - } - long inTheQueue = (System.currentTimeMillis() - d.getTime()) / 1000; - if (inTheQueue > minQueueTime && inTheQueue > 3 * getMean() - && replicationGroups.getRequestedCount(t) < limit) { - return true; - } - else { - return false; - } - } + public synchronized int getN() { + return n; + } + + public synchronized double getMean() { + if (n == 0) { + return INITIAL_QUEUE_TIME_ESTIMATE; + } + else { + return s / n; + } + } + + public synchronized double getStandardDeviation() { + if (n == 0) { + return 0; + } + else { + return Math.sqrt((s2 - s * s / n) / n); + } + } + + public void checkTasks() { + Map m, r; + synchronized (this) { + m = new HashMap(queued); + r = new HashMap(running); + } + Iterator i; + i = m.entrySet().iterator(); + while (i.hasNext()) { + Map.Entry e = (Map.Entry) i.next(); + Task t = (Task) e.getKey(); + Date d = (Date) e.getValue(); + if (shouldBeReplicated(t, d)) { + replicationGroups.requestReplica(t); + } + } + i = r.entrySet().iterator(); + while (i.hasNext()) { + Map.Entry e = (Map.Entry) i.next(); + Task t = (Task) e.getKey(); + Date d = (Date) e.getValue(); + if (isOverDeadline(t, d)) { + logger.info(t + ": deadline passed. Cancelling job."); + cancelTask(t); + } + } + } + + private boolean shouldBeReplicated(Task t, Date d) { + if (t.getStatus().getStatusCode() == STATUS_NEEDS_REPLICATION) { + // don't keep replicating the same job + return false; + } + long inTheQueue = (System.currentTimeMillis() - d.getTime()) / 1000; + if (inTheQueue > minQueueTime && inTheQueue > 3 * getMean() + && replicationGroups.getRequestedCount(t) < limit) { + return true; + } + else { + return false; + } + } + + private boolean isOverDeadline(Task t, Date d) { + if (System.currentTimeMillis() > d.getTime()) { + return true; + } + else { + return false; + } + } + + private void cancelTask(Task t) { + scheduler.cancelTask(t, "Walltime exceeded"); + } } \ No newline at end of file From noreply at svn.ci.uchicago.edu Fri Feb 6 21:08:56 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 6 Feb 2009 21:08:56 -0600 (CST) Subject: [Swift-commit] r2512 - trunk/tests/language-behaviour Message-ID: <20090207030857.137342281E6@www.ci.uchicago.edu> Author: hategan Date: 2009-02-06 21:08:56 -0600 (Fri, 06 Feb 2009) New Revision: 2512 Modified: trunk/tests/language-behaviour/run Log: option to allow attaching a debugger to a test Modified: trunk/tests/language-behaviour/run =================================================================== --- trunk/tests/language-behaviour/run 2009-02-07 03:05:49 UTC (rev 2511) +++ trunk/tests/language-behaviour/run 2009-02-07 03:08:56 UTC (rev 2512) @@ -20,6 +20,10 @@ SUB="y" shift TESTS="$@" +elif [ "$1" == "-d" ]; then + export COG_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,address=9999,server=y,suspend=y" + shift + TESTS="$@" else TESTS="$@" fi From noreply at svn.ci.uchicago.edu Fri Feb 6 21:09:36 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 6 Feb 2009 21:09:36 -0600 (CST) Subject: [Swift-commit] r2513 - trunk/tests/misc Message-ID: <20090207030936.234932281E6@www.ci.uchicago.edu> Author: hategan Date: 2009-02-06 21:09:35 -0600 (Fri, 06 Feb 2009) New Revision: 2513 Added: trunk/tests/misc/wonky-runawayjob.sh Log: test for runaway jobs Added: trunk/tests/misc/wonky-runawayjob.sh =================================================================== --- trunk/tests/misc/wonky-runawayjob.sh (rev 0) +++ trunk/tests/misc/wonky-runawayjob.sh 2009-02-07 03:09:35 UTC (rev 2513) @@ -0,0 +1,14 @@ +#!/bin/bash + +cd ../language-behaviour + +cat ../sites/tc.data > tmp.tc.data.sites + +SITE=wonky/wonky-runawayjob.xml + +echo testing site configuration: $SITE + +export SWIFT_TEST_PARAMS="-sites.file ../sites/${SITE} -tc.file tmp.tc.data.sites" + +./run 001-echo + Property changes on: trunk/tests/misc/wonky-runawayjob.sh ___________________________________________________________________ Name: svn:executable + * From noreply at svn.ci.uchicago.edu Sun Feb 8 13:51:21 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 8 Feb 2009 13:51:21 -0600 (CST) Subject: [Swift-commit] r2514 - trunk/tests/language Message-ID: <20090208195121.E451C2281F3@www.ci.uchicago.edu> Author: benc Date: 2009-02-08 13:51:21 -0600 (Sun, 08 Feb 2009) New Revision: 2514 Modified: trunk/tests/language/run Log: r2496 incorrectly removed a line of code, causing this test to not fail properly Modified: trunk/tests/language/run =================================================================== --- trunk/tests/language/run 2009-02-07 03:09:35 UTC (rev 2513) +++ trunk/tests/language/run 2009-02-08 19:51:21 UTC (rev 2514) @@ -37,6 +37,7 @@ if [ "$?" -ne "0" ]; then echo "OUTPUT DIFFERS" diff -u ../working-base/$b $b + exit 1 # echo press enter to accept this as valid or press ctrl-c to die # read # pushd .. From noreply at svn.ci.uchicago.edu Mon Feb 9 03:05:53 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Feb 2009 03:05:53 -0600 (CST) Subject: [Swift-commit] r2515 - trunk/src/org/griphyn/vdl/karajan/lib Message-ID: <20090209090553.68E3E22819E@www.ci.uchicago.edu> Author: benc Date: 2009-02-09 03:05:52 -0600 (Mon, 09 Feb 2009) New Revision: 2515 Modified: trunk/src/org/griphyn/vdl/karajan/lib/TCProfile.java Log: revert r2509: it breaks gram2 job submission, and appears unnecessary for core swift. Modified: trunk/src/org/griphyn/vdl/karajan/lib/TCProfile.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/TCProfile.java 2009-02-08 19:51:21 UTC (rev 2514) +++ trunk/src/org/griphyn/vdl/karajan/lib/TCProfile.java 2009-02-09 09:05:52 UTC (rev 2515) @@ -67,7 +67,6 @@ attrs = attributesFromTC(tce, attrs); } - attrs = addAttribute(attrs, "tr", tr); addAttributes(named, attrs); return null; } @@ -117,14 +116,6 @@ } named.add(GridExec.A_ATTRIBUTES, attrs); } - - private Map addAttribute(Map attrs, String key, Object value) { - if (attrs == null) { - attrs = new HashMap(); - } - attrs.put(key, value); - return attrs; - } private Map attributesFromTC(TransformationCatalogEntry tce, Map attrs) { List l = tce.getProfiles(Profile.GLOBUS); From noreply at svn.ci.uchicago.edu Mon Feb 9 07:01:29 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Feb 2009 07:01:29 -0600 (CST) Subject: [Swift-commit] r2516 - trunk/src/org/griphyn/vdl/karajan/lib Message-ID: <20090209130129.913C222819E@www.ci.uchicago.edu> Author: benc Date: 2009-02-09 07:01:28 -0600 (Mon, 09 Feb 2009) New Revision: 2516 Modified: trunk/src/org/griphyn/vdl/karajan/lib/RuntimeStats.java Log: tidy inconsistent indentation Modified: trunk/src/org/griphyn/vdl/karajan/lib/RuntimeStats.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/RuntimeStats.java 2009-02-09 09:05:52 UTC (rev 2515) +++ trunk/src/org/griphyn/vdl/karajan/lib/RuntimeStats.java 2009-02-09 13:01:28 UTC (rev 2516) @@ -24,7 +24,7 @@ public static final int MAX_PERIOD_MS=30000; public static final String[] preferredOutputOrder = { - "uninitialized", + "uninitialized", "Initializing", "Selecting site", "Stage in", @@ -94,21 +94,21 @@ public ProgressTicker() { super("Progress ticker"); try { - if ("true".equalsIgnoreCase(VDL2Config.getConfig().getProperty("ticker.disable"))) { - logger.info("Ticker disabled in configuration file"); - disabled = true; - } - } - catch (IOException e) { - logger.debug("Could not read swift properties", e); - } + if ("true".equalsIgnoreCase(VDL2Config.getConfig().getProperty("ticker.disable"))) { + logger.info("Ticker disabled in configuration file"); + disabled = true; + } + } + catch (IOException e) { + logger.debug("Could not read swift properties", e); + } } public void run() { - if (disabled) { - return; - } - while(true) { + if (disabled) { + return; + } + while(true) { dumpState(); try { @@ -121,9 +121,9 @@ } void dumpState() { - if (disabled) { - return; - } + if (disabled) { + return; + } long now = System.currentTimeMillis(); if(lastDumpTime + MIN_PERIOD_MS > now) return; lastDumpTime = now; @@ -131,9 +131,9 @@ } void finalDumpState() { - if (disabled) { - return; - } + if (disabled) { + return; + } printStates("Final status: "); } From noreply at svn.ci.uchicago.edu Mon Feb 9 07:42:54 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Feb 2009 07:42:54 -0600 (CST) Subject: [Swift-commit] r2517 - trunk/src/org/griphyn/vdl/engine Message-ID: <20090209134254.3D82B22819E@www.ci.uchicago.edu> Author: benc Date: 2009-02-09 07:42:53 -0600 (Mon, 09 Feb 2009) New Revision: 2517 Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java Log: xml validation errors shown at warning level - they should never appear in normal operation and they are useful to have very visible for debugging such things Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/Karajan.java 2009-02-09 13:01:28 UTC (rev 2516) +++ trunk/src/org/griphyn/vdl/engine/Karajan.java 2009-02-09 13:42:53 UTC (rev 2517) @@ -135,13 +135,11 @@ logger.info("Validation of XML intermediate file was successful"); } else { logger.warn("Validation of XML intermediate file failed."); - // these errors look rather scary, so output them at - // debug level - logger.debug("Validation errors:"); + logger.warn("Validation errors:"); Iterator i = errors.iterator(); while(i.hasNext()) { XmlError error = (XmlError) i.next(); - logger.debug(error.toString()); + logger.warn(error.toString()); } System.exit(3); } From noreply at svn.ci.uchicago.edu Mon Feb 9 08:12:57 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Feb 2009 08:12:57 -0600 (CST) Subject: [Swift-commit] r2519 - trunk/docs Message-ID: <20090209141257.B4BF62281AD@www.ci.uchicago.edu> Author: benc Date: 2009-02-09 08:12:56 -0600 (Mon, 09 Feb 2009) New Revision: 2519 Modified: trunk/docs/userguide.xml Log: comment that status.mode=provider does not work with coasters Modified: trunk/docs/userguide.xml =================================================================== --- trunk/docs/userguide.xml 2009-02-09 13:49:19 UTC (rev 2518) +++ trunk/docs/userguide.xml 2009-02-09 14:12:56 UTC (rev 2519) @@ -2832,14 +2832,39 @@ from workers to the submit side. In files mode, a file indicating success or failure will be created on the site shared filesystem. In provider mode, the execution provider job status will -be used. Notably, GRAM2 does not return job statuses correctly, and so -provider mode will not work with GRAM2. With other -providers, it can be used to reduce the amount of filesystem access compared -to files mode. (since Swift 0.8) +be used. + +provider mode requires the underlying job execution system +to correctly return exit codes. In at least the cases of GRAM2, and clusters +used with any provider, exit codes are not returned, and so +files mode must be used in those cases. Otherwise, +provider mode can be used to reduce the amount of +filesystem access. (since Swift 0.8) + + + + ticker.disable + + + + Valid values: true, false + + + Default value: false + + + +When set to true, suppresses the output progress ticker that Swift sends +to the console every few seconds during a run (since Swift 0.9) + + + + + From noreply at svn.ci.uchicago.edu Mon Feb 9 08:58:48 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Feb 2009 08:58:48 -0600 (CST) Subject: [Swift-commit] r2520 - in trunk: resources src/org/griphyn/vdl/engine tests/language-behaviour Message-ID: <20090209145848.387762281AD@www.ci.uchicago.edu> Author: benc Date: 2009-02-09 08:58:47 -0600 (Mon, 09 Feb 2009) New Revision: 2520 Added: trunk/tests/language-behaviour/089-nested-proc-call.swift Modified: trunk/resources/Karajan.stg trunk/resources/swiftscript.g trunk/resources/swiftscript.xsd trunk/src/org/griphyn/vdl/engine/Karajan.java trunk/src/org/griphyn/vdl/engine/VariableScope.java Log: permit procedure calls to be used inside expressions when they return a single value. mostly a patch from hategan with a small change by me. Modified: trunk/resources/Karajan.stg =================================================================== --- trunk/resources/Karajan.stg 2009-02-09 14:12:56 UTC (rev 2519) +++ trunk/resources/Karajan.stg 2009-02-09 14:58:47 UTC (rev 2520) @@ -294,6 +294,16 @@ >> +callexpr(call, datatype, callID) ::= << + + + + + $call$ + swift#callintermediate + +>> + array(elements, datatype) ::= << Modified: trunk/resources/swiftscript.g =================================================================== --- trunk/resources/swiftscript.g 2009-02-09 14:12:56 UTC (rev 2519) +++ trunk/resources/swiftscript.g 2009-02-09 14:58:47 UTC (rev 2520) @@ -669,6 +669,30 @@ RPAREN ; +procedureInvocationExpr [StringTemplate code] +{StringTemplate f=null;} + : + id:ID {code.setAttribute("func", id.getText());} + LPAREN ( + f=actualParameter { + code.setAttribute("inputs", f); + } + ( + COMMA f=actualParameter { + code.setAttribute("inputs", f); + } + )* + )? + RPAREN + ; + +procedureCallExpr returns [StringTemplate code=template("call")] +{StringTemplate f=null;} + : + procedureInvocationExpr[code] + ; + + predictProcedurecallDecl : ASSIGN ID LPAREN ; procedurecallDecl [StringTemplate container, StringTemplate type, StringTemplate decl, StringTemplate var] @@ -968,13 +992,17 @@ primExpr returns [StringTemplate code=null] {StringTemplate id=null, exp=null;} - : code=identifier + : (predictProcedureCallExpr) => code=procedureCallExpr + | code=identifier | LPAREN exp=orExpr RPAREN { code=template("paren"); code.setAttribute("exp", exp);} | code=constant | code=functionInvocation ; +predictProcedureCallExpr + : ID LPAREN ; + // TODO - redo identifier parsing to fit in with the XML style // that this patch introduces Modified: trunk/resources/swiftscript.xsd =================================================================== --- trunk/resources/swiftscript.xsd 2009-02-09 14:12:56 UTC (rev 2519) +++ trunk/resources/swiftscript.xsd 2009-02-09 14:58:47 UTC (rev 2520) @@ -556,6 +556,7 @@ + Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/Karajan.java 2009-02-09 14:12:56 UTC (rev 2519) +++ trunk/src/org/griphyn/vdl/engine/Karajan.java 2009-02-09 14:58:47 UTC (rev 2520) @@ -8,8 +8,8 @@ import java.util.Collection; import java.util.HashMap; import java.util.Iterator; -import java.util.List; import java.util.Map; + import javax.xml.namespace.QName; import org.antlr.stringtemplate.StringTemplate; @@ -24,34 +24,12 @@ import org.apache.xmlbeans.XmlObject; import org.apache.xmlbeans.XmlOptions; import org.apache.xmlbeans.XmlString; -import org.globus.swift.language.ActualParameter; -import org.globus.swift.language.ApplicationBinding; -import org.globus.swift.language.Array; -import org.globus.swift.language.Assign; -import org.globus.swift.language.BinaryOperator; -import org.globus.swift.language.Binding; -import org.globus.swift.language.Call; -import org.globus.swift.language.Foreach; -import org.globus.swift.language.FormalParameter; -import org.globus.swift.language.Function; -import org.globus.swift.language.If; -import org.globus.swift.language.Iterate; -import org.globus.swift.language.LabelledBinaryOperator; -import org.globus.swift.language.Procedure; -import org.globus.swift.language.ProgramDocument; -import org.globus.swift.language.Range; -import org.globus.swift.language.Switch; -import org.globus.swift.language.TypeRow; -import org.globus.swift.language.TypeStructure; -import org.globus.swift.language.UnlabelledUnaryOperator; -import org.globus.swift.language.Variable; +import org.globus.swift.language.*; import org.globus.swift.language.Variable.Mapping; import org.globus.swift.language.Variable.Mapping.Param; import org.globus.swift.language.If.Else; import org.globus.swift.language.If.Then; -import org.globus.swift.language.Procedure; import org.globus.swift.language.ProgramDocument.Program; -import org.globus.swift.language.StructureMember; import org.globus.swift.language.Switch.Case; import org.globus.swift.language.Switch.Default; import org.globus.swift.language.TypesDocument.Types; @@ -410,7 +388,7 @@ assign((Assign) child, scope); } else if (child instanceof Call) { - call((Call) child, scope); + call((Call) child, scope, false); } else if (child instanceof Foreach) { foreachStat((Foreach)child, scope); @@ -432,7 +410,7 @@ } } - public void call(Call call, VariableScope scope) throws CompilationException { + public StringTemplate call(Call call, VariableScope scope, boolean inhibitOutput) throws CompilationException { try { StringTemplate callST = template("call"); callST.setAttribute("func", call.getProc().getLocalPart()); @@ -593,7 +571,10 @@ } } - scope.appendStatement(callST); + if (!inhibitOutput) { + scope.appendStatement(callST); + } + return callST; } catch(CompilationException ce) { throw new CompilationException("Compile error in procedure invocation at "+call.getSrc()+": "+ce.getMessage(),ce); } @@ -854,6 +835,7 @@ static final QName ARRAY_EXPR = new QName(SWIFTSCRIPT_NS, "array"); static final QName RANGE_EXPR = new QName(SWIFTSCRIPT_NS, "range"); static final QName FUNCTION_EXPR = new QName(SWIFTSCRIPT_NS, "function"); + static final QName CALL_EXPR = new QName(SWIFTSCRIPT_NS, "call"); /** converts an XML intermediate form expression into a * Karajan expression. @@ -1102,6 +1084,34 @@ } else throw new CompilationException("Function " + name + " is not defined."); return st; + } else if (expressionQName.equals(CALL_EXPR)) { + Call c = (Call) expression; + c.addNewOutput(); + VariableScope subscope = new VariableScope(this, scope); + VariableReferenceDocument ref = VariableReferenceDocument.Factory.newInstance(); + ref.setVariableReference("swift#callintermediate"); + c.getOutputArray(0).set(ref); + String name = c.getProc().getLocalPart(); + ProcedureSignature funcSignature = (ProcedureSignature) proceduresMap.get(name); + + if (funcSignature == null) { + throw new CompilationException("Procedure " + name + " is not defined."); + } + + if (funcSignature.sizeOfOutputArray() != 1) { + throw new CompilationException("Procedure " + name + " must have exactly one " + + "return value to be used in an expression."); + } + + StringTemplate call = template("callexpr"); + + String type = funcSignature.getOutputArray(0).getType(); + subscope.addVariable("swift#callintermediate", type); + + call.setAttribute("datatype", type); + call.setAttribute("call", call(c, subscope, true)); + call.setAttribute("callID", new Integer(callID++)); + return call; } else { throw new CompilationException("unknown expression implemented by class "+expression.getClass()+" with node name "+expressionQName +" and with content "+expression); } Modified: trunk/src/org/griphyn/vdl/engine/VariableScope.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/VariableScope.java 2009-02-09 14:12:56 UTC (rev 2519) +++ trunk/src/org/griphyn/vdl/engine/VariableScope.java 2009-02-09 14:58:47 UTC (rev 2520) @@ -186,6 +186,9 @@ rather then walking the template. */ StringTemplate getLocalDeclaration(String name) { + if (bodyTemplate == null) { + return null; + } Object decls = bodyTemplate.getAttribute("declarations"); if(decls == null) return null; Added: trunk/tests/language-behaviour/089-nested-proc-call.swift =================================================================== --- trunk/tests/language-behaviour/089-nested-proc-call.swift (rev 0) +++ trunk/tests/language-behaviour/089-nested-proc-call.swift 2009-02-09 14:58:47 UTC (rev 2520) @@ -0,0 +1,21 @@ +type messagefile; + +(messagefile t) inner_greeting() { + app { + echo "hello" stdout=@filename(t); + } +} + +(string s) outer_greeting() { + messagefile m <"ssss">; + m = inner_greeting(); + s = readData(m); +} + + +// messagefile outfile <"089-nested-proc-call.swift">; + +// outfile = greeting(); + +trace(outer_greeting()); + From noreply at svn.ci.uchicago.edu Mon Feb 9 09:00:07 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Feb 2009 09:00:07 -0600 (CST) Subject: [Swift-commit] r2521 - trunk Message-ID: <20090209150007.61D6F2281AD@www.ci.uchicago.edu> Author: benc Date: 2009-02-09 09:00:07 -0600 (Mon, 09 Feb 2009) New Revision: 2521 Modified: trunk/CHANGES.txt Log: CHANGES for r2520 Modified: trunk/CHANGES.txt =================================================================== --- trunk/CHANGES.txt 2009-02-09 14:58:47 UTC (rev 2520) +++ trunk/CHANGES.txt 2009-02-09 15:00:07 UTC (rev 2521) @@ -1,4 +1,7 @@ (02/09/09) +*** Procedure invocations can be made in any expression context now, rather + than only directly in an assignment. + *** Mappings can now be made in any declaration, whether it has an assignment or not. Previous a procedure call assignment and a mapping could not be made in the same statement. From noreply at svn.ci.uchicago.edu Mon Feb 9 09:15:00 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Feb 2009 09:15:00 -0600 (CST) Subject: [Swift-commit] r2522 - trunk/docs Message-ID: <20090209151500.BFD542281DC@www.ci.uchicago.edu> Author: benc Date: 2009-02-09 09:14:59 -0600 (Mon, 09 Feb 2009) New Revision: 2522 Modified: trunk/docs/userguide.xml Log: reference from coasters section to coaster-relevant profile keys Modified: trunk/docs/userguide.xml =================================================================== --- trunk/docs/userguide.xml 2009-02-09 15:00:07 UTC (rev 2521) +++ trunk/docs/userguide.xml 2009-02-09 15:14:59 UTC (rev 2522) @@ -3475,6 +3475,22 @@ of a coaster head job, not for file transfer; so for example, a GRAM endpoint should be specified here rather than a GridFTP endpoint. + +Coasters are affected by the following profile settings, which are +documented in the Globus namespace profile +section: + + + + + profile keybrief description + + coastersPerNodeHow many coasters to run per execution node + coasterWorkerMaxwalltimeThe maxwalltime that coaster workers should be submitted with + coasterInternalIPThe IP address of the coaster head node to be used by workers + + +
How-To Tips for Specific User Communities
Saving Logs - for UChicago CI Users From noreply at svn.ci.uchicago.edu Tue Feb 10 13:09:48 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 10 Feb 2009 13:09:48 -0600 (CST) Subject: [Swift-commit] r2523 - SwiftApps/SIDGrid/config Message-ID: <20090210190949.218E9228154@www.ci.uchicago.edu> Author: skenny Date: 2009-02-10 13:09:46 -0600 (Tue, 10 Feb 2009) New Revision: 2523 Modified: SwiftApps/SIDGrid/config/tc.data Log: freesurfer wrappers installed on bsd cluster Modified: SwiftApps/SIDGrid/config/tc.data =================================================================== --- SwiftApps/SIDGrid/config/tc.data 2009-02-09 15:14:59 UTC (rev 2522) +++ SwiftApps/SIDGrid/config/tc.data 2009-02-10 19:09:46 UTC (rev 2523) @@ -295,6 +295,11 @@ BSD Rcmd /disks/gpfs/fmri/cnari/scripts/Rcmd.sh INSTALLED INTEL32::LINUX env::R_LIBS=/disks/gpfs/fmri/cnari/apps/R_library BSD Mediator /disks/gpfs/fmri/cnari/scripts/Mediator.py INSTALLED INTEL32::LINUX null BSD env /usr/env INSTALLED INTEL32::LINUX null +BSD recon1 /disks/ci-gpfs/fmri/cnari/apps/freesurfer/bin/recon1.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/disks/ci-gpfs/fmri/cnari/apps/freesurfer;GLOBUS::host_types=ia32-compute,maxwalltime=1200 +BSD recon2L /disks/ci-gpfs/fmri/cnari/apps/freesurfer/bin/recon2L.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/disks/ci-gpfs/fmri/cnari/apps/freesurfer;GLOBUS::host_types=ia32-compute,maxwalltime=1200 +BSD recon2R /disks/ci-gpfs/fmri/cnari/apps/freesurfer/bin/recon2R.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/disks/ci-gpfs/fmri/cnari/apps/freesurfer;GLOBUS::host_types=ia32-compute,maxwalltime=1200 +BSD autorecon2_BOTH /disks/ci-gpfs/fmri/cnari/apps/freesurfer/bin/autorecon2_BOTH.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/disks/ci-gpfs/fmri/cnari/apps/freesurfer;GLOBUS::host_types=ia32-compute,maxwalltime=1200 +BSD reconFINAL /disks/ci-gpfs/fmri/cnari/apps/freesurfer/bin/reconFINAL.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/disks/ci-gpfs/fmri/cnari/apps/freesurfer;GLOBUS::host_types=ia32-compute,maxwalltime=1200 ####-----------------ANLUCTERAGRID32----------------------------- ANLUCTERAGRID32 AFNI_1dcat /soft/community/SIDGrid/abin/1dcat INSTALLED INTEL32::LINUX null @@ -583,6 +588,7 @@ ANLUCTERAGRID32 recon1 /soft/community/SIDGrid/freesurfer/bin/recon1.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/soft/community/SIDGrid/freesurfer;GLOBUS::host_types=ia32-compute,maxwalltime=1200 ANLUCTERAGRID32 recon2L /soft/community/SIDGrid/freesurfer/bin/recon2L.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/soft/community/SIDGrid/freesurfer;GLOBUS::host_types=ia32-compute,maxwalltime=1200 ANLUCTERAGRID32 recon2R /soft/community/SIDGrid/freesurfer/bin/recon2R.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/soft/community/SIDGrid/freesurfer;GLOBUS::host_types=ia32-compute,maxwalltime=1200 +ANLUCTERAGRID32 autorecon2_BOTH /soft/community/SIDGrid/freesurfer/bin/autorecon2_BOTH.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/soft/community/SIDGrid/freesurfer;GLOBUS::host_types=ia32-compute,maxwalltime=1200 ANLUCTERAGRID32 reconFINAL /soft/community/SIDGrid/freesurfer/bin/reconFINAL.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/soft/community/SIDGrid/freesurfer;GLOBUS::host_types=ia32-compute,maxwalltime=1200 ANLUCTERAGRID32 R /soft/community/SIDGrid/R/bin/R INSTALLED INTEL32::LINUX null ANLUCTERAGRID32 RInvoke /soft/community/SIDGrid/R/bin/RInvoke.sh INSTALLED INTEL32::LINUX null From noreply at svn.ci.uchicago.edu Wed Feb 11 09:17:06 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Wed, 11 Feb 2009 09:17:06 -0600 (CST) Subject: [Swift-commit] r2524 - trunk/src/org/griphyn/vdl/engine Message-ID: <20090211151706.E705C22819E@www.ci.uchicago.edu> Author: benc Date: 2009-02-11 09:17:05 -0600 (Wed, 11 Feb 2009) New Revision: 2524 Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java Log: remove dead code in comment, and fix indentation Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/Karajan.java 2009-02-10 19:09:46 UTC (rev 2523) +++ trunk/src/org/griphyn/vdl/engine/Karajan.java 2009-02-11 15:17:05 UTC (rev 2524) @@ -252,73 +252,63 @@ checkIsTypeDefined(var.getType().getLocalPart()); if(!var.isNil()) { -/* this bit is never used now... but the stuff from dataset should be - here, and also the behaviour if a mapping is not specified needs to - be retained - StringTemplate exprST = expressionToKarajan(var.getAbstractExpression(),scope); - if (!datatype(exprST).equals(var.getType().getLocalPart())) - throw new CompilationException("Could not assign expression of type " + datatype(exprST) + - " to a variable of type " + var.getType().getLocalPart()); - variableST.setAttribute("expr", exprST); -*/ + if (var.getFile() != null) { + StringTemplate fileST = new StringTemplate("file"); + fileST.setAttribute("name", escapeQuotes(var.getFile().getName())); + fileST.defineFormalArgument("params"); + variableST.setAttribute("file", fileST); + } - if (var.getFile() != null) { - StringTemplate fileST = new StringTemplate("file"); - fileST.setAttribute("name", escapeQuotes(var.getFile().getName())); - fileST.defineFormalArgument("params"); - variableST.setAttribute("file", fileST); - } + Mapping mapping = var.getMapping(); - Mapping mapping = var.getMapping(); - - if (mapping != null) { - StringTemplate mappingST = new StringTemplate("mapping"); - mappingST.setAttribute("descriptor", mapping.getDescriptor()); - for (int i = 0; i < mapping.sizeOfParamArray(); i++) { - Param param = mapping.getParamArray(i); - StringTemplate paramST = template("vdl_parameter"); - paramST.setAttribute("name", param.getName()); - Node expressionDOM = param.getAbstractExpression().getDomNode(); - String namespaceURI = expressionDOM.getNamespaceURI(); - String localName = expressionDOM.getLocalName(); - QName expressionQName = new QName(namespaceURI, localName); - if(expressionQName.equals(VARIABLE_REFERENCE_EXPR)) { - paramST.setAttribute("expr",expressionToKarajan(param.getAbstractExpression(),scope)); - } else { - String parameterVariableName="swift#mapper#"+(internedIDCounter++); - // make template for variable declaration (need to compute type of this variable too?) - StringTemplate variableDeclarationST = template("variable"); - variableDeclarationST.setAttribute("waitfor",""); - // TODO factorise this and other code in variable()? - StringTemplate pmappingST = new StringTemplate("mapping"); - pmappingST.setAttribute("descriptor", "concurrent_mapper"); - StringTemplate pparamST = template("vdl_parameter"); - pparamST.setAttribute("name", "prefix"); - pparamST.setAttribute("expr", parameterVariableName + "-" + UUIDGenerator.getInstance().generateRandomBasedUUID().toString()); - pmappingST.setAttribute("params", pparamST); - variableDeclarationST.setAttribute("mapping", pmappingST); - variableDeclarationST.setAttribute("nil", Boolean.TRUE); - variableDeclarationST.setAttribute("name", parameterVariableName); - scope.bodyTemplate.setAttribute("declarations",variableDeclarationST); - StringTemplate paramValueST=expressionToKarajan(param.getAbstractExpression(),scope); - String paramValueType = datatype(paramValueST); - scope.addVariable(parameterVariableName, paramValueType); - variableDeclarationST.setAttribute("type", paramValueType); - StringTemplate variableReferenceST = template("id"); - variableReferenceST.setAttribute("var",parameterVariableName); - StringTemplate variableAssignmentST = template("assign"); - variableAssignmentST.setAttribute("var",variableReferenceST); - variableAssignmentST.setAttribute("value",paramValueST); - scope.appendStatement(variableAssignmentST); - paramST.setAttribute("expr",variableReferenceST); + if (mapping != null) { + StringTemplate mappingST = new StringTemplate("mapping"); + mappingST.setAttribute("descriptor", mapping.getDescriptor()); + for (int i = 0; i < mapping.sizeOfParamArray(); i++) { + Param param = mapping.getParamArray(i); + StringTemplate paramST = template("vdl_parameter"); + paramST.setAttribute("name", param.getName()); + Node expressionDOM = param.getAbstractExpression().getDomNode(); + String namespaceURI = expressionDOM.getNamespaceURI(); + String localName = expressionDOM.getLocalName(); + QName expressionQName = new QName(namespaceURI, localName); + if(expressionQName.equals(VARIABLE_REFERENCE_EXPR)) { + paramST.setAttribute("expr",expressionToKarajan(param.getAbstractExpression(),scope)); + } else { + String parameterVariableName="swift#mapper#"+(internedIDCounter++); + // make template for variable declaration (need to compute type of this variable too?) + StringTemplate variableDeclarationST = template("variable"); + variableDeclarationST.setAttribute("waitfor",""); + // TODO factorise this and other code in variable()? + StringTemplate pmappingST = new StringTemplate("mapping"); + pmappingST.setAttribute("descriptor", "concurrent_mapper"); + StringTemplate pparamST = template("vdl_parameter"); + pparamST.setAttribute("name", "prefix"); + pparamST.setAttribute("expr", parameterVariableName + "-" + UUIDGenerator.getInstance().generateRandomBasedUUID().toString()); + pmappingST.setAttribute("params", pparamST); + variableDeclarationST.setAttribute("mapping", pmappingST); + variableDeclarationST.setAttribute("nil", Boolean.TRUE); + variableDeclarationST.setAttribute("name", parameterVariableName); + scope.bodyTemplate.setAttribute("declarations",variableDeclarationST); + StringTemplate paramValueST=expressionToKarajan(param.getAbstractExpression(),scope); + String paramValueType = datatype(paramValueST); + scope.addVariable(parameterVariableName, paramValueType); + variableDeclarationST.setAttribute("type", paramValueType); + StringTemplate variableReferenceST = template("id"); + variableReferenceST.setAttribute("var",parameterVariableName); + StringTemplate variableAssignmentST = template("assign"); + variableAssignmentST.setAttribute("var",variableReferenceST); + variableAssignmentST.setAttribute("value",paramValueST); + scope.appendStatement(variableAssignmentST); + paramST.setAttribute("expr",variableReferenceST); + } + mappingST.setAttribute("params", paramST); } - mappingST.setAttribute("params", paramST); + variableST.setAttribute("mapping", mappingST); } - variableST.setAttribute("mapping", mappingST); - } } else { // add temporary mapping info StringTemplate mappingST = new StringTemplate("mapping"); From noreply at svn.ci.uchicago.edu Wed Feb 11 19:16:27 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Wed, 11 Feb 2009 19:16:27 -0600 (CST) Subject: [Swift-commit] r2525 - in trunk/src/org/griphyn/vdl/karajan/lib: . replication Message-ID: <20090212011627.4262B2281F4@www.ci.uchicago.edu> Author: hategan Date: 2009-02-11 19:16:26 -0600 (Wed, 11 Feb 2009) New Revision: 2525 Modified: trunk/src/org/griphyn/vdl/karajan/lib/TCProfile.java trunk/src/org/griphyn/vdl/karajan/lib/replication/ReplicationManager.java Log: moved walltime check to TCProfile to avoid passing the tr in an attribute (which causes GT2 to bark) Modified: trunk/src/org/griphyn/vdl/karajan/lib/TCProfile.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/TCProfile.java 2009-02-11 15:17:05 UTC (rev 2524) +++ trunk/src/org/griphyn/vdl/karajan/lib/TCProfile.java 2009-02-12 01:16:26 UTC (rev 2525) @@ -3,11 +3,15 @@ */ package org.griphyn.vdl.karajan.lib; +import java.util.Collections; import java.util.HashMap; +import java.util.HashSet; import java.util.Iterator; import java.util.List; import java.util.Map; +import java.util.Set; +import org.globus.cog.abstraction.impl.common.execution.WallTime; import org.globus.cog.karajan.arguments.Arg; import org.globus.cog.karajan.arguments.ArgUtil; import org.globus.cog.karajan.arguments.NamedArguments; @@ -67,9 +71,50 @@ attrs = attributesFromTC(tce, attrs); } + + attrs = checkWalltime(tr, attrs); + addAttributes(named, attrs); return null; } + + private Map checkWalltime(String tr, Map attrs) { + Object walltime = null; + if (attrs != null) { + walltime = attrs.get("maxwalltime"); + } + if (walltime == null) { + warn(tr, "Warning: missing walltime specification for \"" + tr + + "\". Assuming 10 minutes."); + walltime = "10"; + } + int seconds; + try { + seconds = WallTime.timeToSeconds(walltime.toString()); + } + catch (IllegalArgumentException e) { + warn(tr, "Warning: invalid walltime specification for \"" + tr + + "\" (" + walltime + "). Assuming 10 minutes."); + walltime = "10"; + } + if (attrs == null) { + attrs = Collections.singletonMap("maxwalltime", walltime); + } + else { + attrs.put("maxwalltime", walltime); + } + return attrs; + } + + private static final Set warnedAboutWalltime = new HashSet(); + + private void warn(String tr, String message) { + synchronized (warnedAboutWalltime) { + if (warnedAboutWalltime.add(tr)) { + System.out.println(message); + } + } + } private void addEnvironment(Map m, TransformationCatalogEntry tce) { List l = tce.getProfiles(Profile.ENV); Modified: trunk/src/org/griphyn/vdl/karajan/lib/replication/ReplicationManager.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/replication/ReplicationManager.java 2009-02-11 15:17:05 UTC (rev 2524) +++ trunk/src/org/griphyn/vdl/karajan/lib/replication/ReplicationManager.java 2009-02-12 01:16:26 UTC (rev 2525) @@ -93,38 +93,14 @@ } } - private static final Set warnedAboutWalltime = new HashSet(); - protected void registerRunning(Task task, Date time) { JobSpecification spec = (JobSpecification) task.getSpecification(); Object walltime = spec.getAttribute("maxwalltime"); - String tr = (String) spec.getAttribute("tr"); - if (walltime == null) { - warn(tr, "Warning: missing walltime specification for \"" + tr - + "\". Assuming 10 minutes."); - walltime = "10"; - } - int seconds; - try { - seconds = WallTime.timeToSeconds(walltime.toString()); - } - catch (IllegalArgumentException e) { - warn(tr, "Warning: invalid walltime specification for \"" + tr - + "\" (" + walltime + "). Assuming 10 minutes."); - seconds = 10 * 60; - } + int seconds = WallTime.timeToSeconds(walltime.toString()); Date deadline = new Date(time.getTime() + WALLTIME_DEADLINE_MULTIPLIER * seconds * 1000); running.put(task, deadline); } - private void warn(String tr, String message) { - synchronized (warnedAboutWalltime) { - if (warnedAboutWalltime.add(tr)) { - System.out.println(message); - } - } - } - public synchronized int getN() { return n; } From noreply at svn.ci.uchicago.edu Wed Feb 11 19:32:22 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Wed, 11 Feb 2009 19:32:22 -0600 (CST) Subject: [Swift-commit] r2526 - trunk/src/org/griphyn/vdl/engine Message-ID: <20090212013222.6EA8C228211@www.ci.uchicago.edu> Author: hategan Date: 2009-02-11 19:32:21 -0600 (Wed, 11 Feb 2009) New Revision: 2526 Modified: trunk/src/org/griphyn/vdl/engine/FormalArgumentSignature.java Log: added toString() Modified: trunk/src/org/griphyn/vdl/engine/FormalArgumentSignature.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/FormalArgumentSignature.java 2009-02-12 01:16:26 UTC (rev 2525) +++ trunk/src/org/griphyn/vdl/engine/FormalArgumentSignature.java 2009-02-12 01:32:21 UTC (rev 2526) @@ -49,4 +49,8 @@ public void setOptional(boolean optionalArg) { this.optionalArg = optionalArg; } + + public String toString() { + return type + " " + name; + } } \ No newline at end of file From noreply at svn.ci.uchicago.edu Wed Feb 11 19:32:50 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Wed, 11 Feb 2009 19:32:50 -0600 (CST) Subject: [Swift-commit] r2527 - trunk/src/org/griphyn/vdl/engine Message-ID: <20090212013250.1F11B228211@www.ci.uchicago.edu> Author: hategan Date: 2009-02-11 19:32:49 -0600 (Wed, 11 Feb 2009) New Revision: 2527 Modified: trunk/src/org/griphyn/vdl/engine/ProcedureSignature.java Log: added toString() and fixed get*Array() methods Modified: trunk/src/org/griphyn/vdl/engine/ProcedureSignature.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/ProcedureSignature.java 2009-02-12 01:32:21 UTC (rev 2526) +++ trunk/src/org/griphyn/vdl/engine/ProcedureSignature.java 2009-02-12 01:32:49 UTC (rev 2527) @@ -59,12 +59,17 @@ return outputArgs.size(); } + private static final FormalArgumentSignature[] FORMAL_ARGUMENT_SIGNATURE_ARRAY = + new FormalArgumentSignature[0]; + public FormalArgumentSignature[] getInputArray() { - return (FormalArgumentSignature[]) inputArgs.toArray(); + return (FormalArgumentSignature[]) + inputArgs.toArray(FORMAL_ARGUMENT_SIGNATURE_ARRAY); } public FormalArgumentSignature[] getOutputArray() { - return (FormalArgumentSignature[]) outputArgs.toArray(); + return (FormalArgumentSignature[]) + outputArgs.toArray(FORMAL_ARGUMENT_SIGNATURE_ARRAY); } public FormalArgumentSignature getInputArray(int i) { @@ -201,4 +206,7 @@ return functionsMap; } + public String toString() { + return outputArgs + " " + name + inputArgs; + } } From noreply at svn.ci.uchicago.edu Thu Feb 12 04:01:55 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Thu, 12 Feb 2009 04:01:55 -0600 (CST) Subject: [Swift-commit] r2528 - www/downloads Message-ID: <20090212100155.F077C2281F3@www.ci.uchicago.edu> Author: benc Date: 2009-02-12 04:01:55 -0600 (Thu, 12 Feb 2009) New Revision: 2528 Modified: www/downloads/index.php Log: link to stripped 0.8 release Modified: www/downloads/index.php =================================================================== --- www/downloads/index.php 2009-02-12 01:32:49 UTC (rev 2527) +++ www/downloads/index.php 2009-02-12 10:01:55 UTC (rev 2528) @@ -31,6 +31,7 @@

Swift v0.8 is a development release intended to release functionality and fixes that have gone in to trunk since v0.7. [swift-0.8.tar.gz] +[swift-0.8-stripped.tar.gz - for installation on systems with existing grid stacks] [release-notes-0.8.txt]

- - + + + + + - /disks/ci-gpfs/fmri/swift/work + /disks/ci-gpfs/fmri/swift/work/{username} From noreply at svn.ci.uchicago.edu Tue Feb 17 07:16:03 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 17 Feb 2009 07:16:03 -0600 (CST) Subject: [Swift-commit] r2554 - trunk/libexec Message-ID: <20090217131603.2761D2281A1@www.ci.uchicago.edu> Author: benc Date: 2009-02-17 07:16:01 -0600 (Tue, 17 Feb 2009) New Revision: 2554 Modified: trunk/libexec/wrapper.sh Log: log stdout and stderr to the -info file when a job fails, to aid in debugging those jobs Modified: trunk/libexec/wrapper.sh =================================================================== --- trunk/libexec/wrapper.sh 2009-02-16 21:20:35 UTC (rev 2553) +++ trunk/libexec/wrapper.sh 2009-02-17 13:16:01 UTC (rev 2554) @@ -24,6 +24,14 @@ cat /proc/meminfo 2>&1 >& "$INFO" infosection "command line" echo $COMMANDLINE 2>&1 >& "$INFO" + if [ -f "$STDOUT" ] ; then + infosection "stdout" + cat $STDOUT >& "$INFO" + fi + if [ -f "$STDERR" ] ; then + infosection "stderr" + cat $STDERR >& "$INFO" + fi } logstate() { @@ -245,8 +253,6 @@ fail 254 "The executable $EXEC does not have the executable bit set" fi if [ "$KICKSTART" == "" ]; then -log "command line is $EXEC $@" -log "pwd is $(pwd)" if [ "$STDIN" == "" ]; then "$EXEC" "$@" 1>"$STDOUT" 2>"$STDERR" else From noreply at svn.ci.uchicago.edu Tue Feb 17 07:16:41 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 17 Feb 2009 07:16:41 -0600 (CST) Subject: [Swift-commit] r2555 - trunk/libexec Message-ID: <20090217131641.580D62281A1@www.ci.uchicago.edu> Author: benc Date: 2009-02-17 07:16:40 -0600 (Tue, 17 Feb 2009) New Revision: 2555 Modified: trunk/libexec/seq.sh Log: cluster seq.sh argument list was not being cleared between job invocations in the same cluster. When a job was run with a shorter argument list than a previous job, the present job would incorrectly inherit additional parameters from that previous job. This commit fixes that. Modified: trunk/libexec/seq.sh =================================================================== --- trunk/libexec/seq.sh 2009-02-17 13:16:01 UTC (rev 2554) +++ trunk/libexec/seq.sh 2009-02-17 13:16:40 UTC (rev 2555) @@ -28,9 +28,10 @@ EXEC="$1" shift +# we go round this loop once for each clustered job while [ "$EXEC" != "" ]; do echo `date +%s` LOOP-START >> $WRAPPERLOG - ARGS= + declare -a ARGS INDEX=0 ARG="$1" @@ -56,7 +57,7 @@ echo `date +%s` FAILED >> $WRAPPERLOG exit $EXITCODE fi - + unset ARGS EXEC="$1" shift done From noreply at svn.ci.uchicago.edu Tue Feb 17 08:20:30 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 17 Feb 2009 08:20:30 -0600 (CST) Subject: [Swift-commit] r2556 - log-processing/bin Message-ID: <20090217142030.B9AAC228187@www.ci.uchicago.edu> Author: benc Date: 2009-02-17 08:20:30 -0600 (Tue, 17 Feb 2009) New Revision: 2556 Added: log-processing/bin/swift-log-to-cedps Removed: log-processing/bin/swift2cedps Log: rename CEDPS log formatting command to something more inline with other swift commands Copied: log-processing/bin/swift-log-to-cedps (from rev 2555, log-processing/bin/swift2cedps) =================================================================== --- log-processing/bin/swift-log-to-cedps (rev 0) +++ log-processing/bin/swift-log-to-cedps 2009-02-17 14:20:30 UTC (rev 2556) @@ -0,0 +1,52 @@ +#!/bin/bash + +ORIGDIR=$(pwd) +WORKINGDIR=$(mktemp -d /tmp/swift-plot-log-XXXXXXXXXXXXXXXX) + + +# $1 should be the pathname of the log file to plot + +LOG_CODE_HOME="`dirname $0`/../libexec/" +export SWIFT_PLOT_HOME=$LOG_CODE_HOME + +export PATH=${LOG_CODE_HOME}:${PATH} + +FC="${1:0:1}" + +if [ "$FC" = "/" ]; then # absolute path + LOG_FILE_PATH=$1 +else + LOG_FILE_PATH="`pwd`/$1" +fi + +LOG_DIRECTORY=`dirname $LOG_FILE_PATH` + +LOG_FILE_BASE=`basename $LOG_FILE_PATH .log` + +echo Log file path is $LOG_FILE_PATH +echo Log is in directory $LOG_DIRECTORY +echo Log basename is $LOG_FILE_BASE + +cd $WORKINGDIR +echo Now in directory $(pwd) + +MAKEENV="-f ${LOG_CODE_HOME}/makefile -I ${LOG_CODE_HOME}" +MAKETARGETS="" + +INFODIR=${LOG_DIRECTORY}/${LOG_FILE_BASE}.d + +if [ -d $INFODIR ]; then + MAKEENV="$MAKEENV IDIR=${LOG_DIRECTORY}/${LOG_FILE_BASE}.d/" + MAKETARGETS="webpage.info" +fi + +make $MAKEENV SDL=${ORIGDIR}/report-${LOG_FILE_BASE} LOG=$LOG_FILE_PATH clean execute.cedps execute2.cedps karatasks.cedps + +cp -v *.cedps $ORIGDIR + +# webpage.kara webpage.weights karatasks.JOB_SUBMISSION.Queue.transitions karatasks.JOB_SUBMISSION.Queue.event karatasks.JOB_SUBMISSION.eip $MAKETARGETS webpage distribute + +# mv report-$LOG_FILE_BASE $LOG_DIRECTORY/ + +# rm -r $WORKINGDIR +echo not removing $WORKINGDIR Property changes on: log-processing/bin/swift-log-to-cedps ___________________________________________________________________ Name: svn:executable + * Name: svn:mergeinfo + Deleted: log-processing/bin/swift2cedps =================================================================== --- log-processing/bin/swift2cedps 2009-02-17 13:16:40 UTC (rev 2555) +++ log-processing/bin/swift2cedps 2009-02-17 14:20:30 UTC (rev 2556) @@ -1,52 +0,0 @@ -#!/bin/bash - -ORIGDIR=$(pwd) -WORKINGDIR=$(mktemp -d /tmp/swift-plot-log-XXXXXXXXXXXXXXXX) - - -# $1 should be the pathname of the log file to plot - -LOG_CODE_HOME="`dirname $0`/../libexec/" -export SWIFT_PLOT_HOME=$LOG_CODE_HOME - -export PATH=${LOG_CODE_HOME}:${PATH} - -FC="${1:0:1}" - -if [ "$FC" = "/" ]; then # absolute path - LOG_FILE_PATH=$1 -else - LOG_FILE_PATH="`pwd`/$1" -fi - -LOG_DIRECTORY=`dirname $LOG_FILE_PATH` - -LOG_FILE_BASE=`basename $LOG_FILE_PATH .log` - -echo Log file path is $LOG_FILE_PATH -echo Log is in directory $LOG_DIRECTORY -echo Log basename is $LOG_FILE_BASE - -cd $WORKINGDIR -echo Now in directory $(pwd) - -MAKEENV="-f ${LOG_CODE_HOME}/makefile -I ${LOG_CODE_HOME}" -MAKETARGETS="" - -INFODIR=${LOG_DIRECTORY}/${LOG_FILE_BASE}.d - -if [ -d $INFODIR ]; then - MAKEENV="$MAKEENV IDIR=${LOG_DIRECTORY}/${LOG_FILE_BASE}.d/" - MAKETARGETS="webpage.info" -fi - -make $MAKEENV SDL=${ORIGDIR}/report-${LOG_FILE_BASE} LOG=$LOG_FILE_PATH clean execute.cedps execute2.cedps karatasks.cedps - -cp -v *.cedps $ORIGDIR - -# webpage.kara webpage.weights karatasks.JOB_SUBMISSION.Queue.transitions karatasks.JOB_SUBMISSION.Queue.event karatasks.JOB_SUBMISSION.eip $MAKETARGETS webpage distribute - -# mv report-$LOG_FILE_BASE $LOG_DIRECTORY/ - -# rm -r $WORKINGDIR -echo not removing $WORKINGDIR From noreply at svn.ci.uchicago.edu Tue Feb 17 09:09:18 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 17 Feb 2009 09:09:18 -0600 (CST) Subject: [Swift-commit] r2557 - trunk/src/org/griphyn/vdl/type Message-ID: <20090217150918.2AEF6228187@www.ci.uchicago.edu> Author: benc Date: 2009-02-17 09:09:17 -0600 (Tue, 17 Feb 2009) New Revision: 2557 Modified: trunk/src/org/griphyn/vdl/type/Types.java Log: Remove the instance methods of Types as they are unused. Types is used solely for its static methods. Modified: trunk/src/org/griphyn/vdl/type/Types.java =================================================================== --- trunk/src/org/griphyn/vdl/type/Types.java 2009-02-17 14:20:30 UTC (rev 2556) +++ trunk/src/org/griphyn/vdl/type/Types.java 2009-02-17 15:09:17 UTC (rev 2557) @@ -4,28 +4,13 @@ import java.util.Iterator; import java.util.Map; -public class Types { +public abstract class Types { //TODO: check namespace references in type definitions private static Map types = new HashMap(); private String namespace = null; - public Types() { - } - - public Types(String namespace) { - this.namespace = namespace; - } - - public String getNamespace() { - return namespace; - } - - public void setNamespace(String namespace) { - this.namespace = namespace; - } - public synchronized static Type getType(String name) throws NoSuchTypeException { Type type = (Type) types.get(name); if (type == null) { From noreply at svn.ci.uchicago.edu Tue Feb 17 09:10:22 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 17 Feb 2009 09:10:22 -0600 (CST) Subject: [Swift-commit] r2558 - trunk/src/org/griphyn/vdl/karajan/lib Message-ID: <20090217151022.98C8C228187@www.ci.uchicago.edu> Author: benc Date: 2009-02-17 09:10:21 -0600 (Tue, 17 Feb 2009) New Revision: 2558 Modified: trunk/src/org/griphyn/vdl/karajan/lib/CacheFunction.java Log: Remove unused CacheEntry inner class from CacheFunction. Modified: trunk/src/org/griphyn/vdl/karajan/lib/CacheFunction.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/CacheFunction.java 2009-02-17 15:09:17 UTC (rev 2557) +++ trunk/src/org/griphyn/vdl/karajan/lib/CacheFunction.java 2009-02-17 15:10:21 UTC (rev 2558) @@ -28,26 +28,4 @@ } return cache; } - - public static class CacheEntry { - public String fullPath; - public Object host; - - public CacheEntry(String fullPath, Object host) { - this.fullPath = fullPath; - this.host = host; - } - - public boolean equals(Object other) { - if (other instanceof CacheEntry) { - CacheEntry ce = (CacheEntry) other; - return fullPath.equals(ce.fullPath) && host.equals(ce.host); - } - return false; - } - - public int hashCode() { - return fullPath.hashCode() + host.hashCode(); - } - } } From noreply at svn.ci.uchicago.edu Tue Feb 17 09:11:46 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 17 Feb 2009 09:11:46 -0600 (CST) Subject: [Swift-commit] r2559 - in trunk: libexec src/org/griphyn/vdl/karajan/lib Message-ID: <20090217151146.E7E1C2281AD@www.ci.uchicago.edu> Author: benc Date: 2009-02-17 09:11:43 -0600 (Tue, 17 Feb 2009) New Revision: 2559 Removed: trunk/src/org/griphyn/vdl/karajan/lib/Assign.java Modified: trunk/libexec/vdl-lib.xml Log: Assign karajan function is not used, so remove. Modified: trunk/libexec/vdl-lib.xml =================================================================== --- trunk/libexec/vdl-lib.xml 2009-02-17 15:10:21 UTC (rev 2558) +++ trunk/libexec/vdl-lib.xml 2009-02-17 15:11:43 UTC (rev 2559) @@ -51,7 +51,6 @@ - Deleted: trunk/src/org/griphyn/vdl/karajan/lib/Assign.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/Assign.java 2009-02-17 15:10:21 UTC (rev 2558) +++ trunk/src/org/griphyn/vdl/karajan/lib/Assign.java 2009-02-17 15:11:43 UTC (rev 2559) @@ -1,79 +0,0 @@ -/* - * Created on Dec 26, 2006 - */ -package org.griphyn.vdl.karajan.lib; - -import java.util.Iterator; -import java.util.List; - -import org.globus.cog.karajan.arguments.Arg; -import org.globus.cog.karajan.stack.VariableStack; -import org.globus.cog.karajan.workflow.ExecutionException; -import org.griphyn.vdl.mapping.DSHandle; -import org.griphyn.vdl.mapping.InvalidPathException; -import org.griphyn.vdl.mapping.Path; - -public class Assign extends VDLFunction { - public static final Arg A_DVAR = new Arg.TypedPositional("dvar", DSHandle.class, "handle"); - public static final Arg A_SVAR = new Arg.Positional("svar"); - public static final Arg A_DPATH = new Arg.Optional("dpath", ""); - public static final Arg A_SPATH = new Arg.Optional("spath", ""); - - static { - setArguments(Assign.class, new Arg[] { A_DVAR, A_SVAR, A_DPATH, A_SPATH }); - } - - public Object function(VariableStack stack) throws ExecutionException { - Path dpath = parsePath(A_DPATH.getValue(stack), stack); - Path spath = parsePath(A_SPATH.getValue(stack), stack); - DSHandle dvar = (DSHandle) A_DVAR.getValue(stack); - Object s = A_SVAR.getValue(stack); - if (s == null) { - throw new ExecutionException("Source variable is null"); - } - try { - dvar = dvar.getField(dpath); - if (s instanceof List) { - if (!Path.EMPTY_PATH.equals(spath)) { - throw new ExecutionException( - "If the source is an array there can be no source path (" + spath + ")"); - } - int index = 0; - Iterator i = ((List) s).iterator(); - while (i.hasNext()) { - Object n = i.next(); - Path p = Path.EMPTY_PATH.addLast(String.valueOf(index), true); - if (n instanceof DSHandle) { - dvar.getField(p).set((DSHandle) n); - } - else { - DSHandle field = dvar.getField(p); - field.setValue(n); - closeShallow(stack, field); - } - index++; - } - closeShallow(stack, dvar); - } - else if (s instanceof DSHandle) { - DSHandle svar = (DSHandle) s; - svar = svar.getField(spath); - /* - * Iterator i = svar.getFringePaths().iterator(); while - * (i.hasNext()) { String leafSPath = (String) i.next(); Path - * leafPath = Path.parse(leafSPath); mergeListeners(stack, - * dvar.getField(leafPath), svar.getField(leafPath)); } - */ - dvar.set(svar); - } - else { - dvar.setValue(s); - closeShallow(stack, dvar); - } - } - catch (InvalidPathException e) { - throw new ExecutionException(e); - } - return null; - } -} From noreply at svn.ci.uchicago.edu Tue Feb 17 09:12:47 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 17 Feb 2009 09:12:47 -0600 (CST) Subject: [Swift-commit] r2560 - trunk/src/org/griphyn/vdl/karajan Message-ID: <20090217151248.0D5A722819E@www.ci.uchicago.edu> Author: benc Date: 2009-02-17 09:12:47 -0600 (Tue, 17 Feb 2009) New Revision: 2560 Removed: trunk/src/org/griphyn/vdl/karajan/ClosedHandle.java Log: Remove unused ClosedHandle class Deleted: trunk/src/org/griphyn/vdl/karajan/ClosedHandle.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/ClosedHandle.java 2009-02-17 15:11:43 UTC (rev 2559) +++ trunk/src/org/griphyn/vdl/karajan/ClosedHandle.java 2009-02-17 15:12:47 UTC (rev 2560) @@ -1,17 +0,0 @@ -/* - * Created on Jun 17, 2006 - */ -package org.griphyn.vdl.karajan; - - -public class ClosedHandle extends DSHandleFutureWrapper { - - public ClosedHandle() { - super(null); - } - - public String toString() { - return "Closed"; - } - -} From noreply at svn.ci.uchicago.edu Tue Feb 17 10:42:29 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 17 Feb 2009 10:42:29 -0600 (CST) Subject: [Swift-commit] r2561 - trunk/bin Message-ID: <20090217164229.BB0942281A1@www.ci.uchicago.edu> Author: benc Date: 2009-02-17 10:42:29 -0600 (Tue, 17 Feb 2009) New Revision: 2561 Added: trunk/bin/swift-osg-ress-site-catalog Log: A script which queries the OSG Resource Selection System (ReSS) for site information and builds a Swift site catalog. This commit contributed by Mats Rynge Added: trunk/bin/swift-osg-ress-site-catalog =================================================================== --- trunk/bin/swift-osg-ress-site-catalog (rev 0) +++ trunk/bin/swift-osg-ress-site-catalog 2009-02-17 16:42:29 UTC (rev 2561) @@ -0,0 +1,155 @@ +#!/usr/bin/perl + +use strict; + +use Pod::Usage; +use Getopt::Long; +use File::Temp qw/ tempfile tempdir mktemp /; + +my $opt_help = 0; +my $opt_vo = 'engage'; +my $opt_engage_verified = 0; +my $opt_gt4 = 0; +my $opt_out = '&STDOUT'; + +Getopt::Long::Configure('bundling'); +GetOptions( + "help" => \$opt_help, + "vo=s" => \$opt_vo, + "engage-verified" => \$opt_engage_verified, + "gt4" => \$opt_gt4, + "out=s" => \$opt_out, +) or pod2usage(1); + +if ($opt_help) { + pod2usage(1); +} + +if ($opt_engage_verified && $opt_vo ne "engage") { + die("You can not specify a vo when using --engage-verified\n"); +} + +# make sure condor_status is in the path +my $out = `which condor_status 2>/dev/null`; +if ($out eq "") { + die("This tool depends on condor_status.\n" . + "Please make sure condor_status in your path.\n"); +} + +my %ads; +my %tmp; +my $cmd = "condor_status -any -long -constraint" . + " 'StringlistIMember(\"VO:$opt_vo\";GlueCEAccessControlBaseRule)'" . + " -pool osg-ress-1.fnal.gov"; +# if we want the engage verified sites, ignore opt_vo and query against +# engage central collector +if ($opt_engage_verified) { + $cmd = "condor_status -any -long -constraint" . + " 'SiteVerified==TRUE'" . + " -pool engage-central.renci.org" +} +open(STATUS, "$cmd|"); +while() { + chomp; + if ($_ eq "") { + if ($tmp{'GlueSiteName'} ne "") { + my %copy = %tmp; + $ads{$tmp{'GlueSiteName'}} = \%copy; + undef %tmp; + } + } + else { + my ($key, $value) = split(/ = /, $_, 2); + $value =~ s/^"|"$//g; # remove quotes from Condor strings + $tmp{$key} = $value; + } +} +close(STATUS); + +# lowercase vo +my $lc_vo = lc($opt_vo); + +open(FH, ">$opt_out") or die("Unable to open $opt_out"); +print FH "\n"; +foreach my $sitename (keys %ads) { + my $contact = $ads{$sitename}->{'GlueCEInfoContactString'}; + my $host = $contact; + $host =~ s/:.*//; + my $jm = $contact; + $jm =~ s/.*jobmanager-//; + if ($jm eq "pbs") { + $jm = "PBS"; + } + elsif ($jm eq "lsf") { + $jm = "LSF"; + } + elsif ($jm eq "sge") { + $jm = "SGE"; + } + elsif ($jm eq "condor") { + $jm = "Condor"; + } + my $workdir = $ads{$sitename}->{'GlueCEInfoDataDir'}; + print FH "\n"; + print FH " \n"; + print FH " \n"; + print FH " \n"; + if ($opt_gt4) { + print FH " \n"; + } + else { + print FH " \n"; + } + print FH " $workdir/$lc_vo/tmp\n"; + print FH " \n"; +} +print FH "\n\n"; +close(FH); + +exit(0); + +__END__ + +=head1 NAME + +swift-osg-ress-site-catalog - converts ReSS data to Swift site catalog + +=head1 SYNOPSIS + +swift-osg-ress-site-catalog [options] + +=head1 OPTIONS + +=over 8 + +=item B<--help> + +Show this help message + +=item B<--vo=[name]> + +Set what VO to query ReSS for + +=item B<--engage-verified> + +Only retrieve sites verified by the Engagement VO site verification tests +This can not be used together with --vo, as the query will only work for +sites advertising support for the Engagement VO. + +This option means information will be retrieved from the Engagement collector +instead of the top-level ReSS collector. + +=item B<--out=[filename]> + +Write to [filename] instead of stdout + +=back + +=head1 DESCRIPTION + +B converts ReSS data to Swift site catalog + +=cut + + + Property changes on: trunk/bin/swift-osg-ress-site-catalog ___________________________________________________________________ Name: svn:executable + * From noreply at svn.ci.uchicago.edu Tue Feb 17 10:43:04 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 17 Feb 2009 10:43:04 -0600 (CST) Subject: [Swift-commit] r2562 - trunk Message-ID: <20090217164304.BDD682281A1@www.ci.uchicago.edu> Author: benc Date: 2009-02-17 10:43:04 -0600 (Tue, 17 Feb 2009) New Revision: 2562 Modified: trunk/build.xml Log: chmod swift-osg-ress-site-catalog to executable during build Modified: trunk/build.xml =================================================================== --- trunk/build.xml 2009-02-17 16:42:29 UTC (rev 2561) +++ trunk/build.xml 2009-02-17 16:43:04 UTC (rev 2562) @@ -134,6 +134,7 @@ set COG_OPTS=-DCOG_INSTALL_PATH=%SWIFT_HOME% -Dswift.home=%SWIFT_HOME% -Dvds.home=%SWIFT_HOME% + From noreply at svn.ci.uchicago.edu Tue Feb 17 10:43:48 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 17 Feb 2009 10:43:48 -0600 (CST) Subject: [Swift-commit] r2563 - in trunk: . docs Message-ID: <20090217164348.16BEC2281A1@www.ci.uchicago.edu> Author: benc Date: 2009-02-17 10:43:47 -0600 (Tue, 17 Feb 2009) New Revision: 2563 Modified: trunk/CHANGES.txt trunk/docs/userguide.xml Log: Some details OSG/ReSS interface in CHANGES and userguide. Modified: trunk/CHANGES.txt =================================================================== --- trunk/CHANGES.txt 2009-02-17 16:43:04 UTC (rev 2562) +++ trunk/CHANGES.txt 2009-02-17 16:43:47 UTC (rev 2563) @@ -1,3 +1,9 @@ +(02/09/17) +*** There is a new command swift-osg-ress-site-catalog which will generate + a site catalog based on data gathered from OSG's ReSS information + system. This can be used by OSG users to easily generate a large number + of site catalog entries for sites that are likely to work. + (02/09/14) *** Handling of [*] and . has changed. [*] has been broken for several Swift releases, but now has some approximation of its former behaviour, Modified: trunk/docs/userguide.xml =================================================================== --- trunk/docs/userguide.xml 2009-02-17 16:43:04 UTC (rev 2562) +++ trunk/docs/userguide.xml 2009-02-17 16:43:47 UTC (rev 2563) @@ -3563,6 +3563,36 @@ mpirun command should be explicitly invoked.
+
Generating a sites.xml for OSG + +The Swift distribution contains a command +swift-osg-ress-site-catalog that can generate a site +catalog based on OSG's ReSS information system (since Swift 0.9) + + + +Usage: + swift-osg-ress-site-catalog [options] + +Options: + --help Show this help message + + --vo=[name] + Set what VO to query ReSS for + + --engage-verified + Only retrieve sites verified by the Engagement VO site + verification tests This can not be used together with --vo, as + the query will only work for sites advertising support for the + Engagement VO. + + This option means information will be retrieved from the + Engagement collector instead of the top-level ReSS collector. + + --out=[filename] + Write to [filename] instead of stdout + +
From noreply at svn.ci.uchicago.edu Tue Feb 17 10:57:31 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 17 Feb 2009 10:57:31 -0600 (CST) Subject: [Swift-commit] r2564 - log-processing/bin Message-ID: <20090217165731.820E12281E6@www.ci.uchicago.edu> Author: benc Date: 2009-02-17 10:57:30 -0600 (Tue, 17 Feb 2009) New Revision: 2564 Modified: log-processing/bin/swift-plot-log Log: when making a specific target file, move that target file to the pwd at invocation. remove working directory in /tmp. Modified: log-processing/bin/swift-plot-log =================================================================== --- log-processing/bin/swift-plot-log 2009-02-17 16:43:47 UTC (rev 2563) +++ log-processing/bin/swift-plot-log 2009-02-17 16:57:30 UTC (rev 2564) @@ -44,9 +44,9 @@ make $MAKEENV SDL=${ORIGDIR}/report-${LOG_FILE_BASE} LOG=$LOG_FILE_PATH clean webpage.kara webpage.weights karatasks.JOB_SUBMISSION.Queue.transitions karatasks.JOB_SUBMISSION.Queue.event karatasks.JOB_SUBMISSION.eip $MAKETARGETS webpage distribute else make $MAKEENV SDL=${ORIGDIR}/report-${LOG_FILE_BASE} LOG=$LOG_FILE_PATH clean $2 +cp $2 $ORIGDIR/ fi # mv report-$LOG_FILE_BASE $LOG_DIRECTORY/ -# rm -r $WORKINGDIR -echo not removing $WORKINGDIR +rm -r $WORKINGDIR From noreply at svn.ci.uchicago.edu Tue Feb 17 17:15:19 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 17 Feb 2009 17:15:19 -0600 (CST) Subject: [Swift-commit] r2565 - trunk/libexec Message-ID: <20090217231519.26EE92281AD@www.ci.uchicago.edu> Author: benc Date: 2009-02-17 17:15:17 -0600 (Tue, 17 Feb 2009) New Revision: 2565 Modified: trunk/libexec/svn-revision Log: ' instead of " in SVN version string generation was inhibiting substitution of component name Modified: trunk/libexec/svn-revision =================================================================== --- trunk/libexec/svn-revision 2009-02-17 16:57:30 UTC (rev 2564) +++ trunk/libexec/svn-revision 2009-02-17 23:15:17 UTC (rev 2565) @@ -3,7 +3,7 @@ hereversion() { if [ -d ".svn" ] && [ -x "$(which svn)" ]; then R=$(svn info | grep '^Revision' | sed "s/Revision: /$1-r/") - M=$(svn status | grep --invert-match '^\?' > /dev/null && echo '($1 modified locally)') + M=$(svn status | grep --invert-match '^\?' > /dev/null && echo "($1 modified locally)") elif [ -d ".git" ] && [ -x "$(which git)" ]; then R=$(git svn info | grep '^Revision' | sed "s/Revision: /$1-r/") if git status -a >/dev/null ; then From noreply at svn.ci.uchicago.edu Wed Feb 18 10:49:48 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Wed, 18 Feb 2009 10:49:48 -0600 (CST) Subject: [Swift-commit] r2566 - SwiftApps Message-ID: <20090218164948.DB8252281E6@www.ci.uchicago.edu> Author: houzx Date: 2009-02-18 10:49:47 -0600 (Wed, 18 Feb 2009) New Revision: 2566 Removed: SwiftApps/adem-osg/ Log: Deleting the old version From noreply at svn.ci.uchicago.edu Wed Feb 18 14:13:54 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Wed, 18 Feb 2009 14:13:54 -0600 (CST) Subject: [Swift-commit] r2568 - SwiftApps/SIDGrid/scripts Message-ID: <20090218201355.193FB2281AD@www.ci.uchicago.edu> Author: skenny Date: 2009-02-18 14:13:54 -0600 (Wed, 18 Feb 2009) New Revision: 2568 Added: SwiftApps/SIDGrid/scripts/SNRsimulations.R Log: script for doing SNR simulations Added: SwiftApps/SIDGrid/scripts/SNRsimulations.R =================================================================== --- SwiftApps/SIDGrid/scripts/SNRsimulations.R (rev 0) +++ SwiftApps/SIDGrid/scripts/SNRsimulations.R 2009-02-18 20:13:54 UTC (rev 2568) @@ -0,0 +1,31 @@ +allinputs <- Sys.getenv("R_SWIFT_ARGS") +inputfilename <- noquote(strsplit(allinputs," ")[[1]][1]) +start_num <- as.numeric(noquote(strsplit(allinputs," ")[[1]][2])) +iter <- 1/start_num +tval <- as.numeric(noquote(strsplit(allinputs," ")[[1]][3])) +outname <- paste("results/iter",start_num,"_tval",tval,"_forSNR.txt",sep="") +print(iter) +print(tval) +print(outname) + +data <- read.table(inputfilename); +c <- rowSums(data); +#cnorm <- (c/10000)+1 # this is for modeling 1 % +cnorm <- (c/20000)+1 # this is for modeling 0.5 % + +prob_result <- c(); +tresult <- c(); +simulations <- 10000; + +for (i in 1:simulations){ + #---- 1/SNR = sd for rnorm. e.g., 1/5 SNR = .2. 1/10 SNR = .1. + w <- rnorm(200, 0, iter) # 200 N(0,1) variates + v = filter(w, sides=2, rep(1,3)/3) # moving average + noise.c <- (cnorm+v) + gfit <- lm(noise.c~data[,1]+data[,2]) + currentT <- as.numeric(summary(gfit)$coefficients[8]) + tresult <- c(tresult,currentT) +} +prob <- (length(which(tresult > tval)))/simulations +prob_result <- t(c(iter,prob)) +write.table(prob_result,outname,row.names=FALSE,col.names=FALSE) Property changes on: SwiftApps/SIDGrid/scripts/SNRsimulations.R ___________________________________________________________________ Name: svn:executable + * From noreply at svn.ci.uchicago.edu Wed Feb 18 15:19:09 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Wed, 18 Feb 2009 15:19:09 -0600 (CST) Subject: [Swift-commit] r2569 - SwiftApps/adem-osg Message-ID: <20090218211909.4BB582281A1@www.ci.uchicago.edu> Author: houzx Date: 2009-02-18 15:19:08 -0600 (Wed, 18 Feb 2009) New Revision: 2569 Removed: SwiftApps/adem-osg/setup.sh Log: delete the old setup.sh Deleted: SwiftApps/adem-osg/setup.sh =================================================================== --- SwiftApps/adem-osg/setup.sh 2009-02-18 20:13:54 UTC (rev 2568) +++ SwiftApps/adem-osg/setup.sh 2009-02-18 21:19:08 UTC (rev 2569) @@ -1,66 +0,0 @@ -#!/bin/bash -# -# set-up environment to run Adem - source me -# - -dir=`basename $PWD` -Def_Rep="http://www.ci.uchicago.edu/~houzx/pac-cache" - -if [ "X${GLOBUS_LOCATION}" == "X" ]; then - echo "NO GLOBUS Error!" - -else if [ $dir != "adem-osg" ]; then - echo "Please cd to adem-osg for first setup!" - - else if [ "X${ADEM_HOME}" == "X" ]; then - echo "export ADEM_HOME=$PWD" >> $HOME/.bashrc - echo "export PATH=\$PATH:$PWD/bin" >> $HOME/.bashrc - else if [ "${ADEM_HOME}" != "$PWD" ]; then - sed -i "s,${ADEM_HOME},$PWD,g" $HOME/.bashrc - fi -# source $HOME/.bashrc - fi -fi -fi - -#check the grid proxy -grid-proxy-info 1>/dev/null 2>/dev/null -if [ "$?" != 0 ]; then - echo "ERROR: Couldn not find a valid proxy, Please check your grid proxy!" - exit 1 - else - { - timel=`grid-proxy-info |grep timeleft |cut -d: -f2` - if [ "$timel" -lt 1 ]; then - echo "Proxy has less than 60 minutes remainging - exiting." - exit - fi - } -fi - -#check pacman -if [ "X$PACMAN_LOCATION" == "X" ]; then -/bin/tar -C $ADEM_HOME -zxf $ADEM_HOME/doc/pacman-3.21.tar.gz -cd $ADEM_HOME/pacman-3.21 -source setup.sh 2>/dev/null -echo "cd $ADEM_HOME/pacman-3.21" >> $ADEM_HOME/pac-set.sh -echo "source setup.sh 2>/dev/null" >> $ADEM_HOME/pac-set.sh -echo "source $ADEM_HOME/pac-set.sh" >> $HOME/.bashrc -fi - -#check application software repository -if [ "X$REPOSITORY_HOME" == "X" ]; then - echo "The default application software repository is:" - echo "$Def_Rep" - echo "Do you want to set up your own repository?y or n" - read response - if [ "$response" == "y" ];then - echo "Please input your own application software repository:" - read rep - echo "OK, you should maintain your own repository, to add,update,or remove the pacman files and software packages." - echo "export REPOSITORY_HOME=$rep" >> $HOME/.bashrc - else - echo "export REPOSITORY_HOME=$Def_Rep" >> $HOME/.bashrc - fi - source $HOME/.bashrc -fi From noreply at svn.ci.uchicago.edu Wed Feb 18 15:19:27 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Wed, 18 Feb 2009 15:19:27 -0600 (CST) Subject: [Swift-commit] r2570 - SwiftApps/adem-osg Message-ID: <20090218211927.C94F52281A1@www.ci.uchicago.edu> Author: houzx Date: 2009-02-18 15:19:27 -0600 (Wed, 18 Feb 2009) New Revision: 2570 Removed: SwiftApps/adem-osg/setup.csh Log: delete the old setup.csh Deleted: SwiftApps/adem-osg/setup.csh =================================================================== --- SwiftApps/adem-osg/setup.csh 2009-02-18 21:19:08 UTC (rev 2569) +++ SwiftApps/adem-osg/setup.csh 2009-02-18 21:19:27 UTC (rev 2570) @@ -1,65 +0,0 @@ -#!/bin/csh -# -# set-up environment to run Adem - source me -# - -dir=`basename ${PWD}` -Def_Rep="http://www.ci.uchicago.edu/~houzx/pac-cache" - -if ("X${GLOBUS_LOCATION}" == "X") then - echo "NO GLOBUS Error!" - else if (${dir} != "adem-osg") then - echo "Please cd to adem-osg for first setup!" - - else if ("X${ADEM_HOME}" == "X") then - echo "setenv ADEM_HOME ${PWD}" >> ${HOME}/.cshrc - echo "setenv PATH ${PATH}:${PWD}/bin" >> ${HOME}/.cshrc - else if ("${ADEM_HOME}" != "${PWD}") then - sed -i "s,${ADEM_HOME},${PWD},g" ${HOME}/.cshrc - endif -# source ${HOME}/.cshrc - endif - endif -endif - -#check the grid proxy -grid-proxy-info 1>/dev/null 2>/dev/null -if ("$?" != 0) then - echo "ERROR: Couldn not find a valid proxy, Please check your grid proxy!" - exit 1 - else - { - timel=`grid-proxy-info |grep timeleft |cut -d: -f2` - if ("$timel" -lt 1) then - echo "Proxy has less than 60 minutes remainging - exiting." - exit - endif - } -endif - -#check pacman -if ("X${PACMAN_LOCATION}" == "X") then - /bin/tar -C ${ADEM_HOME} -zxf ${ADEM_HOME}/doc/pacman-3.21.tar.gz - cd ${ADEM_HOME}/pacman-3.21 - source setup.csh 2>/dev/null - echo "cd ${ADEM_HOME}/pacman-3.21" >> ${ADEM_HOME}/pac-set.csh - echo "source setup.sh 2>/dev/null" >> ${ADEM_HOME}/pac-set.csh - echo "source ${ADEM_HOME}/pac-set.csh" >> $HOME/.cshrc -endif - -#check application software repository -if ("X${REPOSITORY_HOME}" == "X") then - echo "The default application software repository is:" - echo "${Def_Rep}" - echo "Do you want to set up your own repository?y or n" - read response - if ("${response}" == "y") then - echo "Please input your own application software repository:" - read rep - echo "OK, you should maintain your own repository, to add,update,or remove the pacman files and software packages." - echo "setenv REPOSITORY_HOME ${rep}" >> ${HOME}/.cshrc - else - echo "setenv REPOSITORY_HOME ${Def_Rep}" >> ${HOME}/.cshrc - endif - source ${HOME}/.cshrc -endif From noreply at svn.ci.uchicago.edu Wed Feb 18 15:20:19 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Wed, 18 Feb 2009 15:20:19 -0600 (CST) Subject: [Swift-commit] r2571 - SwiftApps/adem-osg Message-ID: <20090218212019.E8FDF2281A1@www.ci.uchicago.edu> Author: houzx Date: 2009-02-18 15:20:19 -0600 (Wed, 18 Feb 2009) New Revision: 2571 Added: SwiftApps/adem-osg/setup.sh Log: New setup.sh import Added: SwiftApps/adem-osg/setup.sh =================================================================== --- SwiftApps/adem-osg/setup.sh (rev 0) +++ SwiftApps/adem-osg/setup.sh 2009-02-18 21:20:19 UTC (rev 2571) @@ -0,0 +1,70 @@ +#!/bin/bash +# +# set-up environment to run Adem - source me +# + +dir=`basename $PWD` +Def_Rep="http://www.ci.uchicago.edu/~houzx/pac-cache" + +if [ "X${GLOBUS_LOCATION}" == "X" ]; then + echo "NO GLOBUS Error!" + +else if [ $dir != "adem-osg" ]; then + echo "Please cd to adem-osg for first setup!" + + else if [ "X${ADEM_HOME}" == "X" ]; then + echo "export ADEM_HOME=$PWD" >> $HOME/.bashrc + echo "export PATH=\$PATH:$PWD/bin" >> $HOME/.bashrc + else if [ "${ADEM_HOME}" != "$PWD" ]; then + sed -i "s,${ADEM_HOME},$PWD,g" $HOME/.bashrc + fi +# source $HOME/.bashrc + fi +fi +fi + +#check the grid proxy +grid-proxy-info 1>/dev/null 2>/dev/null +if [ "$?" != 0 ]; then + echo "ERROR: Couldn not find a valid proxy, Please check your grid proxy!" + exit 1 + else + { + timel=`grid-proxy-info |grep timeleft |cut -d: -f2` + if [ "$timel" -lt 1 ]; then + echo "Proxy has less than 60 minutes remainging - exiting." + exit + fi + } +fi + +#check pacman +if [ "X$PACMAN_LOCATION" == "X" ]; then +/bin/tar -C $ADEM_HOME -zxf $ADEM_HOME/doc/pacman-3.21.tar.gz +DIR=$PWD +cd $ADEM_HOME/pacman-3.21 +source setup.sh 2>/dev/null +cd $DIR +echo "DIR=$PWD" >> $ADEM_HOME/pac-set.sh +echo "cd $ADEM_HOME/pacman-3.21" >> $ADEM_HOME/pac-set.sh +echo "source setup.sh 2>/dev/null" >> $ADEM_HOME/pac-set.sh +echo "cd $DIR" >> $ADEM_HOME/pac-set.sh +echo "source $ADEM_HOME/pac-set.sh" >> $HOME/.bashrc +fi + +#check application software repository +if [ "X$REPOSITORY_HOME" == "X" ]; then + echo "The default application software repository is:" + echo "$Def_Rep" + echo "Do you want to set up your own repository?y or n" + read response + if [ "$response" == "y" ];then + echo "Please input your own application software repository:" + read rep + echo "OK, you should maintain your own repository, to add,update,or remove the pacman files and software packages." + echo "export REPOSITORY_HOME=$rep" >> $HOME/.bashrc + else + echo "export REPOSITORY_HOME=$Def_Rep" >> $HOME/.bashrc + fi + source $HOME/.bashrc +fi Property changes on: SwiftApps/adem-osg/setup.sh ___________________________________________________________________ Name: svn:executable + From noreply at svn.ci.uchicago.edu Wed Feb 18 15:21:00 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Wed, 18 Feb 2009 15:21:00 -0600 (CST) Subject: [Swift-commit] r2572 - SwiftApps/adem-osg Message-ID: <20090218212100.5DF562281A1@www.ci.uchicago.edu> Author: houzx Date: 2009-02-18 15:21:00 -0600 (Wed, 18 Feb 2009) New Revision: 2572 Added: SwiftApps/adem-osg/setup.csh Log: New setup.csh import Added: SwiftApps/adem-osg/setup.csh =================================================================== --- SwiftApps/adem-osg/setup.csh (rev 0) +++ SwiftApps/adem-osg/setup.csh 2009-02-18 21:21:00 UTC (rev 2572) @@ -0,0 +1,69 @@ +#!/bin/csh +# +# set-up environment to run Adem - source me +# + +dir=`basename ${PWD}` +Def_Rep="http://www.ci.uchicago.edu/~houzx/pac-cache" + +if ("X${GLOBUS_LOCATION}" == "X") then + echo "NO GLOBUS Error!" + else if (${dir} != "adem-osg") then + echo "Please cd to adem-osg for first setup!" + + else if ("X${ADEM_HOME}" == "X") then + echo "setenv ADEM_HOME ${PWD}" >> ${HOME}/.cshrc + echo "setenv PATH ${PATH}:${PWD}/bin" >> ${HOME}/.cshrc + else if ("${ADEM_HOME}" != "${PWD}") then + sed -i "s,${ADEM_HOME},${PWD},g" ${HOME}/.cshrc + endif +# source ${HOME}/.cshrc + endif + endif +endif + +#check the grid proxy +grid-proxy-info 1>/dev/null 2>/dev/null +if ("$?" != 0) then + echo "ERROR: Couldn not find a valid proxy, Please check your grid proxy!" + exit 1 + else + { + timel=`grid-proxy-info |grep timeleft |cut -d: -f2` + if ("$timel" -lt 1) then + echo "Proxy has less than 60 minutes remainging - exiting." + exit + endif + } +endif + +#check pacman +if ("X${PACMAN_LOCATION}" == "X") then + /bin/tar -C ${ADEM_HOME} -zxf ${ADEM_HOME}/doc/pacman-3.21.tar.gz + DIR=$PWD + cd ${ADEM_HOME}/pacman-3.21 + source setup.csh 2>/dev/null + cd $DIR + echo "DIR=$PWD" >> ${ADEM_HOME}/pac-set.csh + echo "cd ${ADEM_HOME}/pacman-3.21" >> ${ADEM_HOME}/pac-set.csh + echo "source setup.sh 2>/dev/null" >> ${ADEM_HOME}/pac-set.csh + echo "cd $DIR" >> ${ADEM_HOME}/pac-set.csh + echo "source ${ADEM_HOME}/pac-set.csh" >> $HOME/.cshrc +endif + +#check application software repository +if ("X${REPOSITORY_HOME}" == "X") then + echo "The default application software repository is:" + echo "${Def_Rep}" + echo "Do you want to set up your own repository?y or n" + read response + if ("${response}" == "y") then + echo "Please input your own application software repository:" + read rep + echo "OK, you should maintain your own repository, to add,update,or remove the pacman files and software packages." + echo "setenv REPOSITORY_HOME ${rep}" >> ${HOME}/.cshrc + else + echo "setenv REPOSITORY_HOME ${Def_Rep}" >> ${HOME}/.cshrc + endif + source ${HOME}/.cshrc +endif Property changes on: SwiftApps/adem-osg/setup.csh ___________________________________________________________________ Name: svn:executable + From noreply at svn.ci.uchicago.edu Wed Feb 18 15:25:46 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Wed, 18 Feb 2009 15:25:46 -0600 (CST) Subject: [Swift-commit] r2573 - SwiftApps/adem-osg Message-ID: <20090218212546.8AB272281E6@www.ci.uchicago.edu> Author: houzx Date: 2009-02-18 15:25:45 -0600 (Wed, 18 Feb 2009) New Revision: 2573 Removed: SwiftApps/adem-osg/setup.csh Log: delete the old setup.csh Deleted: SwiftApps/adem-osg/setup.csh =================================================================== --- SwiftApps/adem-osg/setup.csh 2009-02-18 21:21:00 UTC (rev 2572) +++ SwiftApps/adem-osg/setup.csh 2009-02-18 21:25:45 UTC (rev 2573) @@ -1,69 +0,0 @@ -#!/bin/csh -# -# set-up environment to run Adem - source me -# - -dir=`basename ${PWD}` -Def_Rep="http://www.ci.uchicago.edu/~houzx/pac-cache" - -if ("X${GLOBUS_LOCATION}" == "X") then - echo "NO GLOBUS Error!" - else if (${dir} != "adem-osg") then - echo "Please cd to adem-osg for first setup!" - - else if ("X${ADEM_HOME}" == "X") then - echo "setenv ADEM_HOME ${PWD}" >> ${HOME}/.cshrc - echo "setenv PATH ${PATH}:${PWD}/bin" >> ${HOME}/.cshrc - else if ("${ADEM_HOME}" != "${PWD}") then - sed -i "s,${ADEM_HOME},${PWD},g" ${HOME}/.cshrc - endif -# source ${HOME}/.cshrc - endif - endif -endif - -#check the grid proxy -grid-proxy-info 1>/dev/null 2>/dev/null -if ("$?" != 0) then - echo "ERROR: Couldn not find a valid proxy, Please check your grid proxy!" - exit 1 - else - { - timel=`grid-proxy-info |grep timeleft |cut -d: -f2` - if ("$timel" -lt 1) then - echo "Proxy has less than 60 minutes remainging - exiting." - exit - endif - } -endif - -#check pacman -if ("X${PACMAN_LOCATION}" == "X") then - /bin/tar -C ${ADEM_HOME} -zxf ${ADEM_HOME}/doc/pacman-3.21.tar.gz - DIR=$PWD - cd ${ADEM_HOME}/pacman-3.21 - source setup.csh 2>/dev/null - cd $DIR - echo "DIR=$PWD" >> ${ADEM_HOME}/pac-set.csh - echo "cd ${ADEM_HOME}/pacman-3.21" >> ${ADEM_HOME}/pac-set.csh - echo "source setup.sh 2>/dev/null" >> ${ADEM_HOME}/pac-set.csh - echo "cd $DIR" >> ${ADEM_HOME}/pac-set.csh - echo "source ${ADEM_HOME}/pac-set.csh" >> $HOME/.cshrc -endif - -#check application software repository -if ("X${REPOSITORY_HOME}" == "X") then - echo "The default application software repository is:" - echo "${Def_Rep}" - echo "Do you want to set up your own repository?y or n" - read response - if ("${response}" == "y") then - echo "Please input your own application software repository:" - read rep - echo "OK, you should maintain your own repository, to add,update,or remove the pacman files and software packages." - echo "setenv REPOSITORY_HOME ${rep}" >> ${HOME}/.cshrc - else - echo "setenv REPOSITORY_HOME ${Def_Rep}" >> ${HOME}/.cshrc - endif - source ${HOME}/.cshrc -endif From noreply at svn.ci.uchicago.edu Wed Feb 18 15:26:09 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Wed, 18 Feb 2009 15:26:09 -0600 (CST) Subject: [Swift-commit] r2574 - SwiftApps/adem-osg Message-ID: <20090218212609.A81092281E6@www.ci.uchicago.edu> Author: houzx Date: 2009-02-18 15:26:09 -0600 (Wed, 18 Feb 2009) New Revision: 2574 Added: SwiftApps/adem-osg/setup.csh Log: New setup.csh import Added: SwiftApps/adem-osg/setup.csh =================================================================== --- SwiftApps/adem-osg/setup.csh (rev 0) +++ SwiftApps/adem-osg/setup.csh 2009-02-18 21:26:09 UTC (rev 2574) @@ -0,0 +1,69 @@ +#!/bin/csh +# +# set-up environment to run Adem - source me +# + +dir=`basename ${PWD}` +Def_Rep="http://www.ci.uchicago.edu/~houzx/pac-cache" + +if ("X${GLOBUS_LOCATION}" == "X") then + echo "NO GLOBUS Error!" + else if (${dir} != "adem-osg") then + echo "Please cd to adem-osg for first setup!" + + else if ("X${ADEM_HOME}" == "X") then + echo "setenv ADEM_HOME ${PWD}" >> ${HOME}/.cshrc + echo "setenv PATH ${PATH}:${PWD}/bin" >> ${HOME}/.cshrc + else if ("${ADEM_HOME}" != "${PWD}") then + sed -i "s,${ADEM_HOME},${PWD},g" ${HOME}/.cshrc + endif +# source ${HOME}/.cshrc + endif + endif +endif + +#check the grid proxy +grid-proxy-info 1>/dev/null 2>/dev/null +if ("$?" != 0) then + echo "ERROR: Couldn not find a valid proxy, Please check your grid proxy!" + exit 1 + else + { + timel=`grid-proxy-info |grep timeleft |cut -d: -f2` + if ("$timel" -lt 1) then + echo "Proxy has less than 60 minutes remainging - exiting." + exit + endif + } +endif + +#check pacman +if ("X${PACMAN_LOCATION}" == "X") then + /bin/tar -C ${ADEM_HOME} -zxf ${ADEM_HOME}/doc/pacman-3.21.tar.gz + DIR=${PWD} + cd ${ADEM_HOME}/pacman-3.21 + source setup.csh 2>/dev/null + cd ${DIR} + echo "DIR=${PWD}" >> ${ADEM_HOME}/pac-set.csh + echo "cd ${ADEM_HOME}/pacman-3.21" >> ${ADEM_HOME}/pac-set.csh + echo "source setup.sh 2>/dev/null" >> ${ADEM_HOME}/pac-set.csh + echo "cd ${DIR}" >> ${ADEM_HOME}/pac-set.csh + echo "source ${ADEM_HOME}/pac-set.csh" >> $HOME/.cshrc +endif + +#check application software repository +if ("X${REPOSITORY_HOME}" == "X") then + echo "The default application software repository is:" + echo "${Def_Rep}" + echo "Do you want to set up your own repository?y or n" + read response + if ("${response}" == "y") then + echo "Please input your own application software repository:" + read rep + echo "OK, you should maintain your own repository, to add,update,or remove the pacman files and software packages." + echo "setenv REPOSITORY_HOME ${rep}" >> ${HOME}/.cshrc + else + echo "setenv REPOSITORY_HOME ${Def_Rep}" >> ${HOME}/.cshrc + endif + source ${HOME}/.cshrc +endif Property changes on: SwiftApps/adem-osg/setup.csh ___________________________________________________________________ Name: svn:executable + From noreply at svn.ci.uchicago.edu Wed Feb 18 16:55:12 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Wed, 18 Feb 2009 16:55:12 -0600 (CST) Subject: [Swift-commit] r2575 - SwiftApps Message-ID: <20090218225512.89E92228187@www.ci.uchicago.edu> Author: houzx Date: 2009-02-18 16:55:10 -0600 (Wed, 18 Feb 2009) New Revision: 2575 Removed: SwiftApps/adem-osg/ Log: Deleting adem-osg again From noreply at svn.ci.uchicago.edu Thu Feb 19 04:35:11 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Thu, 19 Feb 2009 04:35:11 -0600 (CST) Subject: [Swift-commit] r2577 - trunk/libexec Message-ID: <20090219103511.A30C72281AD@www.ci.uchicago.edu> Author: benc Date: 2009-02-19 04:35:10 -0600 (Thu, 19 Feb 2009) New Revision: 2577 Modified: trunk/libexec/wrapper.sh Log: remove debugging find from wrapper that had been accidentally committed Modified: trunk/libexec/wrapper.sh =================================================================== --- trunk/libexec/wrapper.sh 2009-02-18 22:57:00 UTC (rev 2576) +++ trunk/libexec/wrapper.sh 2009-02-19 10:35:10 UTC (rev 2577) @@ -259,8 +259,6 @@ "$EXEC" "$@" 1>"$STDOUT" 2>"$STDERR" <"$STDIN" fi checkError $? "Exit code $?" -find . >& "$INFO" - else if [ ! -f "$KICKSTART" ]; then log "Kickstart executable ($KICKSTART) not found" From noreply at svn.ci.uchicago.edu Thu Feb 19 11:08:02 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Thu, 19 Feb 2009 11:08:02 -0600 (CST) Subject: [Swift-commit] r2578 - trunk/tests/sites Message-ID: <20090219170802.46A082281A1@www.ci.uchicago.edu> Author: benc Date: 2009-02-19 11:08:01 -0600 (Thu, 19 Feb 2009) New Revision: 2578 Modified: trunk/tests/sites/tp-pbs-gram2.xml Log: teraport tests use test queue Modified: trunk/tests/sites/tp-pbs-gram2.xml =================================================================== --- trunk/tests/sites/tp-pbs-gram2.xml 2009-02-19 10:35:10 UTC (rev 2577) +++ trunk/tests/sites/tp-pbs-gram2.xml 2009-02-19 17:08:01 UTC (rev 2578) @@ -3,7 +3,7 @@ - + test /home/benc/swifttest From noreply at svn.ci.uchicago.edu Fri Feb 20 13:35:33 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 20 Feb 2009 13:35:33 -0600 (CST) Subject: [Swift-commit] r2579 - trunk/src/org/griphyn/vdl/mapping Message-ID: <20090220193533.8EECE2281A1@www.ci.uchicago.edu> Author: benc Date: 2009-02-20 13:35:32 -0600 (Fri, 20 Feb 2009) New Revision: 2579 Modified: trunk/src/org/griphyn/vdl/mapping/RootDataNode.java Log: give more details about unexpected exception which results when input files are missing - this error needs more tidying but this is an easy increase in information in the mean-time Modified: trunk/src/org/griphyn/vdl/mapping/RootDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/RootDataNode.java 2009-02-19 17:08:01 UTC (rev 2578) +++ trunk/src/org/griphyn/vdl/mapping/RootDataNode.java 2009-02-20 19:35:32 UTC (rev 2579) @@ -74,7 +74,7 @@ checkInputs(params, mapper, this); } catch (VDL2FutureException e) { - logger.warn("Unexpected VDL2FutureException checking inputs"); + logger.warn("Unexpected VDL2FutureException checking inputs for dataset "+this); throw new RuntimeException("Got a VDL2FutureException but all parameters should have their values",e); } catch (DependentException e) { From noreply at svn.ci.uchicago.edu Fri Feb 20 23:21:21 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 20 Feb 2009 23:21:21 -0600 (CST) Subject: [Swift-commit] r2580 - trunk/bin Message-ID: <20090221052121.A67D522819E@www.ci.uchicago.edu> Author: hategan Date: 2009-02-20 23:21:20 -0600 (Fri, 20 Feb 2009) New Revision: 2580 Modified: trunk/bin/swift-osg-ress-site-catalog Log: patch from Mats to deal with sites with multiple gatekeepers Modified: trunk/bin/swift-osg-ress-site-catalog =================================================================== --- trunk/bin/swift-osg-ress-site-catalog 2009-02-20 19:35:32 UTC (rev 2579) +++ trunk/bin/swift-osg-ress-site-catalog 2009-02-21 05:21:20 UTC (rev 2580) @@ -100,7 +100,7 @@ else { print FH " \n"; } - print FH " $workdir/$lc_vo/tmp\n"; + print FH " $workdir/$lc_vo/tmp/$sitename\n"; print FH " \n"; } print FH "\n\n"; From noreply at svn.ci.uchicago.edu Sun Feb 22 11:00:25 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 22 Feb 2009 11:00:25 -0600 (CST) Subject: [Swift-commit] r2581 - trunk/src/org/griphyn/vdl/karajan Message-ID: <20090222170025.45AC522819E@www.ci.uchicago.edu> Author: benc Date: 2009-02-22 11:00:23 -0600 (Sun, 22 Feb 2009) New Revision: 2581 Modified: trunk/src/org/griphyn/vdl/karajan/Loader.java Log: A TODO for later Modified: trunk/src/org/griphyn/vdl/karajan/Loader.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/Loader.java 2009-02-21 05:21:20 UTC (rev 2580) +++ trunk/src/org/griphyn/vdl/karajan/Loader.java 2009-02-22 17:00:23 UTC (rev 2581) @@ -162,6 +162,7 @@ else { stack.setGlobal(CONST_VDL_OPERATION, VDL_OPERATION_RUN); } + // TODO should not be referring to vds.home any more stack.setGlobal("vds.home", System.getProperty("vds.home")); List arguments = ap.getArguments(); From noreply at svn.ci.uchicago.edu Sun Feb 22 11:01:18 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 22 Feb 2009 11:01:18 -0600 (CST) Subject: [Swift-commit] r2582 - trunk/src/org/griphyn/vdl/karajan Message-ID: <20090222170118.D00BE22819E@www.ci.uchicago.edu> Author: benc Date: 2009-02-22 11:01:18 -0600 (Sun, 22 Feb 2009) New Revision: 2582 Modified: trunk/src/org/griphyn/vdl/karajan/Loader.java Log: fix whitespace in Loader Modified: trunk/src/org/griphyn/vdl/karajan/Loader.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/Loader.java 2009-02-22 17:00:23 UTC (rev 2581) +++ trunk/src/org/griphyn/vdl/karajan/Loader.java 2009-02-22 17:01:18 UTC (rev 2582) @@ -88,7 +88,7 @@ } if (!new File(project).exists()) { - System.err.println("SwiftScript program does not exist: " + project + "\n"); + System.err.println("SwiftScript program does not exist: " + project + "\n"); shortUsage(); System.exit(1); } @@ -165,13 +165,13 @@ // TODO should not be referring to vds.home any more stack.setGlobal("vds.home", System.getProperty("vds.home")); - List arguments = ap.getArguments(); - if (ap.hasValue(ARG_RESUME)) { - arguments.add("-rlog:resume=" + ap.getStringValue(ARG_RESUME)); - } + List arguments = ap.getArguments(); + if (ap.hasValue(ARG_RESUME)) { + arguments.add("-rlog:resume=" + ap.getStringValue(ARG_RESUME)); + } ec.setArguments(arguments); - - ec.start(stack); + + ec.start(stack); ec.waitFor(); if (ec.isFailed()) { runerror = true; From noreply at svn.ci.uchicago.edu Sun Feb 22 11:02:54 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 22 Feb 2009 11:02:54 -0600 (CST) Subject: [Swift-commit] r2583 - in trunk: . libexec resources src/org/griphyn/vdl/engine src/org/griphyn/vdl/karajan Message-ID: <20090222170254.B0CF0228209@www.ci.uchicago.edu> Author: benc Date: 2009-02-22 11:02:54 -0600 (Sun, 22 Feb 2009) New Revision: 2583 Added: trunk/libexec/generate-buildid Modified: trunk/CHANGES.txt trunk/build.xml trunk/resources/Karajan.stg trunk/src/org/griphyn/vdl/engine/Karajan.java trunk/src/org/griphyn/vdl/karajan/Loader.java Log: Recompilation will happen if a .kml file was compiled with a different version of Swift to the version being invoked. This is in addition to the existing behaviour where a .swift file will be recompiled if it is newer than the corresponding .kml file. Modified: trunk/CHANGES.txt =================================================================== --- trunk/CHANGES.txt 2009-02-22 17:01:18 UTC (rev 2582) +++ trunk/CHANGES.txt 2009-02-22 17:02:54 UTC (rev 2583) @@ -1,3 +1,9 @@ +(02/09/22) +*** Recompilation will happen if a .kml file was compiled with a different + version of Swift to the version being invoked. This is in addition to the + existing behaviour where a .swift file will be recompiled if it is newer + than the corresponding .kml file. + (02/09/17) *** There is a new command swift-osg-ress-site-catalog which will generate a site catalog based on data gathered from OSG's ReSS information Modified: trunk/build.xml =================================================================== --- trunk/build.xml 2009-02-22 17:01:18 UTC (rev 2582) +++ trunk/build.xml 2009-02-22 17:02:54 UTC (rev 2583) @@ -273,6 +273,7 @@ Swift ${version} + Added: trunk/libexec/generate-buildid =================================================================== --- trunk/libexec/generate-buildid (rev 0) +++ trunk/libexec/generate-buildid 2009-02-22 17:02:54 UTC (rev 2583) @@ -0,0 +1,11 @@ +#!/bin/bash + +UUIDGEN=$(which uuidgen) + +if [ -x "$UUIDGEN" ]; then + $UUIDGEN +else + # poormans uuidgen + echo $(date +%s) $PPID $HOSTNAME +fi + Property changes on: trunk/libexec/generate-buildid ___________________________________________________________________ Name: svn:executable + * Modified: trunk/resources/Karajan.stg =================================================================== --- trunk/resources/Karajan.stg 2009-02-22 17:01:18 UTC (rev 2582) +++ trunk/resources/Karajan.stg 2009-02-22 17:02:54 UTC (rev 2583) @@ -1,7 +1,7 @@ group Karajan; -program(types,procedures,declarations,statements,constants) ::= << - +program(types,procedures,declarations,statements,constants,buildversion) ::= << + Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/Karajan.java 2009-02-22 17:01:18 UTC (rev 2582) +++ trunk/src/org/griphyn/vdl/engine/Karajan.java 2009-02-22 17:02:54 UTC (rev 2583) @@ -34,6 +34,7 @@ import org.globus.swift.language.Switch.Default; import org.globus.swift.language.TypesDocument.Types; import org.globus.swift.language.TypesDocument.Types.Type; +import org.griphyn.vdl.karajan.Loader; import org.griphyn.vdl.karajan.CompilationException; import org.safehaus.uuid.UUIDGenerator; import org.w3c.dom.Node; @@ -142,6 +143,8 @@ VariableScope scope = new VariableScope(this, null); scope.bodyTemplate = template("program"); + scope.bodyTemplate.setAttribute("buildversion",Loader.buildVersion); + types = prog.getTypes(); if (types != null) { for (int i = 0; i < types.sizeOfTypeArray(); i++) { Modified: trunk/src/org/griphyn/vdl/karajan/Loader.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/Loader.java 2009-02-22 17:01:18 UTC (rev 2582) +++ trunk/src/org/griphyn/vdl/karajan/Loader.java 2009-02-22 17:02:54 UTC (rev 2583) @@ -3,12 +3,15 @@ */ package org.griphyn.vdl.karajan; +import java.io.BufferedReader; import java.io.File; import java.io.FileInputStream; import java.io.FileNotFoundException; import java.io.FileOutputStream; +import java.io.FileReader; import java.io.IOException; import java.io.PrintStream; +import java.io.Reader; import java.security.NoSuchAlgorithmException; import java.security.SecureRandom; import java.text.DateFormat; @@ -61,6 +64,8 @@ public static final String VDL_OPERATION_TYPECHECK = "typecheck"; public static final String VDL_OPERATION_DRYRUN = "dryrun"; + public static String buildVersion; + public static void main(String[] argv) { logger.debug("Loader started"); ArgumentParser ap = buildArgumentParser(); @@ -194,17 +199,50 @@ System.err.print("For usage information: swift -help\n\n"); } - public static String compile(String project) + public static String compile(String project) throws FileNotFoundException, ParsingException, - IncorrectInvocationException, CompilationException { + IncorrectInvocationException, CompilationException, + IOException { File swiftscript = new File(project); File dir = swiftscript.getParentFile(); String projectBase = project.substring(0, project.lastIndexOf('.')); File xml = new File(projectBase + ".xml"); File kml = new File(projectBase + ".kml"); + loadBuildVersion(); + + boolean recompile = false; + if (swiftscript.lastModified() > kml.lastModified()) { logger.info(project + ": source file is new. Recompiling."); + recompile = true; + } + + if (kml.exists()) { + // read first line of kml + Reader fr = new FileReader(kml); + BufferedReader br = new BufferedReader(fr); + String firstLine = br.readLine(); + String prefix = ""); + String kmlversion = cut.substring(0,endOffset); + logger.debug("kmlversion is >"+kmlversion+"<"); + logger.debug("build version is >"+buildVersion+"<"); + if(!(kmlversion.equals(buildVersion))) { + logger.info(project + ": source file was compiled with a different version of Swift. Recompiling."); + recompile=true; + } + } + } + + if (recompile) { VDLt2VDLx.compile(new FileInputStream(swiftscript), new PrintStream(new FileOutputStream(xml))); try { @@ -237,6 +275,12 @@ return kml.getAbsolutePath(); } + private static void loadBuildVersion() throws IOException { + File f = new File(System.getProperty("swift.home")+"/libexec/buildid.txt"); + BufferedReader br = new BufferedReader(new FileReader(f)); + buildVersion = br.readLine(); + } + private static VDL2Config loadConfig(ArgumentParser ap, VariableStack stack) throws IOException { VDL2Config conf; if (ap.hasValue(ARG_INSTANCE_CONFIG)) { From noreply at svn.ci.uchicago.edu Sun Feb 22 11:26:28 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 22 Feb 2009 11:26:28 -0600 (CST) Subject: [Swift-commit] r2584 - trunk/src/org/griphyn/vdl/util Message-ID: <20090222172628.2A38F228187@www.ci.uchicago.edu> Author: benc Date: 2009-02-22 11:26:27 -0600 (Sun, 22 Feb 2009) New Revision: 2584 Modified: trunk/src/org/griphyn/vdl/util/TriStateBoolean.java Log: Tidy whitespace Modified: trunk/src/org/griphyn/vdl/util/TriStateBoolean.java =================================================================== --- trunk/src/org/griphyn/vdl/util/TriStateBoolean.java 2009-02-22 17:02:54 UTC (rev 2583) +++ trunk/src/org/griphyn/vdl/util/TriStateBoolean.java 2009-02-22 17:26:27 UTC (rev 2584) @@ -4,7 +4,7 @@ package org.griphyn.vdl.util; public final class TriStateBoolean { - public static final TriStateBoolean FALSE = new TriStateBoolean(0); + public static final TriStateBoolean FALSE = new TriStateBoolean(0); public static final TriStateBoolean TRUE = new TriStateBoolean(1); public static final TriStateBoolean MAYBE = new TriStateBoolean(2); @@ -14,36 +14,36 @@ this.value = value; } - public boolean equals(Object obj) { - if (obj instanceof TriStateBoolean) { - return value == ((TriStateBoolean) obj).value; + public boolean equals(Object obj) { + if (obj instanceof TriStateBoolean) { + return value == ((TriStateBoolean) obj).value; } else { - return false; + return false; } - } + } - public int hashCode() { - return value; - } + public int hashCode() { + return value; + } - public String toString() { - switch(value) { + public String toString() { + switch(value) { case 0: return "false"; case 1: return "true"; default: return "maybe"; } - } + } public static TriStateBoolean valueOf(String value) { - if ("false".equalsIgnoreCase(value) || "no".equalsIgnoreCase(value)) { - return FALSE; + if ("false".equalsIgnoreCase(value) || "no".equalsIgnoreCase(value)) { + return FALSE; } else if ("true".equalsIgnoreCase(value) || "yes".equalsIgnoreCase(value)) { return TRUE; } else { - return MAYBE; + return MAYBE; } } } From noreply at svn.ci.uchicago.edu Sun Feb 22 11:27:01 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 22 Feb 2009 11:27:01 -0600 (CST) Subject: [Swift-commit] r2585 - trunk/resources Message-ID: <20090222172701.9C2BF228115@www.ci.uchicago.edu> Author: benc Date: 2009-02-22 11:27:01 -0600 (Sun, 22 Feb 2009) New Revision: 2585 Modified: trunk/resources/swiftscript.g Log: Lexer treats "bool" as a token, but it should have "boolean" in that place. In execution, this turns out to not cause a problem because boolean declarations are passed through as if they were user-declared types, and the rest of the stack does not distinguish between inbuilt and user-declared types in a way that causes a problem. Modified: trunk/resources/swiftscript.g =================================================================== --- trunk/resources/swiftscript.g 2009-02-22 17:26:27 UTC (rev 2584) +++ trunk/resources/swiftscript.g 2009-02-22 17:27:01 UTC (rev 2585) @@ -453,13 +453,15 @@ } ; +// TODO unnecessary complexity here - this should work ok matching +// only ID type returns [StringTemplate code=null] {String s = null;} : ( ( "int" {s="int"; } ) | ( "float" {s="float"; } ) | ( "string" {s="string"; } ) - | ( "bool" {s="bool"; } ) + | ( "boolean" {s="boolean"; } ) | ( id:ID {s=id.getText();} ) ) { From noreply at svn.ci.uchicago.edu Sun Feb 22 12:59:26 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 22 Feb 2009 12:59:26 -0600 (CST) Subject: [Swift-commit] r2586 - trunk/resources Message-ID: <20090222185926.C888A2281FB@www.ci.uchicago.edu> Author: benc Date: 2009-02-22 12:59:26 -0600 (Sun, 22 Feb 2009) New Revision: 2586 Modified: trunk/resources/swiftscript.g Log: Remove type tokens from the lexer, as they can all be caught by the general ID case. Modified: trunk/resources/swiftscript.g =================================================================== --- trunk/resources/swiftscript.g 2009-02-22 17:27:01 UTC (rev 2585) +++ trunk/resources/swiftscript.g 2009-02-22 18:59:26 UTC (rev 2586) @@ -453,20 +453,12 @@ } ; -// TODO unnecessary complexity here - this should work ok matching -// only ID type returns [StringTemplate code=null] -{String s = null;} : - ( ( "int" {s="int"; } ) - | ( "float" {s="float"; } ) - | ( "string" {s="string"; } ) - | ( "boolean" {s="boolean"; } ) - | ( id:ID {s=id.getText();} ) - ) + id:ID { code=template("type"); - code.setAttribute("name", s); + code.setAttribute("name", id.getText()); } ; From noreply at svn.ci.uchicago.edu Mon Feb 23 05:51:34 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 23 Feb 2009 05:51:34 -0600 (CST) Subject: [Swift-commit] r2587 - SwiftApps Message-ID: <20090223115134.495BF2281FB@www.ci.uchicago.edu> Author: houzx Date: 2009-02-23 05:51:33 -0600 (Mon, 23 Feb 2009) New Revision: 2587 Removed: SwiftApps/adem-osg/ Log: delete the old version From noreply at svn.ci.uchicago.edu Mon Feb 23 06:17:54 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 23 Feb 2009 06:17:54 -0600 (CST) Subject: [Swift-commit] r2589 - SwiftApps/adem-osg Message-ID: <20090223121755.12E9A2281FB@www.ci.uchicago.edu> Author: houzx Date: 2009-02-23 06:17:54 -0600 (Mon, 23 Feb 2009) New Revision: 2589 Removed: SwiftApps/adem-osg/bin/ Log: delete the old bin version From noreply at svn.ci.uchicago.edu Mon Feb 23 07:11:06 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 23 Feb 2009 07:11:06 -0600 (CST) Subject: [Swift-commit] r2591 - SwiftApps/adem-osg Message-ID: <20090223131106.1E0E9228187@www.ci.uchicago.edu> Author: houzx Date: 2009-02-23 07:11:05 -0600 (Mon, 23 Feb 2009) New Revision: 2591 Removed: SwiftApps/adem-osg/bin/ Log: delete the old bin version From noreply at svn.ci.uchicago.edu Mon Feb 23 09:04:30 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 23 Feb 2009 09:04:30 -0600 (CST) Subject: [Swift-commit] r2593 - in trunk: src/org/griphyn/vdl/engine tests/language-behaviour Message-ID: <20090223150430.485D1228115@www.ci.uchicago.edu> Author: benc Date: 2009-02-23 09:04:29 -0600 (Mon, 23 Feb 2009) New Revision: 2593 Added: trunk/tests/language-behaviour/0851-iterate-using-body-variables.swift Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java Log: bug 177: loop condition in iterate statements can now refer to variables declared within the iteration body. This does not add to the expressiveness of the iterate construction (I think), but does allow a different coding style which may be more convenient. Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/Karajan.java 2009-02-23 13:11:33 UTC (rev 2592) +++ trunk/src/org/griphyn/vdl/engine/Karajan.java 2009-02-23 15:04:29 UTC (rev 2593) @@ -581,14 +581,15 @@ StringTemplate iterateST = template("iterate"); - XmlObject cond = iterate.getAbstractExpression(); - StringTemplate condST = expressionToKarajan(cond, loopScope); - iterateST.setAttribute("cond", condST); iterateST.setAttribute("var", iterate.getVar()); innerScope.bodyTemplate = iterateST; statements(iterate.getBody(), innerScope); + XmlObject cond = iterate.getAbstractExpression(); + StringTemplate condST = expressionToKarajan(cond, innerScope); + iterateST.setAttribute("cond", condST); + Object statementID = new Integer(callID++); Iterator scopeIterator = innerScope.getVariableIterator(); while(scopeIterator.hasNext()) { Added: trunk/tests/language-behaviour/0851-iterate-using-body-variables.swift =================================================================== --- trunk/tests/language-behaviour/0851-iterate-using-body-variables.swift (rev 0) +++ trunk/tests/language-behaviour/0851-iterate-using-body-variables.swift 2009-02-23 15:04:29 UTC (rev 2593) @@ -0,0 +1,18 @@ +type messagefile; + +(messagefile t) greeting() { + app { + echo "hello" stdout=@filename(t); + } +} + +messagefile outfile[] ; + +iterate i { + int j = i; + outfile[i] = greeting(); +} until(j>10); + + From noreply at svn.ci.uchicago.edu Mon Feb 23 09:06:33 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 23 Feb 2009 09:06:33 -0600 (CST) Subject: [Swift-commit] r2594 - trunk/src/org/griphyn/vdl/mapping Message-ID: <20090223150633.3E6CD228115@www.ci.uchicago.edu> Author: benc Date: 2009-02-23 09:06:32 -0600 (Mon, 23 Feb 2009) New Revision: 2594 Modified: trunk/src/org/griphyn/vdl/mapping/ExternalDataNode.java Log: remove unused numeric value handling from ExternalDataNode.toString Modified: trunk/src/org/griphyn/vdl/mapping/ExternalDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/ExternalDataNode.java 2009-02-23 15:04:29 UTC (rev 2593) +++ trunk/src/org/griphyn/vdl/mapping/ExternalDataNode.java 2009-02-23 15:06:32 UTC (rev 2594) @@ -76,23 +76,6 @@ * text description. */ public String toString() { - if (this.value != null && !(this.value instanceof Exception)) { - // special handling for ints... - if (this.getType().equals(Types.INT)) { - try { - Number n = (Number) this.getValue(); - return String.valueOf(n.intValue()); - } - catch (ClassCastException e) { - throw new RuntimeException("Internal type error. Value is not a Number for " - + getDisplayableName() + getPathFromRoot()); - } - } - else { - return this.value.toString(); - } - } - String prefix = this.getClass().getName(); prefix = prefix + " identifier "+this.getIdentifier(); From noreply at svn.ci.uchicago.edu Mon Feb 23 09:10:13 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 23 Feb 2009 09:10:13 -0600 (CST) Subject: [Swift-commit] r2595 - in trunk/tests/language: working working-base Message-ID: <20090223151014.086632281FB@www.ci.uchicago.edu> Author: benc Date: 2009-02-23 09:10:12 -0600 (Mon, 23 Feb 2009) New Revision: 2595 Added: trunk/tests/language/working-base/escapes.xml trunk/tests/language/working-base/operators.xml trunk/tests/language/working-base/underscore-in-id.xml trunk/tests/language/working/escapes.swift trunk/tests/language/working/operators.swift trunk/tests/language/working/underscore-in-id.swift Modified: trunk/tests/language/working-base/struct1.xml trunk/tests/language/working/struct1.swift Log: Additional tests for the lexer (based on information from EMMA coverage test) Added: trunk/tests/language/working/escapes.swift =================================================================== --- trunk/tests/language/working/escapes.swift (rev 0) +++ trunk/tests/language/working/escapes.swift 2009-02-23 15:10:12 UTC (rev 2595) @@ -0,0 +1,16 @@ + +string s = "\n"; + +string t = "\r"; + +string u = "\t"; + +string v = "\b"; + +string w = "\f"; + +string x = "\""; + +string y = "\'"; + +string z = "\\"; Added: trunk/tests/language/working/operators.swift =================================================================== --- trunk/tests/language/working/operators.swift (rev 0) +++ trunk/tests/language/working/operators.swift 2009-02-23 15:10:12 UTC (rev 2595) @@ -0,0 +1,5 @@ + +boolean a = (1 != 2); + +int b = +3; + Modified: trunk/tests/language/working/struct1.swift =================================================================== --- trunk/tests/language/working/struct1.swift 2009-02-23 15:06:32 UTC (rev 2594) +++ trunk/tests/language/working/struct1.swift 2009-02-23 15:10:12 UTC (rev 2595) @@ -4,3 +4,11 @@ int c; int d; } + +type bar { + int a; + int b[], c, d[], e, f, g[]; + int h, i[]; + int j; + int k; +} Added: trunk/tests/language/working/underscore-in-id.swift =================================================================== --- trunk/tests/language/working/underscore-in-id.swift (rev 0) +++ trunk/tests/language/working/underscore-in-id.swift 2009-02-23 15:10:12 UTC (rev 2595) @@ -0,0 +1,2 @@ + +int a_b_c = 3; Added: trunk/tests/language/working-base/escapes.xml =================================================================== --- trunk/tests/language/working-base/escapes.xml (rev 0) +++ trunk/tests/language/working-base/escapes.xml 2009-02-23 15:10:12 UTC (rev 2595) @@ -0,0 +1,44 @@ + + + + s + \n + + + + t + \r + + + + u + \t + + + + v + \b + + + + w + \f + + + + x + " + + + + y + \' + + + + z + \ + + Added: trunk/tests/language/working-base/operators.xml =================================================================== --- trunk/tests/language/working-base/operators.xml (rev 0) +++ trunk/tests/language/working-base/operators.xml 2009-02-23 15:10:12 UTC (rev 2595) @@ -0,0 +1,17 @@ + + + + a + + 1 + 2 + + + + + b + 3 + + Modified: trunk/tests/language/working-base/struct1.xml =================================================================== --- trunk/tests/language/working-base/struct1.xml 2009-02-23 15:06:32 UTC (rev 2594) +++ trunk/tests/language/working-base/struct1.xml 2009-02-23 15:10:12 UTC (rev 2595) @@ -24,6 +24,56 @@ int + + + bar + + + + a + int + + + b + int[] + + + c + int + + + d + int[] + + + e + int + + + f + int + + + g + int[] + + + h + int + + + i + int[] + + + j + int + + + k + int + + Added: trunk/tests/language/working-base/underscore-in-id.xml =================================================================== --- trunk/tests/language/working-base/underscore-in-id.xml (rev 0) +++ trunk/tests/language/working-base/underscore-in-id.xml 2009-02-23 15:10:12 UTC (rev 2595) @@ -0,0 +1,9 @@ + + + + a_b_c + 3 + + From noreply at svn.ci.uchicago.edu Tue Feb 24 06:45:14 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 24 Feb 2009 06:45:14 -0600 (CST) Subject: [Swift-commit] r2596 - trunk/src/org/griphyn/vdl/util Message-ID: <20090224124514.B725D228209@www.ci.uchicago.edu> Author: benc Date: 2009-02-24 06:45:13 -0600 (Tue, 24 Feb 2009) New Revision: 2596 Modified: trunk/src/org/griphyn/vdl/util/VDL2Config.java Log: Host parameter for getConfig, to facilitate more per-host configuration options. Modified: trunk/src/org/griphyn/vdl/util/VDL2Config.java =================================================================== --- trunk/src/org/griphyn/vdl/util/VDL2Config.java 2009-02-23 15:10:12 UTC (rev 2595) +++ trunk/src/org/griphyn/vdl/util/VDL2Config.java 2009-02-24 12:45:13 UTC (rev 2596) @@ -11,9 +11,15 @@ import java.util.List; import java.util.Properties; +import org.apache.log4j.Logger; + +import org.globus.cog.karajan.util.BoundContact; import org.globus.common.CoGProperties; public class VDL2Config extends Properties { + + public static final Logger logger = Logger.getLogger(VDL2Config.class); + public static final String CONFIG_FILE_NAME = "swift.properties"; public static final String[] CONFIG_FILE_SEARCH_PATH = new String[] { System.getProperty("vds.home") + File.separator + "etc" + File.separator @@ -82,6 +88,7 @@ put("replication.min.queue.time", "60"); put("replication.limit", "3"); put("status.mode", "files"); + put("wrapper.invocation.mode", "absolute"); } private VDL2Config(VDL2Config other) { @@ -205,4 +212,21 @@ return conf; } + public String getProperty(String name, BoundContact bc) { +System.err.println("getProperty "+name+" with bc="+bc); + if(bc!=null) { + if(logger.isDebugEnabled()) { + logger.debug("Checking BoundContact "+bc+" for property "+name); + } + String prop = (String) bc.getProperty(name); + if(prop != null) { + return prop; + } + } + if(logger.isDebugEnabled()) { + logger.debug("Getting property "+name+" from global configuration"); + } + return getProperty(name); + } + } From noreply at svn.ci.uchicago.edu Tue Feb 24 06:47:17 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 24 Feb 2009 06:47:17 -0600 (CST) Subject: [Swift-commit] r2597 - in trunk: . docs etc src/org/griphyn/vdl/karajan tests/misc tests/sites/wonky Message-ID: <20090224124717.C4D71228209@www.ci.uchicago.edu> Author: benc Date: 2009-02-24 06:47:17 -0600 (Tue, 24 Feb 2009) New Revision: 2597 Added: trunk/tests/misc/relative-wrapper.sh trunk/tests/misc/wonky-relative-absolute-wrapper.sh trunk/tests/misc/wonky-wrongdir-relative-fail.sh trunk/tests/sites/wonky/relative-absolute-wrapper.xml Modified: trunk/CHANGES.txt trunk/docs/userguide.xml trunk/etc/swift.properties trunk/src/org/griphyn/vdl/karajan/VDSTaskTransformer.java trunk/tests/misc/run trunk/tests/misc/run-wonky Log: Provide a configuration option wrapper.invocation.mode, specifiable either globally in the configuration file or per-site as a profile key, that configures whether wrapper script invocations are made with an absolute path (as was the behaviour in Swift 0.8) or with a relative path (as was behaviour in previous versions of Swift). Modified: trunk/CHANGES.txt =================================================================== --- trunk/CHANGES.txt 2009-02-24 12:45:13 UTC (rev 2596) +++ trunk/CHANGES.txt 2009-02-24 12:47:17 UTC (rev 2597) @@ -1,3 +1,10 @@ +(02/09/24) +*** New configuration option wrapper.invocation.mode, specifiable either + globally in the configuration file or per-site as a profile key, + that configures whether wrapper script invocations are made with an + absolute path (as was the behaviour in Swift 0.8) or with a relative + path (as was behaviour in previous versions of Swift). + (02/09/22) *** Recompilation will happen if a .kml file was compiled with a different version of Swift to the version being invoked. This is in addition to the Modified: trunk/docs/userguide.xml =================================================================== --- trunk/docs/userguide.xml 2009-02-24 12:45:13 UTC (rev 2596) +++ trunk/docs/userguide.xml 2009-02-24 12:47:17 UTC (rev 2597) @@ -2864,7 +2864,33 @@ + + + wrapper.invocation.mode + + + +Valid values: absolute, relative + + +Default value: absolute + + +Determines if Swift remote wrappers will be executed by specifying an +absolute path, or a path relative to the job initial working directory. +In most cases, execution will be successful with either option. However, +some execution sites ignore the specified initial working directory, and +so absolute must be used. Conversely on some sites, +job directories appear in a different place on the worker node file system +than on the filesystem access node, with the execution system handling +translation of the job initial working directory. In such cases, +relative mode must be used. + + + + + Modified: trunk/etc/swift.properties =================================================================== --- trunk/etc/swift.properties 2009-02-24 12:45:13 UTC (rev 2596) +++ trunk/etc/swift.properties 2009-02-24 12:47:17 UTC (rev 2597) @@ -295,3 +295,10 @@ # # status.mode=files + +# Determines if Swift remote wrappers will be executed by specifying an +# absolute path, or a path relative to the job initial working directory +# +# valid values: absolute, relative +# wrapper.invocation.mode=absolute + Modified: trunk/src/org/griphyn/vdl/karajan/VDSTaskTransformer.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/VDSTaskTransformer.java 2009-02-24 12:45:13 UTC (rev 2596) +++ trunk/src/org/griphyn/vdl/karajan/VDSTaskTransformer.java 2009-02-24 12:47:17 UTC (rev 2597) @@ -3,6 +3,7 @@ */ package org.griphyn.vdl.karajan; +import java.io.IOException; import java.util.HashSet; import java.util.LinkedList; import java.util.List; @@ -23,6 +24,7 @@ import org.griphyn.common.catalog.TransformationCatalogEntry; import org.griphyn.common.classes.TCType; import org.griphyn.vdl.util.FQN; +import org.griphyn.vdl.util.VDL2Config; public class VDSTaskTransformer implements TaskTransformer { public static final Logger logger = Logger.getLogger(VDSTaskTransformer.class); @@ -123,11 +125,19 @@ List l = spec.getArgumentsAsList(); // perhaps should check for /bin/bash in the executable, or some other way of detecting we need to do a substitution here... or equally could assume that the second parameter always needs to undergo this substitution... String executable = (String)l.get(0); - if(executable.endsWith("shared/wrapper.sh") || - executable.endsWith("shared/seq.sh")) { + try { + VDL2Config config = VDL2Config.getConfig(); + + if(config.getProperty("wrapper.invocation.mode", bc).equals("absolute") + &&(executable.endsWith("shared/wrapper.sh") + || executable.endsWith("shared/seq.sh"))) { + String s = spec.getDirectory()+"/"+executable; l.set(0,s); + } + } catch(IOException ioe) { + throw new KarajanRuntimeException("Could not determine wrapper invocation mode", ioe); } } Added: trunk/tests/misc/relative-wrapper.sh =================================================================== --- trunk/tests/misc/relative-wrapper.sh (rev 0) +++ trunk/tests/misc/relative-wrapper.sh 2009-02-24 12:47:17 UTC (rev 2597) @@ -0,0 +1,20 @@ +#!/bin/bash + +cd ../language-behaviour + +./generate-tc.data + +cat tc.data ../sites/tc.data > tmp.tc.data.sites + +SITE=wonky/wrongdir.xml + +echo testing site configuration: $SITE + +export CF=swift.properties.wrongdir-relative-fail +cat $(dirname $(which swift))/../etc/swift.properties | grep --invert-match -E '^wrapper.invocation.mode=' > $CF +echo wrapper.invocation.mode=relative >> $CF + +export SWIFT_TEST_PARAMS="-config $CF" + +./run 001-echo + Property changes on: trunk/tests/misc/relative-wrapper.sh ___________________________________________________________________ Name: svn:executable + * Modified: trunk/tests/misc/run =================================================================== --- trunk/tests/misc/run 2009-02-24 12:45:13 UTC (rev 2596) +++ trunk/tests/misc/run 2009-02-24 12:47:17 UTC (rev 2597) @@ -2,6 +2,7 @@ for a in clusters no-retries dryrun typecheck path-prefix restart restart2 restart3 restart4 restart5 restart-iterate workernode-local \ ordering-extern-notlazy restart-extern ordering-extern \ external-mapper-args extract-int-delayed asserts \ +relative-wrapper \ ; do ./${a}.sh R=$? Modified: trunk/tests/misc/run-wonky =================================================================== --- trunk/tests/misc/run-wonky 2009-02-24 12:45:13 UTC (rev 2596) +++ trunk/tests/misc/run-wonky 2009-02-24 12:47:17 UTC (rev 2597) @@ -6,6 +6,7 @@ wonky-replication \ wonky \ wonky80 \ + wonky-wrongdir-relative-fail \ ; do ./${a}.sh R=$? Added: trunk/tests/misc/wonky-relative-absolute-wrapper.sh =================================================================== --- trunk/tests/misc/wonky-relative-absolute-wrapper.sh (rev 0) +++ trunk/tests/misc/wonky-relative-absolute-wrapper.sh 2009-02-24 12:47:17 UTC (rev 2597) @@ -0,0 +1,25 @@ +#!/bin/bash + +cd ../language-behaviour + +./generate-tc.data + +cat tc.data ../sites/tc.data > tmp.tc.data.sites.1 + +cat tmp.tc.data.sites.1 | sed 's/localhost/wonkyA/' >> tmp.tc.data.sites +cat tmp.tc.data.sites.1 | sed 's/localhost/wonkyB/' >> tmp.tc.data.sites + + +SITE=wonky/relative-absolute-wrapper.xml + +echo testing site configuration: $SITE + +export CF=swift.properties.wrongdir-relative-fail +cat $(dirname $(which swift))/../etc/swift.properties | grep --invert-match -E '^execution.retries=' | grep --invert-match -E '^wrapper.invocation.mode=' > $CF +echo execution.retries=0 >> $CF +echo wrapper.invocation.mode=relative >> $CF + +export SWIFT_TEST_PARAMS="-sites.file ../sites/${SITE} -tc.file tmp.tc.data.sites -config $CF" + +./run 130-fmri + Property changes on: trunk/tests/misc/wonky-relative-absolute-wrapper.sh ___________________________________________________________________ Name: svn:executable + * Added: trunk/tests/misc/wonky-wrongdir-relative-fail.sh =================================================================== --- trunk/tests/misc/wonky-wrongdir-relative-fail.sh (rev 0) +++ trunk/tests/misc/wonky-wrongdir-relative-fail.sh 2009-02-24 12:47:17 UTC (rev 2597) @@ -0,0 +1,29 @@ +#!/bin/bash + +cd ../language-behaviour + +./generate-tc.data + +cat tc.data ../sites/tc.data > tmp.tc.data.sites + +SITE=wonky/wrongdir.xml + +echo testing site configuration: $SITE + +export CF=swift.properties.wrongdir-relative-fail +cat $(dirname $(which swift))/../etc/swift.properties | grep --invert-match -E '^wrapper.invocation.mode=' > $CF +echo wrapper.invocation.mode=relative >> $CF + +export SWIFT_TEST_PARAMS="-sites.file ../sites/${SITE} -tc.file tmp.tc.data.sites -config $CF" + +./run 001-echo + +if [ "$?" = "0" ] ; then + echo "wonky-wrongdir-relative-fail test passed when it should have failed" + exit 1 +else + echo "wonky-wrongdir-relative-fail failed as expected, so test passes" + exit 0 +fi + + Property changes on: trunk/tests/misc/wonky-wrongdir-relative-fail.sh ___________________________________________________________________ Name: svn:executable + * Added: trunk/tests/sites/wonky/relative-absolute-wrapper.xml =================================================================== --- trunk/tests/sites/wonky/relative-absolute-wrapper.xml (rev 0) +++ trunk/tests/sites/wonky/relative-absolute-wrapper.xml 2009-02-24 12:47:17 UTC (rev 2597) @@ -0,0 +1,19 @@ + + + + + + /var/tmp/swift-test/A + 0 + + + + + + + /var/tmp/swift-test/B + 0 + absolute + + + From noreply at svn.ci.uchicago.edu Tue Feb 24 08:27:08 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 24 Feb 2009 08:27:08 -0600 (CST) Subject: [Swift-commit] r2598 - trunk/src/org/griphyn/vdl/util Message-ID: <20090224142708.BD7CF228115@www.ci.uchicago.edu> Author: benc Date: 2009-02-24 08:27:06 -0600 (Tue, 24 Feb 2009) New Revision: 2598 Modified: trunk/src/org/griphyn/vdl/util/VDL2Config.java Log: remove accidentally committed println Modified: trunk/src/org/griphyn/vdl/util/VDL2Config.java =================================================================== --- trunk/src/org/griphyn/vdl/util/VDL2Config.java 2009-02-24 12:47:17 UTC (rev 2597) +++ trunk/src/org/griphyn/vdl/util/VDL2Config.java 2009-02-24 14:27:06 UTC (rev 2598) @@ -213,7 +213,6 @@ } public String getProperty(String name, BoundContact bc) { -System.err.println("getProperty "+name+" with bc="+bc); if(bc!=null) { if(logger.isDebugEnabled()) { logger.debug("Checking BoundContact "+bc+" for property "+name); From noreply at svn.ci.uchicago.edu Tue Feb 24 09:16:55 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 24 Feb 2009 09:16:55 -0600 (CST) Subject: [Swift-commit] r2599 - www Message-ID: <20090224151655.EB1F8228209@www.ci.uchicago.edu> Author: benc Date: 2009-02-24 09:16:55 -0600 (Tue, 24 Feb 2009) New Revision: 2599 Removed: www/architecture.shtml www/links.shtml www/menu.html www/projects.shtml Log: remove leftovers from long ago website redesign Deleted: www/architecture.shtml =================================================================== --- www/architecture.shtml 2009-02-24 14:27:06 UTC (rev 2598) +++ www/architecture.shtml 2009-02-24 15:16:55 UTC (rev 2599) @@ -1,21 +0,0 @@ - - - - Swift Architecture - - - - - - - - - -
- - -

Swift Architecture

-

Lorem ipsum again

-
- - Deleted: www/links.shtml =================================================================== --- www/links.shtml 2009-02-24 14:27:06 UTC (rev 2598) +++ www/links.shtml 2009-02-24 15:16:55 UTC (rev 2599) @@ -1,38 +0,0 @@ - - - - Swift Links - - - - - - - - - -
- - -

Swift Links

-

Globus Toolkit

-

- Swift uses the Globus Toolkit as middleware to talk to various resources. -

- -

Java CoG Kit

-

- The CoG Kit provides a high level uniform interface to different versions of the Globus Toolkit. It also contains the Karajan language and interpreter which is used as an underlying execution engine for Swift. -

- -

TeraGrid

- -

- "TeraGrid is an open scientific discovery infrastructure combining leadership class resources at nine partner sites to create an integrated, persistent computational resource." - -

- - -
- - Deleted: www/menu.html =================================================================== --- www/menu.html 2009-02-24 14:27:06 UTC (rev 2598) +++ www/menu.html 2009-02-24 15:16:55 UTC (rev 2599) @@ -1,31 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Home
Documentation
Download
Projects
Links
Contact
 
Deleted: www/projects.shtml =================================================================== --- www/projects.shtml 2009-02-24 14:27:06 UTC (rev 2598) +++ www/projects.shtml 2009-02-24 15:16:55 UTC (rev 2599) @@ -1,45 +0,0 @@ - - - - Swift Projects - - - - - - - - - -
- - -

Swift Projects

- - -

BRIC

- -

-Neuroscience research using the SwiftScript workflow language and the Swift workflow engine. -

- - - - - - - - - -
- - - -
- Flow Graph -
- - -
- - From noreply at svn.ci.uchicago.edu Tue Feb 24 09:17:29 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 24 Feb 2009 09:17:29 -0600 (CST) Subject: [Swift-commit] r2600 - www Message-ID: <20090224151729.B238E228209@www.ci.uchicago.edu> Author: benc Date: 2009-02-24 09:17:29 -0600 (Tue, 24 Feb 2009) New Revision: 2600 Removed: www/swift-logo.jpg Log: remove a logo that is not the swift logo Deleted: www/swift-logo.jpg =================================================================== (Binary files differ) From noreply at svn.ci.uchicago.edu Tue Feb 24 09:18:51 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 24 Feb 2009 09:18:51 -0600 (CST) Subject: [Swift-commit] r2601 - in www: . about apps docs downloads links papers support Message-ID: <20090224151851.27DA7228209@www.ci.uchicago.edu> Author: benc Date: 2009-02-24 09:18:50 -0600 (Tue, 24 Feb 2009) New Revision: 2601 Modified: www/about/index.php www/apps/index.php www/docs/index.php www/downloads/index.php www/index.php www/links/index.php www/papers/index.php www/support/index.php Log: add google analytics to main web pages Modified: www/about/index.php =================================================================== --- www/about/index.php 2009-02-24 15:17:29 UTC (rev 2600) +++ www/about/index.php 2009-02-24 15:18:50 UTC (rev 2601) @@ -50,5 +50,16 @@ + + + + Modified: www/apps/index.php =================================================================== --- www/apps/index.php 2009-02-24 15:17:29 UTC (rev 2600) +++ www/apps/index.php 2009-02-24 15:18:50 UTC (rev 2601) @@ -41,5 +41,14 @@ + + Modified: www/docs/index.php =================================================================== --- www/docs/index.php 2009-02-24 15:17:29 UTC (rev 2600) +++ www/docs/index.php 2009-02-24 15:18:50 UTC (rev 2601) @@ -123,5 +123,14 @@ + + Modified: www/downloads/index.php =================================================================== --- www/downloads/index.php 2009-02-24 15:17:29 UTC (rev 2600) +++ www/downloads/index.php 2009-02-24 15:18:50 UTC (rev 2601) @@ -247,5 +247,14 @@ + + Modified: www/index.php =================================================================== --- www/index.php 2009-02-24 15:17:29 UTC (rev 2600) +++ www/index.php 2009-02-24 15:18:50 UTC (rev 2601) @@ -92,5 +92,16 @@ + + + + Modified: www/links/index.php =================================================================== --- www/links/index.php 2009-02-24 15:17:29 UTC (rev 2600) +++ www/links/index.php 2009-02-24 15:18:50 UTC (rev 2601) @@ -70,5 +70,14 @@ + + Modified: www/papers/index.php =================================================================== --- www/papers/index.php 2009-02-24 15:17:29 UTC (rev 2600) +++ www/papers/index.php 2009-02-24 15:18:50 UTC (rev 2601) @@ -278,5 +278,14 @@ + + Modified: www/support/index.php =================================================================== --- www/support/index.php 2009-02-24 15:17:29 UTC (rev 2600) +++ www/support/index.php 2009-02-24 15:18:50 UTC (rev 2601) @@ -82,5 +82,14 @@ + + From noreply at svn.ci.uchicago.edu Tue Feb 24 10:50:30 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 24 Feb 2009 10:50:30 -0600 (CST) Subject: [Swift-commit] r2602 - trunk/docs/formatting Message-ID: <20090224165030.A1B6E228209@www.ci.uchicago.edu> Author: benc Date: 2009-02-24 10:50:27 -0600 (Tue, 24 Feb 2009) New Revision: 2602 Removed: trunk/docs/formatting/custom_fo.xsl trunk/docs/formatting/custom_html.xsl trunk/docs/formatting/vdl2_html.xsl trunk/docs/formatting/vdl2sh_html.xsl Log: remove vestigial stylesheets Deleted: trunk/docs/formatting/custom_fo.xsl =================================================================== --- trunk/docs/formatting/custom_fo.xsl 2009-02-24 15:18:50 UTC (rev 2601) +++ trunk/docs/formatting/custom_fo.xsl 2009-02-24 16:50:27 UTC (rev 2602) @@ -1,130 +0,0 @@ - -? - -? - - - - - - - - -? - - - - - -left - -? - -1 - -/www/www-unix.globus.org/docbook-images/ -? - -? - - -? - -14pt -? - -? - - - - -? - - -1 - -1 -? - -2 -? - -1 - - - - -? - -underline - - - - - -1 - -? - -true -true - - - -yes - -? - -/www/www-unix.globus.org/toolkit/docs/development/4.2-drafts/glossary.xml - -? - -1 -? - - -? - -? - -0pt -0pt -0pc - - - \ No newline at end of file Deleted: trunk/docs/formatting/custom_html.xsl =================================================================== --- trunk/docs/formatting/custom_html.xsl 2009-02-24 15:18:50 UTC (rev 2601) +++ trunk/docs/formatting/custom_html.xsl 2009-02-24 16:50:27 UTC (rev 2602) @@ -1,236 +0,0 @@ - - - - - -1 - - -1 - -/docbook-images/ - - - - - -/docbook-images/ - -0 - - - -0 - -1 - -2 - -4 - -1 - - - - - - - - - - - - - - - - - - - - - - -1 - -1 - - -yes - - - - - - - - - - - - - - - - - - - - - - - -section-3 - - - - - - - - - - - - - - - - - - - - - - - - - - - include_once("http://www.globus.org/toolkit/docs/4.0/includes/globus_header_docbook.inc"); - ? - - - - - - - - - - - - - - include_once("http://www.globus.org/toolkit/docs/4.0/includes/globus_footer_docbook.inc"); - ? - - - - - - - - - - - - - -title - - - - - - - - - - - - - - \ No newline at end of file Deleted: trunk/docs/formatting/vdl2_html.xsl =================================================================== --- trunk/docs/formatting/vdl2_html.xsl 2009-02-24 15:18:50 UTC (rev 2601) +++ trunk/docs/formatting/vdl2_html.xsl 2009-02-24 16:50:27 UTC (rev 2602) @@ -1,231 +0,0 @@ - - - - - - - - - 1 - - - 1 - - /docbook-images/ - - - - - - /docbook-images/ - - 0 - - - - 0 - - 1 - - 2 - - 4 - - 1 - - - - - - - - - - - - - - - - - - - - - - - 1 - - 1 - - - yes - - - - - - - - - - - - - - - - - - - - - - - - section-3 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - title - - - - - - - - - - - - - - - - - - - - - ]]> - - - - - - - - ]]> - - - Deleted: trunk/docs/formatting/vdl2sh_html.xsl =================================================================== --- trunk/docs/formatting/vdl2sh_html.xsl 2009-02-24 15:18:50 UTC (rev 2601) +++ trunk/docs/formatting/vdl2sh_html.xsl 2009-02-24 16:50:27 UTC (rev 2602) @@ -1,234 +0,0 @@ - - - - - - - - - 1 - - - 1 - - /docbook-images/ - - - - - - /docbook-images/ - - 0 - - - - 0 - - 1 - - 2 - - 4 - - 1 - - - - - - - - - - - - - - - - - - - - - - - 1 - - 1 - - - yes - - - - - - - - - - - - - - - - - - - - - - - - - - - section-3 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - title - - - - - - - - - - - - - - - - - - - - - ]]> - - - - - - - - ]]> - - - From noreply at svn.ci.uchicago.edu Tue Feb 24 10:57:37 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 24 Feb 2009 10:57:37 -0600 (CST) Subject: [Swift-commit] r2603 - trunk/docs/formatting Message-ID: <20090224165737.75528228187@www.ci.uchicago.edu> Author: benc Date: 2009-02-24 10:57:36 -0600 (Tue, 24 Feb 2009) New Revision: 2603 Modified: trunk/docs/formatting/swift_html.xsl trunk/docs/formatting/swiftsh_html.xsl trunk/docs/formatting/swiftsh_html_chunked.xsl Log: add Google Analytics to manuals Modified: trunk/docs/formatting/swift_html.xsl =================================================================== --- trunk/docs/formatting/swift_html.xsl 2009-02-24 16:50:27 UTC (rev 2602) +++ trunk/docs/formatting/swift_html.xsl 2009-02-24 16:57:36 UTC (rev 2603) @@ -180,6 +180,17 @@ + + + + + @@ -242,4 +253,4 @@ ]]> - \ No newline at end of file + Modified: trunk/docs/formatting/swiftsh_html.xsl =================================================================== --- trunk/docs/formatting/swiftsh_html.xsl 2009-02-24 16:50:27 UTC (rev 2602) +++ trunk/docs/formatting/swiftsh_html.xsl 2009-02-24 16:57:36 UTC (rev 2603) @@ -183,6 +183,17 @@ + + + + Modified: trunk/docs/formatting/swiftsh_html_chunked.xsl =================================================================== --- trunk/docs/formatting/swiftsh_html_chunked.xsl 2009-02-24 16:50:27 UTC (rev 2602) +++ trunk/docs/formatting/swiftsh_html_chunked.xsl 2009-02-24 16:57:36 UTC (rev 2603) @@ -198,6 +198,15 @@ + + + From noreply at svn.ci.uchicago.edu Tue Feb 24 11:00:28 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 24 Feb 2009 11:00:28 -0600 (CST) Subject: [Swift-commit] r2604 - www/downloads Message-ID: <20090224170028.DB8E1228187@www.ci.uchicago.edu> Author: benc Date: 2009-02-24 11:00:28 -0600 (Tue, 24 Feb 2009) New Revision: 2604 Modified: www/downloads/index.php Log: google analytics on downloads Modified: www/downloads/index.php =================================================================== --- www/downloads/index.php 2009-02-24 16:57:36 UTC (rev 2603) +++ www/downloads/index.php 2009-02-24 17:00:28 UTC (rev 2604) @@ -30,9 +30,15 @@

Swift 0.8 - 2009/01/30

Swift v0.8 is a development release intended to release functionality and fixes that have gone in to trunk since v0.7. -[swift-0.8.tar.gz] -[swift-0.8-stripped.tar.gz - for installation on systems with existing grid stacks] -[release-notes-0.8.txt] +[swift-0.8.tar.gz] +[swift-0.8-stripped.tar.gz - for installation on systems with existing grid stacks] +[release-notes-0.8.txt]

@@ -248,7 +248,7 @@ - + From noreply at svn.ci.uchicago.edu Thu Feb 26 07:17:33 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Thu, 26 Feb 2009 07:17:33 -0600 (CST) Subject: [Swift-commit] r2619 - www Message-ID: <20090226131733.CB3C6228187@www.ci.uchicago.edu> Author: benc Date: 2009-02-26 07:17:33 -0600 (Thu, 26 Feb 2009) New Revision: 2619 Modified: www/sitemap.xml Log: lower sitemap priority of historical language spec docs, as they are getting highly ranked in search results Modified: www/sitemap.xml =================================================================== --- www/sitemap.xml 2009-02-26 13:01:39 UTC (rev 2618) +++ www/sitemap.xml 2009-02-26 13:17:33 UTC (rev 2619) @@ -4,4 +4,22 @@ http://www.ci.uchicago.edu/swift/ 1.0 + + http://www.ci.uchicago.edu/swift/guides/historical/languagespec.php + 0.2 + + + + http://www.ci.uchicago.edu/swift/guides/historical/languagespec.pdf + 0.2 + + + http://www.ci.uchicago.edu/swift/guides/historical/languagespec-0.6.php + 0.2 + + + + http://www.ci.uchicago.edu/swift/guides/historical/languagespec-0.6.pdf + 0.2 + From noreply at svn.ci.uchicago.edu Fri Feb 27 05:51:52 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 27 Feb 2009 05:51:52 -0600 (CST) Subject: [Swift-commit] r2621 - log-processing/bin Message-ID: <20090227115152.E322D2281FB@www.ci.uchicago.edu> Author: benc Date: 2009-02-27 05:51:51 -0600 (Fri, 27 Feb 2009) New Revision: 2621 Modified: log-processing/bin/swift-log-to-cedps log-processing/bin/swift-plot-log Log: change libexec paths in log-processing bin/ scripts to reflect location change in r2620 Modified: log-processing/bin/swift-log-to-cedps =================================================================== --- log-processing/bin/swift-log-to-cedps 2009-02-27 11:42:31 UTC (rev 2620) +++ log-processing/bin/swift-log-to-cedps 2009-02-27 11:51:51 UTC (rev 2621) @@ -6,7 +6,7 @@ # $1 should be the pathname of the log file to plot -LOG_CODE_HOME="`dirname $0`/../libexec/" +LOG_CODE_HOME="`dirname $0`/../libexec/log-processing/" export SWIFT_PLOT_HOME=$LOG_CODE_HOME export PATH=${LOG_CODE_HOME}:${PATH} Modified: log-processing/bin/swift-plot-log =================================================================== --- log-processing/bin/swift-plot-log 2009-02-27 11:42:31 UTC (rev 2620) +++ log-processing/bin/swift-plot-log 2009-02-27 11:51:51 UTC (rev 2621) @@ -6,7 +6,7 @@ # $1 should be the pathname of the log file to plot -LOG_CODE_HOME="`dirname $0`/../libexec/" +LOG_CODE_HOME="`dirname $0`/../libexec/log-processing/" export SWIFT_PLOT_HOME=$LOG_CODE_HOME export PATH=${LOG_CODE_HOME}:${PATH} From noreply at svn.ci.uchicago.edu Fri Feb 27 08:38:31 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 27 Feb 2009 08:38:31 -0600 (CST) Subject: [Swift-commit] r2622 - log-processing Message-ID: <20090227143831.DB9D622821D@www.ci.uchicago.edu> Author: benc Date: 2009-02-27 08:38:31 -0600 (Fri, 27 Feb 2009) New Revision: 2622 Removed: log-processing/README Log: remove README from here - relevant details move in a different commit into the docs/log-processing document Deleted: log-processing/README =================================================================== --- log-processing/README 2009-02-27 11:51:51 UTC (rev 2621) +++ log-processing/README 2009-02-27 14:38:31 UTC (rev 2622) @@ -1,151 +0,0 @@ -This is the README for the swift log-processing package. - -Some prerequisites -================== -gnuplot 4.0 -gnu m4 -gnu textutils -perl - -Basic use -========= - -To generate a webpage: - -add /bin to your path - -swift-plot-log modelproc-20080918-1644-nyjqiddf.log - -You will be left with a directory - -report-modelproc-20080918-1644-nyjqiddf/ - -in which you will find - -report-modelproc-20080918-1644-nyjqiddf/index.html - -Open that with your favourite web browser. - - - -Other Commandline Tools -======================= - - ./execute2-status-from-log [logfile] - lists every (execute2) job and its final status - - ./execute2-summary-from-log [logfile] - lists the counts of each final job status in log - -There are various event streams, identified by a base name: - execute - an internal procedure in swift - execute2 - " - dostagein - " - dostageout - " - - kickstart - kickstart records (not available as transitions) - falkon - complete falkon job executions (not available as transitions) - - karatasks - karajan level tasks, available as transitions - (there are also four substreams karatasks.FILE_OPERATION, - karatasks.FILE_TRANSFER and karatasks.JOB_SUBMISSION available - as events but not transitions) - - workflow - the workflow execution itself (not available as transitions) - -Streams are generated from their source log files either as .transitions -files or .event files. - -If a .transitions file is available, make knows how to convert that into -an .event file: type make foo.event - -Make knows how to make various plots. For example: - - make foo.png will plot the foo stream - make foo-total.png will plot how many foos are in progress at any time - make foo.sorted-start.png will plot foos (like foo.png) ordered by start time - -Make can also make some statistics: - - make foo.stats will output a text file foo.stats - -CEPDS -===== - -For any event stream that is available in .transition form, - make foo.cedps -will produces a CEDPS-logging form of those transitions - -.event files -============ - -need an event file format of one event per line, with that line -containing start time and duration and other useful data. - -col1 = start, col2 = duration, col3 onwards = event specific data - for -some utilities for now should be column based, but later will maybe -move to attribute based. - -between col 1 and col 2 exactly one space -between col 2 and col 3 exactly one space - -start time is in seconds since unix epoch. start time should *not* be -normalised to start of workflow - -event files should not (for now) be assumed to be in order - -different event streams can be stored in different files. each event -stream should use the extension .event - - -.coloured-event files -===================== -third column is a colour index -first two columns as per .event (thus a coloured-event is a specific -form of .event) - - -t=+inf processing -=================== - -This is not implemented yet. - -For events, the event plots (previously) reflected the time between the -first and last transitions for an event. In the case of an event being -'in-progress' at the end of the known log info, this is misleading when -performing eg. total-events-in-progress graphs - at t=+inf, these events -should still be in-progress - -What is needed is that these events get something like a pseudo-transition -at t=+inf labelled as "INPROGRESS-AS-FOUND"; with t=+inf being some -appropriate value (which can perhaps only be determined after all other -events have been processed such that we can make +inf = maxtime + duration*0.1) - -For each channel, make tables of states. (we maybe need this for cdepds). -On each of these, label events booleanwise with 'in progress' or 'completed' -so 'WHATEVER_STARTED' is an in-progress event, whilst 'JOB_FAILURE' is not -an in-progress event. - -Then for each event that is in-progress, relabel each event whose final state -is an inprogress state as 'INPROGRESS-AS-FOUND' with the end time being -t=+inf - - -for first approx, maxtime is computed by last line of swift logfile - - ---- these are the events: -workflow - execute - execute2 - dostagein - karatasks (fileops and filetrans) - clustering (optional) - karatasks (execution) - cluster-log (optional) - wrapper log (optional) - kickstart log - dostageout - karatasks (fileops and filetrans) - ---- From noreply at svn.ci.uchicago.edu Fri Feb 27 08:39:51 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 27 Feb 2009 08:39:51 -0600 (CST) Subject: [Swift-commit] r2623 - trunk/docs Message-ID: <20090227143951.20A7622821D@www.ci.uchicago.edu> Author: benc Date: 2009-02-27 08:39:50 -0600 (Fri, 27 Feb 2009) New Revision: 2623 Modified: trunk/docs/log-processing.xml Log: information moved from log-processing README Modified: trunk/docs/log-processing.xml =================================================================== --- trunk/docs/log-processing.xml 2009-02-27 14:38:31 UTC (rev 2622) +++ trunk/docs/log-processing.xml 2009-02-27 14:39:50 UTC (rev 2623) @@ -23,6 +23,12 @@
+
Prerequisites + +gnuplot 4.0, gnu m4, gnu textutils, perl + + +
Web page about a run @@ -53,11 +59,114 @@ These streams are then used to provide the data for the various output formats, such as graphs, web pages and CEDPS log format. -The available basic stream names are: execute, execute2, kickstart, -info, karajan, clusters, stageout, -stagein, workflow +The available streams are: + + + + Stream nameDescription + + executeSwift procedure invocations + execute2individual execution attempts + kickstartkickstart records (not available as transitions) + karatasks karajan level tasks, available as transitions (there are also four substreams karatasks.FILE_OPERATION, karatasks.FILE_TRANSFER and karatasks.JOB_SUBMISSION available as events but not transitions) + workflowa single event representing the entire workflow + dostageinstage-in operations for execute2s + dostageoutstage-out operations for execute2s + + +
+ + +Streams are generated from their source log files either as .transitions +or .event files, for example by make foo.event. + + +Various plots are available based on different streams: + + + + makefile targetDescription + + foo.pngPlots the foo event stream + foo-total.pngPlots how many foo events are in progress at any time + foo.sorted-start.pngPlot like foo.png but ordered by start time + + +
+ + + +Text-based statistics are also available with make foo.stats. + + +Event streams are nested something like this: + + +workflow + execute + execute2 + dostagein + karatasks (fileops and filetrans) + clustering (optional) + karatasks (execution) + cluster-log (optional) + wrapper log (optional) + kickstart log + dostageout + karatasks (fileops and filetrans) + + +
+
Internal file formats +The log processing code generates a number of internal files that +follow a standard format. These are used for communication between the +modules that parse various log files to extract relevant information; and +the modules that generate plots and other summary information. + +need an event file format of one event per line, with that line +containing start time and duration and other useful data. + +col1 = start, col2 = duration, col3 onwards = event specific data - for +some utilities for now should be column based, but later will maybe +move to attribute based. + +between col 1 and col 2 exactly one space +between col 2 and col 3 exactly one space + +start time is in seconds since unix epoch. start time should *not* be +normalised to start of workflow + +event files should not (for now) be assumed to be in order + +different event streams can be stored in different files. each event +stream should use the extension .event + + + +.coloured-event files +===================== +third column is a colour index +first two columns as per .event (thus a coloured-event is a specific +form of .event) + + +
+ +
+There are a couple of hacky scripts that aren't made into proper +commandline tools. These are in the libexec/log-processing/ directory: + + + ./execute2-status-from-log [logfile] + lists every (execute2) job and its final status + + ./execute2-summary-from-log [logfile] + lists the counts of each final job status in log + + +
From noreply at svn.ci.uchicago.edu Fri Feb 27 09:15:07 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 27 Feb 2009 09:15:07 -0600 (CST) Subject: [Swift-commit] r2624 - log-processing/libexec Message-ID: <20090227151507.373742281FB@www.ci.uchicago.edu> Author: benc Date: 2009-02-27 09:15:06 -0600 (Fri, 27 Feb 2009) New Revision: 2624 Removed: log-processing/libexec/falkon/ Log: remove empty directory From noreply at svn.ci.uchicago.edu Fri Feb 27 12:22:00 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 27 Feb 2009 12:22:00 -0600 (CST) Subject: [Swift-commit] r2625 - trunk/docs Message-ID: <20090227182200.B29DD22821B@www.ci.uchicago.edu> Author: benc Date: 2009-02-27 12:21:59 -0600 (Fri, 27 Feb 2009) New Revision: 2625 Modified: trunk/docs/log-processing.xml Log: missed title tag Modified: trunk/docs/log-processing.xml =================================================================== --- trunk/docs/log-processing.xml 2009-02-27 15:15:06 UTC (rev 2624) +++ trunk/docs/log-processing.xml 2009-02-27 18:21:59 UTC (rev 2625) @@ -154,7 +154,7 @@
-
+
hacky scripts There are a couple of hacky scripts that aren't made into proper commandline tools. These are in the libexec/log-processing/ directory: From noreply at svn.ci.uchicago.edu Sat Feb 28 13:21:43 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sat, 28 Feb 2009 13:21:43 -0600 (CST) Subject: [Swift-commit] r2626 - SwiftApps/SIDGrid/scripts Message-ID: <20090228192143.1E71722821D@www.ci.uchicago.edu> Author: skenny Date: 2009-02-28 13:21:41 -0600 (Sat, 28 Feb 2009) New Revision: 2626 Added: SwiftApps/SIDGrid/scripts/autorecon2_BOTH.sh Log: wrapper for fs workflow Added: SwiftApps/SIDGrid/scripts/autorecon2_BOTH.sh =================================================================== --- SwiftApps/SIDGrid/scripts/autorecon2_BOTH.sh (rev 0) +++ SwiftApps/SIDGrid/scripts/autorecon2_BOTH.sh 2009-02-28 19:21:41 UTC (rev 2626) @@ -0,0 +1,35 @@ +#!/bin/bash + +# $1 == user +# $2 == subject +# $3 == .tar + +echo "untarring subject dir.................." +echo $3 + +tar -xf $3 + + +echo "freesurfer home is:::::" +echo $FREESURFER_HOME + +export SUBJECTS_DIR=$PWD +source $FREESURFER_HOME/SetUpFreeSurfer.sh + +## end recon1 proper, begin recon2 both hemis + +recon-all -s $2 -gcareg +recon-all -s $2 -canorm +recon-all -s $2 -careg +recon-all -s $2 -rmneck +recon-all -s $2 -skull-lta +recon-all -s $2 -calabel +recon-all -s $2 -segstats +recon-all -s $2 -normalization2 +recon-all -s $2 -segmentation +recon-all -s $2 -fill +recon-all -s $2 -tessellate + +tar cf $3 $2 +cp $3 results/ + Property changes on: SwiftApps/SIDGrid/scripts/autorecon2_BOTH.sh ___________________________________________________________________ Name: svn:executable + * From noreply at svn.ci.uchicago.edu Sat Feb 28 19:56:47 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sat, 28 Feb 2009 19:56:47 -0600 (CST) Subject: [Swift-commit] r2627 - SwiftApps/SIDGrid/swift/swift_scripts/freesurfer Message-ID: <20090301015647.343D8228187@www.ci.uchicago.edu> Author: skenny Date: 2009-02-28 19:56:44 -0600 (Sat, 28 Feb 2009) New Revision: 2627 Modified: SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/reconall.swift Log: Modified: SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/reconall.swift =================================================================== --- SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/reconall.swift 2009-02-28 19:21:41 UTC (rev 2626) +++ SwiftApps/SIDGrid/swift/swift_scripts/freesurfer/reconall.swift 2009-03-01 01:56:44 UTC (rev 2627) @@ -1,4 +1,4 @@ - type file {} +type file {} type mgzfile; (file subj_dir) recon1 (string subj, mgzfile inimage) { From noreply at svn.ci.uchicago.edu Sat Feb 28 20:58:15 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sat, 28 Feb 2009 20:58:15 -0600 (CST) Subject: [Swift-commit] r2628 - SwiftApps/SIDGrid/swift/swift_scripts/afni Message-ID: <20090301025815.A8F1C22821D@www.ci.uchicago.edu> Author: skenny Date: 2009-02-28 20:58:14 -0600 (Sat, 28 Feb 2009) New Revision: 2628 Added: SwiftApps/SIDGrid/swift/swift_scripts/afni/GAvol2surf.swift Log: for the GAproject surface projections Added: SwiftApps/SIDGrid/swift/swift_scripts/afni/GAvol2surf.swift =================================================================== --- SwiftApps/SIDGrid/swift/swift_scripts/afni/GAvol2surf.swift (rev 0) +++ SwiftApps/SIDGrid/swift/swift_scripts/afni/GAvol2surf.swift 2009-03-01 02:58:14 UTC (rev 2628) @@ -0,0 +1,38 @@ +#--- this is for the GAproject surface projections +#--- testing SUMA obj +type file{} + +type AFNI_obj{ + file HEAD; + file BRIK; +} + +type SUMA_obj{ + file smoothwm; + file pial; +} + +app (file out_1d) AFNI_Vol2Surf (file spec_file, SUMA_obj surface, string map, AFNI_obj surfvol, AFNI_obj grid_parent){ + AFNI_3dVol2Surf "-spec" @spec_file "-surf_A" @surface.smoothwm "-surf_B" @surface.pial + "-map_func" map "-f_steps" 15 "-f_index" "voxels" "-oob_index" "-1" "-oob_value" 0 "-no_headers" "-outcols_NSD_format" + "-sv" @surfvol.BRIK "-grid_parent" @grid_parent.BRIK "-out_1D" @filename(out_1d); +} + + +string mapFunctions[] = ["ave"]; +string hemis[] = ["lh","rh"]; +string conditions[] = ["speech"]; + +foreach map in mapFunctions{ + foreach h in hemis{ + foreach cc in conditions{ + file spec_file; + SUMA_obj surface; + AFNI_obj surfvol; + AFNI_obj grid_parent; + file output1D; + output1D = AFNI_Vol2Surf(spec_file, surface, map, surfvol, grid_parent); + } + } +} + From noreply at svn.ci.uchicago.edu Sat Feb 28 21:45:10 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sat, 28 Feb 2009 21:45:10 -0600 (CST) Subject: [Swift-commit] r2629 - SwiftApps/SIDGrid/config Message-ID: <20090301034510.1B130228187@www.ci.uchicago.edu> Author: skenny Date: 2009-02-28 21:45:09 -0600 (Sat, 28 Feb 2009) New Revision: 2629 Modified: SwiftApps/SIDGrid/config/tc.data Log: updates to sites Modified: SwiftApps/SIDGrid/config/tc.data =================================================================== --- SwiftApps/SIDGrid/config/tc.data 2009-03-01 02:58:14 UTC (rev 2628) +++ SwiftApps/SIDGrid/config/tc.data 2009-03-01 03:45:09 UTC (rev 2629) @@ -295,11 +295,11 @@ BSD Rcmd /disks/gpfs/fmri/cnari/scripts/Rcmd.sh INSTALLED INTEL32::LINUX env::R_LIBS=/disks/gpfs/fmri/cnari/apps/R_library BSD Mediator /disks/gpfs/fmri/cnari/scripts/Mediator.py INSTALLED INTEL32::LINUX null BSD env /usr/env INSTALLED INTEL32::LINUX null -BSD recon1 /disks/ci-gpfs/fmri/cnari/apps/freesurfer/bin/recon1.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/disks/ci-gpfs/fmri/cnari/apps/freesurfer;GLOBUS::host_types=ia32-compute,maxwalltime=1200 -BSD recon2L /disks/ci-gpfs/fmri/cnari/apps/freesurfer/bin/recon2L.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/disks/ci-gpfs/fmri/cnari/apps/freesurfer;GLOBUS::host_types=ia32-compute,maxwalltime=1200 -BSD recon2R /disks/ci-gpfs/fmri/cnari/apps/freesurfer/bin/recon2R.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/disks/ci-gpfs/fmri/cnari/apps/freesurfer;GLOBUS::host_types=ia32-compute,maxwalltime=1200 -BSD autorecon2_BOTH /disks/ci-gpfs/fmri/cnari/apps/freesurfer/bin/autorecon2_BOTH.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/disks/ci-gpfs/fmri/cnari/apps/freesurfer;GLOBUS::host_types=ia32-compute,maxwalltime=1200 -BSD reconFINAL /disks/ci-gpfs/fmri/cnari/apps/freesurfer/bin/reconFINAL.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/disks/ci-gpfs/fmri/cnari/apps/freesurfer;GLOBUS::host_types=ia32-compute,maxwalltime=1200 +BSD recon1 /disks/ci-gpfs/fmri/cnari/apps/freesurfer/bin/recon1.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/disks/ci-gpfs/fmri/cnari/apps/freesurfer +BSD recon2L /disks/ci-gpfs/fmri/cnari/apps/freesurfer/bin/recon2L.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/disks/ci-gpfs/fmri/cnari/apps/freesurfer +BSD recon2R /disks/ci-gpfs/fmri/cnari/apps/freesurfer/bin/recon2R.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/disks/ci-gpfs/fmri/cnari/apps/freesurfer +BSD autorecon2_BOTH /disks/ci-gpfs/fmri/cnari/apps/freesurfer/bin/autorecon2_BOTH.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/disks/ci-gpfs/fmri/cnari/apps/freesurfer +BSD reconFINAL /disks/ci-gpfs/fmri/cnari/apps/freesurfer/bin/reconFINAL.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/disks/ci-gpfs/fmri/cnari/apps/freesurfer;env::PATHPREFIX=/disks/ci-gpfs/fmri/SIDGrid/usr/bin ####-----------------ANLUCTERAGRID32----------------------------- ANLUCTERAGRID32 AFNI_1dcat /soft/community/SIDGrid/abin/1dcat INSTALLED INTEL32::LINUX null @@ -589,9 +589,9 @@ ANLUCTERAGRID32 recon2L /soft/community/SIDGrid/freesurfer/bin/recon2L.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/soft/community/SIDGrid/freesurfer;GLOBUS::host_types=ia32-compute,maxwalltime=1200 ANLUCTERAGRID32 recon2R /soft/community/SIDGrid/freesurfer/bin/recon2R.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/soft/community/SIDGrid/freesurfer;GLOBUS::host_types=ia32-compute,maxwalltime=1200 ANLUCTERAGRID32 autorecon2_BOTH /soft/community/SIDGrid/freesurfer/bin/autorecon2_BOTH.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/soft/community/SIDGrid/freesurfer;GLOBUS::host_types=ia32-compute,maxwalltime=1200 -ANLUCTERAGRID32 reconFINAL /soft/community/SIDGrid/freesurfer/bin/reconFINAL.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/soft/community/SIDGrid/freesurfer;GLOBUS::host_types=ia32-compute,maxwalltime=1200 +ANLUCTERAGRID32 reconFINAL /soft/community/SIDGrid/freesurfer/bin/reconFINAL.sh INSTALLED INTEL32::LINUX env::FREESURFER_HOME=/soft/community/SIDGrid/freesurfer,PATHPREFIX=/soft/community/SIDGrid/usr/bin;GLOBUS::host_types=ia32-compute,maxwalltime=1200 ANLUCTERAGRID32 R /soft/community/SIDGrid/R/bin/R INSTALLED INTEL32::LINUX null -ANLUCTERAGRID32 RInvoke /soft/community/SIDGrid/R/bin/RInvoke.sh INSTALLED INTEL32::LINUX null +ANLUCTERAGRID32 RInvoke /soft/community/SIDGrid/scripts/RInvoke.sh INSTALLED INTEL32::LINUX null ANLUCTERAGRID32 Mediator /soft/community/SIDGrid/scripts/Mediator.py INSTALLED INTEL32::LINUX env::PATHPREFIX=/soft/community/SIDGrid/usr/bin;GLOBUS::host_types=ia32-compute,maxwalltime=300 ANLUCTERAGRID32 Vol2Surf /soft/community/SIDGrid/abin/Vol2Surf INSTALLED INTEL32::LINUX null @@ -1718,7 +1718,7 @@ RANGER whereami /scratch/projects/tg/SIDGrid/usr/bin/whereami INSTALLED INTEL64::LINUX GLOBUS::maxwalltime=10;env::PATHPREFIX=/scratch/projects/tg/SIDGrid/usr/bin,LD_LIBRARY_PATH=/scratch/projects/tg/SIDGrid/usr/lib RANGER whirlgif /scratch/projects/tg/SIDGrid/usr/bin/whirlgif INSTALLED INTEL64::LINUX GLOBUS::maxwalltime=10;env::PATHPREFIX=/scratch/projects/tg/SIDGrid/usr/bin,LD_LIBRARY_PATH=/scratch/projects/tg/SIDGrid/usr/lib RANGER Xphace /scratch/projects/tg/SIDGrid/usr/bin/Xphace INSTALLED INTEL64::LINUX GLOBUS::maxwalltime=10;env::PATHPREFIX=/scratch/projects/tg/SIDGrid/usr/bin,LD_LIBRARY_PATH=/scratch/projects/tg/SIDGrid/usr/lib -RANGER RInvoke /scratch/projects/tg/SIDGrid/scripts/RInvoke.sh INSTALLED INTEL64::LINUX env::PATHPREFIX=/scratch/projects/tg/SIDGrid/usr/bin +RANGER RInvoke /scratch/projects/tg/SIDGrid/scripts/RInvoke.sh INSTALLED INTEL64::LINUX env::PATHPREFIX=/scratch/projects/tg/SIDGrid/usr/bin,GLOBUS::maxwalltime=10 RANGER echo /bin/echo INSTALLED INTEL64::LINUX GLOBUS::maxwalltime=10 RANGER env /usr/bin/env INSTALLED INTEL64::LINUX GLOBUS::maxwalltime=10,queue=normal RANGER Mediator /scratch/projects/tg/SIDGrid/scripts/Mediator.py INSTALLED INTEL64::LINUX env::PATHPREFIX=/scratch/projects/tg/SIDGrid/usr/bin,LD_LIBRARY_PATH=/scratch/projects/tg/SIDGrid/usr/lib/mysql @@ -2004,3 +2004,7 @@ BIGRED echo /bin/echo INSTALLED INTEL64::LINUX null BIGRED Mediator /N/u/tg-scommun/BigRed/sidgrid/scripts/Mediator.py INSTALLED INTEL64::LINUX GLOBUS::maxwalltime=120;env::PATHPREFIX=/N/u/tg-scommun/BigRed/sidgrid/usr/bin +#----------------------------localhost---------------------- + +localhost RInvoke /disks/ci-gpfs/fmri/cnari/scripts/RInvoke.sh INSTALLED INTEL64::LINUX null +localhost echo /bin/echo INSTALLED INTEL64::LINUX null