From lgadelha at ci.uchicago.edu Mon Oct 1 14:38:47 2012 From: lgadelha at ci.uchicago.edu (lgadelha at ci.uchicago.edu) Date: Mon, 1 Oct 2012 14:38:47 -0500 (CDT) Subject: [Swift-commit] r5953 - provenancedb Message-ID: <20121001193847.A04869CCAA@svn.ci.uchicago.edu> Author: lgadelha Date: 2012-10-01 14:38:47 -0500 (Mon, 01 Oct 2012) New Revision: 5953 Added: provenancedb/provdb-uml.dia Modified: provenancedb/README.asciidoc Log: Modified: provenancedb/README.asciidoc =================================================================== --- provenancedb/README.asciidoc 2012-09-27 20:04:54 UTC (rev 5952) +++ provenancedb/README.asciidoc 2012-10-01 19:38:47 UTC (rev 5953) @@ -6,9 +6,9 @@ . A set of scripts for extracting provenance information from Swift's log files. The extracted data is imported into a relational database, currently PotgreSQL, where it can queried. -. A query interface for provenance with a built-in query language called SPQL (Swift Provenance Query Language). SPQL is similar to SQL except for not having +FROM+-clauses and join expressions on the +WHERE+-clause, which are automatically computed for the user. A number of functions and stored procedures that abstract common provenance query patterns are available both in SPQL and SQL. +. A query interface for provenance with a built-in query language called SPQL (Swift Provenance Query Language). SPQL is similar to SQL except for not having +FROM+-clauses and join expressions on the +WHERE+-clause, which are automatically computed for the user. A number of functions and stored procedures that abstract common provenance query patterns are available in both SPQL and SQL. -It addresses the characteristics of many-task computing, where concurrent component tasks are submitted to parallel and distributed computational resources. Such resources are subject to failures, and are usually under high demand for executing tasks and transferring data. Science-level performance information, which describes the behavior of an experiment from the point of view of the scientific domain, is critical for the management of such experiments (for instance, by determining how accurate the outcome of a scientific simulation was, and whether accuracy varies between execution environments). Recording the resource-level performance of such workloads can also assist scientists in managing the life cycle of their computational experiments. Features : +The tools for managing provenance information in Swift have the following features: - Gathering of producer-consumer relationships between data sets and processes. @@ -22,8 +22,11 @@ - Provides a usable and useful query interface for provenance information. -A UML diagram of this provenance model is presented in Figure . We simplify the UML notation to abbreviate the information that each annotated entity set (script run, function call, and variable) has one annotation entity set per data type. We define entities that correspond to the Open Provenance Model (OPM) notions of artifact, process, and artifact usage (either being consumed or produced by a process). These are augmented with entities used to represent many-task scientific computations, and to allow for entity annotations. Such annotations, which can be added post-execution, represent information about provenance entities such as object version tags and scientific parameters. +A UML diagram of this provenance model is presented in figure <>. We simplify the UML notation to abbreviate the information that each annotated entity set (script run, function call, and variable) has one annotation entity set per data type. We define entities that correspond to the Open Provenance Model (OPM) notions of artifact, process, and artifact usage (either being consumed or produced by a process). These are augmented with entities used to represent many-task scientific computations, and to allow for entity annotations. Such annotations, which can be added post-execution, represent information about provenance entities such as object version tags and scientific parameters. +[[provdb_schema]] +image::provdb-uml.svg["Swift provenance database schema",width=1000] + +script_run+: refers to the execution (successful or unsuccessful) of an entire many-task scientific computation, which in Swift is specified as the execution of a complete parallel script from start to finish. +function_call+: records calls to Swift functions. These calls take as input data sets, such as values stored in primitive variables or files referenced by mapped variables; perform some computation specified in the respective function declaration; and produce data sets as output. In Swift, function calls can represent invocations of external applications, built-in functions, and operators; each function call is associated with the script run that invoked it. Added: provenancedb/provdb-uml.dia =================================================================== (Binary files differ) Property changes on: provenancedb/provdb-uml.dia ___________________________________________________________________ Added: svn:mime-type + application/octet-stream From wozniak at ci.uchicago.edu Mon Oct 1 19:42:34 2012 From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu) Date: Mon, 1 Oct 2012 19:42:34 -0500 (CDT) Subject: [Swift-commit] r5954 - usertools/plotter/src/plotter Message-ID: <20121002004234.B14D39CCA1@svn.ci.uchicago.edu> Author: wozniak Date: 2012-10-01 19:42:34 -0500 (Mon, 01 Oct 2012) New Revision: 5954 Modified: usertools/plotter/src/plotter/Lines.java Log: Support dotted lines Modified: usertools/plotter/src/plotter/Lines.java =================================================================== --- usertools/plotter/src/plotter/Lines.java 2012-10-01 19:38:47 UTC (rev 5953) +++ usertools/plotter/src/plotter/Lines.java 2012-10-02 00:42:34 UTC (rev 5954) @@ -3,6 +3,7 @@ import gnu.getopt.Getopt; +import java.awt.BasicStroke; import java.awt.Color; import java.awt.geom.Rectangle2D; import java.io.*; @@ -236,10 +237,12 @@ for (int i = 0; i < plot.getSeriesCount(); i++) { Series series = collection.getSeries(i); - if (! showShape(series.getDescription())) + String description = series.getDescription(); + if (! showShape(description)) renderer.setSeriesShapesVisible(i, false); - if (! showLine(series.getDescription())) + if (! showLine(description)) renderer.setSeriesLinesVisible(i, false); + formatLine(description, renderer, i); } Plots.setupLegend(chart, properties); @@ -433,7 +436,19 @@ return false; return true; } - + + static BasicStroke dottedStroke = + new BasicStroke(2.0f, BasicStroke.CAP_ROUND, BasicStroke.JOIN_ROUND, + 1.0f, new float[] {6.0f, 6.0f}, 0.0f); + + static void formatLine(String name, XYLineAndShapeRenderer renderer, int i) + { + System.out.println(name); + String lineStyle = properties.getProperty("line."+name); + if ("dotted".equals(lineStyle)) + renderer.setSeriesStroke(i, dottedStroke); + } + static String toString(double[][] array) { StringBuilder sb = new StringBuilder(); From swift at ci.uchicago.edu Fri Oct 5 16:30:14 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Fri, 5 Oct 2012 16:30:14 -0500 (CDT) Subject: [Swift-commit] cog r3483 Message-ID: <20121005213014.9E3338D000A1@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3483 | hategan | 2012-10-05 16:25:28 -0500 (Fri, 05 Oct 2012) | 1 line fixed potential deadlock (swift bug 841) ------------------------------------------------------------------------ Index: modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Block.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Block.java (revision 3482) +++ modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Block.java (working copy) @@ -81,7 +81,7 @@ this.walltime = walltime; this.bqp = ap; this.creationtime = Time.now(); - this.deadline = Time.now().add(ap.getSettings().getReserve()); + setDeadline(Time.now().add(ap.getSettings().getReserve())); requestedWorkers += workers; } @@ -130,9 +130,9 @@ last = Time.now(); } } - deadline = - Time.min(starttime.add(walltime), + Time deadline = Time.min(starttime.add(walltime), last.add(bqp.getSettings().getMaxWorkerIdleTime())); + setDeadline(deadline); return Time.now().isGreaterThan(deadline); } else { @@ -184,17 +184,17 @@ } public void add(Cpu cpu) { + Cpu last = null; synchronized (cpus) { if (!scpus.add(cpu)) { CoasterService.error(15, "CPU is already in the block", new Throwable()); } - Cpu last = scpus.last(); - if (last != null) { - deadline = - Time.min(last.getTimeLast().add(bqp.getSettings().getReserve()), - getEndTime()); - } + last = scpus.last(); } + if (last != null) { + setDeadline(Time.min(last.getTimeLast().add(bqp.getSettings().getReserve()), + getEndTime())); + } } public void shutdownIfEmpty(Cpu cpu) { @@ -445,7 +445,7 @@ else if (s.getStatusCode() == Status.ACTIVE) { starttime = Time.now(); endtime = starttime.add(walltime); - deadline = starttime.add(bqp.getSettings().getReserve()); + setDeadline(starttime.add(bqp.getSettings().getReserve())); running = true; bqp.getRLogger().log("BLOCK_ACTIVE id=" + getId()); bqp.getSettings().getHook().blockActive(event); @@ -477,11 +477,11 @@ return deadline; } - public void setDeadline(Time t) { + public synchronized void setDeadline(Time t) { this.deadline = t; } - public Time getCreationTime() { + public synchronized Time getCreationTime() { return creationtime; } From lgadelha at ci.uchicago.edu Tue Oct 9 14:04:07 2012 From: lgadelha at ci.uchicago.edu (lgadelha at ci.uchicago.edu) Date: Tue, 9 Oct 2012 14:04:07 -0500 (CDT) Subject: [Swift-commit] r5955 - provenancedb Message-ID: <20121009190407.CE9199CCAA@svn.ci.uchicago.edu> Author: lgadelha Date: 2012-10-09 14:04:07 -0500 (Tue, 09 Oct 2012) New Revision: 5955 Added: provenancedb/provdb.mwb Modified: provenancedb/README.asciidoc Log: Modified: provenancedb/README.asciidoc =================================================================== --- provenancedb/README.asciidoc 2012-10-02 00:42:34 UTC (rev 5954) +++ provenancedb/README.asciidoc 2012-10-09 19:04:07 UTC (rev 5955) @@ -4,7 +4,7 @@ Swift can be configured to gather and store provenance information about script executions. The following tools are available: -. A set of scripts for extracting provenance information from Swift's log files. The extracted data is imported into a relational database, currently PotgreSQL, where it can queried. +. A set of scripts for extracting provenance information from Swift's log files. The extracted data is imported into a relational database, currently PostgreSQL, where it can queried. . A query interface for provenance with a built-in query language called SPQL (Swift Provenance Query Language). SPQL is similar to SQL except for not having +FROM+-clauses and join expressions on the +WHERE+-clause, which are automatically computed for the user. A number of functions and stored procedures that abstract common provenance query patterns are available in both SPQL and SQL. @@ -14,36 +14,35 @@ - Gathering of hierarchical relationships between data sets. -- Gathering of versioned information of the specifications of many-task scientific computations and of their component applications. +- Gathering of script source code used in each execution. - Allows users to enrich their provenance records with annotations. -- Gathering of runtime information about component application executions. +- Gathering of runtime information about application executions. - Provides a usable and useful query interface for provenance information. -A UML diagram of this provenance model is presented in figure <>. We simplify the UML notation to abbreviate the information that each annotated entity set (script run, function call, and variable) has one annotation entity set per data type. We define entities that correspond to the Open Provenance Model (OPM) notions of artifact, process, and artifact usage (either being consumed or produced by a process). These are augmented with entities used to represent many-task scientific computations, and to allow for entity annotations. Such annotations, which can be added post-execution, represent information about provenance entities such as object version tags and scientific parameters. +A UML diagram of this provenance model is presented in figure <>. We simplify the UML notation to abbreviate the information that each annotated entity set (script run, function call, and variable) has one annotation entity set per data type. We define entities that correspond to the Open Provenance Model (OPM) notions of artifact, process, and artifact usage (either being consumed or produced by a process). Annotations, which can be added post-execution, represent information about provenance entities such as object version tags and scientific parameters. [[provdb_schema]] -image::provdb-uml.svg["Swift provenance database schema",width=1000] +image::provdb.svg["Swift provenance database schema",width=1280] -+script_run+: refers to the execution (successful or unsuccessful) of an entire many-task scientific computation, which in Swift is specified as the execution of a complete parallel script from start to finish. ++script+: contains the script source code used and its hash value. -+function_call+: records calls to Swift functions. These calls take as input data sets, such as values stored in primitive variables or files referenced by mapped variables; perform some computation specified in the respective function declaration; and produce data sets as output. In Swift, function calls can represent invocations of external applications, built-in functions, and operators; each function call is associated with the script run that invoked it. ++script_run+: refers to the execution (successful or unsuccessful) of a script, with attributes such as start time, source code filename, and Swift's version. -+app_fun_call+: represents an invocation of one component application of a many-task scientific computation. In Swift, it is generated by an invocation to an external application. External applications are listed in an application catalog along with the computational resources on which they can be executed. ++function_call+: records calls to functions within a script execution. These calls take as input data sets, such as values stored in primitive variables or files referenced by mapped variables; perform some computation specified in the respective function declaration; and produce data sets as output. In Swift, function calls can represent invocations of external applications, built-in functions, and operators; each function call is associated with the script run that invoked it. ++app_fun_call+: represents an invocation of an application function (_app function_). In Swift, it is generated by an invocation to an external application. External applications are listed in an application catalog along with the computational resources on which they can be executed. + +application_execution+: represents execution attempts of an external application. Each application function call triggers one or more execution attempts, where one (or, in the case of retries or replication, several) particular computational resource(s) will be selected to actually execute the application. +runtime_info+: contains information associated with an application execution, such as resource consumption. -+variable+: represents data sets that were assigned to variables in a Swift script. Variable types can be atomic or composite. Atomic types are primitive types, such as integers and strings, recorded in the relation Primitive, or mapped types, recorded in the relation Mapped. Mapped types are used for declaring and accessing data that is stored in files. Composite types are given by structures and arrays. Containment relationships define a hierarchy where each variable may have child variables (when it is a structure or an array), or a parent variable (when it is a member of a collection). A variable may have as attributes a value, when it is a primitive variable; or a filename, when it is a mapped file. ++dataset+: represents data sets that were assigned to variables in a Swift script. +annot+: is a key-value pair associated with either a +variable+, +function_call+, or +script_run+. These annotations are used to store context-specific information about the entities of the provenance data model. Examples include scientific-domain parameters, object versions, and user identities. Annotations can also be used to associate a set of runs of a script related to a particular event or study, which we refer to as a campaign. The +dataset_in+ and +dataset_out+ relationships between +function_call+ and +variable+ define a lineage graph that can be traversed to determine ancestors or descendants of a particular entity. Process dependency and data dependency graphs are derived with transitive queries over these relationships. -The provenance model presented here is a significant refinement of a previous one used by Swift in the Third Provenance Challenge, which was shown to be similar to OPM. This similarity is retained in the current version of the model, which adds support for annotations and runtime information on component application executions. +function_call+ corresponds to OPM processes, and +variable+ corresponds to OPM artifacts as immutable data objects. The OPM entity agent controls OPM processes, e.g. starting or terminating them. While we do not explicitly define an entity type for such agents, this information can be stored in the annotation tables of the +function_call+ or +script_run+ entities. To record which users controlled each script or function call execution, one can gather the associated POSIX userids, when executions are local, or the distinguished name of network security credentials, when executions cross security domains, and store them as annotations for the -respective entity. This is equivalent to establishing an OPM _wasControlledBy_ relationship. The dependency relationships, _used_ and _wasGeneratedBy_, as defined in OPM, correspond to our +dataset\_in+ and +dataset\_out+ relationships, respectively. Our data model has additional entity sets to capture behavior that is specific to parallel and distributed systems, to distinguish, for instance, between application invocations and execution attempts. We currently do not directly support the OPM concept of _account_, which can describe the same computation using different levels of abstraction. However, one could use annotations to associate one or more such accounts with an execution entity. Based on the mapping to OPM described here, Swift Provenance Database provides tools for exporting the provenance database into OPM provenance graph interchange format, which provides interoperability with other OPM-compatible provenance systems. - == Design and Implementation of Swift Provenance Database In this section, we describe the design and implementation of the Swift Provenance Database provenance query framework. It consists of a set of tools used for extracting provenance information from Swift log files, and a query interface. While its log extractor is specific to Swift, the remainder of the system, including the query interface, is applicable to any parallel functional data flow execution model. The Swift Provenance Database system design is influenced by our survey of provenance queries in many-task computing, where a set of query patterns was identified. The _multiple-step relationships_ (R^*) pattern is implemented by queries that follow the transitive closure of basic provenance relationships, such as data containment hierarchies, and data derivation and consumption. The _run correlation_ (RCr) pattern is implemented by queries for correlating attributes from multiple script runs, such as annotation values or the values of function call parameters. Added: provenancedb/provdb.mwb =================================================================== (Binary files differ) Property changes on: provenancedb/provdb.mwb ___________________________________________________________________ Added: svn:mime-type + application/octet-stream From wilde at ci.uchicago.edu Tue Oct 9 19:17:46 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Tue, 9 Oct 2012 19:17:46 -0500 (CDT) Subject: [Swift-commit] r5956 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121010001746.565FC9CFDE@svn.ci.uchicago.edu> Author: wilde Date: 2012-10-09 19:17:46 -0500 (Tue, 09 Oct 2012) New Revision: 5956 Added: SwiftApps/ParVis/HiRAMTools/do_pfrepp.sh SwiftApps/ParVis/HiRAMTools/genpfrepps.sh SwiftApps/ParVis/HiRAMTools/pfrepp.csh SwiftApps/ParVis/HiRAMTools/pfrepp_template.csh SwiftApps/ParVis/HiRAMTools/pfrepps.swift SwiftApps/ParVis/HiRAMTools/runpfrepps.sh Log: Initial pfrepp scripts. Added: SwiftApps/ParVis/HiRAMTools/do_pfrepp.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/do_pfrepp.sh (rev 0) +++ SwiftApps/ParVis/HiRAMTools/do_pfrepp.sh 2012-10-10 00:17:46 UTC (rev 5956) @@ -0,0 +1,9 @@ +tcsh /home/wilde//LZ/HiRAMTools/pfrepp.csh \ + -a /intrepid-fs0/users/lzamboni/persistent/yearly-nco/en1eo12Ic3/run001/run001/ \ + -b /home/wilde/LZ/pfout \ + -s 1950 \ + -i /intrepid-fs0/users/lzamboni/persistent/bin/fms/pfrepp/remap_file/C90_to_180x90/C90_mosaic.nc \ + -x 180 -y 90 \ + -r /intrepid-fs0/users/lzamboni/persistent/bin/fms/pfrepp/remap_file/C90_to_180x90/C90_to_180x90.nc \ + -m 12 -n 50 -q \ + -u \ No newline at end of file Property changes on: SwiftApps/ParVis/HiRAMTools/do_pfrepp.sh ___________________________________________________________________ Added: svn:executable + * Added: SwiftApps/ParVis/HiRAMTools/genpfrepps.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/genpfrepps.sh (rev 0) +++ SwiftApps/ParVis/HiRAMTools/genpfrepps.sh 2012-10-10 00:17:46 UTC (rev 5956) @@ -0,0 +1,27 @@ +#! /bin/sh + +realid=$1 +realdir=$2 +outdir=$3 +scriptlist=$PWD/$4 + +mkdir -p $outdir/$realid/scripts +cd $outdir/$realid/scripts + +tcsh /home/wilde/LZ/HiRAMTools/pfrepp.csh \ + -a $realdir \ + -b $outdir/$realid \ + -s 1950 \ + -i /intrepid-fs0/users/lzamboni/persistent/bin/fms/pfrepp/remap_file/C90_to_180x90/C90_mosaic.nc \ + -x 180 -y 90 \ + -r /intrepid-fs0/users/lzamboni/persistent/bin/fms/pfrepp/remap_file/C90_to_180x90/C90_to_180x90.nc \ + -m 12 -n 50 -q \ + -u + +echo id path >$scriptlist +for f in pfrepp_script.*; do + id=$(echo $f | sed -e 's/pfrepp_script.//') + echo $id $PWD/$f >>$scriptlist +done + +# -a /intrepid-fs0/users/lzamboni/persistent/yearly-nco/en1eo12Ic3/run001/run001/ \ Property changes on: SwiftApps/ParVis/HiRAMTools/genpfrepps.sh ___________________________________________________________________ Added: svn:executable + * Added: SwiftApps/ParVis/HiRAMTools/pfrepp.csh =================================================================== --- SwiftApps/ParVis/HiRAMTools/pfrepp.csh (rev 0) +++ SwiftApps/ParVis/HiRAMTools/pfrepp.csh 2012-10-10 00:17:46 UTC (rev 5956) @@ -0,0 +1,454 @@ +#!/bin/csh + +set echo +set verbose + +set FMSBIN = /home/lzamboni/bin/fms/scripts + +#NOTE : Currently this tool will only postprocess atmos data and assume the name +# of the files starts with "atmos". + + set template = $0:h/pfrepp_template.csh + set environ = INTERACTIVE + set remapfile = "none" + +# ----- parse input argument list ------ + set historydir = "" + set ppdir = "" + set starttime = "" + @ numyears = 5 + @ nummonths = 12 + set zinterp_atm = "ncep" + set zinterp_ocn = "none" + + set do_not_run = 0 + @ ntiles_lnd = 6 + @ ntiles_atm = 6 + @ ntiles_ocn = 1 + set nlon_atm = 0 + set nlat_atm = 0 + set nlon_ocn = 0 + set nlat_ocn = 0 + set mosaic_atm = "" + set mosaic_ocn = "" + set remapfile_atm = "none" + set remapfile_ocn = "none" + set remove_tmp = 1 + set time_average_only = 0 + set use_bbcp = 0 + +# currently there is no horizontal interpolation for ocean/ice data. +#set argv = (`getopt a:s:n:m:t:T:i:I:x:y:r:z:qdh $*`) + +set argv = (`getopt a:b:cs:n:m:t:T:i:I:x:X:y:Y:r:R:z:Z:uqdwh $*`) + +while ("$argv[1]" != "--") + switch ($argv[1]) + case -a: + set historydir = $argv[2]; shift argv; breaksw; + case -b: + set ppdir = $argv[2]; shift argv; breaksw; + case -c: + set use_bbcp = 1; breaksw; + case -s: + set starttime = $argv[2]; shift argv; breaksw; + case -n: + @ numyears = $argv[2]; shift argv; breaksw; + case -m: + @ nummonths = $argv[2]; shift argv; breaksw; + case -t: + @ ntiles_atm = $argv[2]; shift argv; breaksw; + case -T: + @ ntiles_ocn = $argv[2]; shift argv; breaksw; + case -i: + set mosaic_atm = $argv[2]; shift argv; breaksw; + case -I: + set mosaic_ocn = $argv[2]; shift argv; breaksw; + case -x: + set nlon_atm = $argv[2]; shift argv; breaksw; + case -y: + set nlat_atm = $argv[2]; shift argv; breaksw; + case -X: + set nlon_ocn = $argv[2]; shift argv; breaksw; + case -Y: + set nlat_ocn = $argv[2]; shift argv; breaksw; + case -r: + set remapfile_atm = $argv[2]; shift argv; breaksw; + case -R: + set remapfile_ocn = $argv[2]; shift argv; breaksw; + case -z: + set zinterp_atm =$argv[2]; shift argv; breaksw; + case -Z: + set zinterp_ocn =$argv[2]; shift argv; breaksw; + case -u: + set do_not_run = 1; breaksw; + case -q: + set environ = BATCH; breaksw; + case -d: + set remove_tmp = 0; breaksw; + case -w: + set time_average_only = 1; breaksw; + case -h: + set help ; breaksw + endsw + shift argv +end + +#---- print out user guide when -h option is selected. +if( ! $?help) then + if ($historydir == "") then + echo "ERROR from pfrepp.csh: historydir is not set through -a" + set help + endif + + if ($starttime == "") then + echo "ERROR from pfrepp.csh: starttime is not set through -s" + set help + endif + + if($zinterp_atm != "ncep" ) then + echo "ERROR from pfrepp.csh: zinterp_atm set through -z should be 'ncep', contact developer" + exit 1 + endif + + # when mosaic_atm is defined, nlon_atm, nlat_atm should be defined. + if( $mosaic_atm != "" ) then + #first check to make sure mosaic_atm exist + if( ! -e $mosaic_atm ) then + echo "ERROR from pfrepp.csh: $mosaic_atm does not exist" + exit 1 + endif + + if (${ntiles_atm} != 6 ) then + echo "ERROR from pfrepp.csh: ntiles_atm set through -t should be 6 when mosaic_atm is set, contact developer" + exit 1 + endif + + if ($nlon_atm == 0) then + echo "ERROR from pfrepp.csh: nlon_atm is not set through -x" + set help + endif + + if ($nlat_atm == 0) then + echo "ERROR from pfrepp.csh: nlat_atm is not set through -y" + set help + endif + else + if ( $ntiles_atm != 1 ) then + echo "Error from pfrepp.csh: ntiles_atmos should be 1 when mosaic_atm is not set" + exit 1 + endif + + if ($nlon_atm != 0) then + echo "NOTE from pfrepp.csh: there will be no horizontal interpolation when input grid is lat-lon, nlon_atm need not to be set" + @ nlon_atm = 0 + endif + + if ($nlat_atm != 0) then + echo "NOTE from pfrepp.csh: there will be no horizontal interpolation when input grid is lat-lon, nlat_atm need not to be set" + @ nlat_atm = 0 + endif + + endif + + # currently there will be no horizontal and vertical interpolation for ocean/ice data. + if( $ntiles_ocn != 1 ) then + echo "Error from pfrepp.csh: ntiles_ocn should be 1" + exit 1 + endif + + if( $mosaic_ocn != "" ) then + echo "NOTE from pfrepp.csh: there will be no horizontal interpolation for ocean/ice data, mosaic_ocn need not to be set" + set mosaic_ocn = "" + endif + + if ($nlon_ocn != 0) then + echo "NOTE from pfrepp.csh: there will be no horizontal interpolation for ocean/ice data, nlon_ocn need not to be set" + @ nlon_ocn = 0 + endif + + if ($nlat_ocn != 0) then + echo "NOTE from pfrepp.csh: there will be no horizontal interpolation for ocean/ice data, nlat_ocn need not to be set" + @ nlat_ocn = 0 + endif + + if ( $remapfile_ocn != "none" ) then + echo "NOTE from pfrepp.csh: there will be no horizontal interpolation for ocean/ice data, remapfile_ocn need not to be set" + set remapfile_ocn = "none" + endif + + if ($zinterp_ocn != "none") then + echo "NOTE from pfrepp.csh: there will be no vertical interpolation for ocean/ice data, zinterp_ocn need not to be set" + set zinterp_ocn = "none" + endif + + set len = `echo ${starttime} | awk '{print length($0)}'` + + if( $numyears == 0 ) then + if( $len == 4 ) then + set starttime = ${starttime}0101 + endif + else + if( $len != 4 ) then + echo "ERROR from pfrepp.csh: starttime must be in the format 'yyyy' when numyears is not zero" + exit 1 + endif + endif +endif + +if ($?help) then +cat< sed_file + cat $template | sed -f sed_file > $runscript + rm sed_file + chmod +x $runscript + + if( $do_not_run == 1 ) then + echo "pfrepp.csh: -u is set, will not execute ./$runscript" + else if ( $environ == "BATCH" ) then + echo " submitting $runscript" + # qsub -A prec_sense -n 1 -t 480 -M lzamboni at mcs.anl.gov ./$runscript + # qsub -A SU_Climate -n 1 -t 480 -M lzamboni at mcs.anl.gov ./$runscript + qsub -A MTCScienceApps -n 1 -t 480 -M lzamboni at mcs.anl.gov ./$runscript + # qsub -q R.climate -A SU_Climate -n 1 -t 300 -M lzamboni at mcs.anl.gov ./$runscript + else + echo " running $runscript interactively " + ./$runscript + endif + endif + + #when name is month and numyears is greater than 0, need to do time average. + if( $numyears > 0 && $name == "month" ) then + + set mode = "time_average" + set runscript = pfrepp_script.$filename.$mode.$starttime + echo "\ + s|AAAAAAA|${historydir}|g\ + s|BBBBBBB|${starttime}|g \ + s|CCCCCCC|${inputmosaic}|g \ + s|DDDDDDD|${nlon}|g \ + s|EEEEEEE|${nlat}|g \ + s|FFFFFFF|${remapfile}|g \ + s|GGGGGGG|${numyears}|g \ + s|HHHHHHH|${ntiles}|g \ + s|IIIIIII|${component}|g \ + s|JJJJJJJ|${FREGRID}|g \ + s|LLLLLLL|${NCCOMBINE}|g \ + s|MMMMMMM|${TIMEAVG}|g \ + s|NNNNNNN|${PLEVEL}|g \ + s|OOOOOOO|${zinterp}|g \ + s|PPPPPPP|${remove_tmp}|g\ + s|QQQQQQQ|${MERGE}|g\ + s|RRRRRRR|${nummonths}|g\ + s|SSSSSSS|${filename}|g \ + s|TTTTTTT|${mode}|g \ + s|VVVVVVV|${use_bbcp}|g \ + s|WWWWWWW|${interp_method}|g \ + s|UUUUUUU|${ppdir}|g" \ + > sed_file + cat $template | sed -f sed_file > $runscript + rm sed_file + chmod +x $runscript + if( $do_not_run == 1 ) then + echo "pfrepp.csh: -u is set, will not execute ./$runscript" + else if ( $environ == "BATCH" ) then + echo " submitting $runscript" + qsub -A SU_Climate -n 1 -t 480 ./$runscript + else + echo " running $runscript interactively " + ./$runscript + endif + endif + + end +end + +exit 0 Property changes on: SwiftApps/ParVis/HiRAMTools/pfrepp.csh ___________________________________________________________________ Added: svn:executable + * Added: SwiftApps/ParVis/HiRAMTools/pfrepp_template.csh =================================================================== --- SwiftApps/ParVis/HiRAMTools/pfrepp_template.csh (rev 0) +++ SwiftApps/ParVis/HiRAMTools/pfrepp_template.csh 2012-10-10 00:17:46 UTC (rev 5956) @@ -0,0 +1,482 @@ +#!/bin/csh + +### Using modified template! + +set FMS = /home/lzamboni +set FMS = $HOME + +# source $FMS/bin/fms/sys/environ.cshrc - expanded below and modified: + +# set up netcdf-4.1.2.beta2 and nco-3.9.2 + +unlimit stacksize + +# setenv PATH /soft/apps/cdo-1.4.7/bin:/soft/apps/nco-4.0.9/bin:/soft/apps/netcdf-4.1.2-beta2/bin:$PATH +# nco-3.9.2 for pfrepp + +setenv PATH /soft/apps/nco-3.9.2/bin:/soft/apps/netcdf-4.1.2-beta2/bin:$PATH + +setenv LIBRARY_PATH /soft/apps/netcdf-4.1.2-beta2/lib:/soft/apps/intel-fc-11.0.074/lib/intel64:/soft/apps/intel-cc-11.0.074/lib/intel64:/usr/lib:/usr/X11R6/lib:/usr/local/lib +setenv LD_LIBRARY_PATH $LIBRARY_PATH +setenv CPATH /soft/apps/netcdf-4.1.2-beta2/include +setenv FPATH /soft/apps/netcdf-4.1.2-beta2/include + +# end of modified environ.cshrc + +set GCP = /bin/cp + +set init_clock = `date +%s` + +set history_dir = AAAAAAA +set starttime = BBBBBBB +set inputmosaic = CCCCCCC +set nlon = DDDDDDD +set nlat = EEEEEEE +set remapfile = FFFFFFF +set numyears = GGGGGGG +set numtiles = HHHHHHH +set component = IIIIIII +set FREGRID = JJJJJJJ +set nccombine = LLLLLLL +set TIMEAVG = MMMMMMM +set PLEV = NNNNNNN +set zinterp = OOOOOOO +set removetmp = PPPPPPP +set MERGE = QQQQQQQ +set nummonths = RRRRRRR +set filename = SSSSSSS +set mode = TTTTTTT +set root_ppdir = UUUUUUU +set use_bbcp = VVVVVVV +set interp_method = WWWWWWW +set MONTH_IN_A_YEAR = 12 + +#set scratch = /scratch/$USER +#set scratch = /intrepid-fs1/users/$USER/scratch +set scratch = /intrepid-fs0/users/$USER/scratch + +mkdir -p $scratch + +#tmpdir could be changed depending on system +#set tmpdir = /intrepid-fs0/users/$USER/scratch/pfrepp + set tmpdir = `mktemp -d $scratch/pfrepp.XXXXXX` + +echo pfrepp: running in tmpdir $tmpdir + +# print out all the arguments +echo "***** NOTE from pfrepp.csh: Below are the list of arguments ******" +echo "**************************************************************************" +echo "" +echo "history_dir = $history_dir" +echo "starttime = $starttime" +echo "inputmosaic = $inputmosaic" +echo "nlon = $nlon" +echo "nlat = $nlat" +echo "remapfile = $remapfile" +echo "numyears = $numyears" +echo "numtiles = $numtiles" +echo "component = $component" +echo "FREGRID = $FREGRID" +echo "nccombine = $nccombine" +echo "TIMEAVG = $TIMEAVG" +echo "PLEV = $PLEV" +echo "zinterp = $zinterp" +echo "removetmp = $removetmp" +echo "MERGE = $MERGE" +echo "nummonths = $nummonths" +echo "filename = $filename" +echo "mode = $mode" +echo "use_bbcp = $use_bbcp" +echo "" +echo "**************************************************************************" + +set yearname = `echo $starttime | awk '{printf("%04s", $1)}'` +set begindate = ${starttime}0101 +set begindate_dir = $history_dir/$begindate +set workdir = $tmpdir/tmp.$component.$filename.$mode.$yearname +set mydir = $cwd + +if( $mode != "time_series" && $mode != "time_average" ) then + echo "Error from pfrepp_template.csh: mode should be time_series or time_arerage" +endif + +# when inputmosaic is none, there will be no horizontal interpolation. +set do_horiz_interp = 1 +if( $inputmosaic == "" ) set do_horiz_interp = 0 +# when zinterp is "none", there will be no vertical interpolation. +set do_vert_interp = 1 +if( $zinterp == "none" ) set do_vert_interp = 0 +#make sure workdir does not exist +if( -e $workdir ) then + echo "pfrepp: $workdir already exist, please remove this directory before running pfrepp" + exit 1 +endif +mkdir -p $workdir +if( $status != 0 ) then + echo "Failed at make directory $workdir" + exit 1 +endif +cd $workdir + + #first copy the data to workdir +if($numyears != 0) then + @ year = $starttime + @ end_year = $starttime + $numyears - 1 + @ runningtime_cp = 0 + while ( $year <= $end_year ) + set yearname = `echo $year | awk '{printf("%04s", $1)}'` + + @ month = 1 + while ( $month <= ${MONTH_IN_A_YEAR} ) + set monthname = `echo $month | awk '{printf("%02s", $1)}'` + set datename = ${yearname}${monthname}01 + set datedir = $history_dir/${datename} + cd $datedir + + set filelist = (`ls -1 ${filename}.*nc`) + cd $workdir + foreach file ( $filelist) + set clock1 = `date +%s` + if( $use_bbcp == 1 ) then + $GCP $datedir/$file ${datename}.$file + else + ln -s $datedir/$file ${datename}.$file + endif + set clock2 = `date +%s` + @ runningtime_cp = ${runningtime_cp} + $clock2 - $clock1 + if( $status != 0 ) then + echo "Failed at cp $datedir/$file ${datename}.$file" + exit 1 + endif + end + @ month = $month + $nummonths + end + @ year = $year + 1 + end + echo "pfrepp: the running time for cp timeseries is ${runningtime_cp} seconds" +endif + +set name = `echo $filename | sed 's/_/ /' | awk '{printf $2}'` + +if($numyears == 0) then + set datedir = $history_dir/$starttime + if( ! -e $datedir ) then + #nccombine must be executed before pfrepp is executed. + echo "directory $history_dir/$starttime does not exist, which means the data is combined" + exit 1 + endif + cd $datedir + set filelist = (`ls -1 ${filename}.*nc`) + cd $workdir + foreach file ( $filelist ) + if( $use_bbcp == 1 ) then + $GCP $datedir/$file ${starttime}.$file + else + ln -s $datedir/$file ${starttime}.$file + endif + if( $status != 0 ) then + echo "Failed at cp $datedir/$file ${starttime}.$file" + exit 1 + endif + end + #timeseries + set ppdir = $root_ppdir/${component}/ts/$name + if( ! -e $ppdir ) mkdir -p $ppdir + if( $do_vert_interp == 0 ) then + set output_file = ${starttime}.$filename + else + set output_file = ${component}.${starttime} + set clock1 = `date +%s` + $PLEV -a -i ${starttime}.$filename -o ${output_file} -n $numtiles + if( $status != 0 ) then + echo "Failed at running $PLEV -a -i ${starttime}.$filename -o ${output_file} -n $numtiles" + exit 1 + endif + set clock2 = `date +%s` + @ runningtime = $clock2 - $clock1 + echo "pfrepp: the running time for PLEV is $runningtime seconds" + endif + + if( $do_horiz_interp == 0 ) then + set fregrid_out = ${output_file}.nc + else + set fregrid_out = $filename.${starttime}.nc + set clock1 = `date +%s` + $FREGRID -a ${output_file} -b $fregrid_out -g $inputmosaic -n $nlon -m $nlat -r $remapfile -p ${interp_method} + if( $status != 0 ) then + echo "Failed at running $FREGRID -a plev.$filename.${starttime} -b $fregrid_out -g $inputmosaic -n $nlon -m $nlat -r $remapfile -p ${interp_method}" + exit 1 + endif + set clock2 = `date +%s` + @ runningtime = $clock2 - $clock1 + echo "pfrepp: the running time for fregrid is $runningtime seconds" + endif + # mv $fregrid_out $ppdir/${component}.${starttime}.nc + foreach i(1 2 3 4 5 6 7 8 9 0) + dd if=$fregrid_out of=$ppdir/${component}.${starttime}.nc bs=4M + if ( $status == 0 ) then + break + endif + end + +else if( $mode == "time_average" ) then # numyears > 0, time average + @ endtime = $starttime + $numyears - 1 + set start_year = `echo $starttime | awk '{printf("%04s", $1)}'` + set end_year = `echo $endtime | awk '{printf("%04s", $1)}'` + + #merge the files + set clock1 = `date +%s` + $MERGE -s $start_year -n $numyears -m $nummonths -c $filename -t $numtiles + if( $status != 0 ) then + echo "Failed at running $MERGE -a $history_dir -s $start_year -n $numyears -m $nummonths -c $filename -t $numtiles" + exit 1 + endif + set clock2 = `date +%s` + @ runningtime = $clock2 - $clock1 + echo "pfrepp: the running time for merge is $runningtime seconds" + + # timeAverage (monthly averages) + set clock1 = `date +%s` + $TIMEAVG -s $starttime -f monthly -c $component -t $numtiles -n $numyears # the output will be in the format $starttime-$endtime.${component}.$month.tile#.nc. + + if( $status != 0 ) then + echo "Failed at running $TIMEAVG -s $starttime -f monthly -c $component -t $numtiles -n $numyears" + exit 1 + endif + + set clock2 = `date +%s` + @ runningtime = $clock2 - $clock1 + echo "pfrepp: the running time for timeAverage monthly_${numyears}yr is $runningtime seconds" + + set ppdir = $root_ppdir/$component/av/monthly_${numyears}yr/ + if( ! -e $ppdir ) mkdir -p $ppdir + #loop through 12 months + @ month = 0 + @ runningtime_plev = 0 + @ runningtime_fregrid = 0 + while ( $month < $MONTH_IN_A_YEAR ) + @ month = $month + 1 + set monthname = `echo $month | awk '{printf("%02s", $1)}'` + if( $do_vert_interp == 0 ) then + set output_file = $start_year-$end_year.${component}.$monthname + else + set output_file = plev.${component}.$monthname + set clock1 = `date +%s` + $PLEV -a -i $start_year-$end_year.${component}.$monthname -o ${output_file} -n $numtiles + if( $status != 0 ) then + echo "Failed at running $PLEV -a -i $start_year-$end_year.${component}.$monthname -o ${output_file} -n $numtiles" + exit 1 + endif + set clock2 = `date +%s` + @ runningtime_plev = ${runningtime_plev} + $clock2 - $clock1 + endif + + if( $do_horiz_interp == 0 ) then + set fregrid_out = ${output_file}.nc + else + set fregrid_out = ${component}.$start_year-$end_year.$monthname.nc + set clock1 = `date +%s` + $FREGRID -a ${output_file} -b $fregrid_out -g $inputmosaic -n $nlon -m $nlat -r $remapfile -p ${interp_method} + if( $status != 0 ) then + echo "Failed at running $FREGRID -a ${output_file} -b $fregrid_out -g $inputmosaic -n $nlon -m $nlat -r $remapfile -p ${interp_method}" + exit 1 + endif + set clock2 = `date +%s` + @ runningtime_fregrid = ${runningtime_fregrid} + $clock2 - $clock1 + endif + + #move data to pp directory. +# mv ${fregrid_out} $ppdir/${component}.$start_year-$end_year.$monthname.nc + foreach i(1 2 3 4 5 6 7 8 9 0) + dd if=${fregrid_out} of=$ppdir/${component}.$start_year-$end_year.$monthname.nc bs=4M + if ( $status == 0 ) then + break + endif + end + end + + echo "pfrepp: the running time for plev monthly_${numyears}yr is ${runningtime_plev} seconds" + echo "pfrepp: the running time for fregrid monthly_${numyears}yr is ${runningtime_fregrid} seconds" + + # timeAverage (annual averages) + # first check if the nanual average file exist or not. + set clock1 = `date +%s` + + set ppdir = $root_ppdir/${component}/av/annual_1year + if( ! -e $ppdir ) mkdir -p $ppdir + @ year = $start_year - 1 + set annual_exist = 0 + while ( $year < $end_year ) + @ year = $year + 1 + set yearname = `echo $year | awk '{printf("%04s", $1)}'` + set output_file = $ppdir/${component}.$yearname.ann.nc + if( ${annual_exist} == 0 ) then + if( -f $output_file ) then + set annual_exist = 1 + endif + else + if( ! -f $output_file ) then + echo "ERROR from pfrepp.csh: $output_file does not exist" + exit 1 + endif + endif + end + + if( $annual_exist == 1 ) then + echo "NOTE from pfrepp.csh: annual_1year already created" + else + set clock1 = `date +%s` + $TIMEAVG -s $starttime -f annual_1year -c ${component} -t $numtiles -n $numyears # the output will be in the format $starttime-$endtime.${component}.$month.tile#.nc + if( $status != 0 ) then + echo "Failed at running $TIMEAVG -s $starttime -f annual -c ${component} -t $numtiles -n $numyears" + exit 1 + endif + set clock2 = `date +%s` + @ runningtime = $clock2 - $clock1 + echo "pfrepp: the running time for timeAverage annual is $runningtime seconds" + + @ year = $start_year - 1 + @ runningtime_plev = 0 + @ runningtime_fregrid = 0 + + while ( $year < $end_year ) + @ year = $year + 1 + set yearname = `echo $year | awk '{printf("%04s", $1)}'` + if( $do_vert_interp == 0 ) then + set output_file = $yearname.${component} + else + set output_file = plev.${component}.$yearname + set clock1 = `date +%s` + $PLEV -a -i $yearname.${component} -o ${output_file} -n $numtiles + if( $status != 0 ) then + echo "Failed at running $PLEV -a -i $yearname.${component} -o ${output_file} -n $numtiles" + exit 1 + endif + set clock2 = `date +%s` + @ runningtime_plev = ${runningtime_plev} + $clock2 - $clock1 + endif + + if( $do_horiz_interp == 0 ) then + set fregrid_out = ${output_file}.nc + else + set fregrid_out = ${component}.$yearname.ann.nc + set clock1 = `date +%s` + $FREGRID -a ${output_file} -b $fregrid_out -g $inputmosaic -n $nlon -m $nlat -r $remapfile -p ${interp_method} + if( $status != 0 ) then + echo "Failed at running $FREGRID -a ${output_file} -b $fregrid_out -g $inputmosaic -n $nlon -m $nlat -r $remapfile -p ${interp_method}" + exit 1 + endif + set clock2 = `date +%s` + @ runningtime_fregrid = ${runningtime_fregrid} + $clock2 - $clock1 + endif + + #move data to pp directory. +# mv ${fregrid_out} $ppdir/${component}.$yearname.ann.nc + foreach i(1 2 3 4 5 6 7 8 9 0) + dd if=${fregrid_out} of=$ppdir/${component}.$yearname.ann.nc bs=4M + if ( $status == 0 ) then + break + endif + end + end + + echo "pfrepp: the running time for plev annual is ${runningtime_plev} seconds" + echo "pfrepp: the running time for fregrid annual is ${runningtime_fregrid} seconds" + endif + + #timeAverage (annual $numyears averages), only when numyears != 1 + if( $numyears != 1 ) then + set ppdir = $root_ppdir/${component}/av/annual_${numyears}year + if( ! -e $ppdir ) mkdir -p $ppdir + set clock1 = `date +%s` + $TIMEAVG -s $starttime -f annual -i $root_ppdir/${component}/av/annual_1year -c ${component} -t $numtiles -n ${numyears} + if( $status != 0 ) then + echo "Failed at running $TIMEAVG -s $starttime -f annual -i $root_ppdir/${component}/av/annual_1year -c ${component} -t $numtiles -n ${numyears}" + exit 1 + endif + set clock2 = `date +%s` + @ runningtime = $clock2 - $clock1 + echo "pfrepp: the running time for timeAverage annual_${numyears}year is $runningtime seconds" + +# mv ${component}.$start_year-$end_year.ann.nc $ppdir + foreach i(1 2 3 4 5 6 7 8 9 0) + dd if=${component}.$start_year-$end_year.ann.nc of=$ppdir/$start_year-$end_year.ann.nc bs=4M + if ( $status == 0 ) then + break + endif + end + endif +else # timeseries, numyears > 0 + @ runningtime_plev = 0 + @ runningtime_fregrid = 0 + set ppdir = $root_ppdir/$component/ts/$name + if( ! -e $ppdir ) mkdir -p $ppdir + + @ year = $starttime + @ end_year = $starttime + $numyears - 1 + while ( $year <= $end_year ) + set yearname = `echo $year | awk '{printf("%04s", $1)}'` + + @ month = 1 + while ( $month <= ${MONTH_IN_A_YEAR} ) + set monthname = `echo $month | awk '{printf("%02s", $1)}'` + set datename = ${yearname}${monthname}01 + set datedir = $history_dir/${datename} + + if( $do_vert_interp == 0 ) then + set output_file = ${datename}.$filename + else + set output_file = ${component}.${datename} + set clock1 = `date +%s` + $PLEV -a -i ${datename}.$filename -o ${output_file} -n $numtiles + if( $status != 0 ) then + echo "Failed at running $PLEV -a -i ${datename}.$filename -o ${output_file} -n $numtiles" + exit 1 + endif + set clock2 = `date +%s` + @ runningtime_plev = ${runningtime_plev} + $clock2 - $clock1 + endif + + if( $do_horiz_interp == 0 ) then + set fregrid_out = ${output_file}.nc + else + set fregrid_out = $filename.${datename}.nc + set clock1 = `date +%s` + $FREGRID -a ${output_file} -b $fregrid_out -g $inputmosaic -n $nlon -m $nlat -r $remapfile -p ${interp_method} + if( $status != 0 ) then + echo "Failed at running $FREGRID -a ${output_file} -b $fregrid_out -g $inputmosaic -n $nlon -m $nlat -r $remapfile -p ${interp_method}" + exit 1 + endif + set clock2 = `date +%s` + @ runningtime_fregrid = ${runningtime_fregrid} + $clock2 - $clock1 + endif +# mv $fregrid_out $ppdir/${component}.${datename}.nc + foreach i(1 2 3 4 5 6 7 8 9 0) + dd if=$fregrid_out of=$ppdir/${component}.${datename}.nc bs=4M + if ( $status == 0 ) then + break + endif + end + if( $status != 0 ) then + echo "Failed at mv $fregrid_out $ppdir/${component}.${datename}.nc" + exit 1 + endif + + @ month = $month + $nummonths + end + @ year = $year + 1 + end + echo "pfrepp: the running time for plev timeseries is ${runningtime_plev} seconds" + echo "pfrepp: the running time for fregrid timeseries is ${runningtime_fregrid} seconds" +endif + +set end_clock = `date +%s` +@ runningtime = $end_clock - $init_clock +echo "pfrepp: the total running time is $runningtime seconds" + +if( $removetmp == 1 ) then + cd $mydir + rm -rf $workdir +endif + +exit 0 Property changes on: SwiftApps/ParVis/HiRAMTools/pfrepp_template.csh ___________________________________________________________________ Added: svn:executable + * Added: SwiftApps/ParVis/HiRAMTools/pfrepps.swift =================================================================== --- SwiftApps/ParVis/HiRAMTools/pfrepps.swift (rev 0) +++ SwiftApps/ParVis/HiRAMTools/pfrepps.swift 2012-10-10 00:17:46 UTC (rev 5956) @@ -0,0 +1,41 @@ +type file; + +type realization { + string path; + string id; +} + +type script { + string path; + string id; +} + +global string outdir = @arg("outdir","/intrepid-fs0/users/wilde/LZ/pfrepps"); +string reallist = @arg("reallist","NO-REALIZATION-LIST-SPECIFIED"); + +app (file scripts, file std_out, file std_err) genpfrepps (string realid, string realpath) +{ + genpfrepps realid realpath outdir @scripts stdout=@std_out stderr=@std_err; +} + +app (file std_out, file std_err) sh (string script) +{ + sh "-c" script stdout=@std_out stderr=@std_err; +} + +realization reali[] = readData(reallist); + +foreach r,i in reali { + tracef("%s %s\n", r.id, r.path); + file sofile ; + file sefile ; + file scriptlist ; + (scriptlist,sofile,sefile) = genpfrepps(r.id, r.path); + script scripts[] = readData(scriptlist); + foreach s in scripts { + file ssofile ; + file ssefile ; + // (ssofile, ssefile) = sh(s.path); + tracef("Running script: %s: %s\n", s.id, s.path); + } +} Added: SwiftApps/ParVis/HiRAMTools/runpfrepps.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/runpfrepps.sh (rev 0) +++ SwiftApps/ParVis/HiRAMTools/runpfrepps.sh 2012-10-10 00:17:46 UTC (rev 5956) @@ -0,0 +1,79 @@ +#! /bin/bash + +reallist=$1 # full pathname of list of realizations to pfrepp +outdir=$2 # full pathname of output dir + +# reallist must have format: +# path id (with this header line) + +bindir=$(dirname $0) +bindir=$(cd $bindir; pwd) + +echo bindir=$bindir + +# Create new runNNN directory + +rundir=$( echo run??? | sed -e 's/^.*run//' | awk '{ printf("run%03d\n", $1+1)}' ) +mkdir $rundir + +cat >$rundir/cf <$rundir/sites.xml < + + + + .01 + 10000 + + $PWD/swiftwork + + + + + + + SU_Climate + + default + 2.56 + 10000 + 8 + + + 32 + 28800 + 00:20:00 + 100 + 100 + 1 + 1 + $PWD/$rundir/swiftwork + + + + +END + +cat >$rundir/tc <& swift.out Property changes on: SwiftApps/ParVis/HiRAMTools/runpfrepps.sh ___________________________________________________________________ Added: svn:executable + * From wilde at ci.uchicago.edu Tue Oct 9 20:04:22 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Tue, 9 Oct 2012 20:04:22 -0500 (CDT) Subject: [Swift-commit] r5957 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121010010422.DF4269CCAA@svn.ci.uchicago.edu> Author: wilde Date: 2012-10-09 20:04:22 -0500 (Tue, 09 Oct 2012) New Revision: 5957 Modified: SwiftApps/ParVis/HiRAMTools/makeyearly.swift Log: Add stdout/err handling. Modified: SwiftApps/ParVis/HiRAMTools/makeyearly.swift =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly.swift 2012-10-10 00:17:46 UTC (rev 5956) +++ SwiftApps/ParVis/HiRAMTools/makeyearly.swift 2012-10-10 01:04:22 UTC (rev 5957) @@ -9,13 +9,15 @@ string rundir=@arg("rundir","NO-OUTDIR-SPECIFIED"); string ncfile=@arg("ncfile","ncfiles"); -app makeyearly (string dir, string f1, string f2) +app (file sout, file serr) makeyearly (string dir, string f1, string f2) { - makeyearly f1 f2 dir; + makeyearly f1 f2 dir stdout=@sout stderr=@serr; } ncrec nc[] = readData(ncfile); -foreach n in nc { - makeyearly(rundir,n.f1,n.f2); +foreach n,i in nc { + file so ; + file se ; + (so,se) = makeyearly(rundir,n.f1,n.f2); } From wilde at ci.uchicago.edu Tue Oct 9 20:07:58 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Tue, 9 Oct 2012 20:07:58 -0500 (CDT) Subject: [Swift-commit] r5958 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121010010758.70FC29CCAA@svn.ci.uchicago.edu> Author: wilde Date: 2012-10-09 20:07:58 -0500 (Tue, 09 Oct 2012) New Revision: 5958 Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh Log: comment dirname handling conventions Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh 2012-10-10 01:04:22 UTC (rev 5957) +++ SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh 2012-10-10 01:07:58 UTC (rev 5958) @@ -10,9 +10,11 @@ # Enable one of the following patterns to name the output dir: -outsuffix=$(basename $cdir) # rname -outsuffix=$(basename $(dirname $cdir)) # rname/rundir, use rname -outsuffix=$(basename $(dirname $cdir))/$(basename $cdir) # rname/rundir, use both +outsuffix=$(basename $cdir) # cdir ends in rname, use rname +outsuffix=$(basename $(dirname $cdir)) # cdir ends in rname/rundir, use rname +outsuffix=$(basename $(dirname $cdir))/$(basename $cdir) # cdir ends in rname/rundir, use rname/rundir + # Default assumes cdir is a rundir from combine script + # New rundir will be appended to end of outsuffix # generate the list of files to process: do every half-decade From wilde at ci.uchicago.edu Wed Oct 10 08:58:02 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Wed, 10 Oct 2012 08:58:02 -0500 (CDT) Subject: [Swift-commit] r5959 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121010135802.DD5209CCAA@svn.ci.uchicago.edu> Author: wilde Date: 2012-10-10 08:58:01 -0500 (Wed, 10 Oct 2012) New Revision: 5959 Modified: SwiftApps/ParVis/HiRAMTools/makeyearly-cdo.sh Log: Adjust PATH and logging Modified: SwiftApps/ParVis/HiRAMTools/makeyearly-cdo.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly-cdo.sh 2012-10-10 01:07:58 UTC (rev 5958) +++ SwiftApps/ParVis/HiRAMTools/makeyearly-cdo.sh 2012-10-10 13:58:01 UTC (rev 5959) @@ -21,11 +21,12 @@ qtrdec2=$2 # 2nd quarter-decade dir destbase=$3 # base dir for output years -echo destbase=$destbase +echo destbase=$destbase qtrdec1=$1 qtrdec2=$2 # Add cdo and nco tools to PATH and Intel compiler v11 libs to LD_LIBRARY_PATH -export PATH=/soft/apps/cdo-1.4.7/bin:/soft/apps/nco-4.0.9/bin:$PATH +#export PATH=/soft/apps/cdo-1.4.7/bin:/soft/apps/nco-4.0.9/bin:$PATH +export PATH=/soft/apps/nco-4.0.9/bin:$PATH export LD_LIBRARY_PATH=/soft/apps/intel-fc-11.0.074/lib/intel64:/soft/apps/intel-cc-11.0.074/lib/intel64:/soft/apps/netcdf-3.6.3/lib:/usr/lib:/usr/X11R6/lib:/usr/local/lib fname=$(basename $qtrdec1 .nc) From wilde at ci.uchicago.edu Wed Oct 10 09:42:25 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Wed, 10 Oct 2012 09:42:25 -0500 (CDT) Subject: [Swift-commit] r5960 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121010144225.867E69CCB6@svn.ci.uchicago.edu> Author: wilde Date: 2012-10-10 09:42:25 -0500 (Wed, 10 Oct 2012) New Revision: 5960 Modified: SwiftApps/ParVis/HiRAMTools/combine.sh SwiftApps/ParVis/HiRAMTools/combine.swift SwiftApps/ParVis/HiRAMTools/combine_realization.sh Log: Add direct mode, support for handling nodes with problems on /scratch, handling of stdout/err, and change rundir to outdir on swift script. Modified: SwiftApps/ParVis/HiRAMTools/combine.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/combine.sh 2012-10-10 13:58:01 UTC (rev 5959) +++ SwiftApps/ParVis/HiRAMTools/combine.sh 2012-10-10 14:42:25 UTC (rev 5960) @@ -1,29 +1,62 @@ #! /bin/bash -set -x +set -x -mkdir -p /scratch/wilde -tmp=$(mktemp -d /scratch/wilde/combine_XXXX) -cd $tmp - outdir=$1 inpattern=$2 date=$(basename $(dirname $inpattern)) basepattern=$(basename $inpattern) infiles="$2*" -for f in $infiles; do - dd if=$f of=$(basename $f) bs=4M -done -/home/ckerr/bin/fms/tools/x86_64/bin/mppnccombine.exe out.nc ${basepattern}* +mppncombine=/home/ckerr/bin/fms/tools/x86_64/bin/mppnccombine.exe +mode=scratch # default -mkdir -p $outdir/$date -dd if=out.nc of=$outdir/$date/$basepattern.trans bs=4M -mv $outdir/$date/$basepattern.trans $outdir/$date/$basepattern +badnodes="loginx" +thisnode=$(hostname) +if echo $badnodes | grep -q $thisnode ; then + echo Running on node $thisnode - is in badnode list $badnodes. Will run in direct mode. + mode=direct +else + # Try to make a temporary work dir under /scratch - this can fail on bad nodes + mkdir -p /scratch/$USER + tmp=$(mktemp -d /scratch/$USER/combine_XXXXXX) + if [ $? != 0 ]; then + echo Failed to create tmp dir on /scratch/$USER. Will run in direct mode. + mode=direct + fi +fi -cd .. -rm -rf $tmp +set -o errexit # Set this after we know if tmp dir was created ok +export PATH=/soft/apps/nco-3.9.2/bin:$PATH +export LD_LIBRARY_PATH=/soft/apps/intel-fc-11.0.074/lib/intel64:/soft/apps/intel-cc-11.0.074/lib/intel64:/soft/apps/netcdf-3.6.3/lib:/usr/lib:/usr/X11R6/lib:/usr/local/lib + +if [ $mode = scratch ]; then + + cd $tmp + + for f in $infiles; do + dd if=$f of=$(basename $f) bs=4M + done + + # Add nco tools to PATH and Intel compiler v11 libs to LD_LIBRARY_PATH # FIXME: this is precautionary and can likely be removed. + + # FIXME: ^^^ end precaution + + $mppncombine out.nc ${basepattern}* + + mkdir -p $outdir/$date + dd if=out.nc of=$outdir/$date/$basepattern.trans bs=4M + mv $outdir/$date/$basepattern.trans $outdir/$date/$basepattern + + cd .. + rm -rf $tmp + +else # $mode = direct + mkdir -p $outdir/$date + $mppncombine $outdir/$date/$basepattern $infiles + +fi Modified: SwiftApps/ParVis/HiRAMTools/combine.swift =================================================================== --- SwiftApps/ParVis/HiRAMTools/combine.swift 2012-10-10 13:58:01 UTC (rev 5959) +++ SwiftApps/ParVis/HiRAMTools/combine.swift 2012-10-10 14:42:25 UTC (rev 5960) @@ -1,16 +1,18 @@ type file; -string rundir=@arg("rundir","NO-OUTDIR-SPECIFIED"); +string outdir=@arg("outdir","NO-OUTDIR-SPECIFIED"); string groupfile=@arg("groupfile","groups"); -app combine (string dir, string pattern) +app (file sout, file serr) combine (string dir, string pattern) { - combine dir pattern; + combine dir pattern stdout=@sout stderr=@serr; } string group[] = readData(groupfile); -foreach g in group { - combine(rundir,g); +foreach g, i in group { + file so; + file se; + (so,se) = combine(outdir,g); } Modified: SwiftApps/ParVis/HiRAMTools/combine_realization.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/combine_realization.sh 2012-10-10 13:58:01 UTC (rev 5959) +++ SwiftApps/ParVis/HiRAMTools/combine_realization.sh 2012-10-10 14:42:25 UTC (rev 5960) @@ -1,17 +1,35 @@ #! /bin/bash -hdir=$1 -outdir=$2 +realdir=$1 # full pathname of realization dir +outdir=$2 # full pathname of output dir +realname=$(basename $realdir) # realization name (eg en1eo12Ic1) +outdir=$outdir/$realname -/bin/ls -1 -d ${hdir}* >dirs +bindir=$(dirname $0) +bindir=$(cd $bindir; pwd) -echo '=== ' $(basename $0) doing dirs: -cat dirs +echo bindir=$bindir -for d in $(cat dirs); do - find $d | grep .nc.[0-9][0-9][0-9][0-9] | sed -e 's/.....$//' | sort | uniq -done >groups +# patterns for filename matching +yyyy="[0-9][0-9][0-9][0-9]" +yyyymmdd="${yyyy}[0-9][0-9][0-9][0-9]" + +#find $realdir/history/$yyyymmdd | +find $realdir/history/19????01 | + grep .nc.$yyyy | + sed -e 's/.....$//' | sort | uniq > groups +echo Created $real.groups + +# /bin/ls -1 -d ${hdir}* >dirs +# +# echo '=== ' $(basename $0) doing dirs: +# cat dirs +# +# for d in $(cat dirs); do +# find $d | grep .nc.[0-9][0-9][0-9][0-9] | sed -e 's/.....$//' | sort | uniq +# done >groups + # Create new runNNN directory rundir=$( echo run??? | sed -e 's/^.*run//' | awk '{ printf("run%03d\n", $1+1)}' ) @@ -42,7 +60,8 @@ - + + MTCScienceApps default 2.56 10000 @@ -65,18 +84,17 @@ cat >$rundir/tc <& swift.out +swift -config cf -tc.file tc -sites.file sites.xml combine.swift -outdir=$outdir/$runid -groupfile=groups >& swift.out From wilde at ci.uchicago.edu Wed Oct 10 09:43:26 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Wed, 10 Oct 2012 09:43:26 -0500 (CDT) Subject: [Swift-commit] r5961 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121010144326.244DB9CCB6@svn.ci.uchicago.edu> Author: wilde Date: 2012-10-10 09:43:26 -0500 (Wed, 10 Oct 2012) New Revision: 5961 Added: SwiftApps/ParVis/HiRAMTools/STATUS Modified: SwiftApps/ParVis/HiRAMTools/README Log: Updates from work in July 2012. Modified: SwiftApps/ParVis/HiRAMTools/README =================================================================== --- SwiftApps/ParVis/HiRAMTools/README 2012-10-10 14:42:25 UTC (rev 5960) +++ SwiftApps/ParVis/HiRAMTools/README 2012-10-10 14:43:26 UTC (rev 5961) @@ -13,17 +13,21 @@ runall.sh # called by the user, runs the scripts below - makeyearly_realization.sh - makeyearly.swift + combine_realization.sh + combine.swift + combine.sh (app, specified in tc) + + makeyearly_realization.sh + makeyearly.swift makeyearly-cdo.sh (app, specified in tc) - combine_realization.sh - combine.swift - combine.chk.sh (app, specified in tc) + runpfrepps.sh # Called by used, runs the scripts below + pfrepps.swift + genpfrepps.sh + runscript.sh -To process a set of realizations (for example, to combine them), perform these -steps: +To combine or annualize a set of realizations, perform these steps: 1. Add Swift and a recent Sun Java to your path. For now we will use a Swift and Java version maintained by Swift team member Jon Monette: @@ -57,28 +61,83 @@ script=makeyearly_realization.sh # Script to run for each realization -5. Place the list of realizations to process in a file named "real.todo" and - create and empty file "real.done". For example, to process 2 realizations, - Climo_001 and Climo_023): +5. Place the list of *full pathnames* of the realizations to process in a file + named "real.todo" and create and empty file "real.done". For example, to + process 3 realizations, eg en3rc16Ic1, ... do: cat >real.todo - Climo_001 - Climo_023 + /intrepid-fs0/users/lzamboni/persistent/en3rc16Ic1 + /intrepid-fs0/users/lzamboni/persistent/en3rth12Ic2 + /intrepid-fs0/users/lzamboni/persistent/en3rth8Ic2 ^D - >real.done # create an empty real.done file + >real.done # IMPORTANT! create an empty real.done file 6. run: - ./runall.combine_realizations.sh >& runall.out + $HOME/HiRAMTools/runall.sh >& runall.out -Then look for the most recently created run directory, and do: +Then cd to the most recently created run directory, and do: + + cd run042 # for example + tail -f swift.out +=== + +To run pfrep on a set of realizations, do: + +1. Create a list of realizations in a file, starting with a header line, in + the following format: + +--- file pflist --- (next line is first line of file): +path id +/full/path/of/realization/history/dir realizationID +etc +--- + +For example, file "pflist" contains: + +--- Next line is first line of file. "---" is not in file: +path id +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run001/run001/ en1eo14Ic2 +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run002/run003/ en1eo14Ic3 +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run003/run004/ en1eo14Ic4 +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run004/run005/ en1eo16Ic1 +--- + +2. Run the script: + + runpfrepps.sh pflistFile outputDir + + $HOME/HiRAMTools/runpfrepps.sh pflist /intrepid-fs0/users/wilde/persistent/LZ/pfrepps.2012.0630 + +3. Watch the status + + cd run012 tail -f swift.out +4. Manual, local execution of pfrepp atmos average scripts (generated from step 3, above) +cd /intrepid-fs0/users/wilde/persistent/LZ/pfrepps + +# start scripts here ??? + +# show script progress: + +for s in $(cat set02); do echo "$s: "; tail -15 $s.out; done + + + OPEN ISSUES: +- error handling in the leaf scripts is highly suspect: are errors correctly + getting caught??? + - We see straggler scripts on Eureka nodes. Seems to be doing very slow IO on -shared disk for unexplained reasons. Ticket is open with ALCF support, being -investigated by Andrew Cherry. + shared disk for unexplained reasons. Ticket is open with ALCF support, being + investigated by Andrew Cherry. + +- If /scratch is not available, we switch to running entirely on fs0 + +- Naming conventions for outdir now used in combine need to be added to + makeyearly Added: SwiftApps/ParVis/HiRAMTools/STATUS =================================================================== --- SwiftApps/ParVis/HiRAMTools/STATUS (rev 0) +++ SwiftApps/ParVis/HiRAMTools/STATUS 2012-10-10 14:43:26 UTC (rev 5961) @@ -0,0 +1,273 @@ +From LZ: here are the versions we know they work: + + nco-4.0.9 for yearly reorganization + + nco-3.9.2 for pfrepp and for combine (combine might not used nco at all, but + this is our solution today) + + when we know makeyearly, combine and pfrepp work, we may want to test whether + the following version works for all cases (as ALCF Support suggests) + + 3.9.9-udunits2 + +---- + +In: /intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1 + +run001/run001/ run001 en1eo14Ic2 +run002/run003/ run002 en1eo14Ic3 +run003/run004/ run003 en1eo14Ic4 +run004/run005/ run004 en1eo16Ic1 +run005/run006/ run005 en1eo16Ic2 +run006/run007/ run006 en1eo16Ic3 +run007/run008/ run007 en1eo16Ic4 +run008/run009/ run008 en1eo8Ic1 +run009/run010/ run009 en1eo8Ic2 +run010/run011/ run010 en1eo8Ic3 +run011/run012/ run011 en1eo8Ic4 + +run001/run001/ en1eo14Ic2 +run002/run003/ en1eo14Ic3 +run003/run004/ en1eo14Ic4 +run004/run005/ en1eo16Ic1 +run005/run006/ en1eo16Ic2 +run006/run007/ en1eo16Ic3 +run007/run008/ en1eo16Ic4 +run008/run009/ en1eo8Ic1 +run009/run010/ en1eo8Ic2 +run010/run011/ en1eo8Ic3 +run011/run012/ en1eo8Ic4 + +run001/run001/ +run002/run003/ +run003/run004/ +run004/run005/ +run005/run006/ +run006/run007/ +run007/run008/ +run008/run009/ +run009/run010/ +run010/run011/ +run011/run012/ + +740 files per combined dir +740 * 2.5 = 1850 files per annualized dir + +Short yearlies: + +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run001/run001/ 1540 +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run002/run003/ 1405 +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run003/run004/ 1844 +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run011/run012/ 1151 + +cdir=/intrepid-fs0/users/wilde/persistent/LZ/combines.toann.2012.0706 +mkdir -p $cdir +ln -s /intrepid-fs0/users/lzamboni/persistent/combined/many-en1/run001 $cdir/en1eo14Ic2 +ln -s /intrepid-fs0/users/lzamboni/persistent/combined/many-en1/run002 $cdir/en1eo14Ic3 +ln -s /intrepid-fs0/users/lzamboni/persistent/combined/many-en1/run003 $cdir/en1eo14Ic4 +ln -s /intrepid-fs0/users/lzamboni/persistent/combined/many-en1/run011 $cdir/en1eo8Ic4 + + +eur$ for d in $cdir/*; do (echo -n $d: ; find $d/19????01 -type f | wc -l ); done +/intrepid-fs0/users/wilde/persistent/LZ/combines.toann.2012.0706/en1eo14Ic2:740 +/intrepid-fs0/users/wilde/persistent/LZ/combines.toann.2012.0706/en1eo14Ic3:740 +/intrepid-fs0/users/wilde/persistent/LZ/combines.toann.2012.0706/en1eo14Ic4:740 +/intrepid-fs0/users/wilde/persistent/LZ/combines.toann.2012.0706/en1eo8Ic4:740 +eur$ + + +/intrepid-fs0/users/wilde/persistent/LZ/combines.toann.2012.0706/en1eo14Ic2 +/intrepid-fs0/users/wilde/persistent/LZ/combines.toann.2012.0706/en1eo14Ic3 +/intrepid-fs0/users/wilde/persistent/LZ/combines.toann.2012.0706/en1eo14Ic4 +/intrepid-fs0/users/wilde/persistent/LZ/combines.toann.2012.0706/en1eo8Ic4 + + +==== Clean Tracker + +en1 realization IDs: + +en1eo12Ic1 +en1eo12Ic2 +en1eo12Ic3 +en1eo12Ic4 +en1eo14Ic1 +en1eo14Ic2 +en1eo14Ic3 +en1eo14Ic4 +en1eo16Ic1 +en1eo16Ic2 +en1eo16Ic3 +en1eo16Ic4 +en1eo8Ic1 +en1eo8Ic2 +en1eo8Ic3 +en1eo8Ic4 + +en1 HiRAM output: + +/intrepid-fs0/users/lzamboni/persistent/en1eo12Ic1 +/intrepid-fs0/users/lzamboni/persistent/en1eo12Ic2 +/intrepid-fs0/users/lzamboni/persistent/en1eo12Ic3 +/intrepid-fs0/users/lzamboni/persistent/en1eo12Ic4 +/intrepid-fs0/users/lzamboni/persistent/en1eo14Ic1 +/intrepid-fs0/users/lzamboni/persistent/en1eo14Ic2 +/intrepid-fs0/users/lzamboni/persistent/en1eo14Ic3 +/intrepid-fs0/users/lzamboni/persistent/en1eo14Ic4 +/intrepid-fs0/users/lzamboni/persistent/en1eo16Ic1 +/intrepid-fs0/users/lzamboni/persistent/en1eo16Ic2 +/intrepid-fs0/users/lzamboni/persistent/en1eo16Ic3 +/intrepid-fs0/users/lzamboni/persistent/en1eo16Ic4 +/intrepid-fs0/users/lzamboni/persistent/en1eo8Ic1 +/intrepid-fs0/users/lzamboni/persistent/en1eo8Ic2 +/intrepid-fs0/users/lzamboni/persistent/en1eo8Ic3 +/intrepid-fs0/users/lzamboni/persistent/en1eo8Ic4 + +Combined output: + +/intrepid-fs0/users/lzamboni/persistent/combined/en1eo12Ic1/run004 en1eo12Ic1 +/intrepid-fs0/users/lzamboni/persistent/combined/en1eo12Ic2/run001 en1eo12Ic2 +/intrepid-fs0/users/lzamboni/persistent/combined/en1eo12Ic3/run001 en1eo12Ic3 +/intrepid-fs0/users/lzamboni/persistent/combined/run001 en1eo12Ic4 uncertain realID +/intrepid-fs0/users/lzamboni/persistent/combined/run002 en1eo14Ic1 uncertain realID +/intrepid-fs0/users/lzamboni/persistent/combined/many-en1/run001 en1eo14Ic2 BAD +/intrepid-fs0/users/wilde/persistent/LZ/combined.2012.0707/en1eo14Ic2 en1eo14Ic2 +/intrepid-fs0/users/lzamboni/persistent/combined/many-en1/run002 en1eo14Ic3 BAD +/intrepid-fs0/users/wilde/persistent/LZ/combined.2012.0707/en1eo14Ic3 en1eo14Ic3 +/intrepid-fs0/users/lzamboni/persistent/combined/many-en1/run003 en1eo14Ic4 BAD +/intrepid-fs0/users/wilde/persistent/LZ/combined.2012.0707/en1eo14Ic4 en1eo14Ic4 +/intrepid-fs0/users/lzamboni/persistent/combined/many-en1/run004 en1eo16Ic1 +/intrepid-fs0/users/lzamboni/persistent/combined/many-en1/run005 en1eo16Ic2 +/intrepid-fs0/users/lzamboni/persistent/combined/many-en1/run006 en1eo16Ic3 +/intrepid-fs0/users/lzamboni/persistent/combined/many-en1/run007 en1eo16Ic4 +/intrepid-fs0/users/lzamboni/persistent/combined/many-en1/run008 en1eo8Ic1 +/intrepid-fs0/users/lzamboni/persistent/combined/many-en1/run009 en1eo8Ic2 +/intrepid-fs0/users/lzamboni/persistent/combined/many-en1/run010 en1eo8Ic3 +/intrepid-fs0/users/lzamboni/persistent/combined/many-en1/run011 en1eo8Ic4 BAD +/intrepid-fs0/users/wilde/persistent/LZ/combined.2012.0707/en1eo8Ic4 en1eo8Ic4 + +Yearly output: + +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/combined/en1eo12Ic1/run001 en1eo12Ic1 MISSING +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/combined/en1eo12Ic2/run002 en1eo12Ic2 MISSING +COMPLETED - ignore en1eo12Ic3 IGNORE +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/combined/run001/run003 en1eo12Ic4 SHORT by half +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/combined/run002/run004 en1eo14Ic1 +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run001/run001/ en1eo14Ic2 BAD +/intrepid-fs0/users/wilde/persistent/LZ/yearly.2012.0708/en1eo14Ic2/run005/run001 en1eo14Ic2 +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run002/run003/ en1eo14Ic3 BAD +/intrepid-fs0/users/wilde/persistent/LZ/yearly.2012.0708/en1eo14Ic3/run006/run002 en1eo14Ic3 +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run003/run004/ en1eo14Ic4 BAD +/intrepid-fs0/users/wilde/persistent/LZ/yearly.2012.0708/en1eo14Ic4/run007/run004 en1eo14Ic4 +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run004/run005/ en1eo16Ic1 +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run005/run006/ en1eo16Ic2 +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run006/run007/ en1eo16Ic3 +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run007/run008/ en1eo16Ic4 +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run008/run009/ en1eo8Ic1 +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run009/run010/ en1eo8Ic2 +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run010/run011/ en1eo8Ic3 +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run011/run012/ en1eo8Ic4 BAD +/intrepid-fs0/users/wilde/persistent/LZ/yearly.2012.0708/en1eo8Ic4/run008/run004 en1eo8Ic4 running + +Atmos average output from pfrepp: + + en1eo12Ic1 + en1eo12Ic2 +COMPLETED - IGNORE -------------------------------------- en1eo12Ic3 + en1eo12Ic4 +/intrepid-fs0/users/wilde/persistent/LZ/pfrepps/en1eo14Ic1 en1eo14Ic1 +/intrepid-fs0/users/wilde/persistent/LZ/pfrepps/en1eo14Ic2 en1eo14Ic2 +/intrepid-fs0/users/wilde/persistent/LZ/pfrepps/en1eo14Ic3 en1eo14Ic3 + en1eo14Ic4 +/intrepid-fs0/users/wilde/persistent/LZ/pfrepps/en1eo16Ic1 en1eo16Ic1 +/intrepid-fs0/users/wilde/persistent/LZ/pfrepps/en1eo16Ic2 en1eo16Ic2 +/intrepid-fs0/users/wilde/persistent/LZ/pfrepps/en1eo16Ic3 en1eo16Ic3 +/intrepid-fs0/users/wilde/persistent/LZ/pfrepps/en1eo16Ic4 en1eo16Ic4 +/intrepid-fs0/users/wilde/persistent/LZ/pfrepps/en1eo8Ic1 en1eo8Ic1 +/intrepid-fs0/users/wilde/persistent/LZ/pfrepps/en1eo8Ic2 en1eo8Ic2 +/intrepid-fs0/users/wilde/persistent/LZ/pfrepps/en1eo8Ic3 en1eo8Ic3 + en1eo8Ic4 + + + + + +en1eo8Ic1 - not sure why this was omitted from set02??? + + +en1eo16Ic1 - completed atmos avg as part of 3/8 steps that completed in a full-run attempt +en1eo12Ic3 - Laura is running analysis on this reali, so that one must have been pfrepp'ed by her + +Set03 to do: + +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/combined/run002/run004 en1eo14Ic1 +/intrepid-fs0/users/wilde/persistent/LZ/yearly.2012.0708/en1eo14Ic2/run005/run001 en1eo14Ic2 +/intrepid-fs0/users/wilde/persistent/LZ/yearly.2012.0708/en1eo14Ic3/run006/run002 en1eo14Ic3 +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run008/run009/ en1eo8Ic1 + +set04: + +/intrepid-fs0/users/wilde/persistent/LZ/yearly.2012.0708/en1eo14Ic4/run007/run004 en1eo14Ic4 +/intrepid-fs0/users/wilde/persistent/LZ/yearly.2012.0708/en1eo8Ic4/run008/run004 en1eo8Ic4 + +set05 (tentative): + +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/combined/en1eo12Ic1/run001 en1eo12Ic1 MISSING +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/combined/en1eo12Ic2/run002 en1eo12Ic2 MISSING +/intrepid-fs0/users/lzamboni/persistent/yearly-nco/combined/run001/run003 en1eo12Ic4 SHORT by half - Rerun? + + +============= TO FIX ============== + +[x] first of all, if you need to run, make sure you're using the following version +of NCO (which is a package that handles netCDF files) + +nco-4.0.9 for yearly reorganization + +nco-3.9.2 for pfrepp and for combine (combine might not used nco at all, but +this is our solution today) + +when we know makeyearly, combine and pfrepp work, we may want to test whether +the following version works for all cases (as ALCF Support suggests) + +3.9.9-udunits2 + +--- + +You can referred to my working dir /home/lzamboni/SWIFT/output/en1eo12Ic3 to +track these changes (don't look at either en1eo12Ic1 or en1eo12Ic2): + +[no] 1) I renamed combine_realization.sh as combine_realizations.sh (note the "s" +before .sh) since this is what the script expects. + +[x] fixed in README: 2) what is referred to as runall.combine_realizations.sh in the README (see in +6.) is actually runall.sh so I changed that. + +3) On real.todo I added the full path of the input files and not only the name + of the realization. (what in the README is Climo_001 needs to be + /intrepid-fs0/users/lzamboni/persistent/Climo_001/history instead; what + changes between different realizations is only the dirname between + persistent and history) + +[x] used combine.chk.sh version, renamed it to combine.sh: 4) what is referred to as combine.sh is actually combine.chk.sh + +[x] .shk version used $USER: 5) on this combine.chk.sh I changed "wilde" to "lzamboni" + +6) this is the part that needs adjustments: in runall.config I changed outdir +(which is where the output goes) + +if you look at the dir I pointed you to, you'll see that in that case +outdir=/intrepid-fs0/users/lzamboni/persistent/combined/en1eo12Ic3 which +creates "en1eo12Ic3" under /intrepid-fs0/users/lzamboni/persistent/combined/ + +as is, I cannot process 2 realizations (say en1eo12Ic4 and en1eo14Ic1) and +have the output in separate directories. + +At the moment I am running runall.sh to process 2 realizations (en1eo12Ic4 and +en1eo14Ic1). I set outdir=/intrepid-fs0/users/lzamboni/persistent/combined/ in +runall.config + +Even if it completes without problems, I would like to be able to specify just +the name of the realization (real_name) and obtaining the output in +/intrepid-fs0/users/lzamboni/persistent/combined/real_name. The input data +are always in /intrepid-fs0/users/lzamboni/persistent/real_name/history. + From wilde at ci.uchicago.edu Wed Oct 10 09:46:48 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Wed, 10 Oct 2012 09:46:48 -0500 (CDT) Subject: [Swift-commit] r5962 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121010144648.A914B9CCB6@svn.ci.uchicago.edu> Author: wilde Date: 2012-10-10 09:46:48 -0500 (Wed, 10 Oct 2012) New Revision: 5962 Modified: SwiftApps/ParVis/HiRAMTools/runall.sh Log: Cleanups from July 2012. Modified: SwiftApps/ParVis/HiRAMTools/runall.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/runall.sh 2012-10-10 14:43:26 UTC (rev 5961) +++ SwiftApps/ParVis/HiRAMTools/runall.sh 2012-10-10 14:46:48 UTC (rev 5962) @@ -14,13 +14,9 @@ source ./runall.config # set local params fi -echo bindir=$bindir script=$script +echo script=$script -# Helpful patterns for filename matching -yyyy="[0-9][0-9][0-9][0-9]" -yyyymmdd="${yyyy}[0-9][0-9][0-9][0-9]" - mkdir -p $outdir while true; do @@ -32,12 +28,6 @@ echo RunAll: Processing $real -# find $realdir/$real/history/$yyyymmdd | -# grep .nc.$yyyy | -# sed -e 's/.....$//' | sort | uniq > $real.groups -# -# echo Created $real.groups - echo Running: $bindir/$script \ $real \ $outdir From swift at ci.uchicago.edu Sun Oct 14 22:35:14 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Sun, 14 Oct 2012 22:35:14 -0500 (CDT) Subject: [Swift-commit] cog r3484 Message-ID: <20121015033514.638508D0007D@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3484 | hategan | 2012-10-14 22:30:22 -0500 (Sun, 14 Oct 2012) | 1 line fixed two deadlocks ------------------------------------------------------------------------ Index: modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Cpu.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Cpu.java (revision 3483) +++ modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Cpu.java (working copy) @@ -242,13 +242,7 @@ } private boolean checkSuspended(Block block) { - if (block.isSuspended()) { - block.shutdownIfEmpty(this); - return true; - } - else { - return false; - } + return block.isSuspended(); } protected void submit(Job job) { Index: modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Block.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Block.java (revision 3483) +++ modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Block.java (working copy) @@ -116,20 +116,22 @@ } else if (running) { Time last = getStartTime(); + List active = new ArrayList(); synchronized (cpus) { - for (Cpu cpu: cpus) { - if (cpu.getTimeLast().isGreaterThan(last)) { - last = cpu.getTimeLast(); - } + active.addAll(cpus); + } + for (Cpu cpu: active) { + if (cpu.getTimeLast().isGreaterThan(last)) { + last = cpu.getTimeLast(); } - if (cpus.isEmpty()) { - // prevent block from being done when startup of workers is - // really really slow, - // like as on the BGP where it takes a couple of minutes to - // initialize a partition - last = Time.now(); - } } + if (active.isEmpty()) { + // prevent block from being done when startup of workers is + // really really slow, + // like as on the BGP where it takes a couple of minutes to + // initialize a partition + last = Time.now(); + } Time deadline = Time.min(starttime.add(walltime), last.add(bqp.getSettings().getMaxWorkerIdleTime())); setDeadline(deadline); From hategan at ci.uchicago.edu Sun Oct 14 22:35:17 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sun, 14 Oct 2012 22:35:17 -0500 (CDT) Subject: [Swift-commit] r5964 - in trunk: resources src/org/griphyn/vdl/engine Message-ID: <20121015033517.605D79CCB6@svn.ci.uchicago.edu> Author: hategan Date: 2012-10-14 22:35:17 -0500 (Sun, 14 Oct 2012) New Revision: 5964 Modified: trunk/resources/Karajan.stg trunk/src/org/griphyn/vdl/engine/Karajan.java Log: use unbuffered parallel for statements and also drop parallel for call arguments if parameters are passed as variables only Modified: trunk/resources/Karajan.stg =================================================================== --- trunk/resources/Karajan.stg 2012-10-10 14:47:34 UTC (rev 5963) +++ trunk/resources/Karajan.stg 2012-10-15 03:35:17 UTC (rev 5964) @@ -189,11 +189,13 @@ >> +// use unbuffered parallel here +// since calls don't use this macro parallel(statements) ::= << $if(rest(statements))$ - + $statements;separator="\n"$ - + $else$ $statements$ @@ -201,6 +203,7 @@ $endif$ >> + // the 'function' template outputs a karajan code fragment // that calls a function in the 'swiftscript' namespace. @@ -252,15 +255,15 @@ // they are not // $outputs:vdl_log_output();separator="\n"$ -callInternal(func, outputs, inputs, line) ::= << +callInternal(func, outputs, inputs, line, serialize) ::= << $cs()$ <$func$ _traceline="$line$"> - + $if(!serialize)$$endif$ $outputs:callInternal_log_output();separator="\n"$ $inputs:callInternal_log_input();separator="\n"$ - + $if(!serialize)$$endif$ @@ -292,12 +295,12 @@ >> -callUserDefined(func, outputs, inputs, line) ::= << +callUserDefined(func, outputs, inputs, line, serialize) ::= << <$func$ _traceline="$line$"> - + $if(!serialize)$$endif$ $outputs;separator="\n"$ $inputs;separator="\n"$ - + $if(!serialize)$$endif$ >> Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/Karajan.java 2012-10-10 14:47:34 UTC (rev 5963) +++ trunk/src/org/griphyn/vdl/engine/Karajan.java 2012-10-15 03:35:17 UTC (rev 5964) @@ -721,7 +721,7 @@ if (proc.getInputArray(i).isOptional()) noOfOptInArgs++; inArgs.put(proc.getInputArray(i).getName(), proc.getInputArray(i)); - } + } if (!proc.getAnyNumOfInputArgs() && (call.sizeOfInputArray() < proc.sizeOfInputArray() - noOfOptInArgs || call.sizeOfInputArray() > proc.sizeOfInputArray())) @@ -861,12 +861,40 @@ if (!inhibitOutput) { scope.appendStatement(callST); } + if (allVariables(callST.getAttribute("outputs")) && allVariables(callST.getAttribute("inputs"))) { + callST.setAttribute("serialize", Boolean.TRUE); + } return callST; } catch(CompilationException ce) { throw new CompilationException("Compile error in procedure invocation at "+call.getSrc()+": "+ce.getMessage(),ce); } } + private static final Set VAR_TYPES; + + static { + VAR_TYPES = new HashSet(); + VAR_TYPES.add("id"); + VAR_TYPES.add("extractarrayelement"); + VAR_TYPES.add("slicearray"); + VAR_TYPES.add("extractstructureelement"); + } + + private boolean allVariables(Object list) { + @SuppressWarnings("unchecked") + List l = (List) list; + for (StringTemplate pst : l) { + if (!pst.getName().equals("call_arg")) { + return false; + } + StringTemplate expr = (StringTemplate) pst.getAttribute("expr"); + if (!VAR_TYPES.contains(expr.getName())) { + return false; + } + } + return true; + } + private void addWriterToScope(VariableScope scope, XmlObject var) throws CompilationException { String rootvar = abstractExpressionToRootVariable(var); boolean partial = rootVariableIsPartial(var); From hategan at ci.uchicago.edu Sun Oct 14 22:37:49 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sun, 14 Oct 2012 22:37:49 -0500 (CDT) Subject: [Swift-commit] r5965 - in trunk: libexec src/org/griphyn/vdl/karajan/lib Message-ID: <20121015033749.7FEB69CCB6@svn.ci.uchicago.edu> Author: hategan Date: 2012-10-14 22:37:49 -0500 (Sun, 14 Oct 2012) New Revision: 5965 Added: trunk/src/org/griphyn/vdl/karajan/lib/SelfCloseParallelFor.java trunk/src/org/griphyn/vdl/karajan/lib/Throttled.java Removed: trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java Modified: trunk/libexec/execute-default.k trunk/libexec/vdl-lib.xml Log: removed throttling from foreach and added throttling to app invocation Modified: trunk/libexec/execute-default.k =================================================================== --- trunk/libexec/execute-default.k 2012-10-15 03:35:17 UTC (rev 5964) +++ trunk/libexec/execute-default.k 2012-10-15 03:37:49 UTC (rev 5965) @@ -10,7 +10,7 @@ derr == false then( if( sys:not(done) try( - sequential( + throttled( log(LOG:INFO, "START thread={#thread} tr={tr}") vdl:setprogress("Selecting site") restartOnError(".*", vdl:configProperty("execution.retries"), Modified: trunk/libexec/vdl-lib.xml =================================================================== --- trunk/libexec/vdl-lib.xml 2012-10-15 03:35:17 UTC (rev 5964) +++ trunk/libexec/vdl-lib.xml 2012-10-15 03:37:49 UTC (rev 5965) @@ -124,7 +124,8 @@ - + + Added: trunk/src/org/griphyn/vdl/karajan/lib/SelfCloseParallelFor.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/SelfCloseParallelFor.java (rev 0) +++ trunk/src/org/griphyn/vdl/karajan/lib/SelfCloseParallelFor.java 2012-10-15 03:37:49 UTC (rev 5965) @@ -0,0 +1,111 @@ +/* + * Copyright 2012 University of Chicago + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package org.griphyn.vdl.karajan.lib; + +import org.apache.log4j.Logger; +import org.globus.cog.karajan.arguments.Arg; +import org.globus.cog.karajan.stack.VariableStack; +import org.globus.cog.karajan.util.Identifier; +import org.globus.cog.karajan.util.KarajanIterator; +import org.globus.cog.karajan.util.ThreadingContext; +import org.globus.cog.karajan.workflow.ExecutionException; +import org.globus.cog.karajan.workflow.futures.FutureFault; +import org.globus.cog.karajan.workflow.futures.FutureIteratorIncomplete; +import org.globus.cog.karajan.workflow.nodes.AbstractUParallelIterator; +import org.globus.cog.karajan.workflow.nodes.FlowNode; +import org.globus.cog.karajan.workflow.nodes.FlowNode.FNTP; + +public class SelfCloseParallelFor extends AbstractUParallelIterator { + public static final Logger logger = Logger + .getLogger(SelfCloseParallelFor.class); + + public static final Arg A_NAME = new Arg.Positional("name"); + public static final Arg A_IN = new Arg.Positional("in"); + public static final Arg A_SELF_CLOSE = new Arg.Optional("selfclose", Boolean.FALSE); + + static { + setArguments(SelfCloseParallelFor.class, new Arg[] { A_NAME, A_IN, A_SELF_CLOSE }); + } + + protected void partialArgumentsEvaluated(VariableStack stack) + throws ExecutionException { + stack.setVar("selfclose", A_SELF_CLOSE.getValue(stack)); + super.partialArgumentsEvaluated(stack); + } + + protected void citerate(VariableStack stack, Identifier var, KarajanIterator i) + throws ExecutionException { + try { + synchronized(stack.currentFrame()) { + while (i.hasNext()) { + Object value = i.next(); + VariableStack copy = stack.copy(); + copy.enter(); + ThreadingContext.set(copy, ThreadingContext.get(copy).split(i.current())); + setIndex(copy, getArgCount()); + setArgsDone(copy); + copy.setVar(var.getName(), value); + int r = preIncRunning(stack); + startElement(getArgCount(), copy); + } + } + if (FlowNode.debug) { + threadTracker.remove(new FNTP(this, ThreadingContext.get(stack))); + } + // Now make sure all iterations have not completed + int left = preDecRunning(stack); + if (left == 0) { + complete(stack); + } + } + catch (FutureIteratorIncomplete fii) { + synchronized (stack.currentFrame()) { + // if this is defined, then resume from iterate + stack.setVar(ITERATOR, i); + } + fii.getFutureIterator().addModificationAction(this, stack); + } + } + + protected void iterationCompleted(VariableStack stack) throws ExecutionException { + stack.leave(); + int running; + synchronized(stack.currentFrame()) { + running = preDecRunning(stack); + if (running == 1) { + KarajanIterator iter = (KarajanIterator) stack.currentFrame().getVar(ITERATOR); + if (stack.currentFrame().getVar("selfclose").equals(Boolean.TRUE)) { + try { + iter.hasNext(); + } + catch (FutureFault f) { + running = 0; + } + } + } + } + if (running == 0) { + complete(stack); + } + } + + @Override + public String getTextualName() { + return "foreach"; + } +} Added: trunk/src/org/griphyn/vdl/karajan/lib/Throttled.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/Throttled.java (rev 0) +++ trunk/src/org/griphyn/vdl/karajan/lib/Throttled.java 2012-10-15 03:37:49 UTC (rev 5965) @@ -0,0 +1,77 @@ +/* + * Copyright 2012 University of Chicago + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +/* + * Created on Oct 14, 2012 + */ +package org.griphyn.vdl.karajan.lib; + +import java.io.IOException; +import java.util.LinkedList; + +import org.globus.cog.karajan.stack.VariableStack; +import org.globus.cog.karajan.util.TypeUtil; +import org.globus.cog.karajan.workflow.ExecutionException; +import org.globus.cog.karajan.workflow.nodes.Sequential; +import org.griphyn.vdl.util.VDL2Config; + +public class Throttled extends Sequential { + public static final int DEFAULT_MAX_THREADS = 1024; + + private LinkedList waiting; + private int maxThreadCount, current; + + public Throttled() { + try { + maxThreadCount = TypeUtil.toInt(VDL2Config.getConfig() + .getProperty("foreach.max.threads", String.valueOf(DEFAULT_MAX_THREADS))); + } + catch (IOException e) { + maxThreadCount = DEFAULT_MAX_THREADS; + } + current = 0; + waiting = new LinkedList(); + } + + @Override + protected void executeChildren(VariableStack stack) + throws ExecutionException { + synchronized(this) { + if (current == maxThreadCount) { + waiting.addLast(stack); + return; + } + else { + current++; + } + } + super.executeChildren(stack); + } + + @Override + protected void post(VariableStack stack) throws ExecutionException { + synchronized(this) { + if (!waiting.isEmpty()) { + super.executeChildren(waiting.removeFirst()); + } + else { + current--; + } + } + super.post(stack); + } +} Deleted: trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java 2012-10-15 03:35:17 UTC (rev 5964) +++ trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java 2012-10-15 03:37:49 UTC (rev 5965) @@ -1,309 +0,0 @@ -/* - * Copyright 2012 University of Chicago - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - - -package org.griphyn.vdl.karajan.lib; - -import java.io.IOException; -import java.util.ArrayList; -import java.util.Iterator; -import java.util.List; - -import org.apache.log4j.Logger; -import org.globus.cog.karajan.arguments.Arg; -import org.globus.cog.karajan.stack.VariableNotFoundException; -import org.globus.cog.karajan.stack.VariableStack; -import org.globus.cog.karajan.util.Identifier; -import org.globus.cog.karajan.util.KarajanIterator; -import org.globus.cog.karajan.util.ThreadingContext; -import org.globus.cog.karajan.util.TypeUtil; -import org.globus.cog.karajan.workflow.ExecutionException; -import org.globus.cog.karajan.workflow.futures.FutureEvaluationException; -import org.globus.cog.karajan.workflow.futures.FutureFault; -import org.globus.cog.karajan.workflow.futures.FutureIterator; -import org.globus.cog.karajan.workflow.futures.FutureIteratorIncomplete; -import org.globus.cog.karajan.workflow.futures.FutureListener; -import org.globus.cog.karajan.workflow.futures.ListenerStackPair; -import org.globus.cog.karajan.workflow.nodes.AbstractParallelIterator; -import org.griphyn.vdl.util.VDL2Config; - -public class ThrottledParallelFor extends AbstractParallelIterator { - public static final Logger logger = Logger - .getLogger(ThrottledParallelFor.class); - - public static final int DEFAULT_MAX_THREADS = 1024; - - public static final Arg A_NAME = new Arg.Positional("name"); - public static final Arg A_IN = new Arg.Positional("in"); - public static final Arg A_SELF_CLOSE = new Arg.Optional("selfclose", Boolean.FALSE); - - static { - setArguments(ThrottledParallelFor.class, new Arg[] { A_NAME, A_IN, A_SELF_CLOSE }); - } - - public static final String THREAD_COUNT = "#threadcount"; - - private int maxThreadCount = -1; - - protected void partialArgumentsEvaluated(VariableStack stack) - throws ExecutionException { - stack.setVar("selfclose", A_SELF_CLOSE.getValue(stack)); - super.partialArgumentsEvaluated(stack); - } - - public void iterate(VariableStack stack, Identifier var, KarajanIterator i) - throws ExecutionException { - if (elementCount() > 0) { - if (logger.isDebugEnabled()) { - logger.debug("iterateParallel: " + stack.parentFrame()); - } - stack.setVar(VAR, var); - setChildFailed(stack, false); - stack.setCaller(this); - initThreadCount(stack, TypeUtil.toBoolean(stack.currentFrame().getVar("selfclose")), i); - stack.currentFrame().deleteVar("selfclose"); - citerate(stack, var, i); - } - else { - complete(stack); - } - } - - protected void citerate(VariableStack stack, Identifier var, - KarajanIterator i) throws ExecutionException { - ThreadCount tc = getThreadCount(stack); - - // we can bulk operations at the start to avoid contention - // on the counter since at least as many - // threads as reported by available() are available - int available = tc.available(); - try { - int j = 0; - try { - for (; j < available && i.hasNext(); j++) { - VariableStack copy = stack.copy(); - copy.enter(); - ThreadingContext.set(copy, ThreadingContext.get(copy).split( - i.current())); - setIndex(copy, getArgCount()); - setArgsDone(copy); - copy.setVar(var.getName(), i.next()); - startElement(getArgCount(), copy); - } - } - finally { - tc.add(j); - } - while (i.hasNext()) { - Object value = tc.tryIncrement(); - VariableStack copy = stack.copy(); - copy.enter(); - ThreadingContext.set(copy, ThreadingContext.get(copy).split( - i.current())); - setIndex(copy, getArgCount()); - setArgsDone(copy); - copy.setVar(var.getName(), value); - startElement(getArgCount(), copy); - } - - int left; - synchronized(tc) { - // can only have closed and running = 0 in one place - tc.close(); - left = tc.current(); - } - if (left == 0) { - complete(stack); - } - } - catch (FutureIteratorIncomplete fii) { - synchronized (stack.currentFrame()) { - stack.setVar(ITERATOR, i); - } - fii.getFutureIterator().addModificationAction(this, stack); - } - } - - public void failed(VariableStack stack, ExecutionException e) throws ExecutionException { - if (!testAndSetChildFailed(stack)) { - if (stack.parentFrame().isDefined(VAR)) { - stack.leave(); - } - failImmediately(stack, e); - } - } - - protected void iterationCompleted(VariableStack stack) - throws ExecutionException { - stack.leave(); - ThreadCount tc = getThreadCount(stack); - int running; - boolean closed; - boolean iteratorHasValues; - synchronized(tc) { - closed = tc.isClosed(); - running = tc.decrement(); - iteratorHasValues = !tc.selfClose || tc.iteratorHasValues(); - } - if (running == 0) { - if (closed || !iteratorHasValues) { - complete(stack); - } - } - } - - private void initThreadCount(VariableStack stack, boolean selfClose, KarajanIterator i) { - if (maxThreadCount < 0) { - try { - maxThreadCount = TypeUtil.toInt(VDL2Config.getConfig() - .getProperty("foreach.max.threads", String.valueOf(DEFAULT_MAX_THREADS))); - } - catch (IOException e) { - maxThreadCount = DEFAULT_MAX_THREADS; - } - } - stack.setVar(THREAD_COUNT, new ThreadCount(maxThreadCount, selfClose, i)); - } - - private ThreadCount getThreadCount(VariableStack stack) - throws VariableNotFoundException { - return (ThreadCount) stack.getVar(THREAD_COUNT); - } - - @Override - public String getTextualName() { - return "foreach"; - } - - private static class ThreadCount implements FutureIterator { - private int maxThreadCount; - private int crt; - private boolean selfClose, closed; - private List listeners; - private KarajanIterator i; - - public ThreadCount(int maxThreadCount, boolean selfClose, KarajanIterator i) { - this.maxThreadCount = maxThreadCount; - this.selfClose = selfClose; - this.i = i; - crt = 0; - } - - public boolean iteratorHasValues() { - try { - return i.hasNext(); - } - catch (FutureFault e) { - return false; - } - } - - public boolean isSelfClose() { - return selfClose; - } - - public synchronized int available() { - return maxThreadCount - crt; - } - - public synchronized void add(int count) { - crt += count; - } - - public synchronized Object tryIncrement() { - // there is no way that both crt == 0 and i has no values outside this critical section - if (crt < maxThreadCount) { - Object o = i.next(); - crt++; - return o; - } - else { - throw new FutureIteratorIncomplete(this, this); - } - } - - public synchronized int decrement() { - crt--; - notifyListeners(); - return crt; - } - - private void notifyListeners() { - if (listeners != null) { - Iterator i = listeners.iterator(); - listeners = null; - while (i.hasNext()) { - ListenerStackPair etp = i.next(); - i.remove(); - etp.listener.futureModified(this, etp.stack); - } - } - } - - public boolean hasAvailable() { - return false; - } - - public int count() { - return 0; - } - - public synchronized int current() { - return crt; - } - - public Object peek() { - return null; - } - - public boolean hasNext() { - return false; - } - - public Object next() { - return null; - } - - public void remove() { - } - - public synchronized void addModificationAction(FutureListener target, - VariableStack stack) { - if (listeners == null) { - listeners = new ArrayList(); - } - listeners.add(new ListenerStackPair(target, stack)); - if (crt < maxThreadCount) { - notifyListeners(); - } - } - - public synchronized void close() { - this.closed = true; - } - - public void fail(FutureEvaluationException e) { - } - - public Object getValue() { - return null; - } - - public synchronized boolean isClosed() { - return closed; - } - } -} From hategan at ci.uchicago.edu Sun Oct 14 22:41:21 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sun, 14 Oct 2012 22:41:21 -0500 (CDT) Subject: [Swift-commit] r5966 - in trunk/src/org/griphyn/vdl: karajan/lib mapping Message-ID: <20121015034121.806929CCB6@svn.ci.uchicago.edu> Author: hategan Date: 2012-10-14 22:41:21 -0500 (Sun, 14 Oct 2012) New Revision: 5966 Modified: trunk/src/org/griphyn/vdl/karajan/lib/CreateArray.java trunk/src/org/griphyn/vdl/karajan/lib/New.java trunk/src/org/griphyn/vdl/mapping/AbstractDataNode.java trunk/src/org/griphyn/vdl/mapping/RootDataNode.java Log: don't log useless dataset ids if provenance is disabled; don't create dataset identifiers unless needed; fixed bogus logging level test Modified: trunk/src/org/griphyn/vdl/karajan/lib/CreateArray.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/CreateArray.java 2012-10-15 03:37:49 UTC (rev 5965) +++ trunk/src/org/griphyn/vdl/karajan/lib/CreateArray.java 2012-10-15 03:41:21 UTC (rev 5966) @@ -17,15 +17,14 @@ package org.griphyn.vdl.karajan.lib; -import java.util.HashMap; import java.util.List; -import java.util.Map; import org.apache.log4j.Logger; import org.globus.cog.karajan.arguments.Arg; import org.globus.cog.karajan.stack.VariableStack; import org.globus.cog.karajan.workflow.ExecutionException; import org.globus.cog.karajan.workflow.futures.FutureFault; +import org.griphyn.vdl.mapping.AbstractDataNode; import org.griphyn.vdl.mapping.DSHandle; import org.griphyn.vdl.mapping.MappingParam; import org.griphyn.vdl.mapping.MappingParamSet; @@ -44,7 +43,6 @@ setArguments(CreateArray.class, new Arg[] { PA_VALUE }); } - @SuppressWarnings("unchecked") public Object function(VariableStack stack) throws ExecutionException { Object value = PA_VALUE.getValue(stack); try { @@ -61,7 +59,12 @@ setMapper(handle); } - if (logger.isInfoEnabled()) { + /* + * The reason this is disabled without provenance is that the identifier + * is essentially a random number plus a counter. It does not help + * in debugging problems. + */ + if (AbstractDataNode.provenance && logger.isInfoEnabled()) { logger.info("CREATEARRAY START array=" + handle.getIdentifier()); } @@ -78,7 +81,7 @@ SetFieldValue.deepCopy(dst, n, stack, 1); - if (logger.isInfoEnabled()) { + if (AbstractDataNode.provenance && logger.isInfoEnabled()) { logger.info("CREATEARRAY MEMBER array=" + handle.getIdentifier() + " index=" + index + " member=" + n.getIdentifier()); } @@ -87,7 +90,7 @@ handle.closeShallow(); - if (logger.isInfoEnabled()) { + if (AbstractDataNode.provenance && logger.isInfoEnabled()) { logger.info("CREATEARRAY COMPLETED array=" + handle.getIdentifier()); } Modified: trunk/src/org/griphyn/vdl/karajan/lib/New.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/New.java 2012-10-15 03:37:49 UTC (rev 5965) +++ trunk/src/org/griphyn/vdl/karajan/lib/New.java 2012-10-15 03:41:21 UTC (rev 5966) @@ -29,6 +29,7 @@ import org.globus.cog.karajan.stack.VariableStack; import org.globus.cog.karajan.util.TypeUtil; import org.globus.cog.karajan.workflow.ExecutionException; +import org.griphyn.vdl.mapping.AbstractDataNode; import org.griphyn.vdl.mapping.DSHandle; import org.griphyn.vdl.mapping.ExternalDataNode; import org.griphyn.vdl.mapping.MappingParam; @@ -75,8 +76,10 @@ if (line != null) { mps.set(MappingParam.SWIFT_LINE, line); } + + String threadPrefix = getThreadPrefix(stack); - mps.set(MappingParam.SWIFT_RESTARTID, getThreadPrefix(stack) + ":" + dbgname); + mps.set(MappingParam.SWIFT_RESTARTID, threadPrefix + ":" + dbgname); if (waitfor != null) { mps.set(MappingParam.SWIFT_WAITFOR, waitfor); @@ -88,7 +91,6 @@ String mapper = (String) mps.get(MappingParam.SWIFT_DESCRIPTOR); if ("concurrent_mapper".equals(mapper)) { - String threadPrefix = getThreadPrefix(stack); mps.set(ConcurrentMapper.PARAM_THREAD_PREFIX, threadPrefix); } mps.set(MappingParam.SWIFT_BASEDIR, stack.getExecutionContext().getBasedir()); @@ -96,16 +98,16 @@ try { Type type; if (typename == null) { - throw new ExecutionException - ("vdl:new requires a type specification for value: " + value); + throw new ExecutionException("vdl:new requires a type specification for value: " + value); } else { type = Types.getType(typename); } DSHandle handle; - if(typename.equals("external")) { + if (typename.equals("external")) { handle = new ExternalDataNode(); - } else if (type.isArray()) { + } + else if (type.isArray()) { // dealing with array variable handle = new RootArrayDataNode(type); if (value != null) { @@ -114,9 +116,7 @@ } else { if (!(value instanceof List)) { - throw new ExecutionException - ("An array variable can only be initialized " + - "with a list of values"); + throw new ExecutionException("An array variable can only be initialized with a list of values"); } int index = 0; Iterator i = ((List) value).iterator(); @@ -151,8 +151,8 @@ } } - if (logger.isDebugEnabled()) { - logger.debug("NEW id="+handle.getIdentifier()); + if (AbstractDataNode.provenance && logger.isDebugEnabled()) { + logger.debug("NEW id=" + handle.getIdentifier()); } return handle; } Modified: trunk/src/org/griphyn/vdl/mapping/AbstractDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/AbstractDataNode.java 2012-10-15 03:37:49 UTC (rev 5965) +++ trunk/src/org/griphyn/vdl/mapping/AbstractDataNode.java 2012-10-15 03:41:21 UTC (rev 5966) @@ -20,6 +20,7 @@ */ package org.griphyn.vdl.mapping; +import java.io.IOException; import java.util.ArrayList; import java.util.Collection; import java.util.Collections; @@ -44,11 +45,9 @@ static final String DATASET_URI_PREFIX = "dataset:"; - public static final Logger logger = Logger - .getLogger(AbstractDataNode.class); + public static final Logger logger = Logger.getLogger(AbstractDataNode.class); - public static final MappingParam PARAM_PREFIX = new MappingParam("prefix", - null); + public static final MappingParam PARAM_PREFIX = new MappingParam("prefix", null); /** * Datasets are identified within a run by this sequence number and the @@ -67,12 +66,21 @@ * this value - it exists purely for making unique URIs. */ private static final String datasetIDPartialID = Loader.getUUID(); + + public static boolean provenance = false; + static { + try { + provenance = VDL2Config.getConfig().getProvenanceLog(); + } + catch (IOException e) { + } + } private Field field; private Map, DSHandle> handles; private Object value; private boolean closed; - final String identifierURI = makeIdentifierURIString(); + private String identifier; private Path pathFromRoot; protected FutureWrapper wrapper; @@ -222,8 +230,7 @@ } } catch (NoSuchFieldException e) { - logger.warn("could not find variable: " + field.getId() + - " " + path); + logger.warn("could not find variable: " + field.getId() + " " + path); throw new InvalidPathException(path, this); } } @@ -243,8 +250,7 @@ throw new InvalidPathException("getFields([*]) only applies to arrays"); } try { - ((AbstractDataNode) getField(path.getFirst())).getFields( - fields, path.butFirst()); + ((AbstractDataNode) getField(path.getFirst())).getFields(fields, path.butFirst()); } catch (NoSuchFieldException e) { throw new InvalidPathException(path, this); @@ -255,8 +261,7 @@ public void set(DSHandle handle) { // TODO check type if (closed) { - throw new IllegalArgumentException(this.getDisplayableName() - + " is already assigned"); + throw new IllegalArgumentException(this.getDisplayableName() + " is already assigned"); } if (getParent() == null) { /* @@ -292,8 +297,7 @@ public DSHandle createField(Comparable key) throws NoSuchFieldException { if (closed) { - throw new RuntimeException("Cannot write to closed handle: " + this - + " (" + key + ")"); + throw new RuntimeException("Cannot write to closed handle: " + this + " (" + key + ")"); } return addHandle(key, newNode(getChildField(key))); @@ -302,9 +306,7 @@ protected synchronized DSHandle addHandle(Comparable id, DSHandle handle) { Object o = handles.put(id, handle); if (o != null) { - throw new RuntimeException( - "Trying to create a handle that already exists (" - + id + ") in " + this); + throw new RuntimeException("Trying to create a handle that already exists (" + id + ") in " + this); } return handle; } @@ -366,12 +368,12 @@ public void setValue(Object value) { if (this.closed) { - throw new IllegalArgumentException(this.getDisplayableName() - + " is closed with a value of " + this.value); + throw new IllegalArgumentException(this.getDisplayableName() + + " is closed with a value of " + this.value); } if (this.value != null) { - throw new IllegalArgumentException(this.getDisplayableName() - + " is already assigned with a value of " + this.value); + throw new IllegalArgumentException(this.getDisplayableName() + + " is already assigned with a value of " + this.value); } this.value = value; closeShallow(); @@ -397,15 +399,12 @@ child = (AbstractDataNode) this.getField(name); } catch (NoSuchFieldException e) { - throw new RuntimeException - ("Inconsistency between type declaration and " + + throw new RuntimeException("Inconsistency between type declaration and " + "handle for field '" + name + "'"); } Path fullPath = parentPath.addLast(name); Type type = child.getType(); - if (!type.isPrimitive() && - !child.isArray() && - type.getFields().size() == 0) { + if (!type.isPrimitive() && !child.isArray() && type.getFields().size() == 0) { list.add(fullPath); } else { @@ -424,32 +423,27 @@ } // closed notifyListeners(); - if (logger.isInfoEnabled()) { + if (logger.isDebugEnabled()) { logger.debug("closed " + this.getIdentifyingString()); } // so because its closed, we can dump the contents try { - if(VDL2Config.getConfig().getProvenanceLog()) { + if(provenance) { logContent(); } } catch (Exception e) { - logger.warn("Exception whilst logging dataset content for " + this, - e); + logger.warn("Exception whilst logging dataset content for " + this, e); } - // TODO record retrospective provenance information for this dataset - // here + // TODO record retrospective provenance information for this dataset here // we should do it at closing time because that's the point at which we // know the dataset has its values (all the way down the tree) assigned. - // provenance-id for this dataset should have been assigned at creation - // time, - // though, so that we can refer to this dataset elsewhere before it is - // closed. + // provenance-id for this dataset should have been assigned at creation time, + // though, so that we can refer to this dataset elsewhere before it is closed. - // is this method the only one called to set this.closed? or do - // subclasses + // is this method the only one called to set this.closed? or do subclasses // or other methods ever change it? } @@ -458,11 +452,9 @@ Path pathFromRoot = this.getPathFromRoot(); if (this.getPathFromRoot() != null) { if (logger.isInfoEnabled()) { - logger.info("ROOTPATH dataset=" + identifier + " path=" - + pathFromRoot); + logger.info("ROOTPATH dataset=" + identifier + " path=" + pathFromRoot); if (this.getType().isPrimitive()) { - logger.info("VALUE dataset=" + identifier + " VALUE=" - + this.toString()); + logger.info("VALUE dataset=" + identifier + " VALUE=" + this.toString()); } } @@ -498,8 +490,7 @@ if(filemapped) { Object path = m.map(pathFromRoot); if (logger.isInfoEnabled()) { - logger.info("FILENAME dataset=" + identifier + " filename=" - + path); + logger.info("FILENAME dataset=" + identifier + " filename=" + path); } } } @@ -518,8 +509,7 @@ for (DSHandle handle : handles.values()) { AbstractDataNode node = (AbstractDataNode) handle; if (logger.isInfoEnabled()) { - logger.info("CONTAINMENT parent=" + identifier + - " child=" + node.getIdentifier()); + logger.info("CONTAINMENT parent=" + identifier + " child=" + node.getIdentifier()); } node.logContent(); } @@ -554,8 +544,7 @@ synchronized (this) { for (DSHandle handle : handles.values()) { AbstractDataNode child = (AbstractDataNode) handle; - if (child.getType().isArray() || - child.getType().getFields().size() > 0) { + if (child.getType().isArray() || child.getType().getFields().size() > 0) { child.closeArraySizes(); } } @@ -568,8 +557,7 @@ Path myPath; if (parent != null) { myPath = parent.getPathFromRoot(); - pathFromRoot = myPath.addLast(getField().getId(), parent - .getField().getType().isArray()); + pathFromRoot = myPath.addLast(getField().getId(), parent.getField().getType().isArray()); } else { pathFromRoot = Path.EMPTY_PATH; @@ -586,8 +574,11 @@ return handles; } - public String getIdentifier() { - return identifierURI; + public synchronized String getIdentifier() { + if (identifier == null) { + identifier = makeIdentifierURIString(); + } + return identifier; } String makeIdentifierURIString() { Modified: trunk/src/org/griphyn/vdl/mapping/RootDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/RootDataNode.java 2012-10-15 03:37:49 UTC (rev 5965) +++ trunk/src/org/griphyn/vdl/mapping/RootDataNode.java 2012-10-15 03:41:21 UTC (rev 5966) @@ -20,8 +20,6 @@ */ package org.griphyn.vdl.mapping; -import java.util.Map; - import org.apache.log4j.Logger; import org.globus.cog.karajan.stack.VariableStack; import org.globus.cog.karajan.workflow.futures.Future; From hategan at ci.uchicago.edu Sun Oct 14 22:46:49 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sun, 14 Oct 2012 22:46:49 -0500 (CDT) Subject: [Swift-commit] r5967 - trunk/bin Message-ID: <20121015034649.E8AAB9CCB6@svn.ci.uchicago.edu> Author: hategan Date: 2012-10-14 22:46:49 -0500 (Sun, 14 Oct 2012) New Revision: 5967 Modified: trunk/bin/swift Log: dump heap on OOM Modified: trunk/bin/swift =================================================================== --- trunk/bin/swift 2012-10-15 03:41:21 UTC (rev 5966) +++ trunk/bin/swift 2012-10-15 03:46:49 UTC (rev 5967) @@ -6,7 +6,7 @@ # SWIFT_USAGE_STATS EXEC=org.griphyn.vdl.karajan.Loader -OPTIONS= +OPTIONS="-XX:+HeapDumpOnOutOfMemoryError" CYGWIN= CPDELIM=":" From hategan at ci.uchicago.edu Mon Oct 15 13:45:54 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Mon, 15 Oct 2012 13:45:54 -0500 (CDT) Subject: [Swift-commit] r5968 - trunk/src/org/griphyn/vdl/engine Message-ID: <20121015184554.4EA159CCB6@svn.ci.uchicago.edu> Author: hategan Date: 2012-10-15 13:45:53 -0500 (Mon, 15 Oct 2012) New Revision: 5968 Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java Log: fixed parameter check when only one parameter is used Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/Karajan.java 2012-10-15 03:46:49 UTC (rev 5967) +++ trunk/src/org/griphyn/vdl/engine/Karajan.java 2012-10-15 18:45:53 UTC (rev 5968) @@ -25,6 +25,7 @@ import java.io.PrintStream; import java.util.ArrayList; import java.util.Collection; +import java.util.Collections; import java.util.HashMap; import java.util.HashSet; import java.util.LinkedList; @@ -880,9 +881,15 @@ VAR_TYPES.add("extractstructureelement"); } - private boolean allVariables(Object list) { - @SuppressWarnings("unchecked") - List l = (List) list; + @SuppressWarnings("unchecked") + private boolean allVariables(Object st) { + List l; + if (st instanceof List) { + l = (List) st; + } + else { + l = Collections.singletonList((StringTemplate) st); + } for (StringTemplate pst : l) { if (!pst.getName().equals("call_arg")) { return false; From hategan at ci.uchicago.edu Mon Oct 15 16:41:10 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Mon, 15 Oct 2012 16:41:10 -0500 (CDT) Subject: [Swift-commit] r5969 - trunk/src/org/griphyn/vdl/karajan Message-ID: <20121015214110.CE3259CC99@svn.ci.uchicago.edu> Author: hategan Date: 2012-10-15 16:41:10 -0500 (Mon, 15 Oct 2012) New Revision: 5969 Modified: trunk/src/org/griphyn/vdl/karajan/Loader.java Log: xml intermediate file extension renamed to .swiftx (bug 849) Modified: trunk/src/org/griphyn/vdl/karajan/Loader.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/Loader.java 2012-10-15 18:45:53 UTC (rev 5968) +++ trunk/src/org/griphyn/vdl/karajan/Loader.java 2012-10-15 21:41:10 UTC (rev 5969) @@ -287,7 +287,7 @@ File swiftscript = new File(project); debugText("SWIFTSCRIPT", swiftscript); String projectBase = project.substring(0, project.lastIndexOf('.')); - File xml = new File(projectBase + ".xml"); + File xml = new File(projectBase + ".swiftx"); File kml = new File(projectBase + ".kml"); loadBuildVersion(provenanceEnabled); @@ -359,7 +359,7 @@ // error message for the user kml.delete(); throw new CompilationException( - "Failed to convert .xml to .kml for " + project, e); + "Failed to convert .swiftx to .kml for " + project, e); } } else { From wilde at ci.uchicago.edu Tue Oct 16 09:55:15 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Tue, 16 Oct 2012 09:55:15 -0500 (CDT) Subject: [Swift-commit] r5970 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121016145515.63AE79CCA1@svn.ci.uchicago.edu> Author: wilde Date: 2012-10-16 09:55:15 -0500 (Tue, 16 Oct 2012) New Revision: 5970 Modified: SwiftApps/ParVis/HiRAMTools/README Log: Update README Modified: SwiftApps/ParVis/HiRAMTools/README =================================================================== --- SwiftApps/ParVis/HiRAMTools/README 2012-10-15 21:41:10 UTC (rev 5969) +++ SwiftApps/ParVis/HiRAMTools/README 2012-10-16 14:55:15 UTC (rev 5970) @@ -1,7 +1,9 @@ -The top-level script in this toolkit is "runall.sh". This is the only script -that you should need to run directly. +The top-level script in this toolkit is "runall.sh". This is the script +to use to run directly to run combine and annualization procedures. +To run pfrepp, use runpfrepps.sh. + runall.sh takes a config file which specifies what realization processing script to run, where to find the input realizations, and where to place the processed output realizations. @@ -14,14 +16,14 @@ runall.sh # called by the user, runs the scripts below combine_realization.sh - combine.swift - combine.sh (app, specified in tc) + combine.swift + combine.sh (app, specified in tc) makeyearly_realization.sh - makeyearly.swift - makeyearly-cdo.sh (app, specified in tc) + makeyearly.swift + makeyearly-cdo.sh (app, specified in tc) - runpfrepps.sh # Called by used, runs the scripts below + runpfrepps.sh # Called by user, runs the scripts below pfrepps.swift genpfrepps.sh runscript.sh From wilde at ci.uchicago.edu Tue Oct 16 09:58:44 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Tue, 16 Oct 2012 09:58:44 -0500 (CDT) Subject: [Swift-commit] r5971 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121016145844.0A7FE9CCA1@svn.ci.uchicago.edu> Author: wilde Date: 2012-10-16 09:58:43 -0500 (Tue, 16 Oct 2012) New Revision: 5971 Added: SwiftApps/ParVis/HiRAMTools/makeyearly.sh Removed: SwiftApps/ParVis/HiRAMTools/makeyearly-cdo.sh Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh Log: Rename makeyearly-cdo.sh back to makeyearly.sh Deleted: SwiftApps/ParVis/HiRAMTools/makeyearly-cdo.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly-cdo.sh 2012-10-16 14:55:15 UTC (rev 5970) +++ SwiftApps/ParVis/HiRAMTools/makeyearly-cdo.sh 2012-10-16 14:58:43 UTC (rev 5971) @@ -1,83 +0,0 @@ -#! /bin/bash - -# Create a fast (local-filesystem) working dir - -if mkdir -p /scratch/wilde; then - scratch=/scratch/wilde -elif mkdir -p $HOME/scratch; then - scratch=$HOME/scratch -else - echo $0: cant create scratch dir - exit 11 -fi - -tmp=$(mktemp -d $scratch/makeyearly_XXXXXX) -startdir=$PWD -cd $tmp - -echo running in $PWD - -qtrdec1=$1 # 1st quarter-decade dir -qtrdec2=$2 # 2nd quarter-decade dir -destbase=$3 # base dir for output years - -echo destbase=$destbase qtrdec1=$1 qtrdec2=$2 - -# Add cdo and nco tools to PATH and Intel compiler v11 libs to LD_LIBRARY_PATH - -#export PATH=/soft/apps/cdo-1.4.7/bin:/soft/apps/nco-4.0.9/bin:$PATH -export PATH=/soft/apps/nco-4.0.9/bin:$PATH -export LD_LIBRARY_PATH=/soft/apps/intel-fc-11.0.074/lib/intel64:/soft/apps/intel-cc-11.0.074/lib/intel64:/soft/apps/netcdf-3.6.3/lib:/usr/lib:/usr/X11R6/lib:/usr/local/lib - -fname=$(basename $qtrdec1 .nc) -firstyear=$(basename $(dirname $qtrdec1) | sed -e 's/....$//') - -# process only the time-based files - -if echo $fname | egrep -v 'grid_spec|scalar'; then - - # Merge 2 quarter-decade files into one half-decade file - - ncrcat $qtrdec1 $qtrdec2 5years.nc - - # Extract each year from the half-decade file - - for (( y=$firstyear; y<$firstyear+5; y++ )); do - ncks -d time,"${y}-01-01 00:00:00","${y}-12-31 23:59:59" 5years.nc $y.nc - done - -fi - -# Copy the years from temporary to permanent filesystem - -for (( y=$firstyear; y<$firstyear+5; y++ )) -do - destdir=$destbase/${y}0101 - mkdir -p $destdir - if echo $fname | egrep -v 'grid_spec|scalar'; then - dd if=$y.nc of=$destdir/$fname.nc bs=8M - else - dd if=$qtrdec1 of=$destdir/$fname.nc bs=8M - fi -done - -rm -rf $tmp - -# Delete when this works: - - # Old method: - # - # cdo mergetime $qtrdec1 $qtrdec2 ${fname}.nc - # cdo splityear ${fname}.nc ${fname}. - # Yields, eg, from atmos_month.tile1: - # atmos_month.tile1.1950.nc - # atmos_month.tile1.1951.nc - # atmos_month.tile1.1952.nc - # atmos_month.tile1.1953.nc - # atmos_month.tile1.1954.nc - - # New method: - # - # ncrcat input1.nc input2.nc 5years.nc - # ncks -d time,"$yyyy-01-01 00:00:00","$yyyy-12-31 23:59:59" 5years.nc year1.nc - Copied: SwiftApps/ParVis/HiRAMTools/makeyearly.sh (from rev 5969, SwiftApps/ParVis/HiRAMTools/makeyearly-cdo.sh) =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly.sh (rev 0) +++ SwiftApps/ParVis/HiRAMTools/makeyearly.sh 2012-10-16 14:58:43 UTC (rev 5971) @@ -0,0 +1,83 @@ +#! /bin/bash + +# Create a fast (local-filesystem) working dir + +if mkdir -p /scratch/wilde; then + scratch=/scratch/wilde +elif mkdir -p $HOME/scratch; then + scratch=$HOME/scratch +else + echo $0: cant create scratch dir + exit 11 +fi + +tmp=$(mktemp -d $scratch/makeyearly_XXXXXX) +startdir=$PWD +cd $tmp + +echo running in $PWD + +qtrdec1=$1 # 1st quarter-decade dir +qtrdec2=$2 # 2nd quarter-decade dir +destbase=$3 # base dir for output years + +echo destbase=$destbase qtrdec1=$1 qtrdec2=$2 + +# Add cdo and nco tools to PATH and Intel compiler v11 libs to LD_LIBRARY_PATH + +#export PATH=/soft/apps/cdo-1.4.7/bin:/soft/apps/nco-4.0.9/bin:$PATH +export PATH=/soft/apps/nco-4.0.9/bin:$PATH +export LD_LIBRARY_PATH=/soft/apps/intel-fc-11.0.074/lib/intel64:/soft/apps/intel-cc-11.0.074/lib/intel64:/soft/apps/netcdf-3.6.3/lib:/usr/lib:/usr/X11R6/lib:/usr/local/lib + +fname=$(basename $qtrdec1 .nc) +firstyear=$(basename $(dirname $qtrdec1) | sed -e 's/....$//') + +# process only the time-based files + +if echo $fname | egrep -v 'grid_spec|scalar'; then + + # Merge 2 quarter-decade files into one half-decade file + + ncrcat $qtrdec1 $qtrdec2 5years.nc + + # Extract each year from the half-decade file + + for (( y=$firstyear; y<$firstyear+5; y++ )); do + ncks -d time,"${y}-01-01 00:00:00","${y}-12-31 23:59:59" 5years.nc $y.nc + done + +fi + +# Copy the years from temporary to permanent filesystem + +for (( y=$firstyear; y<$firstyear+5; y++ )) +do + destdir=$destbase/${y}0101 + mkdir -p $destdir + if echo $fname | egrep -v 'grid_spec|scalar'; then + dd if=$y.nc of=$destdir/$fname.nc bs=8M + else + dd if=$qtrdec1 of=$destdir/$fname.nc bs=8M + fi +done + +rm -rf $tmp + +# Delete when this works: + + # Old method: + # + # cdo mergetime $qtrdec1 $qtrdec2 ${fname}.nc + # cdo splityear ${fname}.nc ${fname}. + # Yields, eg, from atmos_month.tile1: + # atmos_month.tile1.1950.nc + # atmos_month.tile1.1951.nc + # atmos_month.tile1.1952.nc + # atmos_month.tile1.1953.nc + # atmos_month.tile1.1954.nc + + # New method: + # + # ncrcat input1.nc input2.nc 5years.nc + # ncks -d time,"$yyyy-01-01 00:00:00","$yyyy-12-31 23:59:59" 5years.nc year1.nc + Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh 2012-10-16 14:55:15 UTC (rev 5970) +++ SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh 2012-10-16 14:58:43 UTC (rev 5971) @@ -82,7 +82,7 @@ cat >$rundir/tc < Author: wilde Date: 2012-10-16 10:38:35 -0500 (Tue, 16 Oct 2012) New Revision: 5972 Modified: SwiftApps/ParVis/HiRAMTools/README Log: README fix. Modified: SwiftApps/ParVis/HiRAMTools/README =================================================================== --- SwiftApps/ParVis/HiRAMTools/README 2012-10-16 14:58:43 UTC (rev 5971) +++ SwiftApps/ParVis/HiRAMTools/README 2012-10-16 15:38:35 UTC (rev 5972) @@ -21,7 +21,7 @@ makeyearly_realization.sh makeyearly.swift - makeyearly-cdo.sh (app, specified in tc) + makeyearly.sh (app, specified in tc) runpfrepps.sh # Called by user, runs the scripts below pfrepps.swift From wilde at ci.uchicago.edu Tue Oct 16 11:58:13 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Tue, 16 Oct 2012 11:58:13 -0500 (CDT) Subject: [Swift-commit] r5973 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121016165813.954309CCA1@svn.ci.uchicago.edu> Author: wilde Date: 2012-10-16 11:58:13 -0500 (Tue, 16 Oct 2012) New Revision: 5973 Modified: SwiftApps/ParVis/HiRAMTools/combine.sh SwiftApps/ParVis/HiRAMTools/combine_realization.sh SwiftApps/ParVis/HiRAMTools/get_real_copy.sh SwiftApps/ParVis/HiRAMTools/get_real_copyL.sh SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh SwiftApps/ParVis/HiRAMTools/runall.sh Log: Set all shell scripts to executable. Property changes on: SwiftApps/ParVis/HiRAMTools/combine.sh ___________________________________________________________________ Added: svn:executable + * Property changes on: SwiftApps/ParVis/HiRAMTools/combine_realization.sh ___________________________________________________________________ Added: svn:executable + * Property changes on: SwiftApps/ParVis/HiRAMTools/get_real_copy.sh ___________________________________________________________________ Modified: svn:executable - + * Property changes on: SwiftApps/ParVis/HiRAMTools/get_real_copyL.sh ___________________________________________________________________ Modified: svn:executable - + * Property changes on: SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh ___________________________________________________________________ Added: svn:executable + * Property changes on: SwiftApps/ParVis/HiRAMTools/runall.sh ___________________________________________________________________ Added: svn:executable + * From wilde at ci.uchicago.edu Tue Oct 16 12:05:20 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Tue, 16 Oct 2012 12:05:20 -0500 (CDT) Subject: [Swift-commit] r5974 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121016170520.B644D9CCB6@svn.ci.uchicago.edu> Author: wilde Date: 2012-10-16 12:05:20 -0500 (Tue, 16 Oct 2012) New Revision: 5974 Modified: SwiftApps/ParVis/HiRAMTools/makeyearly.sh Log: Trying to set makeyearly.sh to executable. Modified: SwiftApps/ParVis/HiRAMTools/makeyearly.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly.sh 2012-10-16 16:58:13 UTC (rev 5973) +++ SwiftApps/ParVis/HiRAMTools/makeyearly.sh 2012-10-16 17:05:20 UTC (rev 5974) @@ -1,6 +1,6 @@ #! /bin/bash -# Create a fast (local-filesystem) working dir +# Create a fast (local-filesystem) working dir if mkdir -p /scratch/wilde; then scratch=/scratch/wilde From hategan at ci.uchicago.edu Tue Oct 16 18:46:06 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Tue, 16 Oct 2012 18:46:06 -0500 (CDT) Subject: [Swift-commit] r5975 - trunk/src/org/griphyn/vdl/karajan/lib Message-ID: <20121016234606.4B68D9CCB6@svn.ci.uchicago.edu> Author: hategan Date: 2012-10-16 18:46:06 -0500 (Tue, 16 Oct 2012) New Revision: 5975 Added: trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java Removed: trunk/src/org/griphyn/vdl/karajan/lib/SelfCloseParallelFor.java Modified: trunk/src/org/griphyn/vdl/karajan/lib/Throttled.java Log: restored ThrottledParallelFor Deleted: trunk/src/org/griphyn/vdl/karajan/lib/SelfCloseParallelFor.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/SelfCloseParallelFor.java 2012-10-16 17:05:20 UTC (rev 5974) +++ trunk/src/org/griphyn/vdl/karajan/lib/SelfCloseParallelFor.java 2012-10-16 23:46:06 UTC (rev 5975) @@ -1,111 +0,0 @@ -/* - * Copyright 2012 University of Chicago - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - - -package org.griphyn.vdl.karajan.lib; - -import org.apache.log4j.Logger; -import org.globus.cog.karajan.arguments.Arg; -import org.globus.cog.karajan.stack.VariableStack; -import org.globus.cog.karajan.util.Identifier; -import org.globus.cog.karajan.util.KarajanIterator; -import org.globus.cog.karajan.util.ThreadingContext; -import org.globus.cog.karajan.workflow.ExecutionException; -import org.globus.cog.karajan.workflow.futures.FutureFault; -import org.globus.cog.karajan.workflow.futures.FutureIteratorIncomplete; -import org.globus.cog.karajan.workflow.nodes.AbstractUParallelIterator; -import org.globus.cog.karajan.workflow.nodes.FlowNode; -import org.globus.cog.karajan.workflow.nodes.FlowNode.FNTP; - -public class SelfCloseParallelFor extends AbstractUParallelIterator { - public static final Logger logger = Logger - .getLogger(SelfCloseParallelFor.class); - - public static final Arg A_NAME = new Arg.Positional("name"); - public static final Arg A_IN = new Arg.Positional("in"); - public static final Arg A_SELF_CLOSE = new Arg.Optional("selfclose", Boolean.FALSE); - - static { - setArguments(SelfCloseParallelFor.class, new Arg[] { A_NAME, A_IN, A_SELF_CLOSE }); - } - - protected void partialArgumentsEvaluated(VariableStack stack) - throws ExecutionException { - stack.setVar("selfclose", A_SELF_CLOSE.getValue(stack)); - super.partialArgumentsEvaluated(stack); - } - - protected void citerate(VariableStack stack, Identifier var, KarajanIterator i) - throws ExecutionException { - try { - synchronized(stack.currentFrame()) { - while (i.hasNext()) { - Object value = i.next(); - VariableStack copy = stack.copy(); - copy.enter(); - ThreadingContext.set(copy, ThreadingContext.get(copy).split(i.current())); - setIndex(copy, getArgCount()); - setArgsDone(copy); - copy.setVar(var.getName(), value); - int r = preIncRunning(stack); - startElement(getArgCount(), copy); - } - } - if (FlowNode.debug) { - threadTracker.remove(new FNTP(this, ThreadingContext.get(stack))); - } - // Now make sure all iterations have not completed - int left = preDecRunning(stack); - if (left == 0) { - complete(stack); - } - } - catch (FutureIteratorIncomplete fii) { - synchronized (stack.currentFrame()) { - // if this is defined, then resume from iterate - stack.setVar(ITERATOR, i); - } - fii.getFutureIterator().addModificationAction(this, stack); - } - } - - protected void iterationCompleted(VariableStack stack) throws ExecutionException { - stack.leave(); - int running; - synchronized(stack.currentFrame()) { - running = preDecRunning(stack); - if (running == 1) { - KarajanIterator iter = (KarajanIterator) stack.currentFrame().getVar(ITERATOR); - if (stack.currentFrame().getVar("selfclose").equals(Boolean.TRUE)) { - try { - iter.hasNext(); - } - catch (FutureFault f) { - running = 0; - } - } - } - } - if (running == 0) { - complete(stack); - } - } - - @Override - public String getTextualName() { - return "foreach"; - } -} Modified: trunk/src/org/griphyn/vdl/karajan/lib/Throttled.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/Throttled.java 2012-10-16 17:05:20 UTC (rev 5974) +++ trunk/src/org/griphyn/vdl/karajan/lib/Throttled.java 2012-10-16 23:46:06 UTC (rev 5975) @@ -30,7 +30,7 @@ import org.griphyn.vdl.util.VDL2Config; public class Throttled extends Sequential { - public static final int DEFAULT_MAX_THREADS = 1024; + public static final int DEFAULT_MAX_THREADS = 1000000; private LinkedList waiting; private int maxThreadCount, current; @@ -38,7 +38,7 @@ public Throttled() { try { maxThreadCount = TypeUtil.toInt(VDL2Config.getConfig() - .getProperty("foreach.max.threads", String.valueOf(DEFAULT_MAX_THREADS))); + .getProperty("exec.throttle", String.valueOf(DEFAULT_MAX_THREADS))); } catch (IOException e) { maxThreadCount = DEFAULT_MAX_THREADS; Added: trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java (rev 0) +++ trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java 2012-10-16 23:46:06 UTC (rev 5975) @@ -0,0 +1,309 @@ +/* + * Copyright 2012 University of Chicago + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package org.griphyn.vdl.karajan.lib; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Iterator; +import java.util.List; + +import org.apache.log4j.Logger; +import org.globus.cog.karajan.arguments.Arg; +import org.globus.cog.karajan.stack.VariableNotFoundException; +import org.globus.cog.karajan.stack.VariableStack; +import org.globus.cog.karajan.util.Identifier; +import org.globus.cog.karajan.util.KarajanIterator; +import org.globus.cog.karajan.util.ThreadingContext; +import org.globus.cog.karajan.util.TypeUtil; +import org.globus.cog.karajan.workflow.ExecutionException; +import org.globus.cog.karajan.workflow.futures.FutureEvaluationException; +import org.globus.cog.karajan.workflow.futures.FutureFault; +import org.globus.cog.karajan.workflow.futures.FutureIterator; +import org.globus.cog.karajan.workflow.futures.FutureIteratorIncomplete; +import org.globus.cog.karajan.workflow.futures.FutureListener; +import org.globus.cog.karajan.workflow.futures.ListenerStackPair; +import org.globus.cog.karajan.workflow.nodes.AbstractParallelIterator; +import org.griphyn.vdl.util.VDL2Config; + +public class ThrottledParallelFor extends AbstractParallelIterator { + public static final Logger logger = Logger + .getLogger(ThrottledParallelFor.class); + + public static final int DEFAULT_MAX_THREADS = 1024; + + public static final Arg A_NAME = new Arg.Positional("name"); + public static final Arg A_IN = new Arg.Positional("in"); + public static final Arg A_SELF_CLOSE = new Arg.Optional("selfclose", Boolean.FALSE); + + static { + setArguments(ThrottledParallelFor.class, new Arg[] { A_NAME, A_IN, A_SELF_CLOSE }); + } + + public static final String THREAD_COUNT = "#threadcount"; + + private int maxThreadCount = -1; + + protected void partialArgumentsEvaluated(VariableStack stack) + throws ExecutionException { + stack.setVar("selfclose", A_SELF_CLOSE.getValue(stack)); + super.partialArgumentsEvaluated(stack); + } + + public void iterate(VariableStack stack, Identifier var, KarajanIterator i) + throws ExecutionException { + if (elementCount() > 0) { + if (logger.isDebugEnabled()) { + logger.debug("iterateParallel: " + stack.parentFrame()); + } + stack.setVar(VAR, var); + setChildFailed(stack, false); + stack.setCaller(this); + initThreadCount(stack, TypeUtil.toBoolean(stack.currentFrame().getVar("selfclose")), i); + stack.currentFrame().deleteVar("selfclose"); + citerate(stack, var, i); + } + else { + complete(stack); + } + } + + protected void citerate(VariableStack stack, Identifier var, + KarajanIterator i) throws ExecutionException { + ThreadCount tc = getThreadCount(stack); + + // we can bulk operations at the start to avoid contention + // on the counter since at least as many + // threads as reported by available() are available + int available = tc.available(); + try { + int j = 0; + try { + for (; j < available && i.hasNext(); j++) { + VariableStack copy = stack.copy(); + copy.enter(); + ThreadingContext.set(copy, ThreadingContext.get(copy).split( + i.current())); + setIndex(copy, getArgCount()); + setArgsDone(copy); + copy.setVar(var.getName(), i.next()); + startElement(getArgCount(), copy); + } + } + finally { + tc.add(j); + } + while (i.hasNext()) { + Object value = tc.tryIncrement(); + VariableStack copy = stack.copy(); + copy.enter(); + ThreadingContext.set(copy, ThreadingContext.get(copy).split( + i.current())); + setIndex(copy, getArgCount()); + setArgsDone(copy); + copy.setVar(var.getName(), value); + startElement(getArgCount(), copy); + } + + int left; + synchronized(tc) { + // can only have closed and running = 0 in one place + tc.close(); + left = tc.current(); + } + if (left == 0) { + complete(stack); + } + } + catch (FutureIteratorIncomplete fii) { + synchronized (stack.currentFrame()) { + stack.setVar(ITERATOR, i); + } + fii.getFutureIterator().addModificationAction(this, stack); + } + } + + public void failed(VariableStack stack, ExecutionException e) throws ExecutionException { + if (!testAndSetChildFailed(stack)) { + if (stack.parentFrame().isDefined(VAR)) { + stack.leave(); + } + failImmediately(stack, e); + } + } + + protected void iterationCompleted(VariableStack stack) + throws ExecutionException { + stack.leave(); + ThreadCount tc = getThreadCount(stack); + int running; + boolean closed; + boolean iteratorHasValues; + synchronized(tc) { + closed = tc.isClosed(); + running = tc.decrement(); + iteratorHasValues = !tc.selfClose || tc.iteratorHasValues(); + } + if (running == 0) { + if (closed || !iteratorHasValues) { + complete(stack); + } + } + } + + private void initThreadCount(VariableStack stack, boolean selfClose, KarajanIterator i) { + if (maxThreadCount < 0) { + try { + maxThreadCount = TypeUtil.toInt(VDL2Config.getConfig() + .getProperty("foreach.max.threads", String.valueOf(DEFAULT_MAX_THREADS))); + } + catch (IOException e) { + maxThreadCount = DEFAULT_MAX_THREADS; + } + } + stack.setVar(THREAD_COUNT, new ThreadCount(maxThreadCount, selfClose, i)); + } + + private ThreadCount getThreadCount(VariableStack stack) + throws VariableNotFoundException { + return (ThreadCount) stack.getVar(THREAD_COUNT); + } + + @Override + public String getTextualName() { + return "foreach"; + } + + private static class ThreadCount implements FutureIterator { + private int maxThreadCount; + private int crt; + private boolean selfClose, closed; + private List listeners; + private KarajanIterator i; + + public ThreadCount(int maxThreadCount, boolean selfClose, KarajanIterator i) { + this.maxThreadCount = maxThreadCount; + this.selfClose = selfClose; + this.i = i; + crt = 0; + } + + public boolean iteratorHasValues() { + try { + return i.hasNext(); + } + catch (FutureFault e) { + return false; + } + } + + public boolean isSelfClose() { + return selfClose; + } + + public synchronized int available() { + return maxThreadCount - crt; + } + + public synchronized void add(int count) { + crt += count; + } + + public synchronized Object tryIncrement() { + // there is no way that both crt == 0 and i has no values outside this critical section + if (crt < maxThreadCount) { + Object o = i.next(); + crt++; + return o; + } + else { + throw new FutureIteratorIncomplete(this, this); + } + } + + public synchronized int decrement() { + crt--; + notifyListeners(); + return crt; + } + + private void notifyListeners() { + if (listeners != null) { + Iterator i = listeners.iterator(); + listeners = null; + while (i.hasNext()) { + ListenerStackPair etp = i.next(); + i.remove(); + etp.listener.futureModified(this, etp.stack); + } + } + } + + public boolean hasAvailable() { + return false; + } + + public int count() { + return 0; + } + + public synchronized int current() { + return crt; + } + + public Object peek() { + return null; + } + + public boolean hasNext() { + return false; + } + + public Object next() { + return null; + } + + public void remove() { + } + + public synchronized void addModificationAction(FutureListener target, + VariableStack stack) { + if (listeners == null) { + listeners = new ArrayList(); + } + listeners.add(new ListenerStackPair(target, stack)); + if (crt < maxThreadCount) { + notifyListeners(); + } + } + + public synchronized void close() { + this.closed = true; + } + + public void fail(FutureEvaluationException e) { + } + + public Object getValue() { + return null; + } + + public synchronized boolean isClosed() { + return closed; + } + } +} From hategan at ci.uchicago.edu Tue Oct 16 18:47:05 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Tue, 16 Oct 2012 18:47:05 -0500 (CDT) Subject: [Swift-commit] r5976 - trunk/libexec Message-ID: <20121016234705.4CDC99CCB6@svn.ci.uchicago.edu> Author: hategan Date: 2012-10-16 18:47:05 -0500 (Tue, 16 Oct 2012) New Revision: 5976 Modified: trunk/libexec/vdl-lib.xml Log: restored ThrottledParallelFor Modified: trunk/libexec/vdl-lib.xml =================================================================== --- trunk/libexec/vdl-lib.xml 2012-10-16 23:46:06 UTC (rev 5975) +++ trunk/libexec/vdl-lib.xml 2012-10-16 23:47:05 UTC (rev 5976) @@ -124,7 +124,7 @@ - + From wilde at ci.uchicago.edu Fri Oct 19 12:07:56 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Fri, 19 Oct 2012 12:07:56 -0500 (CDT) Subject: [Swift-commit] r5977 - SwiftApps Message-ID: <20121019170756.09FEE9CCA1@svn.ci.uchicago.edu> Author: wilde Date: 2012-10-19 12:07:55 -0500 (Fri, 19 Oct 2012) New Revision: 5977 Added: SwiftApps/EpiSnp/ Log: Add EpiSnp app From wilde at ci.uchicago.edu Fri Oct 19 12:34:48 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Fri, 19 Oct 2012 12:34:48 -0500 (CDT) Subject: [Swift-commit] r5979 - SwiftApps/EpiSnp Message-ID: <20121019173448.83EC89CCA1@svn.ci.uchicago.edu> Author: wilde Date: 2012-10-19 12:34:48 -0500 (Fri, 19 Oct 2012) New Revision: 5979 Added: SwiftApps/EpiSnp/README SwiftApps/EpiSnp/runepisnp.sh Modified: SwiftApps/EpiSnp/cf Log: Add README and initial run script. Added: SwiftApps/EpiSnp/README =================================================================== --- SwiftApps/EpiSnp/README (rev 0) +++ SwiftApps/EpiSnp/README 2012-10-19 17:34:48 UTC (rev 5979) @@ -0,0 +1,5 @@ +To run: + + ./runepisnp.sh N + +where N is the number of app calls to run. Modified: SwiftApps/EpiSnp/cf =================================================================== --- SwiftApps/EpiSnp/cf 2012-10-19 17:18:04 UTC (rev 5978) +++ SwiftApps/EpiSnp/cf 2012-10-19 17:34:48 UTC (rev 5979) @@ -358,3 +358,4 @@ execution.retries=0 lazy.errors=false status.mode=provider +use.wrapper.staging=false \ No newline at end of file Added: SwiftApps/EpiSnp/runepisnp.sh =================================================================== --- SwiftApps/EpiSnp/runepisnp.sh (rev 0) +++ SwiftApps/EpiSnp/runepisnp.sh 2012-10-19 17:34:48 UTC (rev 5979) @@ -0,0 +1,3 @@ +#! /bin/sh + +swift -config cf -tc.file tc -sites.file sites.xml episnp.swift -nRuns=${1:-1} Property changes on: SwiftApps/EpiSnp/runepisnp.sh ___________________________________________________________________ Added: svn:executable + * From wilde at ci.uchicago.edu Fri Oct 19 15:43:18 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Fri, 19 Oct 2012 15:43:18 -0500 (CDT) Subject: [Swift-commit] r5980 - SwiftApps/EpiSnp Message-ID: <20121019204318.D37DE9CCA1@svn.ci.uchicago.edu> Author: wilde Date: 2012-10-19 15:43:18 -0500 (Fri, 19 Oct 2012) New Revision: 5980 Added: SwiftApps/EpiSnp/cf.midway SwiftApps/EpiSnp/local.xml SwiftApps/EpiSnp/midway.xml Removed: SwiftApps/EpiSnp/cf SwiftApps/EpiSnp/sites.xml Modified: SwiftApps/EpiSnp/README SwiftApps/EpiSnp/episnp.swift SwiftApps/EpiSnp/runepisnp.sh SwiftApps/EpiSnp/tc Log: Add midway capability and extra app parameters Modified: SwiftApps/EpiSnp/README =================================================================== --- SwiftApps/EpiSnp/README 2012-10-19 17:34:48 UTC (rev 5979) +++ SwiftApps/EpiSnp/README 2012-10-19 20:43:18 UTC (rev 5980) @@ -1,5 +1,10 @@ To run: - ./runepisnp.sh N + ./runepisnp.sh sitename nRuns initSingle initPairs -where N is the number of app calls to run. +where: + + site is local or midway + nRuns is the number of EpiSnp application invocations to run (default 1) + initSingle is the base value of parameter output results for single SNP tests (default:1000) + initPairs is the base value of parameter output results for pairwise SNP tests (default:10000) Deleted: SwiftApps/EpiSnp/cf =================================================================== --- SwiftApps/EpiSnp/cf 2012-10-19 17:34:48 UTC (rev 5979) +++ SwiftApps/EpiSnp/cf 2012-10-19 20:43:18 UTC (rev 5980) @@ -1,361 +0,0 @@ -sites.file=${swift.home}/etc/sites.xml -tc.file=${swift.home}/etc/tc.data - -# -# The host name of the submit machine is used by GRAM as a callback -# address to report the status of submitted jobs. In general, Swift -# can automatically detect the host name of the local machine. -# However, if the machine host name is improperly configured or if -# it does not represent a valid DNS entry, certain services (such as -# GRAM) will not be able to send job status notifications back to -# the client. The value of this property can be an IP address. -# -# Format: -# hostname=string -# - - -#hostname=localhost - -# -# A TCP port range can be specified to restrict the ports on which GRAM -# callback services are started. This is likely needed if your submit -# host is behind a firewall, in which case the firewall should be -# configured to allow incoming connections on ports in the range. -# -# Format: -# tcp.port.range=start,end -# - -#tcp.port.range=50000,50100 - -# -# false - means an error will be immediately reported and cause the -# workflow to abort. At this time remote jobs that are already -# running will not be canceled -# true - means that Swift will try to do as much work as possible and -# report all errors encountered at the end. However, "errors" -# here only applies to job execution errors. Certain errors -# that are related to the Swift implementation (should such -# errors occur) will still be reported eagerly. -# -# Default: false -# -lazy.errors=false - -# -# What algorithm to use for caching of remote files. LRU (as in what -# files to purge) is the only implementation right now. One can set -# a target size (in bytes) for a host by using the swift:storagesize -# profile for a host in sites.xml -# -# Default: LRU -# -caching.algorithm=LRU - -# -# true - generate a provenance graph in .dot format (Swift will -# choose a random file name) -# false - do not generate a provenance graph -# - generate a provenange graph in the give file name -# -# Default: false -# -pgraph=false - - -# -# graph properties for the provenance graph (.dot specific) -# -# Default: splines="compound", rankdir="TB" -# -pgraph.graph.options=splines="compound", rankdir="TB" - - -# -# node properties for the provenance graph (.dot specific) -# -# Default: color="seagreen", style="filled" -# -pgraph.node.options=color="seagreen", style="filled" - -# -# true - clustering of small jobs is enabled. Clustering works in the -# following way: If a job is clusterable (meaning that it has the -# GLOBUS::maxwalltime profile specified in tc.data and its value -# is less than the value of the "clustering.min.time" property) it will -# be put in a clustering queue. The queue is processed at intervals -# specified by the "clustering.queue.delay" property. The processing -# of the clustering queue consists of selecting compatible jobs and -# grouping them in clusters whose max wall time does not exceed twice -# the value of the "clustering.min.time" property. Two or more jobs are -# considered compatible if they share the same site and do not have -# conflicting profiles (e.g. different values for the same environment -# variable). -# false - clustering of small jobs is disabled. -# -# Default: false -# -clustering.enabled=false - - -# -# - the intervals at which the clustering queue is processed -# -# Default: 4 -# -clustering.queue.delay=4 - -# -# - the threshold time for clustering -# -# Default: 60 -# -clustering.min.time=60 - -# -# Kickstart is a useful tool that can be used to gather various information -# about a remote process. Before it can be used it must be installed on the -# remote site and the corresponding entry be set in the sites file. -# This option allows controlling of how Swift uses Kickstart. The following -# values are possible: -# false - do not use Kickstart -# true - use Kickstart. If a job is scheduled on a site that does not have -# Kickstart installed, that job will fail. -# maybe - Use Kickstart if installed (i.e. the entry is present in the sites -# file) -# -# Default: maybe -# - -kickstart.enabled=maybe - -# -# Indicates when Kickstart records should be fetched from the remote site: -# true - always transfer Kickstart records if Kickstart was used (see -# kickstart.enabled) -# false - only transfer Kickstart records if the job fails -# -# Default: false -# - -kickstart.always.transfer=false - -# -# Indicates when wrapper logs should be fetched from the remote site: -# true - always transfer wrapper logs -# false - only transfer wrapper logs if the job fails -# -# Default: false -# - -wrapperlog.always.transfer=false - -########################################################################### -# Throttling options # -########################################################################### -# -# For the throttling parameters, valid values are either a positive integer -# or "off" (without the quotes). -# - -# -# Limits the number of concurrent submissions for a workflow instance. This -# throttle only limits the number of concurrent tasks (jobs) that are being -# sent to sites, not the total number of concurrent jobs that can be run. -# The submission stage in GRAM is one of the most CPU expensive stages (due -# mostly to the mutual authentication and delegation). Having too many -# concurrent submissions can overload either or both the submit host CPU -# and the remote host/head node causing degraded performance. -# -# Default: 4 -# - -throttle.submit=4 -#throttle.submit=off - -# -# Limits the number of concurrent submissions for any of the sites Swift will -# try to send jobs to. In other words it guarantees that no more than the -# value of this throttle jobs sent to any site will be concurrently in a state -# of being submitted. -# -# Default: 2 -# - -throttle.host.submit=2 -#throttle.host.submit=off - -# -# The Swift scheduler has the ability to limit the number of concurrent jobs -# allowed on a site based on the performance history of that site. Each site -# is assigned a score (initially 1), which can increase or decrease based -# on whether the site yields successful or faulty job runs. The score for a -# site can take values in the (0.1, 100) interval. The number of allowed jobs -# is calculated using the following formula: -# 2 + score*throttle.score.job.factor -# This means a site will always be allowed at least two concurrent jobs and -# at most 2 + 100*throttle.score.job.factor. With a default of 4 this means -# at least 2 jobs and at most 402. -# -# Default: 4 -# - -throttle.score.job.factor=0.2 -#throttle.score.job.factor=off - - -# -# Limits the total number of concurrent file transfers that can happen at any -# given time. File transfers consume bandwidth. Too many concurrent transfers -# can cause the network to be overloaded preventing various other signalling -# traffic from flowing properly. -# -# Default: 4 -# - -throttle.transfers=4 -#throttle.transfers=off - -# Limits the total number of concurrent file operations that can happen at any -# given time. File operations (like transfers) require an exclusive connection -# to a site. These connections can be expensive to establish. A large number -# of concurrent file operations may cause Swift to attempt to establish many -# such expensive connections to various sites. Limiting the number of concurrent -# file operations causes Swift to use a small number of cached connections and -# achieve better overall performance. -# -# Default: 8 -# - -throttle.file.operations=8 -#throttle.file.operations=off - -# Indicates whether the working directory on the remote site should be -# left intact even when the workflow completes successfully. This can be -# used to inspect the site working directory for debugging purposes. -# -# Default: false -# - -sitedir.keep=false - -# number of time a job will be retried if it fails (giving a maximum of -# 1 + execution.retries attempts at execution) -# - -execution.retries=2 - - -# Enables/disables replication. Replication is used to deal with jobs sitting -# in batch queues for abnormally large amounts of time. If replication is enabled -# and certain conditions are met, Swift creates and submits replicas of jobs, and -# allows multiple instances of a job to compete. -# - -replication.enabled=false - -# If replication is enabled, this value specifies the minimum time, in seconds, -# a job needs to be queued in a batch queue in order to be considered for -# replication -# - -replication.min.queue.time=60 - -# The maximum number of replicas that Swift should attempt. - -replication.limit=3 - -# -# WARNING: This option is deprecated. Please use the hostname option. -# -# The IP address of the submit machine is used by GRAM as a callback -# address to report the status of submitted jobs. In general, Swift -# can automatically detect the IP address of the local machine. -# However, if the machine has more than one network interface, Swift -# will pick the first one, which may not be the right choice. It is -# recommended that this property is set properly before attempting to -# run jobs through GRAM. -# -# Format: -# ip.address=x.y.z.w -# - -#ip.address=127.0.0.1 - - -# Controls how Swift will communicate the result code of running user programs -# from workers to the submit side. In files mode, a file -# indicating success or failure will be created on the site shared filesystem. -# In provider mode, the execution provider job status will -# be used. Notably, GRAM2 does not return job statuses correctly, and so -# provider mode will not work with GRAM2. With other -# providers, it can be used to reduce the amount of filesystem access compared -# to files mode. -# -# status.mode=files - -# Controls how swift will supply parameters to the remote wrapper script. -# 'args' mode will pass parameters on the command line -# 'files' mode will pass parameters through an additional input file -# -# valid values: args, files -# Default: files -# -# wrapper.parameter.mode=args - -# Determines if Swift remote wrappers will be executed by specifying an -# absolute path, or a path relative to the job initial working directory -# -# valid values: absolute, relative -# wrapper.invocation.mode=absolute - -# -# Limits the number of concurrent iterations that each foreach statement -# can have at one time. This conserves memory for swift programs that -# have large numbers of iterations (which would otherwise all be executed -# in parallel). -# -# Default: 1024 -# - -foreach.max.threads=16384 - -# controls whether the log file will contain provenance information -# enabling this will increase the size of log files, sometimes -# significantly. - -provenance.log=false - -# Controls whether file staging is done by swift or by the execution -# provider. If set to false, the standard swift staging mechanism is -# used. If set to true, swift does not stage files. Instead, the -# execution provider is instructed to stage files in and out. -# -# Provider staging is experimental. -# -# When enabled, and when coasters are used as an execution provider, -# a staging mechanism can be selected for each site -# using the swift:stagingMethod site profile in sites.xml. The -# following is a list of accepted mechanisms: -# -# * file: Staging is done from a filesystem accessible to the -# coaster service (typically running on the head node) -# * proxy: Staging is done from a filesystem accessible to the -# client machine that swift is running on, and is proxied -# through the coaster service -# * sfs: (short for "shared filesystem") Staging is done by -# copying files to and from a filesystem accessible -# by the compute node (such as an NFS or GPFS mount). - - -use.provider.staging=false - -# Changed settings: - -wrapperlog.always.transfer=true -sitedir.keep=true -execution.retries=0 -lazy.errors=false -status.mode=provider -use.wrapper.staging=false \ No newline at end of file Copied: SwiftApps/EpiSnp/cf.midway (from rev 5979, SwiftApps/EpiSnp/cf) =================================================================== --- SwiftApps/EpiSnp/cf.midway (rev 0) +++ SwiftApps/EpiSnp/cf.midway 2012-10-19 20:43:18 UTC (rev 5980) @@ -0,0 +1,9 @@ + +use.provider.staging=true +provider.staging.pin.swiftfiles=true +wrapperlog.always.transfer=true +sitedir.keep=true +execution.retries=0 +lazy.errors=false +status.mode=provider +use.wrapper.staging=false \ No newline at end of file Modified: SwiftApps/EpiSnp/episnp.swift =================================================================== --- SwiftApps/EpiSnp/episnp.swift 2012-10-19 17:34:48 UTC (rev 5979) +++ SwiftApps/EpiSnp/episnp.swift 2012-10-19 20:43:18 UTC (rev 5980) @@ -14,11 +14,10 @@ file figout[]; file sigout[]; -int nRuns = @toint(@arg("nRuns","1")); +int nRuns = @toInt(@arg("nRuns","1")); +int initSingles = @toInt(@arg("initSingles","1000")); +int initPairs = @toInt(@arg("initPairs","1000")); foreach incr, i in [0:nRuns-1] { - (figout[i], sigout[i], logout[i]) = episnp(epiwrapper, epiexec, epitrait, epichroms, 1000+i, 10000+i); + (figout[i], sigout[i], logout[i]) = episnp(epiwrapper, epiexec, epitrait, epichroms, initSingles+i, initPairs+i); } - - - Copied: SwiftApps/EpiSnp/local.xml (from rev 5979, SwiftApps/EpiSnp/sites.xml) =================================================================== --- SwiftApps/EpiSnp/local.xml (rev 0) +++ SwiftApps/EpiSnp/local.xml 2012-10-19 20:43:18 UTC (rev 5980) @@ -0,0 +1,9 @@ + + + + .07 + 10000 + + /tmp/swiftwork + + Added: SwiftApps/EpiSnp/midway.xml =================================================================== --- SwiftApps/EpiSnp/midway.xml (rev 0) +++ SwiftApps/EpiSnp/midway.xml 2012-10-19 20:43:18 UTC (rev 5980) @@ -0,0 +1,25 @@ + + + + + 16 + 3500 + 00:10:00 + 20 + 1 + 1 + + 100 + 100 + + + sandyb + + 3.20 + 10000 + + + /tmp/wilde/swiftwork + + + Modified: SwiftApps/EpiSnp/runepisnp.sh =================================================================== --- SwiftApps/EpiSnp/runepisnp.sh 2012-10-19 17:34:48 UTC (rev 5979) +++ SwiftApps/EpiSnp/runepisnp.sh 2012-10-19 20:43:18 UTC (rev 5980) @@ -1,3 +1,5 @@ #! /bin/sh -swift -config cf -tc.file tc -sites.file sites.xml episnp.swift -nRuns=${1:-1} +site=${1:-local} + +swift -config cf.$site -tc.file tc -sites.file $site.xml episnp.swift -nRuns=${2:-1} -initSingles=${3:-1000} -initPairs=${4:-10000} Deleted: SwiftApps/EpiSnp/sites.xml =================================================================== --- SwiftApps/EpiSnp/sites.xml 2012-10-19 17:34:48 UTC (rev 5979) +++ SwiftApps/EpiSnp/sites.xml 2012-10-19 20:43:18 UTC (rev 5980) @@ -1,7 +0,0 @@ - - - - - /home/wilde/swiftwork - - Modified: SwiftApps/EpiSnp/tc =================================================================== --- SwiftApps/EpiSnp/tc 2012-10-19 17:34:48 UTC (rev 5979) +++ SwiftApps/EpiSnp/tc 2012-10-19 20:43:18 UTC (rev 5980) @@ -1,46 +1,3 @@ -#This is the transformation catalog. -# -#It comes pre-configured with a number of simple transformations with -#paths that are likely to work on a linux box. However, on some systems, -#the paths to these executables will be different (for example, sometimes -#some of these programs are found in /usr/bin rather than in /bin) -# -#NOTE WELL: fields in this file must be separated by tabs, not spaces; and -#there must be no trailing whitespace at the end of each line. -# -# sitename transformation path INSTALLED platform profiles -localhost sh /bin/sh INSTALLED INTEL32::LINUX null -localhost echo /bin/echo INSTALLED INTEL32::LINUX null -localhost cat /bin/cat INSTALLED INTEL32::LINUX null -localhost ls /bin/ls INSTALLED INTEL32::LINUX null -localhost grep /bin/grep INSTALLED INTEL32::LINUX null -localhost sort /bin/sort INSTALLED INTEL32::LINUX null -localhost paste /bin/paste INSTALLED INTEL32::LINUX null -localhost pwd /bin/pwd INSTALLED INTEL32::LINUX null -# -ranger cat /bin/cat INSTALLED INTEL32::LINUX null -abe cat /bin/cat INSTALLED INTEL32::LINUX null -qb cat /bin/cat INSTALLED INTEL32::LINUX null -firefly cat /bin/cat INSTALLED INTEL32::LINUX null -teraport cat /bin/cat INSTALLED INTEL32::LINUX null -pbs cat /bin/cat null null null -ssh cat /bin/cat null null null -rssh scat /home/wilde/swift/lab/scat null null null -sico cat /bin/cat null null null -sico scat /home/wilde/swift/lab/scat null null null - -######### - -crush cat /bin/cat null null null -thwomp cat /bin/cat null null null -stomp cat /bin/cat null null null -crank cat /bin/cat null null null -steamroller cat /bin/cat null null null -grind cat /bin/cat null null null -churn cat /bin/cat null null null -trounce cat /bin/cat null null null -thrash cat /bin/cat null null null -vanquish cat /bin/cat null null null -octagon cat /bin/cat null null null -octopus cat /bin/cat null null null -triumph cat /bin/cat null null null +local sh /bin/sh null null null +midway sh /bin/sh null null null +pads sh /bin/sh null null null From wilde at ci.uchicago.edu Fri Oct 19 15:50:07 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Fri, 19 Oct 2012 15:50:07 -0500 (CDT) Subject: [Swift-commit] r5981 - SwiftApps/EpiSnp Message-ID: <20121019205007.31E509CCA1@svn.ci.uchicago.edu> Author: wilde Date: 2012-10-19 15:50:07 -0500 (Fri, 19 Oct 2012) New Revision: 5981 Modified: SwiftApps/EpiSnp/README Log: Update README Modified: SwiftApps/EpiSnp/README =================================================================== --- SwiftApps/EpiSnp/README 2012-10-19 20:43:18 UTC (rev 5980) +++ SwiftApps/EpiSnp/README 2012-10-19 20:50:07 UTC (rev 5981) @@ -8,3 +8,20 @@ nRuns is the number of EpiSnp application invocations to run (default 1) initSingle is the base value of parameter output results for single SNP tests (default:1000) initPairs is the base value of parameter output results for pairwise SNP tests (default:10000) + +Examples: + + module load java + + # Get a more recent version of Swift: + PATH=/home/wilde/swift/src/trunk/cog/modules/swift/dist/swift-svn/bin:$PATH + + ./runepisnp.sh # runs 1 invocation on local host + + ./runepisnp.sh local 10 # runs 10 invocations on local host + + ./runepisnp.sh midway 100 # runs 100 invocations on midway cluster + + ./runepisnp.sh midway 100 10 20 # runs 100 invocations on midway cluster with specific parameters + + From wilde at ci.uchicago.edu Fri Oct 19 16:10:46 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Fri, 19 Oct 2012 16:10:46 -0500 (CDT) Subject: [Swift-commit] r5982 - SwiftApps/EpiSnp Message-ID: <20121019211046.A3B2F9D02D@svn.ci.uchicago.edu> Author: wilde Date: 2012-10-19 16:10:46 -0500 (Fri, 19 Oct 2012) New Revision: 5982 Modified: SwiftApps/EpiSnp/cf.midway Log: Adjust midway parameters to reduce IO Modified: SwiftApps/EpiSnp/cf.midway =================================================================== --- SwiftApps/EpiSnp/cf.midway 2012-10-19 20:50:07 UTC (rev 5981) +++ SwiftApps/EpiSnp/cf.midway 2012-10-19 21:10:46 UTC (rev 5982) @@ -1,8 +1,8 @@ use.provider.staging=true provider.staging.pin.swiftfiles=true -wrapperlog.always.transfer=true -sitedir.keep=true +wrapperlog.always.transfer=false +sitedir.keep=false execution.retries=0 lazy.errors=false status.mode=provider From swift at ci.uchicago.edu Sat Oct 20 13:00:04 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Sat, 20 Oct 2012 13:00:04 -0500 (CDT) Subject: [Swift-commit] cog r3487 Message-ID: <20121020180004.CDB2E8D0008D@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3487 | hategan | 2012-10-20 12:58:18 -0500 (Sat, 20 Oct 2012) | 1 line only update deadline if changed cpu has a chance of affecting it ------------------------------------------------------------------------ Index: modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Block.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Block.java (revision 3486) +++ modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Block.java (working copy) @@ -193,8 +193,8 @@ } last = scpus.last(); } - if (last != null) { - setDeadline(Time.min(last.getTimeLast().add(bqp.getSettings().getReserve()), + if (last == cpu) { + setDeadline(Time.min(cpu.getTimeLast().add(bqp.getSettings().getReserve()), getEndTime())); } } @@ -512,12 +512,14 @@ } public void suspend() { - suspended = true; + synchronized(this) { + suspended = true; + } // ensure we still shut down if no jobs are running shutdownIfEmpty(null); } - public boolean isSuspended() { + public synchronized boolean isSuspended() { return suspended; } From swift at ci.uchicago.edu Sat Oct 20 13:05:04 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Sat, 20 Oct 2012 13:05:04 -0500 (CDT) Subject: [Swift-commit] cog r3488 Message-ID: <20121020180505.040638D0008D@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3488 | hategan | 2012-10-20 13:00:46 -0500 (Sat, 20 Oct 2012) | 1 line sync when instantiating pullthread; more careful syncing on pull ------------------------------------------------------------------------ Index: modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Cpu.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Cpu.java (revision 3487) +++ modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Cpu.java (working copy) @@ -123,9 +123,11 @@ } static PullThread getPullThread(Block block) { - if (pullThread == null) { - pullThread = new PullThread(block.getAllocationProcessor()); - pullThread.start(); + synchronized(Cpu.class) { + if (pullThread == null) { + pullThread = new PullThread(block.getAllocationProcessor()); + pullThread.start(); + } } return pullThread; } @@ -139,6 +141,7 @@ getPullThread(block).enqueue(cpu); } + private synchronized void sleep(Cpu cpu) { getPullThread(cpu.node.getBlock()).sleep(cpu); } @@ -151,7 +154,7 @@ The Cpu requests work from the BlockQueueProcessor The Cpu is awake when calling this (not in PullThread.sleeping) */ - public synchronized void pull() { + public void pull() { boolean success = true; try { if (checkSuspended(block)) { @@ -160,34 +163,36 @@ if (logger.isTraceEnabled()) { logger.trace(block.getId() + ":" + getId() + " pull"); } - if (shutdown) { - return; - } - if (!started()) { - sleep(); - } - else if (running == null) { - lastseq = bqp.getQueueSeq(); - TimeInterval time = endtime.subtract(Time.now()); - int cpus = pullThread.sleepers() + 1; - if (logger.isDebugEnabled()) - logger.debug("requesting work: " + - "block=" + block.getId() + - " id=" + getId() + - " Cpus sleeping: " + cpus); - running = bqp.request(time, cpus); - if (running != null) { - block.jobPulled(); - success = launch(running); + synchronized (this) { + if (shutdown) { + return; } - else { + if (!started()) { sleep(); } + else if (running == null) { + lastseq = bqp.getQueueSeq(); + TimeInterval time = endtime.subtract(Time.now()); + int cpus = pullThread.sleepers() + 1; + if (logger.isDebugEnabled()) + logger.debug("requesting work: " + + "block=" + block.getId() + + " id=" + getId() + + " Cpus sleeping: " + cpus); + running = bqp.request(time, cpus); + if (running != null) { + block.jobPulled(); + success = launch(running); + } + else { + sleep(); + } + } + else { + CoasterService.error(40, "pull called while another job was running", + new Throwable()); + } } - else { - CoasterService.error(40, "pull called while another job was running", - new Throwable()); - } } catch (Exception e) { taskFailed("Failed pull", e); From davidk at ci.uchicago.edu Mon Oct 22 09:52:14 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Mon, 22 Oct 2012 09:52:14 -0500 (CDT) Subject: [Swift-commit] r5983 - SwiftApps/CMTS Message-ID: <20121022145214.48C679CC9B@svn.ci.uchicago.edu> Author: davidk Date: 2012-10-22 09:52:14 -0500 (Mon, 22 Oct 2012) New Revision: 5983 Added: SwiftApps/CMTS/stringmethod/ Log: String method scripts From davidk at ci.uchicago.edu Mon Oct 22 23:06:21 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Mon, 22 Oct 2012 23:06:21 -0500 (CDT) Subject: [Swift-commit] r5985 - branches Message-ID: <20121023040621.859A39CCA1@svn.ci.uchicago.edu> Author: davidk Date: 2012-10-22 23:06:21 -0500 (Mon, 22 Oct 2012) New Revision: 5985 Added: branches/release-0.94/ Log: Creating a branch for 0.94 release From davidk at ci.uchicago.edu Mon Oct 22 23:06:55 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Mon, 22 Oct 2012 23:06:55 -0500 (CDT) Subject: [Swift-commit] r5986 - branches/release-0.94 Message-ID: <20121023040655.BF2A39CCA1@svn.ci.uchicago.edu> Author: davidk Date: 2012-10-22 23:06:55 -0500 (Mon, 22 Oct 2012) New Revision: 5986 Added: branches/release-0.94/trunk/ Log: Creating a 0.94 branch from trunk From davidk at ci.uchicago.edu Mon Oct 22 23:10:31 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Mon, 22 Oct 2012 23:10:31 -0500 (CDT) Subject: [Swift-commit] r5987 - branches Message-ID: <20121023041031.5B73F9CCA1@svn.ci.uchicago.edu> Author: davidk Date: 2012-10-22 23:10:31 -0500 (Mon, 22 Oct 2012) New Revision: 5987 Removed: branches/release-0.94/ Log: From davidk at ci.uchicago.edu Mon Oct 22 23:12:35 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Mon, 22 Oct 2012 23:12:35 -0500 (CDT) Subject: [Swift-commit] r5988 - branches Message-ID: <20121023041235.AC3179CCA1@svn.ci.uchicago.edu> Author: davidk Date: 2012-10-22 23:12:35 -0500 (Mon, 22 Oct 2012) New Revision: 5988 Added: branches/release-0.94/ Log: From hategan at ci.uchicago.edu Wed Oct 24 15:39:03 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Wed, 24 Oct 2012 15:39:03 -0500 (CDT) Subject: [Swift-commit] r5989 - trunk/src/org/griphyn/vdl/engine Message-ID: <20121024203903.7EA9F9CCA1@svn.ci.uchicago.edu> Author: hategan Date: 2012-10-24 15:39:03 -0500 (Wed, 24 Oct 2012) New Revision: 5989 Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java Log: fixed NPE in compiler when proc has no arguments (or no returns) (bug857) Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/Karajan.java 2012-10-23 04:12:35 UTC (rev 5988) +++ trunk/src/org/griphyn/vdl/engine/Karajan.java 2012-10-24 20:39:03 UTC (rev 5989) @@ -884,7 +884,10 @@ @SuppressWarnings("unchecked") private boolean allVariables(Object st) { List l; - if (st instanceof List) { + if (st == null) { + return true; + } + else if (st instanceof List) { l = (List) st; } else { From davidk at ci.uchicago.edu Wed Oct 24 16:20:03 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Wed, 24 Oct 2012 16:20:03 -0500 (CDT) Subject: [Swift-commit] r5990 - branches/release-0.94/src/org/griphyn/vdl/engine Message-ID: <20121024212003.3131A9CCA1@svn.ci.uchicago.edu> Author: davidk Date: 2012-10-24 16:20:03 -0500 (Wed, 24 Oct 2012) New Revision: 5990 Modified: branches/release-0.94/src/org/griphyn/vdl/engine/Karajan.java Log: Applying Mihael's fix for bug 857 to 0.94 branch Modified: branches/release-0.94/src/org/griphyn/vdl/engine/Karajan.java =================================================================== --- branches/release-0.94/src/org/griphyn/vdl/engine/Karajan.java 2012-10-24 20:39:03 UTC (rev 5989) +++ branches/release-0.94/src/org/griphyn/vdl/engine/Karajan.java 2012-10-24 21:20:03 UTC (rev 5990) @@ -884,7 +884,10 @@ @SuppressWarnings("unchecked") private boolean allVariables(Object st) { List l; - if (st instanceof List) { + if (st == null) { + return true; + } + else if(st instanceof List) { l = (List) st; } else { From davidk at ci.uchicago.edu Thu Oct 25 15:04:32 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Thu, 25 Oct 2012 15:04:32 -0500 (CDT) Subject: [Swift-commit] r5991 - in trunk/tests/language-behaviour: arrays cleanup Message-ID: <20121025200432.686579CC92@svn.ci.uchicago.edu> Author: davidk Date: 2012-10-25 15:04:32 -0500 (Thu, 25 Oct 2012) New Revision: 5991 Modified: trunk/tests/language-behaviour/arrays/020-array.clean.sh trunk/tests/language-behaviour/arrays/021-array-in-proc-param.clean.sh trunk/tests/language-behaviour/arrays/022-array-with-exprs.clean.sh trunk/tests/language-behaviour/arrays/025-array-passing.clean.sh trunk/tests/language-behaviour/arrays/027-array-assignment.clean.sh trunk/tests/language-behaviour/cleanup/001-proc-scope-cleanup.check.sh trunk/tests/language-behaviour/cleanup/011-proc-scope-alias-cleanup.check.sh trunk/tests/language-behaviour/cleanup/021-iter-scope-cleanup.check.sh trunk/tests/language-behaviour/cleanup/031-foreach-scope-cleanup.check.sh trunk/tests/language-behaviour/cleanup/041-if-scope-cleanup.check.sh Log: Fixes various tests that were failing due to changes in the test suite Modified: trunk/tests/language-behaviour/arrays/020-array.clean.sh =================================================================== --- trunk/tests/language-behaviour/arrays/020-array.clean.sh 2012-10-24 21:20:03 UTC (rev 5990) +++ trunk/tests/language-behaviour/arrays/020-array.clean.sh 2012-10-25 20:04:32 UTC (rev 5991) @@ -1,4 +1,4 @@ #!/bin/bash set -x -rm -r 020-array-* 020-array.xml 020-array.kml 020-array.out || exit 1 +rm -r 020-array-* 020-array.swiftx 020-array.kml 020-array.out || exit 1 exit 0 Modified: trunk/tests/language-behaviour/arrays/021-array-in-proc-param.clean.sh =================================================================== --- trunk/tests/language-behaviour/arrays/021-array-in-proc-param.clean.sh 2012-10-24 21:20:03 UTC (rev 5990) +++ trunk/tests/language-behaviour/arrays/021-array-in-proc-param.clean.sh 2012-10-25 20:04:32 UTC (rev 5991) @@ -1,4 +1,4 @@ #!/bin/bash set -x -rm -r 021-array-in-proc-param.out 021-array-in-proc-param-* 021-array-in-proc-param.kml 021-array-in-proc-param.xml || exit 1 +rm -r 021-array-in-proc-param.out 021-array-in-proc-param-* 021-array-in-proc-param.kml 021-array-in-proc-param.swiftx || exit 1 exit 0 Modified: trunk/tests/language-behaviour/arrays/022-array-with-exprs.clean.sh =================================================================== --- trunk/tests/language-behaviour/arrays/022-array-with-exprs.clean.sh 2012-10-24 21:20:03 UTC (rev 5990) +++ trunk/tests/language-behaviour/arrays/022-array-with-exprs.clean.sh 2012-10-25 20:04:32 UTC (rev 5991) @@ -1,4 +1,4 @@ #!/bin/bash set -x -rm -r 022-array-with-exprs-* 022-array-with-exprs.kml 022-array-with-exprs.xml 022-array-with-exprs.out || exit 1 +rm -r 022-array-with-exprs-* 022-array-with-exprs.kml 022-array-with-exprs.swiftx 022-array-with-exprs.out || exit 1 exit 0 Modified: trunk/tests/language-behaviour/arrays/025-array-passing.clean.sh =================================================================== --- trunk/tests/language-behaviour/arrays/025-array-passing.clean.sh 2012-10-24 21:20:03 UTC (rev 5990) +++ trunk/tests/language-behaviour/arrays/025-array-passing.clean.sh 2012-10-25 20:04:32 UTC (rev 5991) @@ -1,4 +1,4 @@ #!/bin/bash set -x -rm -r 025-array-passing-* 025-array-passing.kml 025-array-passing.xml 025-array-passing.out || exit 1 +rm -r 025-array-passing-* 025-array-passing.kml 025-array-passing.swiftx 025-array-passing.out || exit 1 exit 0 Modified: trunk/tests/language-behaviour/arrays/027-array-assignment.clean.sh =================================================================== --- trunk/tests/language-behaviour/arrays/027-array-assignment.clean.sh 2012-10-24 21:20:03 UTC (rev 5990) +++ trunk/tests/language-behaviour/arrays/027-array-assignment.clean.sh 2012-10-25 20:04:32 UTC (rev 5991) @@ -1,4 +1,4 @@ #!/bin/bash set -x -rm -r 027-array-assignment.out 027-array-assignment-* 027-array-assignment.kml 027-array-assignment.xml || exit 1 +rm -r 027-array-assignment.out 027-array-assignment-* 027-array-assignment.kml 027-array-assignment.swiftx || exit 1 exit 0 Modified: trunk/tests/language-behaviour/cleanup/001-proc-scope-cleanup.check.sh =================================================================== --- trunk/tests/language-behaviour/cleanup/001-proc-scope-cleanup.check.sh 2012-10-24 21:20:03 UTC (rev 5990) +++ trunk/tests/language-behaviour/cleanup/001-proc-scope-cleanup.check.sh 2012-10-25 20:04:32 UTC (rev 5991) @@ -2,8 +2,7 @@ PREFIX="001-proc-scope-cleanup" - -if [ -n "$(find . -name "$PREFIX.*.tmp" -print -quit)" ]; then +if [ -n "$(find . -maxdepth 1 -name "$PREFIX.*.tmp" -print -quit)" ]; then echo "A temporary file wasn't removed" exit 1 fi @@ -13,12 +12,12 @@ exit 1 fi -if ! grep "Cleaning file.*$PREFIX.t1.tmp.*" $OUTPUT; then +if ! grep "Cleaning file [file://*$PREFIX.t1.tmp*]" $PREFIX.stdout; then echo "Test mapper did not claim to clean t1" exit 1 fi -if ! grep "Not cleaning file.*$PREFIX.t2.out.*" $OUTPUT; then +if ! grep "Not cleaning [file://*$PREFIX.t2.out*]" $PREFIX.stdout; then echo "Test mapper did not identify t2 as persistent" exit 1 fi @@ -29,4 +28,4 @@ fi echo "Everything's ok" -exit 0 \ No newline at end of file +exit 0 Modified: trunk/tests/language-behaviour/cleanup/011-proc-scope-alias-cleanup.check.sh =================================================================== --- trunk/tests/language-behaviour/cleanup/011-proc-scope-alias-cleanup.check.sh 2012-10-24 21:20:03 UTC (rev 5990) +++ trunk/tests/language-behaviour/cleanup/011-proc-scope-alias-cleanup.check.sh 2012-10-25 20:04:32 UTC (rev 5991) @@ -2,7 +2,7 @@ PREFIX="011-proc-scope-alias-cleanup" -if [ -n "$(find . -name "$PREFIX.*.tmp" -print -quit)" ]; then +if [ -n "$(find . -maxdepth 1 -name "$PREFIX.*.tmp" -print -quit)" ]; then echo "A temporary file wasn't removed" exit 1 fi @@ -12,17 +12,17 @@ exit 1 fi -if ! grep "Remapping" $OUTPUT; then +if ! grep "Remapping" $PREFIX.stdout; then echo "Test mapper did not claim to have remapped anything" exit 1 fi -if ! grep "Cleaning file.*$PREFIX.1.tmp.*" $OUTPUT; then +if ! grep "Cleaning file [file://*$PREFIX.1.tmp*]" $PREFIX.stdout; then echo "Test mapper didn't clean what it should have" exit 1 fi -if ! grep "Not cleaning file.*$PREFIX.2.out.*" $OUTPUT; then +if ! grep "Not cleaning [file://*$PREFIX.2.out*]" $PREFIX.stdout; then echo "Test mapper did not identify 2 as persistent" exit 1 fi @@ -33,4 +33,4 @@ fi echo "Everything's ok" -exit 0 \ No newline at end of file +exit 0 Modified: trunk/tests/language-behaviour/cleanup/021-iter-scope-cleanup.check.sh =================================================================== --- trunk/tests/language-behaviour/cleanup/021-iter-scope-cleanup.check.sh 2012-10-24 21:20:03 UTC (rev 5990) +++ trunk/tests/language-behaviour/cleanup/021-iter-scope-cleanup.check.sh 2012-10-25 20:04:32 UTC (rev 5991) @@ -7,10 +7,10 @@ exit 1 fi -if [ `grep "Cleaning" $OUTPUT | wc -l` != "3" ]; then +if [ `grep "Cleaning" $PREFIX.stdout | wc -l` != "3" ]; then echo "Wrong number of files cleaned" exit 1 fi echo "Everything's ok" -exit 0 \ No newline at end of file +exit 0 Modified: trunk/tests/language-behaviour/cleanup/031-foreach-scope-cleanup.check.sh =================================================================== --- trunk/tests/language-behaviour/cleanup/031-foreach-scope-cleanup.check.sh 2012-10-24 21:20:03 UTC (rev 5990) +++ trunk/tests/language-behaviour/cleanup/031-foreach-scope-cleanup.check.sh 2012-10-25 20:04:32 UTC (rev 5991) @@ -7,10 +7,10 @@ exit 1 fi -if [ `grep "Cleaning" $OUTPUT | wc -l` != "4" ]; then +if [ `grep "Cleaning" $PREFIX.stdout | wc -l` != "4" ]; then echo "Wrong number of files cleaned" exit 1 fi echo "Everything's ok" -exit 0 \ No newline at end of file +exit 0 Modified: trunk/tests/language-behaviour/cleanup/041-if-scope-cleanup.check.sh =================================================================== --- trunk/tests/language-behaviour/cleanup/041-if-scope-cleanup.check.sh 2012-10-24 21:20:03 UTC (rev 5990) +++ trunk/tests/language-behaviour/cleanup/041-if-scope-cleanup.check.sh 2012-10-25 20:04:32 UTC (rev 5991) @@ -12,10 +12,10 @@ exit 1 fi -if [ `grep "Cleaning" $OUTPUT | wc -l` != "2" ]; then +if [ `grep "Cleaning" $PREFIX.stdout | wc -l` != "2" ]; then echo "Wrong number of files cleaned" exit 1 fi echo "Everything's ok" -exit 0 \ No newline at end of file +exit 0 From davidk at ci.uchicago.edu Thu Oct 25 16:18:39 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Thu, 25 Oct 2012 16:18:39 -0500 (CDT) Subject: [Swift-commit] r5992 - in branches/release-0.94/tests/language-behaviour: arrays cleanup Message-ID: <20121025211839.6733A9CCA1@svn.ci.uchicago.edu> Author: davidk Date: 2012-10-25 16:18:39 -0500 (Thu, 25 Oct 2012) New Revision: 5992 Modified: branches/release-0.94/tests/language-behaviour/arrays/020-array.clean.sh branches/release-0.94/tests/language-behaviour/arrays/021-array-in-proc-param.clean.sh branches/release-0.94/tests/language-behaviour/arrays/022-array-with-exprs.clean.sh branches/release-0.94/tests/language-behaviour/arrays/025-array-passing.clean.sh branches/release-0.94/tests/language-behaviour/arrays/027-array-assignment.clean.sh branches/release-0.94/tests/language-behaviour/cleanup/001-proc-scope-cleanup.check.sh branches/release-0.94/tests/language-behaviour/cleanup/011-proc-scope-alias-cleanup.check.sh branches/release-0.94/tests/language-behaviour/cleanup/021-iter-scope-cleanup.check.sh branches/release-0.94/tests/language-behaviour/cleanup/031-foreach-scope-cleanup.check.sh branches/release-0.94/tests/language-behaviour/cleanup/041-if-scope-cleanup.check.sh Log: Fixes various tests that were failing Modified: branches/release-0.94/tests/language-behaviour/arrays/020-array.clean.sh =================================================================== --- branches/release-0.94/tests/language-behaviour/arrays/020-array.clean.sh 2012-10-25 20:04:32 UTC (rev 5991) +++ branches/release-0.94/tests/language-behaviour/arrays/020-array.clean.sh 2012-10-25 21:18:39 UTC (rev 5992) @@ -1,4 +1,4 @@ #!/bin/bash set -x -rm -r 020-array-* 020-array.xml 020-array.kml 020-array.out || exit 1 +rm -r 020-array-* 020-array.swiftx 020-array.kml 020-array.out || exit 1 exit 0 Modified: branches/release-0.94/tests/language-behaviour/arrays/021-array-in-proc-param.clean.sh =================================================================== --- branches/release-0.94/tests/language-behaviour/arrays/021-array-in-proc-param.clean.sh 2012-10-25 20:04:32 UTC (rev 5991) +++ branches/release-0.94/tests/language-behaviour/arrays/021-array-in-proc-param.clean.sh 2012-10-25 21:18:39 UTC (rev 5992) @@ -1,4 +1,4 @@ #!/bin/bash set -x -rm -r 021-array-in-proc-param.out 021-array-in-proc-param-* 021-array-in-proc-param.kml 021-array-in-proc-param.xml || exit 1 +rm -r 021-array-in-proc-param.out 021-array-in-proc-param-* 021-array-in-proc-param.kml 021-array-in-proc-param.swiftx || exit 1 exit 0 Modified: branches/release-0.94/tests/language-behaviour/arrays/022-array-with-exprs.clean.sh =================================================================== --- branches/release-0.94/tests/language-behaviour/arrays/022-array-with-exprs.clean.sh 2012-10-25 20:04:32 UTC (rev 5991) +++ branches/release-0.94/tests/language-behaviour/arrays/022-array-with-exprs.clean.sh 2012-10-25 21:18:39 UTC (rev 5992) @@ -1,4 +1,4 @@ #!/bin/bash set -x -rm -r 022-array-with-exprs-* 022-array-with-exprs.kml 022-array-with-exprs.xml 022-array-with-exprs.out || exit 1 +rm -r 022-array-with-exprs-* 022-array-with-exprs.kml 022-array-with-exprs.swiftx 022-array-with-exprs.out || exit 1 exit 0 Modified: branches/release-0.94/tests/language-behaviour/arrays/025-array-passing.clean.sh =================================================================== --- branches/release-0.94/tests/language-behaviour/arrays/025-array-passing.clean.sh 2012-10-25 20:04:32 UTC (rev 5991) +++ branches/release-0.94/tests/language-behaviour/arrays/025-array-passing.clean.sh 2012-10-25 21:18:39 UTC (rev 5992) @@ -1,4 +1,4 @@ #!/bin/bash set -x -rm -r 025-array-passing-* 025-array-passing.kml 025-array-passing.xml 025-array-passing.out || exit 1 +rm -r 025-array-passing-* 025-array-passing.kml 025-array-passing.swiftx 025-array-passing.out || exit 1 exit 0 Modified: branches/release-0.94/tests/language-behaviour/arrays/027-array-assignment.clean.sh =================================================================== --- branches/release-0.94/tests/language-behaviour/arrays/027-array-assignment.clean.sh 2012-10-25 20:04:32 UTC (rev 5991) +++ branches/release-0.94/tests/language-behaviour/arrays/027-array-assignment.clean.sh 2012-10-25 21:18:39 UTC (rev 5992) @@ -1,4 +1,4 @@ #!/bin/bash set -x -rm -r 027-array-assignment.out 027-array-assignment-* 027-array-assignment.kml 027-array-assignment.xml || exit 1 +rm -r 027-array-assignment.out 027-array-assignment-* 027-array-assignment.kml 027-array-assignment.swiftx || exit 1 exit 0 Modified: branches/release-0.94/tests/language-behaviour/cleanup/001-proc-scope-cleanup.check.sh =================================================================== --- branches/release-0.94/tests/language-behaviour/cleanup/001-proc-scope-cleanup.check.sh 2012-10-25 20:04:32 UTC (rev 5991) +++ branches/release-0.94/tests/language-behaviour/cleanup/001-proc-scope-cleanup.check.sh 2012-10-25 21:18:39 UTC (rev 5992) @@ -2,8 +2,7 @@ PREFIX="001-proc-scope-cleanup" - -if [ -n "$(find . -name "$PREFIX.*.tmp" -print -quit)" ]; then +if [ -n "$(find . -maxdepth 1 -name "$PREFIX.*.tmp" -print -quit)" ]; then echo "A temporary file wasn't removed" exit 1 fi @@ -13,12 +12,12 @@ exit 1 fi -if ! grep "Cleaning file.*$PREFIX.t1.tmp.*" $OUTPUT; then +if ! grep "Cleaning file [file://*$PREFIX.t1.tmp*]" $PREFIX.stdout; then echo "Test mapper did not claim to clean t1" exit 1 fi -if ! grep "Not cleaning file.*$PREFIX.t2.out.*" $OUTPUT; then +if ! grep "Not cleaning [file://*$PREFIX.t2.out*]" $PREFIX.stdout; then echo "Test mapper did not identify t2 as persistent" exit 1 fi @@ -29,4 +28,4 @@ fi echo "Everything's ok" -exit 0 \ No newline at end of file +exit 0 Modified: branches/release-0.94/tests/language-behaviour/cleanup/011-proc-scope-alias-cleanup.check.sh =================================================================== --- branches/release-0.94/tests/language-behaviour/cleanup/011-proc-scope-alias-cleanup.check.sh 2012-10-25 20:04:32 UTC (rev 5991) +++ branches/release-0.94/tests/language-behaviour/cleanup/011-proc-scope-alias-cleanup.check.sh 2012-10-25 21:18:39 UTC (rev 5992) @@ -2,7 +2,7 @@ PREFIX="011-proc-scope-alias-cleanup" -if [ -n "$(find . -name "$PREFIX.*.tmp" -print -quit)" ]; then +if [ -n "$(find . -maxdepth 1 -name "$PREFIX.*.tmp" -print -quit)" ]; then echo "A temporary file wasn't removed" exit 1 fi @@ -12,17 +12,17 @@ exit 1 fi -if ! grep "Remapping" $OUTPUT; then +if ! grep "Remapping" $PREFIX.stdout; then echo "Test mapper did not claim to have remapped anything" exit 1 fi -if ! grep "Cleaning file.*$PREFIX.1.tmp.*" $OUTPUT; then +if ! grep "Cleaning file [file://*$PREFIX.1.tmp*]" $PREFIX.stdout; then echo "Test mapper didn't clean what it should have" exit 1 fi -if ! grep "Not cleaning file.*$PREFIX.2.out.*" $OUTPUT; then +if ! grep "Not cleaning [file://*$PREFIX.2.out*]" $PREFIX.stdout; then echo "Test mapper did not identify 2 as persistent" exit 1 fi @@ -33,4 +33,4 @@ fi echo "Everything's ok" -exit 0 \ No newline at end of file +exit 0 Modified: branches/release-0.94/tests/language-behaviour/cleanup/021-iter-scope-cleanup.check.sh =================================================================== --- branches/release-0.94/tests/language-behaviour/cleanup/021-iter-scope-cleanup.check.sh 2012-10-25 20:04:32 UTC (rev 5991) +++ branches/release-0.94/tests/language-behaviour/cleanup/021-iter-scope-cleanup.check.sh 2012-10-25 21:18:39 UTC (rev 5992) @@ -7,10 +7,10 @@ exit 1 fi -if [ `grep "Cleaning" $OUTPUT | wc -l` != "3" ]; then +if [ `grep "Cleaning" $PREFIX.stdout | wc -l` != "3" ]; then echo "Wrong number of files cleaned" exit 1 fi echo "Everything's ok" -exit 0 \ No newline at end of file +exit 0 Modified: branches/release-0.94/tests/language-behaviour/cleanup/031-foreach-scope-cleanup.check.sh =================================================================== --- branches/release-0.94/tests/language-behaviour/cleanup/031-foreach-scope-cleanup.check.sh 2012-10-25 20:04:32 UTC (rev 5991) +++ branches/release-0.94/tests/language-behaviour/cleanup/031-foreach-scope-cleanup.check.sh 2012-10-25 21:18:39 UTC (rev 5992) @@ -7,10 +7,10 @@ exit 1 fi -if [ `grep "Cleaning" $OUTPUT | wc -l` != "4" ]; then +if [ `grep "Cleaning" $PREFIX.stdout | wc -l` != "4" ]; then echo "Wrong number of files cleaned" exit 1 fi echo "Everything's ok" -exit 0 \ No newline at end of file +exit 0 Modified: branches/release-0.94/tests/language-behaviour/cleanup/041-if-scope-cleanup.check.sh =================================================================== --- branches/release-0.94/tests/language-behaviour/cleanup/041-if-scope-cleanup.check.sh 2012-10-25 20:04:32 UTC (rev 5991) +++ branches/release-0.94/tests/language-behaviour/cleanup/041-if-scope-cleanup.check.sh 2012-10-25 21:18:39 UTC (rev 5992) @@ -12,10 +12,10 @@ exit 1 fi -if [ `grep "Cleaning" $OUTPUT | wc -l` != "2" ]; then +if [ `grep "Cleaning" $PREFIX.stdout | wc -l` != "2" ]; then echo "Wrong number of files cleaned" exit 1 fi echo "Everything's ok" -exit 0 \ No newline at end of file +exit 0 From lgadelha at ci.uchicago.edu Fri Oct 26 12:10:18 2012 From: lgadelha at ci.uchicago.edu (lgadelha at ci.uchicago.edu) Date: Fri, 26 Oct 2012 12:10:18 -0500 (CDT) Subject: [Swift-commit] r5993 - provenancedb Message-ID: <20121026171018.54C2A9CC92@svn.ci.uchicago.edu> Author: lgadelha Date: 2012-10-26 12:10:18 -0500 (Fri, 26 Oct 2012) New Revision: 5993 Modified: provenancedb/README.asciidoc provenancedb/provdb-uml.dia Log: Modified: provenancedb/README.asciidoc =================================================================== --- provenancedb/README.asciidoc 2012-10-25 21:18:39 UTC (rev 5992) +++ provenancedb/README.asciidoc 2012-10-26 17:10:18 UTC (rev 5993) @@ -25,7 +25,7 @@ A UML diagram of this provenance model is presented in figure <>. We simplify the UML notation to abbreviate the information that each annotated entity set (script run, function call, and variable) has one annotation entity set per data type. We define entities that correspond to the Open Provenance Model (OPM) notions of artifact, process, and artifact usage (either being consumed or produced by a process). Annotations, which can be added post-execution, represent information about provenance entities such as object version tags and scientific parameters. [[provdb_schema]] -image::provdb.svg["Swift provenance database schema",width=1280] +image::provdb.svg["Swift provenance database schema",width=1150] +script+: contains the script source code used and its hash value. @@ -41,29 +41,28 @@ +dataset+: represents data sets that were assigned to variables in a Swift script. -+annot+: is a key-value pair associated with either a +variable+, +function_call+, or +script_run+. These annotations are used to store context-specific information about the entities of the provenance data model. Examples include scientific-domain parameters, object versions, and user identities. Annotations can also be used to associate a set of runs of a script related to a particular event or study, which we refer to as a campaign. The +dataset_in+ and +dataset_out+ relationships between +function_call+ and +variable+ define a lineage graph that can be traversed to determine ancestors or descendants of a particular entity. Process dependency and data dependency graphs are derived with transitive queries over these relationships. ++annot+: is a key-value pair associated with either a +variable+, +function_call+, or +script_run+. The annotations are free-form and can be used, for instance, to record scientific-domain parameters, object versions, and user identities. +The +dataset_in+ and +dataset_out+ relationships between +function_call+ and +variable+ define a lineage graph that can be traversed to determine ancestors or descendants of a particular entity. Process dependency and data dependency graphs are derived with transitive queries over these relationships. + == Design and Implementation of Swift Provenance Database -In this section, we describe the design and implementation of the Swift Provenance Database provenance query framework. It consists of a set of tools used for extracting provenance information from Swift log files, and a query interface. While its log extractor is specific to Swift, the remainder of the system, including the query interface, is applicable to any parallel functional data flow execution model. The Swift Provenance Database system design is influenced by our survey of provenance queries in many-task computing, where a set of query patterns was identified. The _multiple-step relationships_ (R^*) pattern is implemented by queries that follow the transitive closure of basic provenance relationships, such as data containment hierarchies, and data derivation and consumption. The _run correlation_ (RCr) pattern is implemented by queries for correlating attributes from multiple script runs, such as annotation values or the values of function call parameters. +The Swift Provenance Database design is influenced by our survey of provenance queries in many-task computing. Built-in functions and stored procedures can be used to follow the transitive closure of basic provenance relationships, such as data containment hierarchies, and data derivation and consumption; or for correlating attributes from multiple script runs, such as annotation values or the values of function call parameters. === Provenance Gathering and Storage -Swift can be configured to add both prospective and retrospective provenance information to the log file it creates to track the behavior of each script run. The provenance extraction mechanism processes these log files, filters the entries that contain provenance data, and exports this information to a relational SQL database. Each application execution is launched by a wrapper script that sets up the execution environment. We modified these scripts to also gather runtime information, such as memory consumption and processor load. Additionally, one can define a script that generates annotations in the form of key-value pairs, to be executed immediately before the actual application. These annotations can be exported to the provenance database and associated with the respective application execution. Swift Provenance Database processes the data logged by each wrapper to extract both the runtime information and the annotations, storing them in the provenance database. Addit ional annotations can be generated per script run +Swift can be configured to add both prospective and retrospective provenance information to its log files. The provenance extraction mechanism processes these log files, filters the entries that contain provenance data, and exports this information to a relational database. Each application execution is started by a wrapper script that sets up the execution environment. We modified these scripts to also gather runtime information, such as memory consumption and processor load. Additionally, one can define a script that generates annotations in the form of key-value pairs, to be executed immediately before the actual application. These annotations can be exported to the provenance database and associated with the respective application execution. The data logged by each wrapper is processed to extract both the runtime information and the annotations, storing them in the provenance database. Additional annotations can be generated per script run using _ad-hoc_ annotator scripts. In addition to retrospective provenance, Swift Provenance Database keeps prospective provenance by recording the Swift script source code, the application catalog, and the site catalog used in each script run. - - - === Query Interface -During the Third Provenance Challenge, we observed that expressing provenance queries in SQL is often cumbersome. For example, such queries require extensive use of complex relational joins, for instance, which are beyond the level of complexity that most domain scientists are willing, or have the time, to master and write. Such usability barriers are increasingly being seen as a critical issue in database management systems. Jagadish et al. propose that ease of use should be a requirement as important as functionality and performance. They observe that, even though general-purpose query languages such as SQL and XQuery allow for the design of powerful queries, they require detailed knowledge of the database schema and rather complex programming to express queries in terms of such schemas. Since databases are often normalized, data is spread through different relations requiring even more extensive use of database join operations when designing -queries. Some of the approaches used to improve usability are forms-based query interfaces, visual query builders, and schema summarization. +During the Third Provenance Challenge, we observed that expressing provenance queries in SQL is often cumbersome. For example, such queries require extensive use of complex relational joins which are beyond the level of complexity that most domain scientists are willing, or have the time, to master and write. Swift Provenance Query Language, SPQL for short, was designed to allow for easier design of provenance queries for common query patterns than can be accomplished with general purpose query languages, such as SQL. +In the query interface, every SPQL query is translated into a SQL query that is processed by the underlying relational database. While the syntax of SPQL is by design similar to SQL, it does not require detailed knowledge of the underlying database schema for designing queries, but rather only of the entities in a simpler, higher-level abstract provenance schema, and their respective attributes. -In this section, we describe our structured provenance query language, SPQL for short, a component of Swift Provenance Database. It was designed to meet the requirements listed in section \ref{provmodel} and to allow for easier formation of provenance queries for the patterns identified than can be accomplished with general purpose query languages, such as SQL. SPQL supports exploratory queries, where the user seeks information through a sequence of queries that progressively refine previous outputs, instead of having to compose many subqueries into a single complex query, as it is often the case with SQL. Even though our current implementation uses a relational database as the underlying data model for storing provenance information, it should not be dependent on it, we plan to evaluate alternative underlying data models such as graph databases, Datalog, and distributed column-oriented stores. Therefore, in the current implementation, every SPQL query is -translated into a SQL query that is processed by the underlying relational database. While the syntax of SPQL is by design similar to SQL, it does not require detailed knowledge of the underlying database schema for designing queries, but rather only of the entities in a simpler, higher-level abstract provenance schema, and their respective attributes. +//SPQL supports exploratory queries, where the user seeks information through a sequence of queries that progressively refine previous outputs, instead of having to compose many subqueries into a single complex query, as it is often the case with SQL. + The basic building block of a SPQL query consists of a selection query with the following format: -------------------------------------- @@ -73,13 +72,15 @@ (order by orderByClause))) -------------------------------------- -This syntax is very similar to a selection query in SQL, with a critical usability benefit: hide the complexity of designing extensive join expressions. One does not need to provide all tables of the from clause. Instead, only the entity name is given and the translator reconstructs the underlying entity that was broken apart to produce the normalized schema. As in the relational data model, every query or built-in function results in a table, to preserve the power of SQL in querying results of another query. Selection queries can be composed using the usual set operations: union, intersect, and difference. A +select+ clause is a list with elements of the form +(.)+ or +(.)+. If attribute names are omitted, the query returns all the existing attributes of the entity set. SPQL supports the same aggregation, grouping, set -operation and ordering constructs provided by SQL. +Where the optional parts are within parentheses. As in the relational data model, every query or built-in function results in a table, to preserve the power of SQL in querying results of another query. Selection queries can be composed using the usual set operations: union, intersection, and difference. A +select+ clause is a list with elements of the form +(.)+ or +(.)+. If attribute names are omitted, the query returns all the existing attributes of the entity set. SPQL supports the same aggregation, grouping, set operation and ordering constructs provided by SQL. -To simplify the schema that the user needs to understand to design queries, we used database views to define the higher-level schema presentation shown in Figure. This abstract, OPM-compliant provenance schema, is a simplified view of the physical database schema detailed in section. It groups information related to a provenance entity set in a single relation. The annotation entity set shown is the union of the annotation entity sets of the underlying database, presented in Figure. To avoid defining one annotation table per data type, we use dynamic expression evaluation in the SPQL to SQL translator to determine the required type-specific annotation table of the underlying provenance database. +To simplify the schema that the user needs to understand to design queries, we used database views to define the higher-level schema presentation shown in <>. This abstract, OPM-compliant provenance schema, is a simplified view of the physical database schema detailed in section. It groups information related to a provenance entity set in a single relation. The annotation entity set shown is the union of the annotation entity sets of the underlying database, presented in Figure. To avoid defining one annotation table per data type, we use dynamic expression evaluation in the SPQL to SQL translator to determine the required type-specific annotation table of the underlying provenance database. -Most of the query patterns identified in \cite{gadelha_provenance_2011-1} are relatively straightforward to express in a relational query language such as SQL, except for the R^* and RCr patterns, which require either recursion or extensive use of relational joins. To abstract queries that match these patterns, we included in SPQL the following built-in functions to make these common provenance queries easier to express: +[[provdb_schema_summary]] +image::provdb-uml-summary.svg["Summary of Swift provenance database schema",width=800] +To simplify query design, we included in SPQL the following built-in functions to make these common provenance queries easier to express: + - +ancestors(object_id})+ returns a table with a single column containing the identifiers of variables and function calls that precede a particular node in a provenance graph stored in the database. - +data_dependencies(variable_id})+, related to the previous built-in function, returns the identifiers of variables upon which +variable_id+ depends. @@ -107,7 +108,7 @@ -------------------------------------- To further simplify query specification, SPQL uses a generic mechanism for computing the {\em from} clauses and the join expressions of the +where+ clause for the target SQL query. The SPQL to SQL query translator first scans all the entities present in the SPQL query. A shortest path containing all these entities is computed in the graph defined by the schema of the provenance database. All the entities present in this shortest path are listed in the +from+ clause of the target SQL query. The join expressions of the +where+ clause of the target query are computed using the edges of the shortest path, where each edge derives an expression that equates the attributes involved in the foreign key constraint of the entities that define the edge. While this automated join computation facilitates query design, it does somewhat reduce the expressivity of SPQL, as one is not able to perform other types of joins, such as self-joins, explicitly. However, many such queries can be expre ssed using subqueries, -which are supported by SPQL. While some of the expressive power of SQL is thus lost, we show in the sections that follow that SPQL is able to express, with far less effort and complexity, most important and useful queries that provenance query patterns require. As a quick taste, this SPQL query returns the identifiers of the script runs that either produced or consumed the file +nr+: +which are supported by SPQL. While some of the expressive power of SQL is thus lost, we show in the sections that follow that SPQL is able to express, with far less effort and complexity, most important and useful queries that provenance query patterns require. For example, this SPQL query returns the value of the parameter +proteinId+ per script run that consumed a file named +nr+: -------------------------------------- select compare_run(parameter='proteinId').run_id where file.name='nr'; @@ -164,6 +165,7 @@ === Swift Configuration To enable the generation of provenance information in Swift's log files the option +provenance.log+ should be set to true in +etc/swift.properties+: + -------------------------------------- provenance.log=true -------------------------------------- @@ -205,33 +207,30 @@ SELECT script_filename, swift_version, cog_version, final_state, start_time, duration FROM script_run; - script_filename | swift_version | cog_version | final_state | start_time | duration ---------------------+---------------+-------------+-------------+----------------------------+---------- - modis.swift | 5746 | 3371 | FAIL | 2012-09-19 17:26:19.221-03 | 2.168 - modis-vortex.swift | 5746 | 3371 | FAIL | 2012-09-19 17:28:24.809-03 | 180.542 - modis-vortex.swift | 5746 | 3371 | FAIL | 2012-09-19 17:31:55.706-03 | 312.249 + script_filename | swift_version | cog_version | final_state | start_time | duration +-----------------+---------------+-------------+-------------+----------------------------+---------- + modis.swift | 5483 | 3339 | SUCCESS | 2012-10-26 11:46:51.282-02 | 100.724 + modis.swift | 5483 | 3339 | SUCCESS | 2012-10-26 11:44:59.909-02 | 85.050 -------------------------------------- -------------------------------------- -select * from ancestors('dataset:20120919-1731-06svjllb:720000000654'); +select * from ancestors('dataset:20121026-1146-jng6bir4:720000001604'); ancestors -+++-----------------------------------------------------------+++ - modis-vortex-20120919-1731-6fa0kk03:0 - modis-vortex-20120919-1731-6fa0kk03:0-6 - dataset:20120919-1731-06svjllb:720000000335 - dataset:20120919-1731-06svjllb:720000000653 - dataset:20120919-1731-06svjllb:720000000007 - modis-vortex-20120919-1731-6fa0kk03:06svjllb:720000000335 - dataset:20120919-1731-06svjllb:720000000336 - dataset:20120919-1731-06svjllb:720000000337 +---------------------------------------------------------+- + modis-20121026-1146-yp9rbbx5:0 + modis-20121026-1146-yp9rbbx5:0-6 + dataset:20121026-1146-jng6bir4:720000000335 + dataset:20121026-1146-jng6bir4:720000000653 + dataset:20121026-1146-jng6bir4:720000000007 + modis-20121026-1146-yp9rbbx5:jng6bir4:720000000335 + dataset:20121026-1146-jng6bir4:720000000336 ... - dataset:20120919-1731-06svjllb:720000000042 - dataset:20120919-1731-06svjllb:720000000229 - dataset:20120919-1731-06svjllb:720000000006 -(958 rows) + dataset:20121026-1146-jng6bir4:720000000107 + dataset:20121026-1146-jng6bir4:720000000015 + dataset:20121026-1146-jng6bir4:720000000006 -------------------------------------- Modified: provenancedb/provdb-uml.dia =================================================================== (Binary files differ) From wilde at ci.uchicago.edu Fri Oct 26 12:41:25 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Fri, 26 Oct 2012 12:41:25 -0500 (CDT) Subject: [Swift-commit] r5995 - SwiftApps/SciColSim Message-ID: <20121026174125.A93869CC92@svn.ci.uchicago.edu> Author: wilde Date: 2012-10-26 12:41:25 -0500 (Fri, 26 Oct 2012) New Revision: 5995 Added: SwiftApps/SciColSim/big_graph.v01.txt.gz Log: Add first version of Andrey's large-scale graph. Added: SwiftApps/SciColSim/big_graph.v01.txt.gz =================================================================== (Binary files differ) Property changes on: SwiftApps/SciColSim/big_graph.v01.txt.gz ___________________________________________________________________ Added: svn:mime-type + application/octet-stream From wilde at ci.uchicago.edu Sat Oct 27 08:57:03 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Sat, 27 Oct 2012 08:57:03 -0500 (CDT) Subject: [Swift-commit] r5996 - SwiftApps/SciColSim Message-ID: <20121027135703.A61E79CD00@svn.ci.uchicago.edu> Author: wilde Date: 2012-10-27 08:57:02 -0500 (Sat, 27 Oct 2012) New Revision: 5996 Added: SwiftApps/SciColSim/trimgraph.sh Log: Tool to create test subsets of Andrey's big 1.3M node graph. Added: SwiftApps/SciColSim/trimgraph.sh =================================================================== --- SwiftApps/SciColSim/trimgraph.sh (rev 0) +++ SwiftApps/SciColSim/trimgraph.sh 2012-10-27 13:57:02 UTC (rev 5996) @@ -0,0 +1,22 @@ +#! /bin/sh + +vertices=${1:-100} +fraction=${2:-1.0} + +tmp=$(mktemp) + +awk ' +NR == 1 { } + +(NR > 1) && ($1 < n) && ($2 < n) && (rand() < fraction) { print $1, $2 } +' n=$vertices fraction=$fraction $tmp + +edges=$(wc -l <$tmp) + +echo $vertices $edges +cat $tmp +rm $tmp + + + + Property changes on: SwiftApps/SciColSim/trimgraph.sh ___________________________________________________________________ Added: svn:executable + * From wilde at ci.uchicago.edu Sat Oct 27 09:31:40 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Sat, 27 Oct 2012 09:31:40 -0500 (CDT) Subject: [Swift-commit] r5997 - SwiftApps/SciColSim Message-ID: <20121027143140.F1E939CD24@svn.ci.uchicago.edu> Author: wilde Date: 2012-10-27 09:31:40 -0500 (Sat, 27 Oct 2012) New Revision: 5997 Added: SwiftApps/SciColSim/gengraphs.sh SwiftApps/SciColSim/graphsim.cpp SwiftApps/SciColSim/testgraph.py Modified: SwiftApps/SciColSim/Makefile Log: Changes to make test graphs and single-graph simulation. Modified: SwiftApps/SciColSim/Makefile =================================================================== --- SwiftApps/SciColSim/Makefile 2012-10-27 13:57:02 UTC (rev 5996) +++ SwiftApps/SciColSim/Makefile 2012-10-27 14:31:40 UTC (rev 5997) @@ -2,11 +2,14 @@ ifeq ($(UNAME), Linux) -all: openmp-optimizer +all: openmp-optimizer graphsim openmp-optimizer: optimizer.cpp g++ -DP_OPENMP -static -O -fopenmp -I boost_1_47_0 -o openmp-optimizer optimizer.cpp +graphsim: graphsim.cpp + g++ -DP_OPENMP -static -O -fopenmp -I boost_1_47_0 -o graphsim graphsim.cpp + clean: @rm -rvf openmp-optimizer Added: SwiftApps/SciColSim/gengraphs.sh =================================================================== --- SwiftApps/SciColSim/gengraphs.sh (rev 0) +++ SwiftApps/SciColSim/gengraphs.sh 2012-10-27 14:31:40 UTC (rev 5997) @@ -0,0 +1,9 @@ +#! /bin/sh + +mkdir -p graph + +for s in $(seq 1000 1000 10000); do + ./trimgraph.sh $s 1.0 > graph/$s.100 + ./trimgraph.sh $s 0.1 > graph/$s.10 + ./trimgraph.sh $s 0.5 > graph/$s.50 +done \ No newline at end of file Property changes on: SwiftApps/SciColSim/gengraphs.sh ___________________________________________________________________ Added: svn:executable + * Added: SwiftApps/SciColSim/graphsim.cpp =================================================================== --- SwiftApps/SciColSim/graphsim.cpp (rev 0) +++ SwiftApps/SciColSim/graphsim.cpp 2012-10-27 14:31:40 UTC (rev 5997) @@ -0,0 +1,1866 @@ +// +// main.cpp +// optimizer +// +// Created by Andrey Rzhetsky on 4/11/11. +// Copyright 2011 University of Chicago. All rights reserved. +// + +// Select OpenMP or Grand Central Dispatch for multithreading: + +#define MAXNworkers 1 +int Nworkers=MAXNworkers; + +#define NUniverse 1 + +// Add operation code to enable existing code to be used at lower level from Swift scripts: + +char operation = 'n'; // n: normal; m: do one multi_loss (with n_reruns). + // Not used: a: analyze and generate next annealing parameter set. g: tbd + +unsigned initSeed = 0; + +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#ifdef P_DISPATCH +#include +#endif + +#include + + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#define BOOST_MATH_OVERFLOW_ERROR_POLICY ignore_error +#define BOOST_MATH_DISCRETE_QUANTILE_POLICY real +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define INT_INFINITY 2147483647 + +#define FIX_VARIABLES 1 + +#define MAX_LOSS 160 + +using namespace boost; +using namespace std; +using namespace boost::numeric::ublas; + +static int max_dist=0; + +typedef boost::adjacency_matrix Graph; +typedef std::pair Edge; +typedef boost::graph_traits GraphTraits; +typedef boost::numeric::ublas::triangular_matrix prob; +typedef boost::numeric::ublas::triangular_matrix pathlength; +typedef boost::graph_traits::vertex_descriptor vertex_descriptor; + +namespace std { + using ::time; +} + +static int var_fixed[5] = {1, 0, 1, 1, 0}; + +typedef boost::minstd_rand base_generator_type; +typedef adjacency_list < listS, vecS, directedS, +no_property, property < edge_weight_t, int > > graph_t; +typedef graph_traits < graph_t >::vertex_descriptor vertex_descriptor; +typedef graph_traits < graph_t >::edge_descriptor edge_descriptor; + + +//================================================ +string strDouble(double number) +{ + stringstream ss;//create a stringstream + ss << number;//add number to the stream + return ss.str();//return a string with the contents of the stream +} + +//================================================ + +double gaussian(double sigma) +{ + double GaussNum = 0.0; + int NumInSum = 10; + for(int i = 0; i < NumInSum; i++) + { + GaussNum += ((double)rand()/(double)RAND_MAX - 0.5); + } + GaussNum = GaussNum*sqrt((double)12/(double)NumInSum); + + + return GaussNum; + +} + + + +//================================================= +double diffclock(clock_t clock1,clock_t clock2) +{ + double diffticks=clock1-clock2; + double diffms=(diffticks)/CLOCKS_PER_SEC; + return diffms; +} + +//================================================ +//================================================================ +double get_new_x(double x, double dx){ + + double new_x; + // boost::variate_generator > uni(generator, uni_dist); + double r = rand()/(double)(pow(2.,31)-1.); + + if (r > 0.5){ + new_x = x + rand()*dx/(double)(pow(2.,31)-1.); + } else { + new_x = x - rand()*dx/(double)(pow(2.,31)-1.); + } + + return new_x; + +} + + +//=============================================== +string string_wrap(string ins, int mode){ + + std::ostringstream s; + + switch(mode){ + case 0: + s << "\033[1;29m" << ins << "\033[0m"; + break; + case 1: + s << "\033[1;34m" << ins << "\033[0m"; + break; + case 2: + s << "\033[1;44m" << ins << "\033[0m"; + break; + case 3: + s << "\033[1;35m" << ins << "\033[0m"; + break; + case 4: + s << "\033[1;33;44m" << ins << "\033[0m"; + break; + case 5: + s << "\033[1;47;34m" << ins << "\033[0m"; + break; + case 6: + s << "\033[1;1;31m" << ins << "\033[0m"; + break; + case 7: + s << "\033[1;1;33m" << ins << "\033[0m"; + break; + case 8: + s << "\033[1;1;43;34m" << ins << "\033[0m"; + break; + case 9: + s << "\033[1;1;37m" << ins << "\033[0m"; + break; + case 10: + s << "\033[1;30;47m" << ins << "\033[0m"; + break; + default: + s << ins; + } + + return s.str(); +} + + +//=============================================== +string wrap_double(double val, int mode){ + + std::ostringstream s; + s << string_wrap(strDouble(val),mode); + + return s.str(); +} + + + +//=============================================== +const +string i2string(int i){ + + std::ostringstream s; + s << "worker" + << lexical_cast(i); + + return s.str(); + +} + +//=============================================== +char* i2char(int i){ + + std::ostringstream s; + s << "worker" + << lexical_cast(i); + + char* a=new char[s.str().size()+1]; + memcpy(a,s.str().c_str(), s.str().size()); + + return a; +} + +//================================================ +class Universe { + +private: + + double alpha_i; + double alpha_m; + double beta; + double gamma; + double delta; + + double TargetNovelty; + double CumulativeRelativeLoss; + double CRLsquare; + string id; + + + int N_nodes; + int M_edges; + + int N_epochs; + int N_steps; + int N_repeats; + + int current_epoch; + double current_loss; + int current_repeat; + double current_novelty; + + int mode_identify_failed; + int verbose_level; // 0 is silent, higher is more + + double k_max; + + graph_t Full_g; + + double **Prob; + double **Tried; + double **Dist; + double **Final; + double **EdgeIndex; + double *Rank; + + base_generator_type generator; + boost::uniform_real<> uni_dist; + boost::geometric_distribution geo; + +public: + + + + //====== Constructor ====== + Universe(const std::string FileToOpen, int Epochs, int Steps, int Repeats, int identify_failed, double target, const std::string idd) + { + //typedef array_type2::index index2; + + + std::ifstream inFile; + //string line; + + //------------------------------- + + base_generator_type gene(42u); + generator = gene; + generator.seed(static_cast(std::time(0))); + boost::uniform_real<> uni_d(0,1); + uni_dist = uni_d; + + //-------------------------------- + + int i, k; + int x, y; + Edge* edge_array_mine; + int num_arcs_mine, num_nodes_mine; + int* weights_mine; + + TargetNovelty = target; + CumulativeRelativeLoss = 0.; + CRLsquare = 0.; + + + N_epochs = Epochs; + N_steps = Steps; + N_repeats = Repeats; + + current_epoch = 0; + current_loss = 0.; + current_repeat = 0; + + id = idd; + + verbose_level = 1; + + mode_identify_failed = identify_failed; + + + //------------------------------- + // The first pass though file with the graph + inFile.open(FileToOpen.c_str()); + if (inFile.fail()) { + cout << "Unable to open file"; + exit(1); // terminate with error + }else { + + if (verbose_level > 2){ + std::cout << " Opened <" << FileToOpen << ">"<> x; + inFile >> y; + + if (verbose_level > 2){ + std::cout << " x: " << x; + std::cout << " y: " << y << std::endl; + } + + if (i==0){ + N_nodes=x; + M_edges=y; + break; + } + i++; + + + } + inFile.close(); + + if (verbose_level == 2){ + std::cout << N_nodes << " nodes, " << M_edges << " edges"< 2){ + std::cout << " Opened <" << FileToOpen << ">"<> x && inFile >>y) { + if (i > 0) { + Final[x][y]=1.; + Final[y][x]=1.; + + + if (verbose_level == 2){ + std::cout << "."; + } + } + i++; + + } + if (verbose_level == 2){ + std::cout << std::endl; + } + inFile.close(); + + k=0; + for (int i=0; i 0.){ + EdgeIndex[i][j]=k; + k++; + } + } + } + + + + //+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + // create graph -- hopefully, we can keep it, just modifying edge weights + + + edge_array_mine = new Edge[2*M_edges]; + num_arcs_mine = 2*M_edges; + num_nodes_mine = N_nodes; + weights_mine = new int[2*M_edges]; + for (int i=0; i<2*M_edges; i++){ weights_mine[i]=1;} + + k=0; + for(int i=0; i0.){ + edge_array_mine[2*k] =Edge(i,j); + edge_array_mine[2*k+1]=Edge(j,i); + k++; + } + } + } + graph_t g(edge_array_mine, edge_array_mine + num_arcs_mine, weights_mine, num_nodes_mine); + + Full_g = g; + delete edge_array_mine; + delete weights_mine; + + //=========================================================================== + std::vector p(num_edges(Full_g)); + std::vector d(num_edges(Full_g)); + edge_descriptor s; + boost::graph_traits::vertex_descriptor u, v; + + for (int i=0; i 0.){ + u = vertex(i, Full_g); + v = vertex(j, Full_g); + remove_edge(u,v,Full_g); + remove_edge(v,u,Full_g); + + } + } + } + + + } + + + //===================================================================== + int sample_failed_number(double pfail){ + + //boost::geometric_distribution geo(pfail); + //boost::variate_generator > geom(generator, geo); + + double r, u, g; + + r=0.; + for(int i=0; i=3){ + std::cout << id << " failed " << r << std::endl; + } + return (int) round(r); // FIXME: Andrey: please verify that round() is correct. + + } + + //============================================= + double get_target(void){ + return TargetNovelty; + } + + //============================================= + void set_target(double target){ + TargetNovelty=target; + } + + //============================================= + int sample(){ + + //boost::variate_generator > uni(generator, uni_dist); + // double r = uni(), Summa = 0.; + + + + double r = rand(), Summa = 0.; + r /= (double)(pow(2.,31)-1.); + int result = 0; + int finished = 0; + + if (verbose_level==4){ + std::cout << id << " sampled " << r << std::endl; + } + + for(int i=0; i r){ + + Tried[i][j]+=1.; + + if (Final[i][j] > 0.){ + result = 1; + } + finished = 1; + } + } + } + + return result; + + } + + //=============================== + void update_current_graph(void){ + + std::vector p(num_edges(Full_g)); + std::vector d(num_edges(Full_g)); + edge_descriptor s; + boost::graph_traits::vertex_descriptor u, v; + + //property_map::type weightmap = get(edge_weight, Full_g); + for (int i=0; i 0. && Tried[i][j]>0){ + //s = edge(i, j, Full_g); + boost::graph_traits::edge_descriptor e1,e2; + bool found1, found2; + u = vertex(i, Full_g); + v = vertex(j, Full_g); + tie(e1, found1) = edge(u, v, Full_g); + tie(e2, found2) = edge(v, u, Full_g); + if (!found1 && !found2){ + add_edge(u,v,1,Full_g); + add_edge(v,u,1,Full_g); + } + + } + } + + } + } + + //=============================== + void update_distances(void){ + // put shortest paths to the *Dist[][] + std::vector p(num_vertices(Full_g)); + std::vector d(num_vertices(Full_g)); + vertex_descriptor s; + + + // put shortest paths to the *Dist[][] + for (int j=0; j 0.){ + s = vertex(j, Full_g); + dijkstra_shortest_paths(Full_g, s, predecessor_map(&p[0]).distance_map(&d[0])); + + //std::cout <<" Vertex "<< j << std::endl; + graph_traits < graph_t >::vertex_iterator vi, vend; + + for (boost::tie(vi, vend) = vertices(Full_g); vi != vend; ++vi) { + + if (p[*vi]!=*vi){ + Dist[*vi][j]=d[*vi]; + Dist[j][*vi]=d[*vi]; + + if ( (int)round(Dist[*vi][j]>max_dist)) { + // FIXME: Andrey: please verify that (int) cast is correct. Do we need to round()? + // also, the indent on this iff statement was way off - + // perhaps due to space v. tab? + max_dist=(int)round(Dist[*vi][j]); + } + + + } else { + Dist[*vi][j]=-1.; + Dist[j][*vi]=-1.; + } + } + } + + } + + + } + + //====================================================== + void update_ranks(void){ + + for(int i=0; i0. && Final[i][j] >0.){ + Rank[i]++; + Rank[j]++; + } + } + } + + } + + //==================================================================== + void set_world(double a_i, double a_m, double b, double g, double d){ + + alpha_i=a_i; + alpha_m=a_m; + gamma=g; + beta=b; + delta=d; + + } + + //==================================================================== + void reset_world(){ + + //==================================================== + std::vector p(num_edges(Full_g)); + std::vector d(num_edges(Full_g)); + edge_descriptor s; + boost::graph_traits::vertex_descriptor u, v; + + + for (int i=0; i 0. && Tried[i][j] > 0){ + u = vertex(i, Full_g); + v = vertex(j, Full_g); + remove_edge(u,v,Full_g); + remove_edge(v,u,Full_g); + + } + } + } + + //================================================== + + current_loss=0; + current_epoch=0; + current_repeat++; + current_novelty=0; + + for(int i = 0; i < N_nodes; ++i) { + Rank[i]=0.; + for(int j = 0; j < N_nodes; ++j) { + Prob[i][j]=0.; + Dist[i][j]=-1.; + Tried[i][j]=0.; + } + } + } + + + //============================================== + void show_parameters(void){ + + std::cout << "Parameters: " + << alpha_i << " " + << alpha_m << " | " + << beta << " " + << gamma << " | " + << delta << std::endl; + + } + + + + //=============================================== + string file_name(){ + + std::ostringstream s; + s << "world_" + << lexical_cast(alpha_i) << "_" + << lexical_cast(alpha_m) << "_" + << lexical_cast(beta) << "_" + << lexical_cast(gamma) << "_" + << lexical_cast(delta) << "_" + << lexical_cast(N_epochs) << "_" + << lexical_cast(N_steps) << "_" + << lexical_cast(N_repeats) << ".txt"; + + return s.str(); + + } + + + + + //================================================= + void set_verbose(int verbose){ + + verbose_level = verbose; + } + + + //============================================================= + void update_probabilities(void){ + + + //========================= + // Compute sampling probabilities + // first pass: \xi_i,j + for(int i=0; i 0.){ + + double k = Dist[i][j]; + if (k >= k_max){ + k = k_max-1; + } + + bg = beta * log(k/k_max) + gamma * log(1. - k/k_max); + + } else { + bg = delta; + } + + Prob[i][j] = exp(Prob[i][j] + bg); + } + } + + + // second pass: sum + double Summa = 0.; + + for(int i=0; i0. && Final[i][j]>0.){ + novel+=1.; + } + } + } + + } + //^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + else { + + double pfail=0.; + int n_failed; + //, n_check = 0; + + for(int i=0; i0. && Final[i][j]>0.){ + novel+=1.; + } + } + } + } + + current_novelty = novel; + + + //^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + if (verbose_level == 2){ + std::cout << (current_repeat+1) << " epoch=" << (current_epoch+1) + + << " cost=" << cost + << " novel=" << novel + << " rel_loss=" << cost/novel + << std::endl; + } + + current_epoch++; + } + + + //====== Destructor ====== + ~Universe(){ + + delete_2Dmatrix(Final, N_nodes); + delete_2Dmatrix(Dist, N_nodes); + delete_2Dmatrix(Tried, N_nodes); + delete_2Dmatrix(Prob, N_nodes); + delete_2Dmatrix(EdgeIndex, N_nodes); + delete_1Dmatrix(Rank); + } + + //================================================ + // Allocate memory + double** allocate_2Dmatrix(int N, int M) + { + double **pointer; + + if (verbose_level == 2){ + std::cout<< "["< 0){ + + pointer = new double[N]; + + }else { + + pointer = NULL; + } + + return pointer; + + } + + //============================================== + // De-Allocate memory to prevent memory leak + void delete_2Dmatrix(double **pointer, int N){ + + if (pointer != NULL){ + + for (int i = 0; i < N; ++i){ + delete [] pointer[i]; + } + delete [] pointer; + } + } + //==================== + void delete_1Dmatrix(double *pointer){ + + delete [] pointer; + } + + //=========================================== + double get_rel_loss(){ + + return CumulativeRelativeLoss ; + } + + //=========================================== + double get_rel_loss_err(){ + + return CRLsquare ; + } + + + + //================================================================================== + void evolve_to_target_and_save(int istart, int iend, double* storage, int* counters){ + + double ALOT=100000000000.; + double ratio = 0.0; + int check = 0; + + reset_world(); + + for (int k = istart; k < iend; k++){ + + for(int i=0; i< N_epochs && current_novelty < TargetNovelty; i++){ + + if(ratio < max_loss_value((int)TargetNovelty)){ + update_world(); + } + else{ + check = 1; + break; + } + ratio = current_loss/current_novelty; + } + if( check ){ + storage[k]=ALOT; + } + else{ + storage[k]=current_loss/current_novelty; + } + counters[k]=1; + + reset_world(); + } + + } + + //============================================== + double Omax_loss_value(int tn){ + + if( tn == 58 ) return 128.286; + else if( tn == 108 ) return 131.866; + else if( tn == 158 ) return 135.551; + else if( tn == 208 ) return 139.694; + else if( tn == 258 ) return 144.163; + else if( tn == 308 ) return 148.967; + else if( tn == 358 ) return 154.201; + else if( tn == 408 ) return 159.962; + else if( tn == 458 ) return 166.441; + else if( tn == 508 ) return 173.655; + else if( tn == 558 ) return 181.921; + else if( tn == 608 ) return 191.246; + else if( tn == 658 ) return 202.150; + else if( tn == 708 ) return 215.197; + else if( tn == 758 ) return 202.150; // Verify with Andrey. Same as TargetNovelty of 658 + else if( tn == 808 ) return 251.698; + else if( tn == 858 ) return 279.201; + else if( tn == 908 ) return 320.112; + else if( tn == 958 ) return 394.774; + else if( tn == 1008 ) return 1052.38; // Huge number here. Why? + else return 10000.; /* Pray that this does not occur. Ask Andrey what to do */ + } + double max_loss_value(int tn){ + + switch(tn) { + case 58: return 128.286; + case 108: return 131.866; + case 158: return 135.551; + case 208: return 139.694; + case 258: return 144.163; + case 308: return 148.967; + case 358: return 154.201; + case 408: return 159.962; + case 458: return 166.441; + case 508: return 173.655; + case 558: return 181.921; + case 608: return 191.246; + case 658: return 202.150; + case 708: return 215.197; + case 758: return 202.150; // Verify with Andrey. Same as TargetNovelty of 658 + case 808: return 251.698; + case 858: return 279.201; + case 908: return 320.112; + case 958: return 394.774; + case 1008: return 1052.38; // Huge number here. Why? + default: return 10000.; /* Pray that this does not occur. Ask Andrey what to do */ + } + } + + + //============================================== + int get_reruns(void){ + return N_repeats; + } + + //============================================== + double get_parameter(int i){ + + switch(i){ + case 0: + return alpha_i; + case 1: + return alpha_m; + case 2: + return beta; + case 3: + return gamma; + case 4: + return delta; + default: + + std::cout << "Erroneous parameter id!!!!\n\n\n"; + return 0.; + } + } + + + //============================================== + void evolve_to_target(){ + + reset_world(); + if (beta < -1. || gamma < -1.){ + CumulativeRelativeLoss = 100000000000.; + CRLsquare = 0.; + return; + } + + + for (int k=0; k< N_repeats; k++){ + + + for(int i=0; i 4) {return 0;} + + else { + + switch(position){ + case 0: + alpha_i=value; + return 1; + case 1: + alpha_m=value; + return 1; + case 2: + beta=value; + return 1; + case 3: + gamma=value; + return 1; + case 4: + delta=value; + return 1; + } + + } + + return 0; + } + +#ifdef notdef + //================================================================= + void try_annealing(double starting_jump, int iterations, + double temp_start, double temp_end, double target_rejection){ + + double dx[5]={0.,0.,0.,0.,0}; + double x[5]={0.,0.,0.,0.,0}; + double rejection[5]={0., 0., 0., 0., 0.}; + double curr_x, curr_err, x_tmp; + double temperature; + double ratio, r; + int cycle=10; + boost::variate_generator > uni(generator, uni_dist); + + // set up parameter for annealing + + x[0]=alpha_i; + x[1]=alpha_m; + x[2]=beta; + x[3]=gamma; + x[4]=delta; + + for(int i=0;i<5;i++){ + dx[i] = starting_jump; + } + + // establish the current value + + //.......................................... + evolve_to_target(); + std::cout << CumulativeRelativeLoss << " +- " << CRLsquare << std::endl; + + curr_x = CumulativeRelativeLoss; + curr_err = CRLsquare; + CumulativeRelativeLoss = 0; + CRLsquare = 0; + //........................................... + + // optimization cycle + for(int i=0; i ratio){ + + std::cout << string_wrap(id, 4) <<" "<< (i+1) << ","<< (j) + <<" "<< (i+1) << " Did not accept " + << x_tmp << "(" << j << ")" << std::endl; + std::cout << alpha_i << " "<< alpha_m << " " + << beta << " " << gamma << " " + << delta << " " << std::endl; + set_parameter(x[j], j); + CumulativeRelativeLoss = 0; + CRLsquare = 0; + + rejection[j]+=1.; + } + + else { + + curr_x = CumulativeRelativeLoss; + curr_err = CRLsquare; + x[j] = x_tmp; + CumulativeRelativeLoss = 0; + CRLsquare = 0; + std::cout << (i+1) << string_wrap((string) " Rejection counts: ", 8) + << wrap_double(rejection[0],2) + << " "<< wrap_double(rejection[1], 7) << " " + << wrap_double(rejection[2],5) << " " << wrap_double(rejection[2],9) << " " + << wrap_double(rejection[4],6) << " " + << std::endl << std::endl; + + std::cout << string_wrap(id, 4) <<" "<< (i+1) <<","<< (j) + <<" " + << string_wrap((string) "***** Did accept! ", 3) + << wrap_double(alpha_i,2) + << " "<< wrap_double(alpha_m, 7) << " " + << wrap_double(beta,5) << " " + << wrap_double(gamma,9) << " " + << wrap_double(delta,6) << " " + << std::endl << std::endl; + + } + //........................................................ + + } + + } + + } + +#endif + +}; + +//============================================================ + +#ifdef P_DISPATCH +std::pair multi_loss(dispatch_group_t group, + Universe* un[], + dispatch_queue_t* CustomQueues, + double* Results, + int* Counters, + double* params){ +#else +std::pair multi_loss(Universe* un[], + double* Results, + int* Counters, + double* params){ + +#endif + + int N = un[0]->get_reruns(); + int step = (int)floor((double)N/(double)(Nworkers)); // FIXME: Andrey: please check change in cast grouping and use of floor + int istart=0; + int iend = istart+step; + + double Loss=0., LossSquare=0.; + + timeval startTime, endTime; + double elapsedTime; + gettimeofday(&startTime, NULL); + + if ( N > NUniverse ) { + std::cerr << "Error: Number of reruns=" << N << " is greater than max NUniverse=" << NUniverse << "\n"; + exit(1); + } + + for(int i=0; iset_parameter(params[j],j); + } + } + +#ifdef P_DISPATCH + for(int i=0; ievolve_to_target_and_save(istart, iend, Results, Counters); + un[i]->evolve_to_target_and_save(i*step, min((i+1)*step,N), Results, Counters); + }); + + std::cout << "queued: i=" << i << " N=" << N << " istart=" << istart << " iend=" << iend << "\n"; + // istart += step; + // iend = min(istart+step,N); + + } + dispatch_group_wait(group, DISPATCH_TIME_FOREVER); + //dispatch_release(group); +#else + int i; + + // Execute rerun loop in parallel + + #pragma omp parallel for private (i) + for(i=0; ievolve_to_target_and_save(i, i+1, Results, Counters); + } +#endif + + for (int i=0; i Res; + Res.first=Loss; + Res.second=two_std; + + gettimeofday(&endTime, NULL); + elapsedTime = (endTime.tv_sec - startTime.tv_sec) * 1000.0; // sec to ms + elapsedTime += (endTime.tv_usec - startTime.tv_usec) / 1000.0; // us to ms + elapsedTime /= 1000.; + cout << "multi_loss(N=" << N << ", target=" << un[0]->get_target() << ") elapsed time: " << elapsedTime << " seconds " << elapsedTime/60. << " minutes\n\n"; + + return Res; +} +//============================================================ + + +//============================================================ +#ifdef P_DISPATCH +void multi_annealing( dispatch_group_t group, + Universe* un[], + dispatch_queue_t* CustomQueues, + double T_start, double T_end, + double Target_rejection, + int Annealing_repeats, + double starting_jump, + double* Results, + int* Counters, + double* params0, + double annealing_cycles){ +#else +void multi_annealing(Universe* un[], + double T_start, double T_end, + double Target_rejection, + int Annealing_repeats, + double starting_jump, + double* Results, + int* Counters, + double* params0, + double annealing_cycles){ +#endif + //................................. + // re-implement annealing + + double dx[5]={0.,0.,0.,0.,0}; + double x[5]={0.,0.,0.,0.,0}; + double rejection[5]={0., 0., 0., 0., 0.}; + double curr_x, curr_err, x_tmp; + double temperature; + double ratio, r; + int cycle=10; + //boost::variate_generator > uni(generator, uni_dist); + + // set up parameter for annealing + + x[0]=params0[0]; + x[1]=params0[1]; + x[2]=params0[2]; + x[3]=params0[3]; + x[4]=params0[4]; + + for(int i=0;i<5;i++){ + dx[i] = starting_jump; + } + + // establish the current value + std::pairRes; + +#ifdef P_DISPATCH + Res = multi_loss(group, un, CustomQueues, Results, Counters, x); +#else + Res = multi_loss( un, Results, Counters, x); +#endif + std::cout << "Returned from initial multi_loss:" << std::endl; + std::cout << Res.first << " +- " << Res.second << std::endl; + + if ( operation == 'm' ) { + FILE *f; + int N = un[0]->get_reruns(); + + f = fopen("multi_loss.data","w"); + for(int i=0; iset_parameter(x_tmp, j); + } + +#ifdef P_DISPATCH + Res = multi_loss(group, un, CustomQueues, Results, Counters, x); +#else + Res = multi_loss( un, Results, Counters, x); +#endif + std::cout << Res.first << " +- " << Res.second << std::endl; + + ratio = min(1.,exp(-(Res.first-curr_x)/temperature)); + r = rand()/(double)(pow(2.,31)-1.); + std::cout << r << " vs " << ratio << std::endl; + + double ALOT=100000000000.; + + if (Res.first < ALOT) + { + ofstream filestr; + + filestr.open ("best_opt_some.txt", ofstream::app); + + // >> i/o operations here << + filestr + + << "N, " << i << ", " << j << ", " << dx[j] << ", " << rejection[j] << ", |, " // FIXME: MW-DEBUGGING + + << un[0]->get_target() << ", " + << Res.first + << ", " << un[0]->get_parameter(0) + << ", " << un[0]->get_parameter(1) + << ", " << un[0]->get_parameter(2) + << ", " << un[0]->get_parameter(3) + << ", " << un[0]->get_parameter(4) << ", " << Res.second << ",\n"; + + filestr.close(); + + + filestr.open ("max_dist.txt", ofstream::app); + + // >> i/o operations here << + filestr << max_dist << ",\n"; + + filestr.close(); + + FILE *bf; + bf = fopen("bestdb.txt","a"); + fprintf(bf, "N %2d %2d %10.5f %5.2f | %5.2f %10.5f [ %5.2f %5.2f %10.5f %10.5f %10.5f ] %10.5f\n", i, j, dx[j], rejection[j], + un[0]->get_target(), + Res.first, + un[0]->get_parameter(0), + un[0]->get_parameter(1), + un[0]->get_parameter(2), + un[0]->get_parameter(3), + un[0]->get_parameter(4), + Res.second); + fclose(bf); + } + + + if (r > ratio){ + + std::cout << " "<< (i+1) << ","<< (j) + <<" "<< (i+1) << " Did not accept " + << x_tmp << "(" << j << ")" << std::endl; + std::cout << un[0]->get_parameter(0) + << " " << un[0]->get_parameter(1) + << " " << un[0]->get_parameter(2) + << " " << un[0]->get_parameter(3) + << " " << un[0]->get_parameter(4) << " " << std::endl; + + x[j]=x_hold; + for(int w=0; wset_parameter(x[j], j); + } + + + //set_parameter(x[j], j); + rejection[j]+=1.; + } + + else { + + curr_x = Res.first; + curr_err = Res.second; + x[j] = x_tmp; + + for(int w=0; wset_parameter(x[j], j); + } + + std::cout << (i+1) << string_wrap((string) " Rejection counts: ", 8) + << wrap_double(rejection[0],2) << " " + << wrap_double(rejection[1],7) << " " + << wrap_double(rejection[2],5) << " " + << wrap_double(rejection[3],9) << " " + << wrap_double(rejection[4],6) << " " + << std::endl << std::endl; + + std::cout << " "<< (i+1) <<","<< (j) + <<" " + << string_wrap((string) "***** Did accept! ", 3) + << wrap_double(un[0]->get_parameter(0),2) << " " + << wrap_double(un[0]->get_parameter(1),7) << " " + << wrap_double(un[0]->get_parameter(2),5) << " " + << wrap_double(un[0]->get_parameter(3),9) << " " + << wrap_double(un[0]->get_parameter(4),6) << " " + << std::endl << std::endl; + + + + } + //........................................................ + + } + } + + } + +} + + + +//================================================ +int +main(int argc, char* argv[]) +{ + + double params0[6] = {0., 0., 0., 0., 0., 0.2}, target=50., range; + string par_names0[6] = {"alpha_i", "alpha_m", "beta", "gamma", "delta", "target"}; + string par_names1[4] = {"n_epochs", "n_steps", "n_reruns", "range"}; + string par_names2[5] = {"T_start", "T_end", "Annealing_steps","Target_rejection","Starting_jump"}; + string par_names3[5] = {"FREEZE_alpha_i", "FREEZE_alpha_m", "FREEZE_beta", "FREEZE_gamma", "FREEZE_delta"}; + string par_names4[3] = {"Operation", "Nworkers", "initSeed"}; + + int params1[4] = {300, 50, 1000, 10}; + int params3[5] = { 0, 0, 0, 0, 0}; + + // temperature_start, temperature_end, annealing_steps target_rejection Starting_jump + double params2[5] = {1, 0.001, 100, 0.3, 1.5}; + + int verbose_level = 2; + const std::string one="one", two="two"; + static Universe* un[NUniverse]; +#ifdef P_DISPATCH + static dispatch_queue_t CustomQueues[MAXNworkers]; +#endif + static double* Results; + static int* Counters; + + timeval t1, t2; + double elapsedTime; + // start timer + gettimeofday(&t1, NULL); + + + if (argc < 8) { + std::cout << "Usage: super_optimizer alpha_i alpha_m beta gamma delta target_innov [n_epochs n_steps n_reruns] [range] [verbose_level]\n"; + std::cout << " [T_start T_end Annealing_steps Target_rejection Starting_jump]\n"; + std::cout << " [FREEZE_alpha_i FREEZE_alpha_m FREEZE_beta FREEZE_gamma FREEZE_delta]\n"; + + system("pwd"); + + + return(1); + } + else { + for (int nArg=0; nArg < argc; nArg++){ + //std::cout << nArg << " " << argv[nArg] << std::endl; + if (nArg > 0 && nArg < 7){ + params0[nArg-1]= atof(argv[nArg]); + std::cout << par_names0[nArg-1] << ": " << params0[nArg-1] << std::endl; + } + if (nArg > 6 && nArg < 11){ + params1[nArg-7]= atoi(argv[nArg]); + std::cout << par_names1[nArg-7] << ": " << params1[nArg-7] << std::endl; + } + if (nArg == 11){ + verbose_level = atoi(argv[nArg]); + std::cout << "verbose level: " << verbose_level << std::endl; + } + if (nArg > 11 && nArg < 17){ + params2[nArg-12]= atof(argv[nArg]); + std::cout << par_names2[nArg-12] << ": " << params2[nArg-12] << std::endl; + } + if (nArg > 16 && nArg < 22){ + params3[nArg-17]= (int)round(atof(argv[nArg])); // FIXME: Andrey: please verify that round() is correct. + var_fixed[nArg-17]= (int)round(atof(argv[nArg])); // FIXME: ditto + std::cout << par_names3[nArg-17] << ": " << var_fixed[nArg-17] << std::endl; + } + if (nArg == 22 ){ + operation = *argv[nArg]; + std::cout << par_names4[0] << ": " << operation << std::endl; + } + if (nArg == 23 ){ + Nworkers = atoi(argv[nArg]); + std::cout << par_names4[1] << ": " << Nworkers << std::endl; + } + if (nArg == 24 ){ + initSeed = atoi(argv[nArg]); + std::cout << par_names4[2] << ": " << initSeed << std::endl; + } + + + } + + } + + for (int j=0; j<5; j++){ + + cout << j << " | " << var_fixed[j] << " (fixed) \n"; + } + + target=params0[5]; + range = (double)params1[3]; + int identify_failed = 0; + char* filename= (char *)"movie_graph.txt"; + int n_ep=params1[0], n_st=params1[1], n_rep=params1[2]; + + //............................... + + for(int i=0; i 0){ + + Results = new double[n_rep]; + Counters = new int[n_rep]; + + }else { + + Results = NULL; + Counters = NULL; + std::cout << " Number of reruns should be positive! " << std::endl; + return 0; + + } + //............................... + //srand(time(0)); + //srandomdev(); + + if ( initSeed != 0.0 ) { + srand(initSeed); + } + else { + timeval t; + gettimeofday(&t, NULL); + srand(t.tv_usec); + } + + { + double r=0; + for (int j=0; j<100; j++){ + + + + r = rand()/(double)(pow(2.,31)-1.); + std::cout << r << " "; + } + std::cout << "\n"; + } + //random initiation of starting parameters + + if (range > 0.){ + + for (int i=0; i < 5; i++){ + + if (params0[i]==-100.){ + + double r1 = (rand()/(double)(pow(2.,31)-1.)); + double r2 = (rand()/(double)(pow(2.,31)-1.)); + double sign = 1.; + + if(r1 > 0.5){ + sign=-1.; + } + + params0[i] = sign*r2*range; + + std::cout << par_names0[i] << ": " << params0[i] << std::endl; + } + } + + } + + + double T_start=params2[0], T_end=params2[1], Target_rejection=params2[3], starting_jump=params2[4]; + int Annealing_repeats = (int) params2[2]; + +#ifdef P_DISPATCH + dispatch_group_t group = dispatch_group_create(); + + //............................. + multi_annealing(group, un, CustomQueues, T_start, T_end, Target_rejection, Annealing_repeats, + starting_jump, Results, Counters, params0, Annealing_repeats); + + //dispatch_group_wait(group, DISPATCH_TIME_FOREVER); + dispatch_release(group); + //............................. +#else + multi_annealing( un, T_start, T_end, Target_rejection, Annealing_repeats, + starting_jump, Results, Counters, params0, Annealing_repeats); +#endif + + + // stop timer + gettimeofday(&t2, NULL); + + // compute and print the elapsed time in millisec + elapsedTime = (t2.tv_sec - t1.tv_sec) * 1000.0; // sec to ms + elapsedTime += (t2.tv_usec - t1.tv_usec) / 1000.0; // us to ms + elapsedTime /= 1000.; + cout << "\n*** optimizer completed, elapsed time=" << elapsedTime << " seconds " << elapsedTime/60. << " minutes)\n\n"; + + //..................... + + for(int i=0; i 0){ + + delete [] Results; + delete [] Counters; + + } + + return 0; + + + +} + Added: SwiftApps/SciColSim/testgraph.py =================================================================== --- SwiftApps/SciColSim/testgraph.py (rev 0) +++ SwiftApps/SciColSim/testgraph.py 2012-10-27 14:31:40 UTC (rev 5997) @@ -0,0 +1,223 @@ +#! /usr/bin/env python + +import os, sys + +unbuffered = os.fdopen(sys.stdout.fileno(), 'w', 0) +sys.stdout = unbuffered + +app = "./orig-optimizer"; # For Mac only: original code (+1 loop fix) using Grnd Central Dispatch +app = "./dispatch-optimizer"; # For Mac only: sing Grand Central Dispatch +app = "./openmp-optimizer"; # For Mac or Linux: Using OpenMP (Default) + +#paramset="default" +paramset="mw3" +#paramset="VERYFASTTEST" +#paramset="VERYFASTTEST" + +startTarget = int(sys.argv[1]) +print("Running with startTarget " + str(startTarget) + "\n\n"); + +print("Running with parameter set " + paramset + "\n\n"); + +if paramset == "beta": + startTarget = 58 + endTarget = 59 + incrTarget = 50 + optimizerRepeats = 100 + evolveReruns = 32 + annealingSteps = 100 + NWorkers = "8" + seed = "" + openmp = "OMP_NUM_THREADS=" + NWorkers + operation = "n" + + +elif paramset == "default": + + # Parameter names and optimizer command line positions + + startTarget = 58 # target (target innovation) + endTarget = 1009 # " + incrTarget = 50 # " + optimizerRepeats = 15 # wrapper control var (# of repeats of optimizer to do for each target loss) + evolveReruns = 1000 # n_reruns = + annealingSteps = 100 + +# Following params are for modified optimizer openmp-optimizer + +# NWorkers = "2" +# operation = "n" # n=normal, m=manual (runs 1 multi_loss call) +# seed = "1234567" +# openmp = "OMP_NUM_THREADS=24" + + NWorkers = "" + operation = "" + seed = "" + openmp = "" + +elif paramset == "FULLSCALE": # FULL INITIAL TEST PARAMETERS from Andrey + + startTarget = 58 + endTarget = 1009 + incrTarget = 50 + optimizerRepeats = 15 + evolveReruns = 1000 + annealingSteps = 100 + +elif paramset == "FASTTEST": # FAST TEST PARAMETERS by Mike + + startTarget = 58 + endTarget = 209 + incrTarget = 50 + optimizerRepeats = 2 + evolveReruns = 10 + annealingSteps = 2 + NWorkers = "2" + openmp = "OMP_NUM_THREADS=" + NWorkers + operation = "n" # n=normal, m=manual (runs 1 multi_loss call) + seed = "1234567" + +elif paramset == "VERYFASTTEST": #VERY FAST TEST PARAMETERS by Ketan + + startTarget = 58 + endTarget = 209 + incrTarget = 50 + optimizerRepeats = 2 + evolveReruns = 1 + annealingSteps = 2 + NWorkers = "1" + openmp = "OMP_NUM_THREADS=" + NWorkers + operation = "n" # n=normal, m=manual (runs 1 multi_loss call) + seed = "1234567" + +elif paramset == "MANUALTEST": # for manual (one multi_loss call) mode + + startTarget = 258 # runtime increases with greater target + endTarget = 259 + incrTarget = 50 + optimizerRepeats = 1 + evolveReruns = 60 + annealingSteps = 5 + NWorkers = "4" + openmp = "OMP_NUM_THREADS=" + NWorkers + operation = "m" # n=normal, m=manual (runs 1 multi_loss call) + seed = "1234567" + +elif paramset == "mw": # FAST TEST PARAMETERS by Mike + + startTarget = 58 + endTarget = 59 + incrTarget = 50 + optimizerRepeats = 1 + evolveReruns = 100 + annealingSteps = 5 + NWorkers = "4" + openmp = "OMP_NUM_THREADS=" + NWorkers + operation = "n" # n=normal, m=manual (runs 1 multi_loss call) + seed = "1234567" + app = "./openmp-optimizer"; + +elif paramset == "mw2": # Timing TEST PARAMETERS by Mike + + startTarget = 58 + endTarget = 209 + incrTarget = 50 + optimizerRepeats = 1 + evolveReruns = 240 + annealingSteps = 1 + NWorkers = "8" + openmp = "OMP_NUM_THREADS=" + NWorkers + operation = "m" # n=normal, m=manual (runs 1 multi_loss call) + seed = "123456" + app = "./openmp-optimizer"; + +elif paramset == "mw3": # Timing TEST PARAMETERS by Mike + + # startTarget = 9800 + endTarget = startTarget+1 + incrTarget = 50 + optimizerRepeats = 1 + evolveReruns = 1 + annealingSteps = 1 + NWorkers = "8" + openmp = "OMP_NUM_THREADS=" + NWorkers + operation = "m" # n=normal, m=manual (runs 1 multi_loss call) + seed = "123456" + app = "./openmp-optimizer"; + +# Ensure we dont pass new parameters to original optimizer versions +# (but they would be ignored) + +if app == "./orig-optimizer": + NWorkers = "" + operation = "" + seed = "" + +# Run the test + +for target in range(startTarget,endTarget,incrTarget): + for i in range(optimizerRepeats): + args = "rm -f bestdb.txt; " + \ + openmp + " " + app + " 0 0 0 0 0 " + str(target) + " 40000 20 " + str(evolveReruns) + \ + " 2 1 2. 0.01 " + str(annealingSteps) + " 0.3 2.3 0 0 0 0 0 " + operation + " " + NWorkers + " " + seed + \ + " # >& out.T"+str(target)+".i"+str(i) + "; #mv bestdb.txt best.T" + str(target) + ".i" + str(i) + print("\n**** Calling optimizer: "+args+"\n") + os.system(args); + +print sys.argv[0] + " Done!" + +# openmp + " " + app + " 0 0 4 50 -1 " + str(target) + " 40000 20 " + str(evolveReruns) + \ + +### Argument worksheet: + +# string fastargs1[] = [ +# "0", "0", "4", "50", "-1", @strcat(target), +# "40000", "20", "1000", "2", +# "1", +# "2.", "0.01", "100", "0.3", "2.3", +# "1", "1", "0", "0", "0"]; +# string fastargs2[] = [ +# "0", "0", "4", "50", "-1", @strcat(target), +# "40000", "20", "1000", "2", +# "1", +# "2.", "0.01", "5", "0.3", "2.3", +# "1", "1", "0", "0", "0", "m"]; +# string fastargs3[] = [ +# "0", "0", "4", "50", "-1", @strcat(target), +# "40000", "20", @strcat(repeats), "2", +# "1", +# "2.", "0.01", "2", "0.3", "2.3", +# "1", "1", "0", "0", "0", "m"]; +# +# +# long running args = "0 0 4 50 -1 target | 40000 20 1000 | 2 | 1 | 2. 0.01 100 0.3 2.3 | 1 1 0 0 0 | m 24? 12345 +# +# [ alpha_i alpha_m beta gamma delta target_innov ] +# 0 0 4 50 -1 target +# +# [ n_epochs n_steps n_reruns ] [ range ] [ verbose_level] +# 40000 20 1000 2 1 +# +# [ T_start T_end Annealing_steps Target_rejection Starting_jump ] +# 2. 0.01 100 0.3 2.3 +# +# [ FREEZE_alpha_i FREEZE_alpha_m FREEZE_beta FREEZE_gamma FREEZE_delta ] +# 1 1 0 0 0 +# +# [ operation-code:(n,m) Nworkers seed ] +# n 24 12345 +# + +# From older test, using separate file per run: +# args = app + " 0 0 4 50 -1 " + str(target) + " 40000 20 " + str(evolveReruns) + \ +# " 2 1 2. 0.01 " + str(annealingSteps) + " 0.3 2.3 1 1 0 0 0 > out.T"+str(target)+".i"+str(i) + +### Optimization loop logic: + +# a) 20 targets (parallel) +# b) 15 repeats (parallel) of optimizer (optimizer == multi_annealing): +# 1 multi_loss to initialize +# c) 100 Annealing_cycles (serial) +# d) 3 repeats (1 per non-fixed param, serial) of multi_loss: +# e) 1000 to 10000 evolve_reruns (parallel) == multi_loss +# f) evolve() Property changes on: SwiftApps/SciColSim/testgraph.py ___________________________________________________________________ Added: svn:executable + * From swift at ci.uchicago.edu Sat Oct 27 18:00:39 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Sat, 27 Oct 2012 18:00:39 -0500 (CDT) Subject: [Swift-commit] cog r3490 Message-ID: <20121027230040.63D198D0008D@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3490 | hategan | 2012-10-27 17:57:34 -0500 (Sat, 27 Oct 2012) | 1 line fixed deadlock in shutdown ------------------------------------------------------------------------ Index: modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Block.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Block.java (revision 3489) +++ modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Block.java (working copy) @@ -256,54 +256,57 @@ } public void shutdown(boolean now) { + List cpusToShutDown; synchronized (cpus) { if (shutdown) { return; } - logger.info("Shutting down block " + this); - bqp.getRLogger().log("BLOCK_SHUTDOWN id=" + getId()); shutdown = true; - long busyTotal = 0; - long idleTotal = 0; - int count = 0; - if (running) { - for (Cpu cpu : cpus) { - idleTotal = cpu.idleTime; - busyTotal = cpu.busyTime; - if (!failed) { - cpu.shutdown(); - } - count++; + cpusToShutDown = new ArrayList(cpus); + cpus.clear(); + } + logger.info("Shutting down block " + this); + bqp.getRLogger().log("BLOCK_SHUTDOWN id=" + getId()); + + long busyTotal = 0; + long idleTotal = 0; + int count = 0; + if (running) { + for (Cpu cpu : cpusToShutDown) { + idleTotal = cpu.idleTime; + busyTotal = cpu.busyTime; + if (!failed) { + cpu.shutdown(); } - if (!failed) { - if (count < workers || now) { - if (logger.isInfoEnabled()) { - logger.info("Adding short shutdown watchdog: count = " + - count + ", workers = " + workers + ", now = " + now); - } - addForcedShutdownWatchdog(100); - } - else { - if (logger.isInfoEnabled()) { - logger.info("Adding normal shutdown watchdog"); - } - addForcedShutdownWatchdog(SHUTDOWN_WATCHDOG_DELAY); - } + count++; + } + if (!failed) { + if (count < workers || now) { + if (logger.isInfoEnabled()) { + logger.info("Adding short shutdown watchdog: count = " + + count + ", workers = " + workers + ", now = " + now); + } + addForcedShutdownWatchdog(100); + } + else { + if (logger.isInfoEnabled()) { + logger.info("Adding normal shutdown watchdog"); + } + addForcedShutdownWatchdog(SHUTDOWN_WATCHDOG_DELAY); } + } - if (idleTotal > 0) { - double u = (busyTotal * 10000) / (busyTotal + idleTotal); - u /= 100; - logger.info("Average utilization: " + u + "%"); - bqp.getRLogger().log("BLOCK_UTILIZATION id=" + getId() + ", u=" + u); - } + if (idleTotal > 0) { + double u = (busyTotal * 10000) / (busyTotal + idleTotal); + u /= 100; + logger.info("Average utilization: " + u + "%"); + bqp.getRLogger().log("BLOCK_UTILIZATION id=" + getId() + ", u=" + u); } - else { - logger.info("Block " + this + " not running. Cancelling job."); - forceShutdown(); - } - cpus.clear(); } + else { + logger.info("Block " + this + " not running. Cancelling job."); + forceShutdown(); + } } private void addForcedShutdownWatchdog(long delay) { From swift at ci.uchicago.edu Sun Oct 28 21:05:16 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Sun, 28 Oct 2012 21:05:16 -0500 (CDT) Subject: [Swift-commit] cog r3491 Message-ID: <20121029020516.8FB418D0009A@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3491 | hategan | 2012-10-28 21:01:32 -0500 (Sun, 28 Oct 2012) | 1 line create file appender if one does not exist instead of failing ------------------------------------------------------------------------ Index: modules/util/src/org/globus/cog/util/Misc.java =================================================================== --- modules/util/src/org/globus/cog/util/Misc.java (revision 3490) +++ modules/util/src/org/globus/cog/util/Misc.java (working copy) @@ -12,13 +12,17 @@ import org.apache.log4j.Appender; import org.apache.log4j.FileAppender; +import org.apache.log4j.Layout; +import org.apache.log4j.Level; import org.apache.log4j.Logger; +import org.apache.log4j.PatternLayout; public class Misc { /** - Set the FileAppender output from the given Logger - to the given file name + * Set the FileAppender output from the given Logger + * to the given file name. If no file appender exists, one + * is created. */ public static void setFileAppenderOutput(Logger logger, String filename) { @@ -26,8 +30,11 @@ FileAppender fa = (FileAppender) Misc.getAppender(FileAppender.class); if (fa == null) { - logger.fatal("Failed to configure log file name"); - System.exit(2); + fa = new FileAppender(); + Layout l = new PatternLayout("%d{yyyy-MM-dd HH:mm:ss,SSSZZZZZ} %-5p %c{1} %m%n"); + fa.setLayout(l); + fa.setThreshold(Level.DEBUG); + Logger.getRootLogger().addAppender(fa); } fa.setFile(file.getAbsolutePath()); fa.activateOptions(); From swift at ci.uchicago.edu Sun Oct 28 21:35:10 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Sun, 28 Oct 2012 21:35:10 -0500 (CDT) Subject: [Swift-commit] cog r3494 Message-ID: <20121029023510.D5E918D0009A@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3494 | hategan | 2012-10-28 21:33:33 -0500 (Sun, 28 Oct 2012) | 1 line added command line client ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/configure.ac =================================================================== --- modules/provider-coaster-c-client/configure.ac (revision 3493) +++ modules/provider-coaster-c-client/configure.ac (working copy) @@ -23,7 +23,7 @@ CXXFLAGS= # Checks for header files. -AC_CHECK_HEADERS([arpa/inet.h fcntl.h netdb.h netinet/in.h stdlib.h string.h sys/socket.h sys/time.h unistd.h]) +AC_CHECK_HEADERS([arpa/inet.h fcntl.h netdb.h netinet/in.h stdlib.h string.h sys/socket.h sys/time.h unistd.h getopt.h]) # Libs AC_CHECK_LIB(pthread, pthread_create) Index: modules/provider-coaster-c-client/.autotools =================================================================== --- modules/provider-coaster-c-client/.autotools (revision 3493) +++ modules/provider-coaster-c-client/.autotools (working copy) @@ -30,11 +30,6 @@