From wilde at ci.uchicago.edu Sat Nov 3 15:51:23 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Sat, 3 Nov 2012 15:51:23 -0500 (CDT) Subject: [Swift-commit] r6004 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121103205123.3998E178884@svn.ci.uchicago.edu> Author: wilde Date: 2012-11-03 15:51:22 -0500 (Sat, 03 Nov 2012) New Revision: 6004 Modified: SwiftApps/ParVis/HiRAMTools/README Log: Update pfrepp inststructions. Modified: SwiftApps/ParVis/HiRAMTools/README =================================================================== --- SwiftApps/ParVis/HiRAMTools/README 2012-10-30 19:50:17 UTC (rev 6003) +++ SwiftApps/ParVis/HiRAMTools/README 2012-11-03 20:51:22 UTC (rev 6004) @@ -97,39 +97,48 @@ etc --- +The line "path id" should be entered exactly like that; these are headers for a swift +read() function. + For example, file "pflist" contains: ---- Next line is first line of file. "---" is not in file: -path id -/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run001/run001/ en1eo14Ic2 -/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run002/run003/ en1eo14Ic3 -/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run003/run004/ en1eo14Ic4 -/intrepid-fs0/users/lzamboni/persistent/yearly-nco/many-en1/run004/run005/ en1eo16Ic1 +--- Next line is first line of file. "---" is not in file, 5 lines total: +path id +/intrepid-fs0/projects/SU_Climate/post/data/annual/en1eo12Ic1 en1eo12Ic1 +/intrepid-fs0/projects/SU_Climate/post/data/annual/en1eo12Ic2 en1eo12Ic2 +/intrepid-fs0/projects/SU_Climate/post/data/annual/en1eo12Ic3 en1eo12Ic3 +/intrepid-fs0/projects/SU_Climate/post/data/annual/en1eo12Ic4 en1eo12Ic4 --- 2. Run the script: - runpfrepps.sh pflistFile outputDir + # syntax: runpfrepps.sh pflistFile outputDir # eg: - $HOME/HiRAMTools/runpfrepps.sh pflist /intrepid-fs0/users/wilde/persistent/LZ/pfrepps.2012.0630 + $HOME/HiRAMTools/runpfrepps.sh pflist /intrepid-fs0/users/wilde/persistent/LZ/pfrepps 3. Watch the status cd run012 tail -f swift.out -4. Manual, local execution of pfrepp atmos average scripts (generated from step 3, above) +4. Then do manual, local (i.e., interactive-node) execution of pfrepp atmos + average scripts (generated from step 3, above) -cd /intrepid-fs0/users/wilde/persistent/LZ/pfrepps + ssh login2 # (or do a qsubi to get a dedicated interactive node) -# start scripts here ??? + cd /intrepid-fs0/users/wilde/persistent/LZ/pfrepps -# show script progress: + # start scripts: -for s in $(cat set02); do echo "$s: "; tail -15 $s.out; done + for f in $(cat set06); do $f >& $f.out & done + # "monitor" script progress: + for s in $(cat set02); do echo "$s: "; tail -15 $s.out; done +--- + + OPEN ISSUES: - error handling in the leaf scripts is highly suspect: are errors correctly From wilde at ci.uchicago.edu Sun Nov 4 21:42:30 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Sun, 4 Nov 2012 21:42:30 -0600 (CST) Subject: [Swift-commit] r6005 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121105034230.B70919CCE6@svn.ci.uchicago.edu> Author: wilde Date: 2012-11-04 21:42:29 -0600 (Sun, 04 Nov 2012) New Revision: 6005 Added: SwiftApps/ParVis/HiRAMTools/genmap.sh Log: Add experimental generic ext mapper script. Not yet used here; consider moving to Swift release. Added: SwiftApps/ParVis/HiRAMTools/genmap.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/genmap.sh (rev 0) +++ SwiftApps/ParVis/HiRAMTools/genmap.sh 2012-11-05 03:42:29 UTC (rev 6005) @@ -0,0 +1,5 @@ +#! /bin/sh + +script=$2 + +sh -c "$script" | awk '{ printf("[%d] %s\n",NR-1,$0) }' Property changes on: SwiftApps/ParVis/HiRAMTools/genmap.sh ___________________________________________________________________ Added: svn:executable + * From wilde at ci.uchicago.edu Sun Nov 4 21:44:51 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Sun, 4 Nov 2012 21:44:51 -0600 (CST) Subject: [Swift-commit] r6006 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121105034451.7AB009CCE6@svn.ci.uchicago.edu> Author: wilde Date: 2012-11-04 21:44:51 -0600 (Sun, 04 Nov 2012) New Revision: 6006 Modified: SwiftApps/ParVis/HiRAMTools/README SwiftApps/ParVis/HiRAMTools/genpfrepps.sh SwiftApps/ParVis/HiRAMTools/pfrepps.swift SwiftApps/ParVis/HiRAMTools/runpfrepps.sh Log: Update pfrepp scripts and README. Now supports running the generated pfrepp scripts for N realizations in parallel, from a single runpfrepps.sh invocation and a single swift script. The set of generated scripts to run is selectable with a shell scriptlet passed as an argument. Modified: SwiftApps/ParVis/HiRAMTools/README =================================================================== --- SwiftApps/ParVis/HiRAMTools/README 2012-11-05 03:42:29 UTC (rev 6005) +++ SwiftApps/ParVis/HiRAMTools/README 2012-11-05 03:44:51 UTC (rev 6006) @@ -86,6 +86,8 @@ === +=== OLD PFREPP + To run pfrep on a set of realizations, do: 1. Create a list of realizations in a file, starting with a header line, in @@ -138,7 +140,84 @@ --- +=== NEW PFREPP: +To run pfrep on a set of realizations, do: + +1. Create a list of realizations in a file, starting with a header line, in + the following format: + +--- file pflist --- (next line is first line of file): +path id +/full/path/of/annualized_realization/history_dir realizationID +etc +--- + +The header (first) line "path id" should be entered exactly like that; these +are headers for a swift read() function. Extra spaces are allowed within this +header line. + +For example, to run 4 realizations through pfrepp, file "pflist" could contain +the lines between the --- lines: + +--- Next line is first line of file. "---" is not in file, 5 lines total: +path id +/intrepid-fs0/projects/SU_Climate/post/data/annual/en1eo12Ic1 en1eo12Ic1 +/intrepid-fs0/projects/SU_Climate/post/data/annual/en1eo12Ic2 en1eo12Ic2 +/intrepid-fs0/projects/SU_Climate/post/data/annual/en1eo12Ic3 en1eo12Ic3 +/intrepid-fs0/projects/SU_Climate/post/data/annual/en1eo12Ic4 en1eo12Ic4 +--- + +2. Run the script: + + mkdir /intrepid-fs0/projects/SU_Climate/post/data/pfrepp/swift + cd /intrepid-fs0/projects/SU_Climate/post/data/pfrepp/swift + emacs pfset01 # per above format + htools=/intrepid-fs0/projects/SU_Climate/post/swift/HiRAMTools + + # syntax: runpfrepps.sh pflistFile outputDir optionalScriptlet optionalRunType # eg: + + # Dryrun, using 6 atmos scripts: + + $htools/runpfrepps.sh pfset01 /intrepid-fs0/users/wilde/persistent/LZ/pfrepps "" dryrun + + # Real run, using 6 atmos scripts: + + $htools/runpfrepps.sh pfset01 /intrepid-fs0/users/wilde/persistent/LZ/pfrepps "" realrun + + # Real run, using 5 atmos scripts, omitting the (currently) long daily run: + + $htools/runpfrepps.sh pfset01 /intrepid-fs0/users/wilde/persistent/LZ/pfrepps \ + "grep pfrepp_script.atmos_* | grep -v atmos_daily" realrun + + # Real run, doing 3 specific atmos scripts: + + $htools/runpfrepps.sh pfset01 /intrepid-fs0/users/wilde/persistent/LZ/pfrepps \ + "egrep '4xdaily|8xdaily.time|8xdaily_instant'" realrun # FIXME: verify that '' works! + + +3. Watch the status + + cd run012 + tail -f swift.out + +4. Examine script output: + + # Generated scripts + + /intrepid-fs0/projects/SU_Climate/post/data/pfrepp/en1eo12Ic1/scripts + + # pfrepp output (summary) data: + + /intrepid-fs0/projects/SU_Climate/post/data/pfrepp/en1eo12Ic1/atmos/{ts,???} + + +5. (Manually) promote the accepted output to its final position in the campaign directories: + + + +--- + OPEN ISSUES: - error handling in the leaf scripts is highly suspect: are errors correctly Modified: SwiftApps/ParVis/HiRAMTools/genpfrepps.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/genpfrepps.sh 2012-11-05 03:42:29 UTC (rev 6005) +++ SwiftApps/ParVis/HiRAMTools/genpfrepps.sh 2012-11-05 03:44:51 UTC (rev 6006) @@ -4,6 +4,7 @@ realdir=$2 outdir=$3 scriptlist=$PWD/$4 +scriptfilter=$5 mkdir -p $outdir/$realid/scripts cd $outdir/$realid/scripts @@ -18,10 +19,41 @@ -m 12 -n 50 -q \ -u +echo $0: scriptfilter=$scriptfilter + +if [ "_$scriptfilter" != _ ]; then + echo $0: using scriptfilter + scripts=$(/bin/ls -1 pfrepp_script* | sh -c "$scriptfilter" ) +else + scripts=$(/bin/ls -1 pfrepp_script* ) +fi + echo id path >$scriptlist -for f in pfrepp_script.*; do - id=$(echo $f | sed -e 's/pfrepp_script.//') - echo $id $PWD/$f >>$scriptlist +for f in $scripts; do + echo $realid $f >>$scriptlist done +exit + # -a /intrepid-fs0/users/lzamboni/persistent/yearly-nco/en1eo12Ic3/run001/run001/ \ + +# datasets needed by LZ: + +Pfrepp-month.time_average +Pfrepp-month.time_series +Pfrepp-daily.time_series (4X???) +Pfrepp-8xdaily.time_series +Pfrepp-8xdaily_instant.time_series +Pfrepp-4xdaily.time_series + +which means: + +pfrepp_script.atmos_month.time_average.1950* +pfrepp_script.atmos_month.time_series.1950* +pfrepp_script.atmos_4xdaily.time_series.1950* +pfrepp_script.atmos_8xdaily.time_series.1950* +pfrepp_script.atmos_8xdaily_instant.time_series.1950* +pfrepp_script.atmos_daily.time_series.1950* + +pfrepp_script.ice_month.time_average.1950* +pfrepp_script.ice_month.time_series.1950* Modified: SwiftApps/ParVis/HiRAMTools/pfrepps.swift =================================================================== --- SwiftApps/ParVis/HiRAMTools/pfrepps.swift 2012-11-05 03:42:29 UTC (rev 6005) +++ SwiftApps/ParVis/HiRAMTools/pfrepps.swift 2012-11-05 03:44:51 UTC (rev 6006) @@ -10,12 +10,14 @@ string id; } -global string outdir = @arg("outdir","/intrepid-fs0/users/wilde/LZ/pfrepps"); -string reallist = @arg("reallist","NO-REALIZATION-LIST-SPECIFIED"); +global string outdir = @arg("outdir", ""); + string reallist = @arg("reallist", ""); + string scriptfilter = @arg("scriptfilter","grep pfrepp_script.atmos_|grep -v atmos_daily.SKIPME"); + string dryrun = @arg("dryrun","true"); # default to doing a "dry run" -app (file scripts, file std_out, file std_err) genpfrepps (string realid, string realpath) +app (file scripts, file std_out, file std_err) genpfrepps (string realid, string realpath, string filter) { - genpfrepps realid realpath outdir @scripts stdout=@std_out stderr=@std_err; + genpfrepps realid realpath outdir @scripts filter stdout=@std_out stderr=@std_err; } app (file std_out, file std_err) sh (string script) @@ -23,19 +25,36 @@ sh "-c" script stdout=@std_out stderr=@std_err; } -realization reali[] = readData(reallist); +if(outdir == "") { + tracef("pfrepps.swift: ERROR: output directory (swift argument outdir) not specified."); +} -foreach r,i in reali { - tracef("%s %s\n", r.id, r.path); - file sofile ; - file sefile ; - file scriptlist ; - (scriptlist,sofile,sefile) = genpfrepps(r.id, r.path); - script scripts[] = readData(scriptlist); - foreach s in scripts { - file ssofile ; - file ssefile ; - // (ssofile, ssefile) = sh(s.path); - tracef("Running script: %s: %s\n", s.id, s.path); +if(reallist == "") { + tracef("pfrepps.swift: ERROR: realization list (swift argument reallist) not specified"); +} + +if(reallist != "" && outdir != "") { + realization reali[] = readData(reallist); + foreach r,i in reali { + tracef("%s %s\n", r.id, r.path); + file sofile ; + file sefile ; + file scriptlist ; + (scriptlist,sofile,sefile) = genpfrepps(r.id, r.path, scriptfilter); + script scripts[] = readData(scriptlist); + foreach s in scripts { + string cmd = @strcat(outdir,"/",r.id,"/scripts/",s.path); + file ssofile ; + file ssefile ; + if( dryrun != "true" ) { + (ssofile, ssefile) = sh(cmd); + } + else { + tracef("Running script: %s: %s\n", s.id, s.path); + } + } } } +else { + tracef("pfrepps.swift: EXIT: exiting due to argument errors."); +} \ No newline at end of file Modified: SwiftApps/ParVis/HiRAMTools/runpfrepps.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/runpfrepps.sh 2012-11-05 03:42:29 UTC (rev 6005) +++ SwiftApps/ParVis/HiRAMTools/runpfrepps.sh 2012-11-05 03:44:51 UTC (rev 6006) @@ -1,17 +1,28 @@ #! /bin/bash reallist=$1 # full pathname of list of realizations to pfrepp + # reallist must have format: + # path id (with this header line) + outdir=$2 # full pathname of output dir -# reallist must have format: -# path id (with this header line) + # scriptlet to filter lists of generated pfrepp scripts to run: +scriptfilter=${3:-"grep pfrepp_script.atmos_ | grep -v atmos_daily"} +runtype=${4:-dryrun} # User should specify dryrun | realrun for this arg + +if [ $runtype = dryrun ]; then + dryrun=true +else # dryrun=anything else requests generated scripts to be run ("real run") + dryrun=false +fi + bindir=$(dirname $0) bindir=$(cd $bindir; pwd) echo bindir=$bindir -# Create new runNNN directory +# Create new runNNN directory below current working dir where this script was run from rundir=$( echo run??? | sed -e 's/^.*run//' | awk '{ printf("run%03d\n", $1+1)}' ) mkdir $rundir @@ -20,8 +31,9 @@ wrapperlog.always.transfer=true sitedir.keep=true -execution.retries=10 -lazy.errors=true +execution.retries=0 +lazy.errors=false +use.wrapper.staging=false END @@ -34,7 +46,7 @@ .01 10000 - $PWD/swiftwork + $PWD/$rundir @@ -51,12 +63,12 @@ 32 28800 - 00:20:00 + 07:45:00 100 100 1 1 - $PWD/$rundir/swiftwork + $PWD/$rundir @@ -76,4 +88,14 @@ echo Running in directory $rundir runid=$(basename $rundir) -swift -config cf -tc.file tc -sites.file sites.xml pfrepps.swift -outdir=$outdir -reallist=$reallist >& swift.out +echo RUNNING: swift -config cf -tc.file tc -sites.file sites.xml pfrepps.swift \ + -outdir=$outdir \ + -reallist=$reallist \ + -scriptfilter="$scriptfilter" \ + -dryrun=$dryrun + +swift -config cf -tc.file tc -sites.file sites.xml pfrepps.swift >& swift.out \ + -outdir=$outdir \ + -reallist=$reallist \ + -scriptfilter="$scriptfilter" \ + -dryrun=$dryrun From wilde at ci.uchicago.edu Tue Nov 6 19:46:18 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Tue, 6 Nov 2012 19:46:18 -0600 (CST) Subject: [Swift-commit] r6008 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121107014618.188099D10A@svn.ci.uchicago.edu> Author: wilde Date: 2012-11-06 19:46:17 -0600 (Tue, 06 Nov 2012) New Revision: 6008 Modified: SwiftApps/ParVis/HiRAMTools/combine_realization.sh SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh Log: Add use.wrapper.staging parameter for latest Swift trunk rev. Modified: SwiftApps/ParVis/HiRAMTools/combine_realization.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/combine_realization.sh 2012-11-06 21:17:42 UTC (rev 6007) +++ SwiftApps/ParVis/HiRAMTools/combine_realization.sh 2012-11-07 01:46:17 UTC (rev 6008) @@ -41,6 +41,7 @@ sitedir.keep=true execution.retries=10 lazy.errors=true +use.wrapper.staging=false END Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh 2012-11-06 21:17:42 UTC (rev 6007) +++ SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh 2012-11-07 01:46:17 UTC (rev 6008) @@ -45,6 +45,7 @@ execution.retries=10 lazy.errors=true status.mode=provider +use.wrapper.staging=false END From swift at ci.uchicago.edu Wed Nov 7 15:10:09 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Wed, 7 Nov 2012 15:10:09 -0600 (CST) Subject: [Swift-commit] cog r3498 Message-ID: <20121107211009.6D6F18D0009C@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3498 | davidkelly999 | 2012-11-07 15:05:14 -0600 (Wed, 07 Nov 2012) | 2 lines Initial commit of LSF provider ------------------------------------------------------------------------ Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/lsf/QueuePoller.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/lsf/QueuePoller.java (revision 0) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/lsf/QueuePoller.java (revision 3498) @@ -0,0 +1,175 @@ +//---------------------------------------------------------------------- +//This code is developed as part of the Java CoG Kit project +//The terms of the license can be found at http://www.cogkit.org/license +//This message may not be removed or altered. +//---------------------------------------------------------------------- + +package org.globus.cog.abstraction.impl.scheduler.lsf; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.io.StringReader; +import java.util.HashSet; +import java.util.Iterator; +import java.util.Map; +import java.util.Set; + +import org.apache.log4j.Logger; +import org.globus.cog.abstraction.impl.scheduler.common.AbstractProperties; +import org.globus.cog.abstraction.impl.scheduler.common.AbstractQueuePoller; +import org.globus.cog.abstraction.impl.scheduler.common.Job; + +public class QueuePoller extends AbstractQueuePoller { + public static final Logger logger = Logger.getLogger(QueuePoller.class); + public static final int FULL_LIST_THRESHOLD = 16; + + private Set processed; + + public QueuePoller(AbstractProperties properties) { + super("PBS provider queue poller", properties); + processed = new HashSet(); + } + + private static String[] CMDARRAY; + + protected synchronized String[] getCMDArray() { + if (getJobs().size() <= FULL_LIST_THRESHOLD) { + String[] cmda = new String[2 + getJobs().size()]; + cmda[0] = getProperties().getPollCommand(); + cmda[1] = "-f"; + int i = 2; + for (Job j : getJobs().values()) { + cmda[i++] = j.getJobID(); + } + return cmda; + } + else { + if (CMDARRAY == null) { + CMDARRAY = new String[] { getProperties().getPollCommand(), "-f" }; + } + } + return CMDARRAY; + } + + @Override + protected int getError(int ec, String stderr) { + if (ec != 0) { + BufferedReader sr = new BufferedReader(new StringReader(stderr)); + try { + String line = sr.readLine(); + while (line != null) { + if (!line.contains("Unknown Job Id")) { + return ec; + } + line = sr.readLine(); + } + } + catch (IOException e) { + // should not occur while reading from a string reader + e.printStackTrace(); + } + return 0; + } + else { + return ec; + } + } + + protected void processStdout(InputStream is) throws IOException { + BufferedReader br = new BufferedReader(new InputStreamReader(is)); + processed.clear(); + String line; + String currentJobID = null; + Job currentJob = null; + do { + line = br.readLine(); + if (line != null) { + try { + line = line.trim(); + if (line.startsWith("Job Id: ")) { + currentJobID = line.substring("Job Id: ".length()); + processed.add(currentJobID); + currentJob = getJob(currentJobID); + continue; + } + if (currentJob != null) { + if (line.startsWith("job_state = ")) { + if (logger.isDebugEnabled()) { + logger.debug("Status line: " + line); + } + switch (line.substring("job_state = ".length()) + .charAt(0)) { + case 'Q': { + if (logger.isDebugEnabled()) { + logger.debug("Status for " + + currentJobID + " is Q"); + } + currentJob.setState(Job.STATE_QUEUED); + break; + } + case 'R': { + if (logger.isDebugEnabled()) { + logger.debug("Status for " + + currentJobID + " is R"); + } + currentJob.setState(Job.STATE_RUNNING); + break; + } + case 'C': { + // for sites where keep_completed is there, + // don't wait + // for the job to be removed from the queue + if (logger.isDebugEnabled()) { + logger.debug("Status for " + + currentJobID + " is C"); + } + addDoneJob(currentJob.getJobID()); + break; + } + } + } + else if (line.startsWith("exit_status = ")) { + try { + int ec = Integer.parseInt(line.substring( + "exit_status = ".length()).trim()); + currentJob.setExitcode(ec); + } + catch (Exception e) { + if (logger.isDebugEnabled()) { + logger.debug("Could not parse exit_status", + e); + } + } + } + } + } + catch (Exception e) { + logger.warn("Exception caught while handling " + + getProperties().getPollCommandName() + + " output: " + line, e); + } + } + } while (line != null); + Iterator i = getJobs().entrySet().iterator(); + while (i.hasNext()) { + Map.Entry e = (Map.Entry) i.next(); + String id = (String) e.getKey(); + if (!processed.contains(id)) { + Job job = (Job) e.getValue(); + if (logger.isDebugEnabled()) { + logger.debug("Status for " + id + " is Done"); + } + job.setState(Job.STATE_DONE); + if (job.getState() == Job.STATE_DONE) { + addDoneJob(id); + } + } + } + } + + protected void processStderr(InputStream is) throws IOException { + } +} + Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/lsf/LSFExecutor.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/lsf/LSFExecutor.java (revision 0) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/lsf/LSFExecutor.java (revision 3498) @@ -0,0 +1,381 @@ +package org.globus.cog.abstraction.impl.scheduler.lsf; + +//---------------------------------------------------------------------- +//This code is developed as part of the Java CoG Kit project +//The terms of the license can be found at http://www.cogkit.org/license +//This message may not be removed or altered. +//---------------------------------------------------------------------- + +import java.io.IOException; +import java.io.Writer; +import java.text.DecimalFormat; +import java.text.NumberFormat; +import java.util.Collection; +import java.util.Date; +import java.util.Iterator; + +import org.apache.log4j.Logger; +import org.globus.cog.abstraction.impl.common.execution.WallTime; +import org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor; +import org.globus.cog.abstraction.impl.scheduler.common.AbstractProperties; +import org.globus.cog.abstraction.impl.scheduler.common.AbstractQueuePoller; +import org.globus.cog.abstraction.impl.scheduler.common.Job; +import org.globus.cog.abstraction.impl.scheduler.common.ProcessListener; +import org.globus.cog.abstraction.interfaces.FileLocation; +import org.globus.cog.abstraction.interfaces.JobSpecification; +import org.globus.cog.abstraction.interfaces.Task; + +public class LSFExecutor extends AbstractExecutor { + public static final Logger logger = Logger.getLogger(LSFExecutor.class); + + + /** + Number of program invocations + */ + int count = 1; + + /** + PBS processes-per-node + */ + int ppn = 1; + + /** + PBS mppdepth: number of available threads per node + */ + int depth = 1; + + /** + Unique number for automatic task names + */ + private static int unique = 0; + + public LSFExecutor(Task task, ProcessListener listener) { + super(task, listener); + } + + private static NumberFormat IDF = new DecimalFormat("000000"); + + /** + The job name is limited to 15 characters: + http://doesciencegrid.org/public/pbs/qsub.html + */ + protected void validate(Task task) { + String name = task.getName(); + if (name == null) { + int i = 0; + synchronized(LSFExecutor.class) { + i = unique++; + } + name = "cog-" + IDF.format(i); + if (logger.isDebugEnabled()) { + logger.debug("PBS name: for: " + task.getIdentity() + + " is: " + name); + } + } + else if (name.length() > 15) { + task.setName(name.substring(0, 15)); + } + } + + /** + Write attribute if non-null + @throws IOException + */ + protected void writeAttr(String attrName, String arg, Writer wr) + throws IOException { + Object value = getSpec().getAttribute(attrName); + if (value != null) { + wr.write("#PBS " + arg + String.valueOf(value) + '\n'); + } + } + + /** + Write attribute if non-null and non-empty + @throws IOException + */ + protected void writeNonEmptyAttr(String attrName, String arg, + Writer wr) + throws IOException { + Object value = getSpec().getAttribute(attrName); + if (value != null) { + String v = String.valueOf(value); + if (v.length() > 0 ) + wr.write("#PBS " + arg + v + '\n'); + } + } + + protected void writeWallTime(Writer wr) throws IOException { + Object walltime = getSpec().getAttribute("maxwalltime"); + if (walltime != null) { + wr.write("#PBS -l walltime=" + + WallTime.normalize(walltime.toString(), "pbs-native") + + '\n'); + } + } + + private int parseAndValidateInt(Object obj, String name) { + try { + assert(obj != null); + return Integer.parseInt(obj.toString()); + } + catch (NumberFormatException e) { + throw new IllegalArgumentException("Illegal value for " + name + ". Must be an integer."); + } + } + + /** + Obtains profile settings regarding job size from + JobSpecification and writes them into the PBS file. + Looks for profiles count, ppn, ppts, and pbs.mpp + count: mandatory, default 1 (number of processes) + depth: default 1 (number of threads per node) + ppn: optional, default 1 (processes per node) + pbs.mpp: output mppwidth/mppnppn instead of nodes/ppn + pbs.properties: extra PBS properties + pbs.resource_list: extra PBS -l line + + Note that the semantics are different for the pbs.mpp setting: + mppwidth is the total number of cores while nodes is the number + of nodes. + + http://www.clusterresources.com/torquedocs/2.1jobsubmission.shtml + @return true if this is a multi-core job + */ + protected boolean writeCountAndPPN(JobSpecification spec, + Writer wr) + throws IOException { + boolean result = false; + + Object o; + + // Number of program invocations + o = getSpec().getAttribute("count"); + if (o != null) + count = parseAndValidateInt(o, "count"); + if (count != 1) + result = true; + + o = spec.getAttribute("ppn"); + if (o != null) + ppn = parseAndValidateInt(o, "ppn"); + + o = spec.getAttribute("depth"); + if (o != null) + depth = parseAndValidateInt(o, "depth"); + + String pbsProperties = + (String) getSpec().getAttribute("lsf.properties"); + + boolean mpp = false; + if (spec.getAttribute("lsf.mpp") != null) + mpp = true; + + StringBuilder sb = new StringBuilder(512); + sb.append("#PBS -l "); + if (mpp) { + sb.append("mppwidth=").append(count); + sb.append(","); + sb.append("mppnppn=").append(ppn); + sb.append(","); + sb.append("mppdepth=").append(depth); + } + else { + sb.append("nodes="); + sb.append(count); + sb.append(":"); + sb.append("ppn="); + sb.append(ppn); + } + + if (pbsProperties != null && + pbsProperties.length() > 0 ) { + sb.append(":"); + sb.append(pbsProperties); + } + + sb.append('\n'); + + wr.write(sb.toString()); + + return result; + } + + /* + private boolean parseAndValidateBool(Object obj, String name) + { + try { + return Boolean.parseBoolean(obj.toString()); + } + catch (NumberFormatException e) { + throw new IllegalArgumentException + ("Illegal value for " + name + ". Must be true/false."); + } + } + */ + + @Override + protected void writeScript(Writer wr, String exitcodefile, String stdout, + String stderr) + throws IOException { + Task task = getTask(); + JobSpecification spec = getSpec(); + Properties properties = Properties.getProperties(); + + getSpec().unpackProviderAttributes(); + + validate(task); + writeHeader(wr); + + wr.write("#PBS -S /bin/bash\n"); + wr.write("#PBS -N " + task.getName() + '\n'); + wr.write("#PBS -m n\n"); + writeNonEmptyAttr("project", "-A ", wr); + boolean multiple = writeCountAndPPN(spec, wr); + writeWallTime(wr); + writeNonEmptyAttr("queue", "-q ", wr); + wr.write("#PBS -o " + quote(stdout) + '\n'); + wr.write("#PBS -e " + quote(stderr) + '\n'); + + for (String name : spec.getEnvironmentVariableNames()) { + // "export" is necessary on the Cray XT5 Crow + wr.write("export "); + wr.write(name); + wr.write('='); + wr.write(quote(spec.getEnvironmentVariable(name))); + wr.write('\n'); + } + + if (spec.getEnvironmentVariableNames().size() > 0) { + wr.write("#PBS -v " + makeList(spec.getEnvironmentVariableNames()) + '\n'); + } + + String resources = + (String) spec.getAttribute("lsf.resource_list"); + if (resources != null && resources.length() > 0) { + if (logger.isDebugEnabled()) + logger.debug("lsf.resource_list: " + resources); + wr.write("#PBS -l " + resources + '\n'); + } + + // aprun option specifically for Cray Beagle, Franklin + boolean aprun = false; + if (spec.getAttribute("lsf.aprun") != null) + aprun = true; + + String type = (String) spec.getAttribute("jobType"); + if (logger.isDebugEnabled()) + logger.debug("Job type: " + type); + if ("multiple".equals(type)) + multiple = true; + else if("single".equals(type)) + multiple = false; + if (aprun) + multiple = false; + if (multiple) + writeMultiJobPreamble(wr, exitcodefile); + + if (type != null) { + String wrapper = + properties.getProperty("wrapper." + type); + if (logger.isDebugEnabled()) { + logger.debug("Wrapper: " + wrapper); + } + if (wrapper != null) { + wrapper = replaceVars(wrapper); + wr.write(wrapper); + wr.write(' '); + } + if (logger.isDebugEnabled()) { + logger.debug("Wrapper after variable substitution: " + wrapper); + } + } + if (spec.getDirectory() != null) { + wr.write("cd " + quote(spec.getDirectory()) + " && "); + } + + if (aprun) + wr.write("aprun -n " + count + " -N 1 -cc none -d " + + depth + " -F exclusive /bin/sh -c '"); + + wr.write(quote(spec.getExecutable())); + writeQuotedList(wr, spec.getArgumentsAsList()); + + if (aprun) + wr.write("'"); + + if (spec.getStdInput() != null) { + wr.write(" < " + quote(spec.getStdInput())); + } + if (multiple) { + writeMultiJobPostamble(wr); + } + else { + wr.write('\n'); + wr.write("/bin/echo $? >" + exitcodefile + '\n'); + } + wr.close(); + } + + void writeHeader(Writer writer) + throws IOException { + writer.write("#CoG This script generated by CoG\n"); + writer.write("#CoG by class: " + LSFExecutor.class + '\n'); + writer.write("#CoG on date: " + new Date() + "\n\n"); + } + + + private String makeList(Collection names) { + StringBuilder sb = new StringBuilder(); + Iterator i = names.iterator(); + while (i.hasNext()) { + sb.append(i.next()); + if (i.hasNext()) { + sb.append(", "); + } + } + return sb.toString(); + } + + protected void writeMultiJobPreamble(Writer wr, String exitcodefile) + throws IOException { + wr.write("NODES=`cat $PBS_NODEFILE`\n"); + wr.write("ECF=" + exitcodefile + "\n"); + wr.write("INDEX=0\n"); + wr.write("for NODE in $NODES; do\n"); + wr.write(" echo \"N\" >$ECF.$INDEX\n"); + wr.write(" ssh $NODE /bin/bash -c \\\" \""); + } + + + @Override +protected String getName() { + return "PBS"; + } + + @Override +protected AbstractProperties getProperties() { + return Properties.getProperties(); + } + + @Override +protected Job createJob(String jobid, String stdout, + FileLocation stdOutputLocation, String stderr, + FileLocation stdErrorLocation, String exitcode, + AbstractExecutor executor) { + return new Job(jobid, stdout, stdOutputLocation, stderr, + stdErrorLocation, exitcode, executor); + } + + private static QueuePoller poller; + + @Override +protected AbstractQueuePoller getQueuePoller() { + synchronized(LSFExecutor.class) { + if (poller == null) { + poller = new QueuePoller(getProperties()); + poller.start(); + } + return poller; + } + } +} Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/lsf/Properties.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/lsf/Properties.java (revision 0) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/lsf/Properties.java (revision 3498) @@ -0,0 +1,59 @@ +package org.globus.cog.abstraction.impl.scheduler.lsf; + +//---------------------------------------------------------------------- +//This code is developed as part of the Java CoG Kit project +//The terms of the license can be found at http://www.cogkit.org/license +//This message may not be removed or altered. +//---------------------------------------------------------------------- + +//import org.apache.log4j.Logger; +import org.globus.cog.abstraction.impl.scheduler.common.AbstractProperties; + +public class Properties extends AbstractProperties { + + private static final long serialVersionUID = 1L; + + // private static Logger logger = + // Logger.getLogger(Properties.class); + + public static final String PROPERTIES = "provider-lsf.properties"; + + public static final String POLL_INTERVAL = "poll.interval"; + public static final String QSUB = "qsub"; + public static final String QSTAT = "qstat"; + public static final String QDEL = "qdel"; + public static final String USE_MPPWIDTH = "use.mppwidth"; + + private static Properties properties; + + public static synchronized Properties getProperties() { + if (properties == null) { + properties = new Properties(); + properties.load(PROPERTIES); + } + return properties; + } + + protected void setDefaults() { + setPollInterval(5); + setSubmitCommand("qsub"); + setPollCommand("qstat"); + setRemoveCommand("qdel"); + } + + + public String getPollCommandName() { + return QSTAT; + } + + + public String getRemoveCommandName() { + return QDEL; + } + + + public String getSubmitCommandName() { + return QSUB; + } +} + Index: modules/provider-localscheduler/resources/cog-provider.properties =================================================================== --- modules/provider-localscheduler/resources/cog-provider.properties (revision 3497) +++ modules/provider-localscheduler/resources/cog-provider.properties (working copy) @@ -29,3 +29,7 @@ executionTaskHandler=org.globus.cog.abstraction.impl.scheduler.slurm.execution.TaskHandlerImpl securityContext=org.globus.cog.abstraction.impl.common.task.SecurityContextImpl +provider=lsf +sandbox=false +executionTaskHandler=org.globus.cog.abstraction.impl.scheduler.lsf.execution.TaskHandlerImpl +securityContext=org.globus.cog.abstraction.impl.common.task.SecurityContextImpl Index: modules/provider-localscheduler/etc/provider-lsf.properties =================================================================== --- modules/provider-localscheduler/etc/provider-lsf.properties (revision 0) +++ modules/provider-localscheduler/etc/provider-lsf.properties (revision 3498) @@ -0,0 +1,33 @@ +# +# The interval, in seconds, at which the provider will poll the PBS +# queue for status updates. There is at most one poll thread per JVM, +# which is shared by all the jobs submitted through the PBS provider. +# +poll.interval=5 + +# +# The path to qsub. The default assumes that qsub is in PATH +# +qsub=lsf-qsub + +# +# The path to qstat. The default assumes that qstat is in PATH +# +qstat=lsf-qstat + +# +# The path to qdel. The default assumes that qdel is in PATH +# +qdel=lsf-qdel + +# If true, use "#PBS -l mppwidth=" instead of "#PBS -l nodes=" +# in PBS script +use.mppwidth=false + +# If the jobType attribute is specified, then the PBS provider +# will look for a property named "wrapper." and prepend +# that to the executable line in the PBS script. It will also +# substitute value of attributes in the job specification, using +# the "$attrName" notation. +# +wrapper.mpi=mpirun -np $count From swift at ci.uchicago.edu Fri Nov 9 23:05:13 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Fri, 9 Nov 2012 23:05:13 -0600 (CST) Subject: [Swift-commit] cog r3501 Message-ID: <20121110050513.C01A18D0009E@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3501 | hategan | 2012-11-09 23:04:59 -0600 (Fri, 09 Nov 2012) | 1 line no need for java stack traces here ------------------------------------------------------------------------ Index: modules/karajan/src/org/globus/cog/karajan/stack/VariableNotFoundException.java =================================================================== --- modules/karajan/src/org/globus/cog/karajan/stack/VariableNotFoundException.java (revision 3500) +++ modules/karajan/src/org/globus/cog/karajan/stack/VariableNotFoundException.java (working copy) @@ -43,4 +43,9 @@ public String getMessage() { return "Variable not found: " + super.getMessage(); } + + @Override + public synchronized Throwable fillInStackTrace() { + return this; + } } \ No newline at end of file From swift at ci.uchicago.edu Fri Nov 9 23:10:05 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Fri, 9 Nov 2012 23:10:05 -0600 (CST) Subject: [Swift-commit] cog r3505 Message-ID: <20121110051005.6919B8D0009E@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3505 | hategan | 2012-11-09 23:09:31 -0600 (Fri, 09 Nov 2012) | 1 line better errors when communicating with the C client ------------------------------------------------------------------------ Index: modules/karajan/src/org/globus/cog/karajan/workflow/service/ProtocolException.java =================================================================== --- modules/karajan/src/org/globus/cog/karajan/workflow/service/ProtocolException.java (revision 3504) +++ modules/karajan/src/org/globus/cog/karajan/workflow/service/ProtocolException.java (working copy) @@ -25,7 +25,7 @@ } public ProtocolException(Throwable cause) { - super(cause); + super(cause.getMessage(), cause); } } From swift at ci.uchicago.edu Fri Nov 9 23:15:04 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Fri, 9 Nov 2012 23:15:04 -0600 (CST) Subject: [Swift-commit] cog r3507 Message-ID: <20121110051505.3C66A8D0009E@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3507 | hategan | 2012-11-09 23:12:47 -0600 (Fri, 09 Nov 2012) | 1 line fixed NPE and added some generics ------------------------------------------------------------------------ Index: modules/karajan/src/org/globus/cog/karajan/workflow/nodes/functions/Misc.java =================================================================== --- modules/karajan/src/org/globus/cog/karajan/workflow/nodes/functions/Misc.java (revision 3506) +++ modules/karajan/src/org/globus/cog/karajan/workflow/nodes/functions/Misc.java (working copy) @@ -66,8 +66,10 @@ String expanded = TypeUtil.toString(PA_VALUE.getValue(stack)); do { line = br.readLine(); - if (line.indexOf(expanded) != -1) { - return true; + if (line != null) { + if (line.indexOf(expanded) != -1) { + return true; + } } } while (line != null); } @@ -375,10 +377,10 @@ setArguments("str_split", new Arg[] { PA_STRING, PA_SEPARATOR }); } - public List str_split(VariableStack stack) throws ExecutionException { + public List str_split(VariableStack stack) throws ExecutionException { String str = TypeUtil.toString(PA_STRING.getValue(stack)); String sep = TypeUtil.toString(PA_SEPARATOR.getValue(stack)); - List list = new ArrayList(); + List list = new ArrayList(); int index = -1; int last = 0; do { @@ -436,7 +438,7 @@ Pattern pattern = Pattern.compile(TypeUtil.toString(PA_REGEXP.getValue(stack))); boolean invert = TypeUtil.toBoolean(OA_INVERT.getValue(stack)); Object[] args = Arg.VARGS.asArray(stack); - ArrayList ret = new ArrayList(); + ArrayList ret = new ArrayList(); if (args.length > 1) { for (int i = 0; i < args.length; i++) { String value = TypeUtil.toString(args[i]); @@ -447,7 +449,7 @@ return ret.toArray(); } else if (args.length == 1) { - Iterator i = TypeUtil.toIterator(args[0]); + Iterator i = TypeUtil.toIterator(args[0]); while (i.hasNext()) { String value = TypeUtil.toString(i.next()); if (pattern.matcher(value).matches() ^ invert) { From hategan at ci.uchicago.edu Fri Nov 9 23:15:21 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 9 Nov 2012 23:15:21 -0600 (CST) Subject: [Swift-commit] r6011 - trunk/resources Message-ID: <20121110051521.832629CC9F@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-09 23:15:21 -0600 (Fri, 09 Nov 2012) New Revision: 6011 Modified: trunk/resources/Karajan.stg Log: annotate foreach with the names of the k and v variables Modified: trunk/resources/Karajan.stg =================================================================== --- trunk/resources/Karajan.stg 2012-11-08 22:52:29 UTC (rev 6010) +++ trunk/resources/Karajan.stg 2012-11-10 05:15:21 UTC (rev 6011) @@ -226,7 +226,8 @@ >> foreach(var,in,indexVar,indexVarType,declarations,statements,line,selfClose,cleanups,trace) ::= << - + $in$ From hategan at ci.uchicago.edu Fri Nov 9 23:17:38 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 9 Nov 2012 23:17:38 -0600 (CST) Subject: [Swift-commit] r6012 - trunk/libexec Message-ID: <20121110051738.49EDE9CC9F@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-09 23:17:38 -0600 (Fri, 09 Nov 2012) New Revision: 6012 Modified: trunk/libexec/vdl-int-staging.k trunk/libexec/vdl-int.k Log: nice app exceptions Modified: trunk/libexec/vdl-int-staging.k =================================================================== --- trunk/libexec/vdl-int-staging.k 2012-11-10 05:15:21 UTC (rev 6011) +++ trunk/libexec/vdl-int-staging.k 2012-11-10 05:17:38 UTC (rev 6012) @@ -180,10 +180,9 @@ exception( concat( "Exception in {tr}:", nl(), - maybe("Arguments: {arguments}", nl()), - "Host: {rhost}", nl(), - "Directory: {tmpdir}", nl(), - "- - -", nl() + maybe(" Arguments: {arguments}", nl()), + " Host: {rhost}", nl(), + " Directory: {tmpdir}", nl(), ) exception ) Modified: trunk/libexec/vdl-int.k =================================================================== --- trunk/libexec/vdl-int.k 2012-11-10 05:15:21 UTC (rev 6011) +++ trunk/libexec/vdl-int.k 2012-11-10 05:17:38 UTC (rev 6012) @@ -368,16 +368,15 @@ for(f, list(list("stderr.txt", stderr), list("stdout.txt", stdout)) [name, file] := each(f) destfile := "{jobid}-{file}" - nl() - "{name}: " try( sequential( task:transfer(srchost=rhost, srcdir=tmpdir, srcfile=file, destfile=destfile) - + nl() + " {name}: " file:read(destfile) ) - nl() + "" ) maybe(file:remove(destfile)) ) @@ -599,11 +598,10 @@ exception( concat( "Exception in {tr}:", nl(), - maybe("Arguments: ", arguments, nl()), - "Host: {rhost}", nl(), - "Directory: {tmpdir}", - "{outs}", nl(), - "----", nl(), + maybe(" Arguments: ", arguments, nl()), + " Host: {rhost}", nl(), + " Directory: {tmpdir}", + "{outs}", kickstartRec ) exception From hategan at ci.uchicago.edu Fri Nov 9 23:18:56 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 9 Nov 2012 23:18:56 -0600 (CST) Subject: [Swift-commit] r6013 - trunk/src/org/griphyn/vdl/karajan Message-ID: <20121110051856.C765A9CC9F@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-09 23:18:56 -0600 (Fri, 09 Nov 2012) New Revision: 6013 Modified: trunk/src/org/griphyn/vdl/karajan/VDL2ExecutionContext.java Log: print nice karajan/swift level stack trace Modified: trunk/src/org/griphyn/vdl/karajan/VDL2ExecutionContext.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/VDL2ExecutionContext.java 2012-11-10 05:17:38 UTC (rev 6012) +++ trunk/src/org/griphyn/vdl/karajan/VDL2ExecutionContext.java 2012-11-10 05:18:56 UTC (rev 6013) @@ -20,6 +20,8 @@ */ package org.griphyn.vdl.karajan; +import java.util.List; + import org.apache.log4j.Logger; import org.globus.cog.karajan.stack.VariableStack; import org.globus.cog.karajan.workflow.ElementTree; @@ -58,6 +60,13 @@ else { getStderr().append(ProcessBulkErrors.getMessageChain(e)); } + if (e.getStack() != null) { + List l = Monitor.getSwiftTrace(e.getStack()); + for (String s : l) { + getStderr().append("\n\t"); + getStderr().append(s); + } + } getStderr().append("\n"); } else { From hategan at ci.uchicago.edu Fri Nov 9 23:19:44 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 9 Nov 2012 23:19:44 -0600 (CST) Subject: [Swift-commit] r6014 - trunk/src/org/griphyn/vdl/karajan Message-ID: <20121110051944.275CF9CC9F@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-09 23:19:44 -0600 (Fri, 09 Nov 2012) New Revision: 6014 Modified: trunk/src/org/griphyn/vdl/karajan/VDSAdaptiveScheduler.java Log: moved message from karajan to swift since karajan has nothing to do with sites files and tc.data Modified: trunk/src/org/griphyn/vdl/karajan/VDSAdaptiveScheduler.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/VDSAdaptiveScheduler.java 2012-11-10 05:18:56 UTC (rev 6013) +++ trunk/src/org/griphyn/vdl/karajan/VDSAdaptiveScheduler.java 2012-11-10 05:19:44 UTC (rev 6014) @@ -40,6 +40,7 @@ import org.globus.cog.abstraction.interfaces.Status; import org.globus.cog.abstraction.interfaces.Task; import org.globus.cog.karajan.scheduler.AbstractScheduler; +import org.globus.cog.karajan.scheduler.NoSuchResourceException; import org.globus.cog.karajan.scheduler.ResourceConstraintChecker; import org.globus.cog.karajan.scheduler.TaskConstraints; import org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler; @@ -388,6 +389,10 @@ } protected void failTask(Task t, String message, Exception e) { + if (e instanceof NoSuchResourceException) { + message = "The application \"" + getTaskConstraints(t).getConstraint("tr") + + "\" is not available for any site/pool in your tc.data catalog "; + } if (logger.isDebugEnabled()) { logger.debug("Failing task " + t.getIdentity()); } From hategan at ci.uchicago.edu Fri Nov 9 23:21:45 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 9 Nov 2012 23:21:45 -0600 (CST) Subject: [Swift-commit] r6015 - trunk/src/org/griphyn/vdl/karajan/lib/swiftscript Message-ID: <20121110052145.8DD279CC9F@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-09 23:21:45 -0600 (Fri, 09 Nov 2012) New Revision: 6015 Modified: trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/FnArg.java Log: attach stack to exception such that a trace can be printed by swift Modified: trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/FnArg.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/FnArg.java 2012-11-10 05:19:44 UTC (rev 6014) +++ trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/FnArg.java 2012-11-10 05:21:45 UTC (rev 6015) @@ -82,7 +82,7 @@ value = P_VALUE.getValue(stack); } if (value == null) { - throw new ExecutionException("Missing command line argument: " + name); + throw new ExecutionException(stack, "Missing command line argument: " + name); } else { DSHandle result = new RootDataNode(Types.STRING, value); From hategan at ci.uchicago.edu Fri Nov 9 23:23:08 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 9 Nov 2012 23:23:08 -0600 (CST) Subject: [Swift-commit] r6016 - trunk/src/org/griphyn/vdl/mapping Message-ID: <20121110052308.EA1349CC9F@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-09 23:23:08 -0600 (Fri, 09 Nov 2012) New Revision: 6016 Modified: trunk/src/org/griphyn/vdl/mapping/InvalidPathException.java Log: "Invalid path" is not very good wording. Changed exception to be specific. If an array index is invalid, then say so. If a struct field is invalid, then say so Modified: trunk/src/org/griphyn/vdl/mapping/InvalidPathException.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/InvalidPathException.java 2012-11-10 05:21:45 UTC (rev 6015) +++ trunk/src/org/griphyn/vdl/mapping/InvalidPathException.java 2012-11-10 05:23:08 UTC (rev 6016) @@ -20,16 +20,45 @@ */ package org.griphyn.vdl.mapping; + public class InvalidPathException extends Exception { public InvalidPathException(String path, DSHandle source) { - super("Invalid path (" + path + ") for " - + source.toString()); + super(getMessage(path, source)); } - public InvalidPathException(Path path, DSHandle source) { + private static String getMessage(String path, DSHandle source) { + if (source.getType().isArray()) { + return "Array index '" + path + "' not found for " + getName(source) + " of size " + getSize(source); + } + else if (source.getType().isComposite()) { + return "Invalid field name '" + path + "' for " + getName(source) + " of type " + source.getType(); + } + else { + return "Invalid path (" + path + ") for " + source.toString(); + } + } + + private static int getSize(DSHandle source) { + return source.getArrayValue().size(); + } + + private static String getName(DSHandle source) { + if (source instanceof AbstractDataNode) { + return ((AbstractDataNode) source).getDisplayableName(); + } + else { + return source.toString(); + } + } + + public InvalidPathException(Path path, DSHandle source) { this(path.toString(), source); } - + + public InvalidPathException(Object path, DSHandle source) { + this(path.toString(), source); + } + public InvalidPathException(String string) { super(string); } From hategan at ci.uchicago.edu Fri Nov 9 23:24:17 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 9 Nov 2012 23:24:17 -0600 (CST) Subject: [Swift-commit] r6017 - trunk/src/org/griphyn/vdl/karajan Message-ID: <20121110052417.A99219CC9F@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-09 23:24:17 -0600 (Fri, 09 Nov 2012) New Revision: 6017 Modified: trunk/src/org/griphyn/vdl/karajan/DSHandleFutureWrapper.java trunk/src/org/griphyn/vdl/karajan/FutureWrapper.java Log: added method to handle wrappers to access the handle Modified: trunk/src/org/griphyn/vdl/karajan/DSHandleFutureWrapper.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/DSHandleFutureWrapper.java 2012-11-10 05:23:08 UTC (rev 6016) +++ trunk/src/org/griphyn/vdl/karajan/DSHandleFutureWrapper.java 2012-11-10 05:24:17 UTC (rev 6017) @@ -30,10 +30,11 @@ import org.globus.cog.karajan.workflow.futures.FutureListener; import org.globus.cog.karajan.workflow.futures.ListenerStackPair; import org.griphyn.vdl.mapping.AbstractDataNode; +import org.griphyn.vdl.mapping.DSHandle; public class DSHandleFutureWrapper implements FutureWrapper { private LinkedList listeners; - private AbstractDataNode node; + private final AbstractDataNode node; public DSHandleFutureWrapper(AbstractDataNode node) { this.node = node; @@ -61,6 +62,10 @@ // closed == true; notifyListeners(); } + + public DSHandle getHandle() { + return node; + } public void notifyListeners() { List l; Modified: trunk/src/org/griphyn/vdl/karajan/FutureWrapper.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/FutureWrapper.java 2012-11-10 05:23:08 UTC (rev 6016) +++ trunk/src/org/griphyn/vdl/karajan/FutureWrapper.java 2012-11-10 05:24:17 UTC (rev 6017) @@ -19,6 +19,7 @@ import org.globus.cog.karajan.workflow.events.EventTargetPair; import org.globus.cog.karajan.workflow.futures.Future; +import org.griphyn.vdl.mapping.DSHandle; public interface FutureWrapper extends Future { void notifyListeners(); @@ -26,4 +27,6 @@ int listenerCount(); EventTargetPair[] getListenerEvents(); + + DSHandle getHandle(); } From hategan at ci.uchicago.edu Fri Nov 9 23:27:58 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 9 Nov 2012 23:27:58 -0600 (CST) Subject: [Swift-commit] r6018 - trunk/src/org/griphyn/vdl/karajan/lib Message-ID: <20121110052758.282429CC9F@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-09 23:27:57 -0600 (Fri, 09 Nov 2012) New Revision: 6018 Modified: trunk/src/org/griphyn/vdl/karajan/lib/VDLFunction.java Log: set stack for exceptions if not already there and replace strangely worded dependent exception Modified: trunk/src/org/griphyn/vdl/karajan/lib/VDLFunction.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/VDLFunction.java 2012-11-10 05:24:17 UTC (rev 6017) +++ trunk/src/org/griphyn/vdl/karajan/lib/VDLFunction.java 2012-11-10 05:27:57 UTC (rev 6018) @@ -39,12 +39,14 @@ import org.globus.cog.karajan.util.TypeUtil; import org.globus.cog.karajan.workflow.ExecutionException; import org.globus.cog.karajan.workflow.KarajanRuntimeException; +import org.globus.cog.karajan.workflow.futures.Future; import org.globus.cog.karajan.workflow.nodes.SequentialWithArguments; import org.globus.cog.karajan.workflow.nodes.restartLog.RestartLog; import org.globus.swift.catalog.TCEntry; import org.globus.swift.catalog.transformation.File; import org.globus.swift.catalog.types.TCType; import org.griphyn.vdl.karajan.AssertFailedException; +import org.griphyn.vdl.karajan.FutureWrapper; import org.griphyn.vdl.karajan.Loader; import org.griphyn.vdl.karajan.TCCache; import org.griphyn.vdl.karajan.functions.ConfigProperty; @@ -86,10 +88,16 @@ logger.fatal("swift: assert failed: " + e.getMessage()); stack.getExecutionContext().failedQuietly(stack, e); } + catch (ExecutionException e) { + if (e.getStack() == null) { + e.setStack(stack); + } + throw e; + } catch (DependentException e) { // This would not be the primal fault so in non-lazy errors mode it // should not matter - throw new ExecutionException("Wrapping a dependent exception in VDLFunction.post() - errors in data dependencies",e); + throw new ExecutionException(stack, e); } } @@ -443,6 +451,7 @@ return tc; } } + private static int provenanceIDCount = 451000; From hategan at ci.uchicago.edu Fri Nov 9 23:28:59 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 9 Nov 2012 23:28:59 -0600 (CST) Subject: [Swift-commit] r6019 - trunk/src/org/griphyn/vdl/mapping Message-ID: <20121110052859.8085D9CC9F@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-09 23:28:59 -0600 (Fri, 09 Nov 2012) New Revision: 6019 Modified: trunk/src/org/griphyn/vdl/mapping/DependentException.java Log: user friendly variable name in dependent exceptions Modified: trunk/src/org/griphyn/vdl/mapping/DependentException.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/DependentException.java 2012-11-10 05:27:57 UTC (rev 6018) +++ trunk/src/org/griphyn/vdl/mapping/DependentException.java 2012-11-10 05:28:59 UTC (rev 6019) @@ -41,6 +41,15 @@ public DSHandle getHandle() { return handle; } + + public String getVariableName() { + if (handle instanceof AbstractDataNode) { + return ((AbstractDataNode) handle).getDisplayableName(); + } + else { + return handle.toString(); + } + } public String toString() { return getMessage(); From hategan at ci.uchicago.edu Fri Nov 9 23:33:48 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 9 Nov 2012 23:33:48 -0600 (CST) Subject: [Swift-commit] r6020 - trunk/src/org/griphyn/vdl/mapping Message-ID: <20121110053348.5F7239CC9F@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-09 23:33:48 -0600 (Fri, 09 Nov 2012) New Revision: 6020 Modified: trunk/src/org/griphyn/vdl/mapping/AbstractDataNode.java Log: fixed a race condition where a procedure like readData would create a struct in an array before populating the fields; the array would notify listeners of the newly created array element, and code trying to access fields would fail because the fields are not there yet. The fix changes the behaviour of createField as follows: if the new field is a struct, then its fields are also automatically created so that concurrent threads writing and reading on those fields can work with the same objects Modified: trunk/src/org/griphyn/vdl/mapping/AbstractDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/AbstractDataNode.java 2012-11-10 05:28:59 UTC (rev 6019) +++ trunk/src/org/griphyn/vdl/mapping/AbstractDataNode.java 2012-11-10 05:33:48 UTC (rev 6020) @@ -95,6 +95,18 @@ } } + protected void populateStructFields() { + for (String name : getType().getFieldNames()) { + try { + createField(name); + } + catch (NoSuchFieldException e) { + throw new RuntimeException("Internal inconsistency found: field '" + name + + "' is listed by the type but createField() claims it is invalid"); + } + } + } + public void init(MappingParamSet params) { throw new UnsupportedOperationException(); } @@ -214,6 +226,26 @@ } return prefix; } + + public String getDeclarationLine() { + String line = getRoot().getParam(MappingParam.SWIFT_LINE); + if (line == null || line.length() == 0) { + return null; + } + else { + return line; + } + } + + public String getThread() { + String restartId = getRoot().getParam(MappingParam.SWIFT_RESTARTID); + if (restartId != null) { + return restartId.substring(0, restartId.lastIndexOf(":")); + } + else { + return null; + } + } public DSHandle getField(Path path) throws InvalidPathException { if (path.isEmpty()) { @@ -230,7 +262,6 @@ } } catch (NoSuchFieldException e) { - logger.warn("could not find variable: " + field.getId() + " " + path); throw new InvalidPathException(path, this); } } @@ -316,7 +347,11 @@ return new ArrayDataNode(f, getRoot(), this); } else { - return new DataNode(f, getRoot(), this); + DataNode dn = new DataNode(f, getRoot(), this); + if (field.getType().isComposite()) { + dn.populateStructFields(); + } + return dn; } } From hategan at ci.uchicago.edu Fri Nov 9 23:34:58 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 9 Nov 2012 23:34:58 -0600 (CST) Subject: [Swift-commit] r6021 - trunk/src/org/griphyn/vdl/mapping Message-ID: <20121110053458.EBBF59CC9F@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-09 23:34:58 -0600 (Fri, 09 Nov 2012) New Revision: 6021 Modified: trunk/src/org/griphyn/vdl/mapping/DataDependentException.java Log: better error message for dependent exceptions Modified: trunk/src/org/griphyn/vdl/mapping/DataDependentException.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/DataDependentException.java 2012-11-10 05:33:48 UTC (rev 6020) +++ trunk/src/org/griphyn/vdl/mapping/DataDependentException.java 2012-11-10 05:34:58 UTC (rev 6021) @@ -35,7 +35,7 @@ } public String getMessage() { - return getHandle() + " not derived due to errors in data dependencies"; + return getVariableName() + " not derived due to errors in data dependencies"; } } From hategan at ci.uchicago.edu Fri Nov 9 23:42:36 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 9 Nov 2012 23:42:36 -0600 (CST) Subject: [Swift-commit] r6022 - trunk/src/org/griphyn/vdl/mapping Message-ID: <20121110054236.AC31F9CC9F@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-09 23:42:36 -0600 (Fri, 09 Nov 2012) New Revision: 6022 Modified: trunk/src/org/griphyn/vdl/mapping/ArrayDataNode.java Log: delegate the exact message construction to the exception Modified: trunk/src/org/griphyn/vdl/mapping/ArrayDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/ArrayDataNode.java 2012-11-10 05:34:58 UTC (rev 6021) +++ trunk/src/org/griphyn/vdl/mapping/ArrayDataNode.java 2012-11-10 05:42:36 UTC (rev 6022) @@ -141,7 +141,7 @@ fields, path.butFirst()); } catch (NoSuchFieldException e) { - throw new InvalidPathException("Invalid index (" + path.getKey(0) + ") for " + this); + throw new InvalidPathException(path.getKey(0), this); } } } From hategan at ci.uchicago.edu Fri Nov 9 23:43:55 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 9 Nov 2012 23:43:55 -0600 (CST) Subject: [Swift-commit] r6023 - in trunk/src/org/griphyn/vdl: karajan/lib karajan/lib/swiftscript mapping util Message-ID: <20121110054355.C30879CC9F@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-09 23:43:55 -0600 (Fri, 09 Nov 2012) New Revision: 6023 Added: trunk/src/org/griphyn/vdl/karajan/lib/Tracer.java Modified: trunk/src/org/griphyn/vdl/karajan/lib/InfiniteCountingWhile.java trunk/src/org/griphyn/vdl/karajan/lib/New.java trunk/src/org/griphyn/vdl/karajan/lib/SetFieldValue.java trunk/src/org/griphyn/vdl/karajan/lib/Stagein.java trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java trunk/src/org/griphyn/vdl/karajan/lib/UnitStart.java trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/ReadData.java trunk/src/org/griphyn/vdl/mapping/MappingParamSet.java trunk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java trunk/src/org/griphyn/vdl/mapping/RootDataNode.java trunk/src/org/griphyn/vdl/util/VDL2Config.java trunk/src/org/griphyn/vdl/util/VDL2ConfigProperties.java Log: added tracing Modified: trunk/src/org/griphyn/vdl/karajan/lib/InfiniteCountingWhile.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/InfiniteCountingWhile.java 2012-11-10 05:42:36 UTC (rev 6022) +++ trunk/src/org/griphyn/vdl/karajan/lib/InfiniteCountingWhile.java 2012-11-10 05:43:55 UTC (rev 6023) @@ -34,17 +34,29 @@ public static final String COUNTER_NAME = "$"; public static final Arg.Positional VAR = new Arg.Positional("var"); + + private Tracer tracer; public InfiniteCountingWhile() { setOptimize(false); } + + @Override + protected void initializeStatic() { + super.initializeStatic(); + tracer = Tracer.getTracer(this); + } - public void pre(VariableStack stack) throws ExecutionException { + public void pre(VariableStack stack) throws ExecutionException { ThreadingContext tc = (ThreadingContext)stack.getVar("#thread"); stack.setVar("#iteratethread", tc); stack.setVar("#thread", tc.split(0)); stack.setVar(COUNTER_NAME, Collections.singletonList(0)); - stack.setVar((String) VAR.getStatic(this), new RootDataNode(Types.INT, 0)); + String var = (String) VAR.getStatic(this); + if (tracer.isEnabled()) { + tracer.trace(tc.toString(), var + " = 0"); + } + stack.setVar(var, new RootDataNode(Types.INT, 0)); super.pre(stack); } @@ -69,9 +81,14 @@ int i = c.get(0).intValue(); i++; ThreadingContext tc = (ThreadingContext)stack.getVar("#iteratethread"); - stack.setVar("#thread", tc.split(i)); + ThreadingContext ntc = tc.split(i); + stack.setVar("#thread", ntc); stack.setVar(COUNTER_NAME, Collections.singletonList(i)); - stack.setVar((String) VAR.getStatic(this), new RootDataNode(Types.INT, i)); + String var = (String) VAR.getStatic(this); + if (tracer.isEnabled()) { + tracer.trace(ntc.toString(), var + " = " + i); + } + stack.setVar(var, new RootDataNode(Types.INT, i)); } if (index >= elementCount()) { // starting new iteration Modified: trunk/src/org/griphyn/vdl/karajan/lib/New.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/New.java 2012-11-10 05:42:36 UTC (rev 6022) +++ trunk/src/org/griphyn/vdl/karajan/lib/New.java 2012-11-10 05:43:55 UTC (rev 6023) @@ -55,8 +55,16 @@ setArguments(New.class, new Arg[] { OA_TYPE, OA_MAPPING, OA_VALUE, OA_DBGNAME, OA_WAITFOR}); } + + private Tracer tracer; - public Object function(VariableStack stack) throws ExecutionException { + @Override + protected void initializeStatic() { + super.initializeStatic(); + tracer = Tracer.getTracer(this); + } + + public Object function(VariableStack stack) throws ExecutionException { String typename = TypeUtil.toString(OA_TYPE.getValue(stack)); Object value = OA_VALUE.getValue(stack); @SuppressWarnings("unchecked") @@ -105,6 +113,9 @@ } DSHandle handle; if (typename.equals("external")) { + if (tracer.isEnabled()) { + tracer.trace(threadPrefix, dbgname + " = external"); + } handle = new ExternalDataNode(); } else if (type.isArray()) { @@ -112,12 +123,18 @@ handle = new RootArrayDataNode(type); if (value != null) { if (value instanceof RootArrayDataNode) { + if (tracer.isEnabled()) { + tracer.trace(threadPrefix, dbgname + " = " + Tracer.getVarName((RootDataNode) value)); + } handle = (RootArrayDataNode) value; } else { if (!(value instanceof List)) { throw new ExecutionException("An array variable can only be initialized with a list of values"); } + if (tracer.isEnabled()) { + tracer.trace(threadPrefix, dbgname + " = " + formatList((List) value)); + } int index = 0; Iterator i = ((List) value).iterator(); while (i.hasNext()) { @@ -137,18 +154,34 @@ } handle.closeShallow(); } + else { + if (tracer.isEnabled()) { + tracer.trace(threadPrefix, dbgname); + } + } handle.init(mps); } else if (value instanceof DSHandle) { + if (tracer.isEnabled()) { + tracer.trace(threadPrefix, dbgname + " = " + Tracer.getVarName((DSHandle) value)); + } handle = (DSHandle) value; } else { handle = new RootDataNode(type); handle.init(mps); if (value != null) { + if (tracer.isEnabled()) { + tracer.trace(threadPrefix, dbgname + " = " + value); + } handle.setValue(internalValue(type, value)); } + else { + if (tracer.isEnabled()) { + tracer.trace(threadPrefix, dbgname + formatMPS(mps)); + } + } } if (AbstractDataNode.provenance && logger.isDebugEnabled()) { @@ -160,4 +193,55 @@ throw new ExecutionException(e); } } + + private String formatMPS(MappingParamSet mps) { + Object desc = mps.get(MappingParam.SWIFT_DESCRIPTOR); + if (desc == null) { + return ""; + } + StringBuilder sb = new StringBuilder(); + sb.append(" <"); + sb.append(desc); + sb.append("; "); + boolean first = true; + for (String name : mps.names()) { + if (name.indexOf('#') >= 0) { + // skip internal parameters + continue; + } + if (first) { + first = false; + } + else { + sb.append(", "); + } + sb.append(name); + sb.append(" = \""); + sb.append(Tracer.unwrapHandle(mps._get(name))); + sb.append("\""); + } + sb.append('>'); + return sb.toString(); + } + + private String formatList(List value) { + StringBuilder sb = new StringBuilder(); + sb.append('['); + unfoldList(value, sb); + sb.append(']'); + return sb.toString(); + } + + private void unfoldList(List value, StringBuilder sb) { + boolean first = true; + for (Object v : value) { + if (first) { + first = false; + } + else { + sb.append(", "); + } + sb.append(v); + } + } } Modified: trunk/src/org/griphyn/vdl/karajan/lib/SetFieldValue.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/SetFieldValue.java 2012-11-10 05:42:36 UTC (rev 6022) +++ trunk/src/org/griphyn/vdl/karajan/lib/SetFieldValue.java 2012-11-10 05:43:55 UTC (rev 6023) @@ -26,10 +26,15 @@ import org.apache.log4j.Logger; import org.globus.cog.karajan.arguments.Arg; +import org.globus.cog.karajan.stack.VariableNotFoundException; import org.globus.cog.karajan.stack.VariableStack; +import org.globus.cog.karajan.util.ThreadingContext; import org.globus.cog.karajan.workflow.ExecutionException; +import org.globus.cog.karajan.workflow.futures.Future; import org.globus.cog.karajan.workflow.futures.FutureFault; import org.globus.cog.karajan.workflow.futures.FutureNotYetAvailable; +import org.griphyn.vdl.karajan.DSHandleFutureWrapper; +import org.griphyn.vdl.karajan.FutureWrapper; import org.griphyn.vdl.karajan.Pair; import org.griphyn.vdl.karajan.PairIterator; import org.griphyn.vdl.karajan.WaitingThreadsMonitor; @@ -37,7 +42,6 @@ import org.griphyn.vdl.mapping.DSHandle; import org.griphyn.vdl.mapping.InvalidPathException; import org.griphyn.vdl.mapping.Mapper; -import org.griphyn.vdl.mapping.MappingParam; import org.griphyn.vdl.mapping.Path; import org.griphyn.vdl.type.Type; @@ -51,8 +55,15 @@ } private String src, dest; + private Tracer tracer; - public Object function(VariableStack stack) throws ExecutionException { + @Override + protected void initializeStatic() { + super.initializeStatic(); + tracer = Tracer.getTracer(this); + } + + public Object function(VariableStack stack) throws ExecutionException { DSHandle var = (DSHandle) PA_VAR.getValue(stack); try { Path path = parsePath(OA_PATH.getValue(stack), stack); @@ -60,11 +71,13 @@ AbstractDataNode value = (AbstractDataNode) PA_VALUE.getValue(stack); if (src == null) { - dest = getVarName(var); - src = getVarName(value); + dest = Tracer.getVarName(var); + src = Tracer.getVarName(value); } - log(leaf, value); + if (tracer.isEnabled()) { + log(leaf, value, stack); + } // TODO want to do a type check here, for runtime type checking // and pull out the appropriate internal value from value if it @@ -76,6 +89,9 @@ return null; } catch (FutureFault f) { + if (tracer.isEnabled()) { + tracer.trace(stack, var + " waiting for " + Tracer.getFutureName(f.getFuture())); + } WaitingThreadsMonitor.addOutput(stack, Collections.singletonList(var)); throw f; } @@ -84,56 +100,10 @@ } } - private String getVarName(DSHandle var) { - String name = var.getRoot().getParam(MappingParam.SWIFT_DBGNAME); - if (var == var.getRoot()) { - return name; - } - else { - Path p = var.getPathFromRoot(); - if (p.isArrayIndex(0)) { - return name + var.getPathFromRoot(); - } - else { - return name + "." + var.getPathFromRoot(); - } - } + private void log(DSHandle leaf, DSHandle value, VariableStack stack) throws VariableNotFoundException { + tracer.trace(stack, dest + " = " + Tracer.unwrapHandle(value)); } - private void log(DSHandle leaf, DSHandle value) { - if (logger.isDebugEnabled()) { - logger.debug("Setting " + leaf + " to " + value); - } - else if (logger.isInfoEnabled()) { - if (leaf instanceof AbstractDataNode) { - AbstractDataNode data = (AbstractDataNode) leaf; - Path path = data.getPathFromRoot(); - String p; - if (path.isEmpty()) { - p = ""; - } - else if (path.isArrayIndex(0)) { - p = path.toString(); - } - else { - p = "." + path.toString(); - } - String name = data.getDisplayableName() + p; - if (value.getType().isArray()) { - if (logger.isInfoEnabled()) { - logger.info("Set: " + name + "=" + - unpackHandles(value, value.getArrayValue())); - } - } - else { - if (logger.isInfoEnabled()) { - logger.info("Set: " + name + "=" + value.getValue()); - } - } - } - } - } - String unpackHandles(DSHandle handle, Map, DSHandle> handles) { StringBuilder sb = new StringBuilder(); sb.append("{"); Modified: trunk/src/org/griphyn/vdl/karajan/lib/Stagein.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/Stagein.java 2012-11-10 05:42:36 UTC (rev 6022) +++ trunk/src/org/griphyn/vdl/karajan/lib/Stagein.java 2012-11-10 05:43:55 UTC (rev 6023) @@ -30,6 +30,8 @@ import org.globus.cog.karajan.stack.VariableStack; import org.globus.cog.karajan.workflow.ExecutionException; import org.globus.cog.karajan.workflow.futures.FutureFault; +import org.globus.cog.karajan.workflow.nodes.FlowNode; +import org.griphyn.vdl.engine.Karajan; import org.griphyn.vdl.mapping.AbstractDataNode; import org.griphyn.vdl.mapping.DSHandle; import org.griphyn.vdl.mapping.DependentException; @@ -42,10 +44,21 @@ public static final Arg VAR = new Arg.Positional("var"); public static final Channel STAGEIN = new Channel("stagein"); + + private Tracer tracer; + private String procName; static { setArguments(Stagein.class, new Arg[] { VAR }); } + + @Override + protected void initializeStatic() { + super.initializeStatic(); + FlowNode def = (FlowNode) getParent().getParent(); + procName = Karajan.demangle(def.getTextualName()); + tracer = Tracer.getTracer(def, "APPCALL"); + } private boolean isPrimitive(DSHandle var) { return (var instanceof AbstractDataNode && ((AbstractDataNode) var) @@ -61,7 +74,11 @@ Collection fp = var.getFringePaths(); try { for (Path p : fp) { - ((AbstractDataNode) var.getField(p)).waitFor(); + AbstractDataNode n = (AbstractDataNode) var.getField(p); + n.waitFor(); + if (tracer.isEnabled()) { + tracer.trace(stack, procName + " available " + Tracer.getVarName(n)); + } } } catch (DependentException e) { @@ -72,6 +89,9 @@ } } catch (FutureFault f) { + if (tracer.isEnabled()) { + tracer.trace(stack, procName + " wait " + Tracer.getFutureName(f.getFuture())); + } throw f; } catch (MappingDependentException e) { @@ -90,7 +110,18 @@ } else { // we still wait until the primitive value is there - var.waitFor(); + if (tracer.isEnabled()) { + try { + var.waitFor(); + } + catch (FutureFault f) { + tracer.trace(stack, procName + " waiting for " + Tracer.getFutureName(f.getFuture())); + throw f; + } + } + else { + var.waitFor(); + } } return null; } Modified: trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java 2012-11-10 05:42:36 UTC (rev 6022) +++ trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java 2012-11-10 05:43:55 UTC (rev 6023) @@ -38,6 +38,8 @@ import org.globus.cog.karajan.workflow.futures.FutureListener; import org.globus.cog.karajan.workflow.futures.ListenerStackPair; import org.globus.cog.karajan.workflow.nodes.AbstractParallelIterator; +import org.griphyn.vdl.karajan.Pair; +import org.griphyn.vdl.mapping.DSHandle; import org.griphyn.vdl.util.VDL2Config; public class ThrottledParallelFor extends AbstractParallelIterator { @@ -57,10 +59,24 @@ public static final String THREAD_COUNT = "#threadcount"; private int maxThreadCount = -1; + private Tracer forTracer, iterationTracer; + private String kvar, vvar; + + @Override + protected void initializeStatic() { + super.initializeStatic(); + forTracer = Tracer.getTracer(this, "FOREACH"); + iterationTracer = Tracer.getTracer(this, "ITERATION"); + kvar = (String) getProperty("_kvar"); + vvar = (String) getProperty("_vvar"); + } protected void partialArgumentsEvaluated(VariableStack stack) throws ExecutionException { stack.setVar("selfclose", A_SELF_CLOSE.getValue(stack)); + if (forTracer.isEnabled()) { + forTracer.trace(ThreadingContext.get(stack).toString()); + } super.partialArgumentsEvaluated(stack); } @@ -96,11 +112,15 @@ for (; j < available && i.hasNext(); j++) { VariableStack copy = stack.copy(); copy.enter(); - ThreadingContext.set(copy, ThreadingContext.get(copy).split( - i.current())); + ThreadingContext ntc = ThreadingContext.get(copy).split(i.current()); + ThreadingContext.set(copy, ntc); setIndex(copy, getArgCount()); setArgsDone(copy); - copy.setVar(var.getName(), i.next()); + Object value = i.next(); + if (iterationTracer.isEnabled()) { + iterationTracer.trace(ntc.toString(), unwrap(value)); + } + copy.setVar(var.getName(), value); startElement(getArgCount(), copy); } } @@ -137,7 +157,22 @@ } } - public void failed(VariableStack stack, ExecutionException e) throws ExecutionException { + private Object unwrap(Object value) { + if (value instanceof Pair) { + Pair p = (Pair) value; + if (kvar != null) { + return kvar + "=" + p.get(0) + ", " + vvar + "=" + Tracer.unwrapHandle(p.get(1)); + } + else { + return vvar + "=" + Tracer.unwrapHandle(p.get(1)); + } + } + else { + return "!"; + } + } + + public void failed(VariableStack stack, ExecutionException e) throws ExecutionException { if (!testAndSetChildFailed(stack)) { if (stack.parentFrame().isDefined(VAR)) { stack.leave(); Added: trunk/src/org/griphyn/vdl/karajan/lib/Tracer.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/Tracer.java (rev 0) +++ trunk/src/org/griphyn/vdl/karajan/lib/Tracer.java 2012-11-10 05:43:55 UTC (rev 6023) @@ -0,0 +1,294 @@ +//---------------------------------------------------------------------- +//This code is developed as part of the Java CoG Kit project +//The terms of the license can be found at http://www.cogkit.org/license +//This message may not be removed or altered. +//---------------------------------------------------------------------- + +/* + * Created on Nov 9, 2012 + */ +package org.griphyn.vdl.karajan.lib; + +import java.io.IOException; +import java.util.HashMap; +import java.util.Map; + +import org.apache.log4j.Logger; +import org.globus.cog.karajan.stack.VariableNotFoundException; +import org.globus.cog.karajan.stack.VariableStack; +import org.globus.cog.karajan.util.ThreadingContext; +import org.globus.cog.karajan.workflow.futures.Future; +import org.globus.cog.karajan.workflow.nodes.FlowElement; +import org.globus.cog.karajan.workflow.nodes.FlowNode; +import org.griphyn.vdl.engine.Karajan; +import org.griphyn.vdl.karajan.FutureWrapper; +import org.griphyn.vdl.mapping.AbstractDataNode; +import org.griphyn.vdl.mapping.DSHandle; +import org.griphyn.vdl.mapping.Mapper; +import org.griphyn.vdl.mapping.Path; +import org.griphyn.vdl.type.Types; +import org.griphyn.vdl.util.VDL2Config; + +public class Tracer { + public static final Logger logger = Logger.getLogger("TRACE"); + private static boolean globalTracingEnabled; + private static final Map NAME_MAPPINGS; + private ThreadLocal thread = new ThreadLocal(); + + static { + try { + globalTracingEnabled = VDL2Config.getConfig().isTracingEnabled(); + } + catch (IOException e) { + globalTracingEnabled = false; + } + NAME_MAPPINGS = new HashMap(); + NAME_MAPPINGS.put("assignment", "ASSIGN"); + NAME_MAPPINGS.put("iterate", "ITERATE"); + NAME_MAPPINGS.put("vdl:new", "DECLARE"); + } + + private final String source; + private final boolean enabled; + + private Tracer(boolean enabled) { + source = null; + this.enabled = enabled; + } + + private Tracer(FlowNode fe, String name) { + source = buildSource(fe, name); + if (source == null) { + enabled = false; + } + else { + enabled = true; + } + } + + private Tracer(String line, String name) { + source = buildSource(line, name); + enabled = true; + } + + private Tracer(String name) { + source = name; + enabled = true; + } + + private Tracer(FlowNode fe) { + this(fe, null); + } + + private String buildSource(FlowNode fe, String name) { + String line = findLine(fe); + if (line == null) { + return null; + } + if (name == null) { + name = getType(fe); + } + return buildSource(line, name); + } + + private String buildSource(String line, String name) { + StringBuilder sb = new StringBuilder(); + sb.append(name); + sb.append(", line "); + sb.append(line); + return sb.toString(); + } + + private String getType(FlowNode fe) { + String t = Karajan.demangle(fe.getTextualName()); + String nt = NAME_MAPPINGS.get(t); + if (nt == null) { + return t; + } + else { + return nt; + } + } + + private String findLine(FlowElement fe) { + String line; + if (fe.hasProperty("_traceline")) { + line = (String) fe.getProperty("_traceline"); + } + else if (fe.hasProperty("_defline")) { + line = (String) fe.getProperty("_defline"); + } + else { + line = null; + } + if (line == null || line.equals("-1") || line.equals("")) { + return null; + } + else { + return line; + } + } + + public boolean isEnabled() { + return enabled; + } + + public void trace(VariableStack stack, Object msg) throws VariableNotFoundException { + trace(ThreadingContext.get(stack).toString(), msg); + } + + public void trace(String thread, Object msg) { + String str = source + ", thread " + thread + ", " + msg; + logger.info(str); + System.out.println(str); + } + + public void trace(String thread, String name, String line, Object msg) { + if (line == null) { + return; + } + String str = name + ", line " + line + ", thread " + thread + ", "+ msg; + logger.info(str); + System.out.println(str); + } + + public void trace(String thread, String line, Object msg) { + if (line == null) { + return; + } + String str = source + ", line " + line + ", thread " + thread + ", " + msg; + logger.info(str); + System.out.println(str); + } + + public void trace(String thread) { + logger.info(source + ", thread " + thread); + System.out.println(source + ", thread " + thread); + } + + private static Tracer disabledTracer, enabledTracer; + + public static Tracer getTracer(FlowNode fe) { + return getTracer(fe, null); + } + + public static Tracer getTracer(FlowNode fe, String name) { + if (globalTracingEnabled) { + return new Tracer(fe, name); + } + else { + return getGenericTracer(false); + } + } + + public static Tracer getTracer(String line, String name) { + if (globalTracingEnabled) { + return new Tracer(line, name); + } + else { + return getGenericTracer(false); + } + } + + public static Tracer getTracer(String name) { + if (globalTracingEnabled) { + return new Tracer(name); + } + else { + return getGenericTracer(false); + } + } + + public static Tracer getTracer() { + return getGenericTracer(globalTracingEnabled); + } + + private synchronized static Tracer getGenericTracer(boolean enabled) { + if (enabled) { + if (enabledTracer == null) { + enabledTracer = new Tracer(true); + } + return enabledTracer; + } + else { + if (disabledTracer == null) { + disabledTracer = new Tracer(false); + } + return disabledTracer; + } + } + + public static String getVarName(DSHandle var) { + if (var instanceof AbstractDataNode) { + AbstractDataNode data = (AbstractDataNode) var; + Path path = data.getPathFromRoot(); + String p; + if (path.isEmpty()) { + p = ""; + } + else if (path.isArrayIndex(0)) { + p = path.toString(); + } + else { + p = "." + path.toString(); + } + return data.getDisplayableName() + p; + } + else { + return String.valueOf(var); + } + } + + public static String getFutureName(Future future) { + if (future instanceof FutureWrapper) { + return getVarName(((FutureWrapper) future).getHandle()); + } + else { + return future.toString(); + } + } + + public static Object unwrapHandle(Object o) { + if (o instanceof DSHandle) { + DSHandle h = (DSHandle) o; + if (h.isClosed()) { + if (h.getType().isPrimitive()) { + if (Types.STRING.equals(h.getType())) { + return "\"" + h.getValue() + '"'; + } + else { + return h.getValue(); + } + } + else if (h.getType().isComposite()){ + return getVarName(h); + } + else { + Mapper m = h.getMapper(); + return "<" + m.map(h.getPathFromRoot()) + ">"; + } + } + else { + return "?" + getVarName(h); + } + } + else { + return o; + } + } + + public static Object fileName(AbstractDataNode n) { + if (Types.STRING.equals(n.getType())) { + return unwrapHandle(n); + } + else { + Mapper m = n.getActualMapper(); + if (m == null) { + return "?" + getVarName(n); + } + else { + return "<" + m.map(n.getPathFromRoot()) + ">"; + } + } + } +} Modified: trunk/src/org/griphyn/vdl/karajan/lib/UnitStart.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/UnitStart.java 2012-11-10 05:42:36 UTC (rev 6022) +++ trunk/src/org/griphyn/vdl/karajan/lib/UnitStart.java 2012-11-10 05:43:55 UTC (rev 6023) @@ -9,6 +9,8 @@ */ package org.griphyn.vdl.karajan.lib; +import java.util.ArrayList; +import java.util.Arrays; import java.util.Collections; import java.util.LinkedList; import java.util.List; @@ -20,6 +22,7 @@ import org.globus.cog.karajan.util.ThreadingContext; import org.globus.cog.karajan.workflow.ExecutionException; import org.globus.cog.karajan.workflow.nodes.FlowNode; +import org.griphyn.vdl.engine.Karajan; import org.griphyn.vdl.karajan.WaitingThreadsMonitor; import org.griphyn.vdl.mapping.DSHandle; @@ -33,7 +36,44 @@ public static final Arg.Optional LINE = new Arg.Optional("line", null); public static final Arg.Optional OUTPUTS = new Arg.Optional("outputs", null); + private Tracer tracer; + private List inputArgs, outputArgs; + @Override + protected void initializeStatic() { + super.initializeStatic(); + String type = (String) TYPE.getStatic(this); + if (type.equals("PROCEDURE")) { + tracer = Tracer.getTracer((FlowNode) getParent(), "APPCALL"); + } + else if (type.equals("COMPOUND")) { + tracer = Tracer.getTracer((FlowNode) getParent(), "CALL"); + } + if (tracer != null && tracer.isEnabled()) { + populateArgNames(); + } + } + + private void populateArgNames() { + String outs = (String) getStaticArguments().get("outputs"); + if (outs != null && outs.length() > 0) { + outputArgs = Arrays.asList(outs.split(",")); + } + else { + outputArgs = Collections.emptyList(); + } + String args = (String) getParent().getStaticArguments().get("arguments"); + if (args != null && args.length() > 0) { + inputArgs = new ArrayList(Arrays.asList(args.split(","))); + inputArgs.removeAll(outputArgs); + } + else { + inputArgs = Collections.emptyList(); + } + + } + + @Override public void execute(VariableStack stack) throws ExecutionException { executeSimple(stack); complete(stack); @@ -51,6 +91,10 @@ String name = (String) NAME.getStatic(this); String line = (String) LINE.getStatic(this); + if (tracer != null && tracer.isEnabled()) { + tracer.trace(thread.toString(), Karajan.demangle(name) + "(" + formatArguments(stack) + ")"); + } + log(true, type, thread, name, line); String outputs = (String) OUTPUTS.getStatic(this); @@ -59,6 +103,23 @@ } } + private String formatArguments(VariableStack stack) { + StringBuilder sb = new StringBuilder(); + boolean first = true; + for (String name : inputArgs) { + if (first) { + first = false; + } + else { + sb.append(", "); + } + sb.append(Karajan.demangle(name)); + sb.append(" = "); + sb.append(Tracer.unwrapHandle(stack.parentFrame().getVar(name))); + } + return sb.toString(); + } + private static final List EMPTY_OUTPUTS = Collections.emptyList(); private void trackOutputs(VariableStack stack, String outputs, boolean deep) { Modified: trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/ReadData.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/ReadData.java 2012-11-10 05:42:36 UTC (rev 6022) +++ trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/ReadData.java 2012-11-10 05:43:55 UTC (rev 6023) @@ -32,6 +32,7 @@ import org.globus.cog.karajan.arguments.Arg; import org.globus.cog.karajan.stack.VariableStack; import org.globus.cog.karajan.workflow.ExecutionException; +import org.griphyn.vdl.karajan.lib.Tracer; import org.griphyn.vdl.karajan.lib.VDLFunction; import org.griphyn.vdl.mapping.AbsFile; import org.griphyn.vdl.mapping.AbstractDataNode; @@ -48,14 +49,25 @@ public static final Arg DEST = new Arg.Positional("dest"); public static final Arg SRC = new Arg.Positional("src"); public static boolean warning; + + public static Tracer tracer; static { setArguments(ReadData.class, new Arg[] { DEST, SRC }); } + + @Override + protected void initializeStatic() { + super.initializeStatic(); + tracer = Tracer.getTracer(this); + } - protected Object function(VariableStack stack) throws ExecutionException { + protected Object function(VariableStack stack) throws ExecutionException { DSHandle dest = (DSHandle) DEST.getValue(stack); AbstractDataNode src = (AbstractDataNode) SRC.getValue(stack); + if (tracer.isEnabled()) { + tracer.trace(stack, Tracer.fileName(src)); + } src.waitFor(); if (src.getType().equals(Types.STRING)) { readData(dest, (String) src.getValue()); @@ -175,8 +187,8 @@ } else { String[] header = line.split("\\s+"); - Set t = new HashSet(type.getFieldNames()); - Set h = new HashSet(Arrays.asList(header)); + Set t = new HashSet(type.getFieldNames()); + Set h = new HashSet(Arrays.asList(header)); if (t.size() != h.size()) { throw new ExecutionException("File header does not match type. " + "Expected " + t.size() + " whitespace separated items. Got " + h.size() + " instead."); Modified: trunk/src/org/griphyn/vdl/mapping/MappingParamSet.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/MappingParamSet.java 2012-11-10 05:42:36 UTC (rev 6022) +++ trunk/src/org/griphyn/vdl/mapping/MappingParamSet.java 2012-11-10 05:43:55 UTC (rev 6023) @@ -29,6 +29,10 @@ public Object get(MappingParam p) { return params.get(p.getName()); } + + public Object _get(String name) { + return params.get(name); + } public boolean isPresent(MappingParam p) { return params.containsKey(p.getName()); Modified: trunk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java 2012-11-10 05:42:36 UTC (rev 6022) +++ trunk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java 2012-11-10 05:43:55 UTC (rev 6023) @@ -17,16 +17,15 @@ package org.griphyn.vdl.mapping; -import java.util.Map; +import org.apache.log4j.Logger; +import org.globus.cog.karajan.stack.VariableStack; +import org.globus.cog.karajan.workflow.futures.Future; +import org.globus.cog.karajan.workflow.futures.FutureListener; +import org.globus.cog.karajan.workflow.futures.FutureNotYetAvailable; +import org.griphyn.vdl.karajan.lib.Tracer; +import org.griphyn.vdl.type.Field; +import org.griphyn.vdl.type.Type; -import org.apache.log4j.Logger; -import org.globus.cog.karajan.stack.VariableStack; -import org.globus.cog.karajan.workflow.futures.Future; -import org.globus.cog.karajan.workflow.futures.FutureListener; -import org.globus.cog.karajan.workflow.futures.FutureNotYetAvailable; -import org.griphyn.vdl.type.Field; -import org.griphyn.vdl.type.Type; - public class RootArrayDataNode extends ArrayDataNode implements FutureListener { Logger logger = Logger.getLogger(RootArrayDataNode.class); @@ -34,7 +33,9 @@ private boolean initialized = false; private Mapper mapper; private MappingParamSet params; - private AbstractDataNode waitingMapperParam; + private AbstractDataNode waitingMapperParam; + + private static final Tracer tracer = Tracer.getTracer("VARIABLE"); /** * Instantiate a root array data node with specified type. @@ -61,6 +62,10 @@ waitingMapperParam = params.getFirstOpenParamValue(); if (waitingMapperParam != null) { waitingMapperParam.getFutureWrapper().addModificationAction(this, null); + if (tracer.isEnabled()) { + tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " wait " + + Tracer.getVarName(waitingMapperParam)); + } return; } @@ -93,7 +98,11 @@ } } - public void futureModified(Future f, VariableStack stack) { + public void futureModified(Future f, VariableStack stack) { + if (tracer.isEnabled()) { + tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " available " + + Tracer.getFutureName(f)); + } innerInit(); } @@ -139,6 +148,9 @@ private synchronized void initialized() { initialized = true; - waitingMapperParam = null; + waitingMapperParam = null; + if (tracer.isEnabled()) { + tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " initialized"); + } } } Modified: trunk/src/org/griphyn/vdl/mapping/RootDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/RootDataNode.java 2012-11-10 05:42:36 UTC (rev 6022) +++ trunk/src/org/griphyn/vdl/mapping/RootDataNode.java 2012-11-10 05:43:55 UTC (rev 6023) @@ -25,6 +25,8 @@ import org.globus.cog.karajan.workflow.futures.Future; import org.globus.cog.karajan.workflow.futures.FutureListener; import org.globus.cog.karajan.workflow.futures.FutureNotYetAvailable; +import org.griphyn.vdl.karajan.lib.Tracer; +import org.griphyn.vdl.karajan.lib.VDLFunction; import org.griphyn.vdl.type.Field; import org.griphyn.vdl.type.Type; @@ -36,6 +38,8 @@ private Mapper mapper; private MappingParamSet params; private AbstractDataNode waitingMapperParam; + + private static final Tracer tracer = Tracer.getTracer("VARIABLE"); public RootDataNode(Type type) { super(Field.Factory.createField(null, type)); @@ -62,6 +66,10 @@ waitingMapperParam = params.getFirstOpenParamValue(); if (waitingMapperParam != null) { waitingMapperParam.getFutureWrapper().addModificationAction(this, null); + if (tracer.isEnabled()) { + tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " wait " + + Tracer.getVarName(waitingMapperParam)); + } return; } @@ -101,6 +109,10 @@ } public void futureModified(Future f, VariableStack stack) { + if (tracer.isEnabled()) { + tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " available " + + Tracer.getFutureName(f)); + } innerInit(); } @@ -238,5 +250,8 @@ private synchronized void initialized() { initialized = true; waitingMapperParam = null; + if (tracer.isEnabled()) { + tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " initialized"); + } } } Modified: trunk/src/org/griphyn/vdl/util/VDL2Config.java =================================================================== --- trunk/src/org/griphyn/vdl/util/VDL2Config.java 2012-11-10 05:42:36 UTC (rev 6022) +++ trunk/src/org/griphyn/vdl/util/VDL2Config.java 2012-11-10 05:43:55 UTC (rev 6023) @@ -231,10 +231,19 @@ public boolean getSitedirKeep() { return Boolean.valueOf(getProperty(VDL2ConfigProperties.SITEDIR_KEEP, "true")).booleanValue(); } + + private Boolean provenanceLogCached; public boolean getProvenanceLog() { - return Boolean.valueOf(getProperty(VDL2ConfigProperties.PROVENANCE_LOG, "false")).booleanValue(); + if (provenanceLogCached == null) { + provenanceLogCached = Boolean.valueOf(getProperty(VDL2ConfigProperties.PROVENANCE_LOG, "false")); + } + return provenanceLogCached; } + + public boolean isTracingEnabled() { + return Boolean.valueOf(getProperty(VDL2ConfigProperties.TRACING_ENABLED, "false")).booleanValue(); + } public String getTickerDateFormat() { return getProperty("ticker.date.format"); Modified: trunk/src/org/griphyn/vdl/util/VDL2ConfigProperties.java =================================================================== --- trunk/src/org/griphyn/vdl/util/VDL2ConfigProperties.java 2012-11-10 05:42:36 UTC (rev 6022) +++ trunk/src/org/griphyn/vdl/util/VDL2ConfigProperties.java 2012-11-10 05:43:55 UTC (rev 6023) @@ -43,6 +43,7 @@ public static final String SITEDIR_KEEP = "sitedir.keep"; public static final String PROVENANCE_LOG = "provenance.log"; public static final String FILE_GC_ENABLED = "file.gc.enabled"; + public static final String TRACING_ENABLED = "tracing.enabled"; public static final Map PROPERTIES; static { @@ -137,6 +138,11 @@ PROPERTIES.put(FILE_GC_ENABLED, new PropInfo("", "Allows disabling the file garbage collector. " + "If set to false, files mapped by collectable mappers (such as the concurrent mapper) will not be " + "deleted when their swift variables go out of scope.")); + + PROPERTIES.put(TRACING_ENABLED, new PropInfo("", "Enables execution tracing. If set to 'true', " + + "operations within swift such as iterations, invocations, assignments, and declarations, as well " + + "as data dependencies will be logged. This comes at a cost in performance. It is therefore " + + "disabled by default.")); } public static Map getPropertyDescriptions() { From hategan at ci.uchicago.edu Fri Nov 9 23:56:59 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 9 Nov 2012 23:56:59 -0600 (CST) Subject: [Swift-commit] r6024 - trunk/src/org/griphyn/vdl/karajan/lib Message-ID: <20121110055659.7BF529CC9F@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-09 23:56:59 -0600 (Fri, 09 Nov 2012) New Revision: 6024 Modified: trunk/src/org/griphyn/vdl/karajan/lib/Tracer.java Log: removed trace output to console Modified: trunk/src/org/griphyn/vdl/karajan/lib/Tracer.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/Tracer.java 2012-11-10 05:43:55 UTC (rev 6023) +++ trunk/src/org/griphyn/vdl/karajan/lib/Tracer.java 2012-11-10 05:56:59 UTC (rev 6024) @@ -140,7 +140,6 @@ public void trace(String thread, Object msg) { String str = source + ", thread " + thread + ", " + msg; logger.info(str); - System.out.println(str); } public void trace(String thread, String name, String line, Object msg) { @@ -149,7 +148,6 @@ } String str = name + ", line " + line + ", thread " + thread + ", "+ msg; logger.info(str); - System.out.println(str); } public void trace(String thread, String line, Object msg) { @@ -158,12 +156,10 @@ } String str = source + ", line " + line + ", thread " + thread + ", " + msg; logger.info(str); - System.out.println(str); } public void trace(String thread) { logger.info(source + ", thread " + thread); - System.out.println(source + ", thread " + thread); } private static Tracer disabledTracer, enabledTracer; From hategan at ci.uchicago.edu Sat Nov 10 14:23:24 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sat, 10 Nov 2012 14:23:24 -0600 (CST) Subject: [Swift-commit] r6025 - in trunk/src/org/griphyn/vdl: karajan/lib karajan/lib/swiftscript mapping Message-ID: <20121110202324.479DC9CC9B@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-10 14:23:23 -0600 (Sat, 10 Nov 2012) New Revision: 6025 Modified: trunk/src/org/griphyn/vdl/karajan/lib/Stagein.java trunk/src/org/griphyn/vdl/karajan/lib/Tracer.java trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/ReadData.java trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/ReadStructured.java trunk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java trunk/src/org/griphyn/vdl/mapping/RootDataNode.java Log: rename main thread to "main" instead of an empty string when tracing; formalized messages according to bug867 Modified: trunk/src/org/griphyn/vdl/karajan/lib/Stagein.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/Stagein.java 2012-11-10 05:56:59 UTC (rev 6024) +++ trunk/src/org/griphyn/vdl/karajan/lib/Stagein.java 2012-11-10 20:23:23 UTC (rev 6025) @@ -76,9 +76,6 @@ for (Path p : fp) { AbstractDataNode n = (AbstractDataNode) var.getField(p); n.waitFor(); - if (tracer.isEnabled()) { - tracer.trace(stack, procName + " available " + Tracer.getVarName(n)); - } } } catch (DependentException e) { @@ -90,7 +87,7 @@ } catch (FutureFault f) { if (tracer.isEnabled()) { - tracer.trace(stack, procName + " wait " + Tracer.getFutureName(f.getFuture())); + tracer.trace(stack, procName + " WAIT " + Tracer.getFutureName(f.getFuture())); } throw f; } @@ -115,7 +112,7 @@ var.waitFor(); } catch (FutureFault f) { - tracer.trace(stack, procName + " waiting for " + Tracer.getFutureName(f.getFuture())); + tracer.trace(stack, procName + " WAIT " + Tracer.getFutureName(f.getFuture())); throw f; } } Modified: trunk/src/org/griphyn/vdl/karajan/lib/Tracer.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/Tracer.java 2012-11-10 05:56:59 UTC (rev 6024) +++ trunk/src/org/griphyn/vdl/karajan/lib/Tracer.java 2012-11-10 20:23:23 UTC (rev 6025) @@ -134,11 +134,11 @@ } public void trace(VariableStack stack, Object msg) throws VariableNotFoundException { - trace(ThreadingContext.get(stack).toString(), msg); + trace(threadName(stack), msg); } public void trace(String thread, Object msg) { - String str = source + ", thread " + thread + ", " + msg; + String str = source + ", thread " + threadName(thread) + ", " + msg; logger.info(str); } @@ -146,7 +146,7 @@ if (line == null) { return; } - String str = name + ", line " + line + ", thread " + thread + ", "+ msg; + String str = name + ", line " + line + ", thread " + threadName(thread) + ", "+ msg; logger.info(str); } @@ -154,14 +154,27 @@ if (line == null) { return; } - String str = source + ", line " + line + ", thread " + thread + ", " + msg; + String str = source + ", line " + line + ", thread " + threadName(thread) + ", " + msg; logger.info(str); } public void trace(String thread) { - logger.info(source + ", thread " + thread); + logger.info(source + ", thread " + threadName(thread)); } + private String threadName(String thread) { + if (thread.isEmpty()) { + return "main"; + } + else { + return thread; + } + } + + private String threadName(VariableStack stack) throws VariableNotFoundException { + return threadName(ThreadingContext.get(stack).toString()); + } + private static Tracer disabledTracer, enabledTracer; public static Tracer getTracer(FlowNode fe) { @@ -245,8 +258,8 @@ } public static Object unwrapHandle(Object o) { - if (o instanceof DSHandle) { - DSHandle h = (DSHandle) o; + if (o instanceof AbstractDataNode) { + AbstractDataNode h = (AbstractDataNode) o; if (h.isClosed()) { if (h.getType().isPrimitive()) { if (Types.STRING.equals(h.getType())) { @@ -260,8 +273,7 @@ return getVarName(h); } else { - Mapper m = h.getMapper(); - return "<" + m.map(h.getPathFromRoot()) + ">"; + return fileName(h); } } else { @@ -274,17 +286,12 @@ } public static Object fileName(AbstractDataNode n) { - if (Types.STRING.equals(n.getType())) { - return unwrapHandle(n); + Mapper m = n.getActualMapper(); + if (m == null) { + return "?" + getVarName(n); } else { - Mapper m = n.getActualMapper(); - if (m == null) { - return "?" + getVarName(n); - } - else { - return "<" + m.map(n.getPathFromRoot()) + ">"; - } + return "<" + m.map(n.getPathFromRoot()) + ">"; } } } Modified: trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/ReadData.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/ReadData.java 2012-11-10 05:56:59 UTC (rev 6024) +++ trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/ReadData.java 2012-11-10 20:23:23 UTC (rev 6025) @@ -50,7 +50,7 @@ public static final Arg SRC = new Arg.Positional("src"); public static boolean warning; - public static Tracer tracer; + public Tracer tracer; static { setArguments(ReadData.class, new Arg[] { DEST, SRC }); @@ -59,14 +59,14 @@ @Override protected void initializeStatic() { super.initializeStatic(); - tracer = Tracer.getTracer(this); + tracer = Tracer.getTracer(this, "SWIFTCALL"); } protected Object function(VariableStack stack) throws ExecutionException { DSHandle dest = (DSHandle) DEST.getValue(stack); AbstractDataNode src = (AbstractDataNode) SRC.getValue(stack); if (tracer.isEnabled()) { - tracer.trace(stack, Tracer.fileName(src)); + tracer.trace(stack, "readData(" + Tracer.unwrapHandle(src) + ")"); } src.waitFor(); if (src.getType().equals(Types.STRING)) { Modified: trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/ReadStructured.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/ReadStructured.java 2012-11-10 05:56:59 UTC (rev 6024) +++ trunk/src/org/griphyn/vdl/karajan/lib/swiftscript/ReadStructured.java 2012-11-10 20:23:23 UTC (rev 6025) @@ -29,11 +29,11 @@ import org.globus.cog.karajan.arguments.Arg; import org.globus.cog.karajan.stack.VariableStack; import org.globus.cog.karajan.workflow.ExecutionException; +import org.griphyn.vdl.karajan.lib.Tracer; import org.griphyn.vdl.karajan.lib.VDLFunction; import org.griphyn.vdl.mapping.AbsFile; import org.griphyn.vdl.mapping.AbstractDataNode; import org.griphyn.vdl.mapping.DSHandle; -import org.griphyn.vdl.mapping.HandleOpenException; import org.griphyn.vdl.mapping.Path; import org.griphyn.vdl.mapping.PhysicalFormat; import org.griphyn.vdl.type.Types; @@ -48,10 +48,21 @@ static { setArguments(ReadStructured.class, new Arg[] { DEST, SRC }); } + + public Tracer tracer; + + @Override + protected void initializeStatic() { + super.initializeStatic(); + tracer = Tracer.getTracer(this, "SWIFTCALL"); + } protected Object function(VariableStack stack) throws ExecutionException { DSHandle dest = (DSHandle) DEST.getValue(stack); AbstractDataNode src = (AbstractDataNode) SRC.getValue(stack); + if (tracer.isEnabled()) { + tracer.trace(stack, "readData2(" + Tracer.unwrapHandle(src) + ")"); + } src.waitFor(); if (src.getType().equals(Types.STRING)) { readData(dest, (String) src.getValue()); Modified: trunk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java 2012-11-10 05:56:59 UTC (rev 6024) +++ trunk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java 2012-11-10 20:23:23 UTC (rev 6025) @@ -63,7 +63,7 @@ if (waitingMapperParam != null) { waitingMapperParam.getFutureWrapper().addModificationAction(this, null); if (tracer.isEnabled()) { - tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " wait " + tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " WAIT " + Tracer.getVarName(waitingMapperParam)); } return; @@ -98,11 +98,7 @@ } } - public void futureModified(Future f, VariableStack stack) { - if (tracer.isEnabled()) { - tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " available " - + Tracer.getFutureName(f)); - } + public void futureModified(Future f, VariableStack stack) { innerInit(); } @@ -150,7 +146,7 @@ initialized = true; waitingMapperParam = null; if (tracer.isEnabled()) { - tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " initialized"); + tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " INITIALIZED"); } } } Modified: trunk/src/org/griphyn/vdl/mapping/RootDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/RootDataNode.java 2012-11-10 05:56:59 UTC (rev 6024) +++ trunk/src/org/griphyn/vdl/mapping/RootDataNode.java 2012-11-10 20:23:23 UTC (rev 6025) @@ -67,7 +67,7 @@ if (waitingMapperParam != null) { waitingMapperParam.getFutureWrapper().addModificationAction(this, null); if (tracer.isEnabled()) { - tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " wait " + tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " WAIT " + Tracer.getVarName(waitingMapperParam)); } return; @@ -109,10 +109,6 @@ } public void futureModified(Future f, VariableStack stack) { - if (tracer.isEnabled()) { - tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " available " - + Tracer.getFutureName(f)); - } innerInit(); } @@ -251,7 +247,7 @@ initialized = true; waitingMapperParam = null; if (tracer.isEnabled()) { - tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " initialized"); + tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " INITIALIZED"); } } } From hategan at ci.uchicago.edu Sat Nov 10 19:53:55 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sat, 10 Nov 2012 19:53:55 -0600 (CST) Subject: [Swift-commit] r6026 - trunk/src/org/griphyn/vdl/engine Message-ID: <20121111015355.BC7079CC90@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-10 19:53:55 -0600 (Sat, 10 Nov 2012) New Revision: 6026 Added: trunk/src/org/griphyn/vdl/engine/CompilerUtils.java trunk/src/org/griphyn/vdl/engine/Warnings.java Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java trunk/src/org/griphyn/vdl/engine/VariableScope.java Log: make error messages about variables use line numbers instead of cryptic java hash codes of some random object; warnings only displayed once per message Added: trunk/src/org/griphyn/vdl/engine/CompilerUtils.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/CompilerUtils.java (rev 0) +++ trunk/src/org/griphyn/vdl/engine/CompilerUtils.java 2012-11-11 01:53:55 UTC (rev 6026) @@ -0,0 +1,41 @@ +//---------------------------------------------------------------------- +//This code is developed as part of the Java CoG Kit project +//The terms of the license can be found at http://www.cogkit.org/license +//This message may not be removed or altered. +//---------------------------------------------------------------------- + +/* + * Created on Nov 10, 2012 + */ +package org.griphyn.vdl.engine; + +import org.apache.xmlbeans.XmlObject; +import org.w3c.dom.Node; + +public class CompilerUtils { + public static String getLine(Node n) { + if (n == null) { + return "line unknown"; + } + Node src = n.getAttributes().getNamedItem("src"); + if (src == null) { + return getLine(n.getParentNode()); + } + else { + return src.getNodeValue(); + } + } + + public static String getLine(XmlObject src) { + if (src == null) { + return null; + } + String loc = src.getDomNode().getAttributes().getNamedItem("src").getNodeValue(); + if (loc == null) { + return null; + } + else { + return loc.substring(loc.indexOf(' ') + 1); + } + } +} Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/Karajan.java 2012-11-10 20:23:23 UTC (rev 6025) +++ trunk/src/org/griphyn/vdl/engine/Karajan.java 2012-11-11 01:53:55 UTC (rev 6026) @@ -64,6 +64,8 @@ import org.safehaus.uuid.UUIDGenerator; import org.w3c.dom.Node; +import static org.griphyn.vdl.engine.CompilerUtils.*; + public class Karajan { public static final Logger logger = Logger.getLogger(Karajan.class); @@ -172,26 +174,7 @@ protected StringTemplate template(String name) { return m_templates.getInstanceOf(name); } - - private void warn(XmlObject obj, String msg) { - msg = "Warning: " + msg + ", at " + getLine(obj.getDomNode()); - logger.info(msg); - System.err.println(msg); - } - private String getLine(Node n) { - if (n == null) { - return "line unknown"; - } - Node src = n.getAttributes().getNamedItem("src"); - if (src == null) { - return getLine(n.getParentNode()); - } - else { - return src.getNodeValue(); - } - } - private void processImports(Program prog) throws CompilationException { Imports imports = prog.getImports(); @@ -351,10 +334,6 @@ } } - private String getLine(String src) { - return src.substring(src.indexOf(' ') + 1); - } - private void setVariableUsed(String s) { usedVariables.add(s); } @@ -364,7 +343,7 @@ VariableScope innerScope = new VariableScope(this, outerScope, VariableScope.ENCLOSURE_NONE); StringTemplate procST = template("procedure"); containingScope.bodyTemplate.setAttribute("procedures", procST); - procST.setAttribute("line", getLine(proc.getSrc())); + procST.setAttribute("line", getLine(proc)); procST.setAttribute("name", mangle(proc.getName())); for (int i = 0; i < proc.sizeOfOutputArray(); i++) { FormalParameter param = proc.getOutputArray(i); @@ -376,7 +355,7 @@ procST.setAttribute("arguments", paramST); String type = normalize(param.getType().getLocalPart()); checkIsTypeDefined(type); - innerScope.addVariable(param.getName(), type); + innerScope.addVariable(param.getName(), type, param); } for (int i = 0; i < proc.sizeOfInputArray(); i++) { FormalParameter param = proc.getInputArray(i); @@ -388,7 +367,7 @@ procST.setAttribute("arguments", paramST); String type = normalize(param.getType().getLocalPart()); checkIsTypeDefined(type); - outerScope.addVariable(param.getName(), type); + outerScope.addVariable(param.getName(), type, param); } Binding bind; @@ -464,7 +443,7 @@ public void variableForSymbol(Variable var, VariableScope scope) throws CompilationException { checkIsTypeDefined(var.getType().getLocalPart()); - scope.addVariable(var.getName(), var.getType().getLocalPart(), var.getIsGlobal()); + scope.addVariable(var.getName(), var.getType().getLocalPart(), var.getIsGlobal(), var); } public void variable(Variable var, VariableScope scope) throws CompilationException { @@ -472,7 +451,7 @@ variableST.setAttribute("name", var.getName()); variableST.setAttribute("type", var.getType().getLocalPart()); variableST.setAttribute("isGlobal", Boolean.valueOf(var.getIsGlobal())); - variableST.setAttribute("line", getLine(var.getSrc())); + variableST.setAttribute("line", getLine(var)); variables.add(variableST); if(!var.isNil()) { @@ -519,7 +498,7 @@ scope.bodyTemplate.setAttribute("declarations",variableDeclarationST); StringTemplate paramValueST=expressionToKarajan(param.getAbstractExpression(),scope); String paramValueType = datatype(paramValueST); - scope.addVariable(parameterVariableName, paramValueType); + scope.addVariable(parameterVariableName, paramValueType, param); variableDeclarationST.setAttribute("type", paramValueType); StringTemplate variableReferenceST = template("id"); variableReferenceST.setAttribute("var",parameterVariableName); @@ -575,7 +554,7 @@ " to a variable of type " + datatype(varST)); assignST.setAttribute("var", varST); assignST.setAttribute("value", valueST); - assignST.setAttribute("line", getLine(assign.getSrc())); + assignST.setAttribute("line", getLine(assign)); String rootvar = abstractExpressionToRootVariable(assign.getAbstractExpressionArray(0)); scope.addWriter(rootvar, new Integer(callID++), rootVariableIsPartial(assign.getAbstractExpressionArray(0))); scope.appendStatement(assignST); @@ -716,7 +695,7 @@ ("Unknown procedure invocation mode "+proc.getInvocationMode()); } callST.setAttribute("func", mangle(procName)); - callST.setAttribute("line", getLine(call.getSrc())); + callST.setAttribute("line", getLine(call)); /* Does number of input arguments match */ for (int i = 0; i < proc.sizeOfInputArray(); i++) { if (proc.getInputArray(i).isOptional()) @@ -919,10 +898,10 @@ VariableScope loopScope = new VariableScope(this, scope, VariableScope.ENCLOSURE_LOOP); VariableScope innerScope = new VariableScope(this, loopScope, VariableScope.ENCLOSURE_LOOP); - loopScope.addVariable(iterate.getVar(), "int"); + loopScope.addVariable(iterate.getVar(), "int", iterate); StringTemplate iterateST = template("iterate"); - iterateST.setAttribute("line", getLine(iterate.getSrc())); + iterateST.setAttribute("line", getLine(iterate)); iterateST.setAttribute("var", iterate.getVar()); innerScope.bodyTemplate = iterateST; @@ -959,11 +938,11 @@ if (itemType == null) { throw new CompilationException("You can iterate through an array structure only"); } - innerScope.addVariable(foreach.getVar(), itemType); + innerScope.addVariable(foreach.getVar(), itemType, foreach); foreachST.setAttribute("indexVar", foreach.getIndexVar()); foreachST.setAttribute("indexVarType", keyType); if(foreach.getIndexVar() != null) { - innerScope.addVariable(foreach.getIndexVar(), keyType); + innerScope.addVariable(foreach.getIndexVar(), keyType, foreach); } innerScope.bodyTemplate = foreachST; @@ -992,7 +971,7 @@ StringTemplate ifST = template("if"); StringTemplate conditionST = expressionToKarajan(ifstat.getAbstractExpression(), scope); ifST.setAttribute("condition", conditionST.toString()); - ifST.setAttribute("line", getLine(ifstat.getSrc())); + ifST.setAttribute("line", getLine(ifstat)); if (!datatype(conditionST).equals("boolean")) throw new CompilationException ("Condition in if statement has to be of boolean type."); @@ -1157,7 +1136,7 @@ public StringTemplate function(Function func, VariableScope scope) throws CompilationException { StringTemplate funcST = template("function"); funcST.setAttribute("name", mangle(func.getName())); - funcST.setAttribute("line", getLine(func.getSrc())); + funcST.setAttribute("line", getLine(func)); ProcedureSignature funcSignature = functionsMap.get(func.getName()); if(funcSignature == null) { throw new CompilationException("Unknown function: @"+func.getName()); @@ -1506,7 +1485,7 @@ /* Functions have only one output parameter */ st.setAttribute("datatype", funcSignature.getOutputArray(0).getType()); if (funcSignature.isDeprecated()) { - warn(f, "Function " + name + " is deprecated"); + Warnings.warn(f, "Function " + name + " is deprecated"); } } else throw new CompilationException("Function " + name + " is not defined."); @@ -1530,12 +1509,12 @@ "return value to be used in an expression."); } - warn(c, "Procedure " + name + " is deprecated"); + Warnings.warn(c, "Procedure " + name + " is deprecated"); StringTemplate call = template("callexpr"); String type = funcSignature.getOutputArray(0).getType(); - subscope.addInternalVariable("swift#callintermediate", type); + subscope.addInternalVariable("swift#callintermediate", type, null); call.setAttribute("datatype", type); call.setAttribute("call", call(c, subscope, true)); Modified: trunk/src/org/griphyn/vdl/engine/VariableScope.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/VariableScope.java 2012-11-10 20:23:23 UTC (rev 6025) +++ trunk/src/org/griphyn/vdl/engine/VariableScope.java 2012-11-11 01:53:55 UTC (rev 6026) @@ -17,14 +17,7 @@ package org.griphyn.vdl.engine; -import org.antlr.stringtemplate.StringTemplate; - -import org.apache.log4j.Logger; - -import org.griphyn.vdl.karajan.CompilationException; - import java.util.ArrayList; -import java.util.Collections; import java.util.HashMap; import java.util.HashSet; import java.util.Iterator; @@ -32,7 +25,12 @@ import java.util.Map; import java.util.Set; +import org.antlr.stringtemplate.StringTemplate; +import org.apache.log4j.Logger; +import org.apache.xmlbeans.XmlObject; +import org.griphyn.vdl.karajan.CompilationException; + public class VariableScope { /** permit array up-assignment, but not entire variables */ @@ -76,7 +74,7 @@ private Set inhibitClosing = new HashSet(); public VariableScope(Karajan c, VariableScope parent) { - this(c,parent,ENCLOSURE_ALL); + this(c, parent, ENCLOSURE_ALL); } /** Creates a new variable scope. @@ -87,11 +85,14 @@ scopes will be handled. */ public VariableScope(Karajan c, VariableScope parent, int a) { - if(parentScope!=null) { - logger.debug("New scope "+hashCode()+" with parent scope "+parentScope.hashCode()); - } else { - logger.debug("New scope "+hashCode()+" with no parent."); - } + if (logger.isDebugEnabled()) { + if (parentScope != null) { + logger.debug("New scope " + hashCode() + " with parent scope " + parentScope.hashCode()); + } + else { + logger.debug("New scope " + hashCode() + " with no parent."); + } + } compiler = c; parentScope = parent; enclosureType = a; @@ -105,13 +106,13 @@ /** Set of variables (String token names) that are declared in this scope - not variables in parent scopes, though. */ - Set variables = new HashSet(); + Map variables = new HashMap(); Map varTypes = new HashMap(); /** Set of variables (String token names) which are global and declared in this scope (which must be a root scope). Variables in this set must also appear in the variables set. */ - Set globals = new HashSet(); + Map globals = new HashMap(); /** Asserts that a named variable is declared in this scope. Might also eventually contain more information about the @@ -119,41 +120,41 @@ declaration already exists. Perhaps error in same scope and warning if it shadows an outer scope? */ - public void addVariable(String name, String type) throws CompilationException { - addVariable(name,type,false); + public void addVariable(String name, String type, XmlObject src) throws CompilationException { + addVariable(name, type, false, src); } public void inhibitClosing(String name) { inhibitClosing.add(name); } - public void addVariable(String name, String type, boolean global) throws CompilationException { - logger.debug("Adding variable "+name+" of type "+type+" to scope "+hashCode()); + public void addVariable(String name, String type, boolean global, XmlObject src) throws CompilationException { + if (logger.isDebugEnabled()) { + logger.debug("Adding variable " + name + " of type " + type + " to scope " + hashCode()); + } if(isVariableDefined(name)) { - throw new CompilationException("Variable "+name+" is already defined."); + throw new CompilationException("Variable " + name + ", on line " + + CompilerUtils.getLine(src) + ", was already defined on line " + getDefinitionLine(name)); } // TODO does this if() ever fire? or is it always taken // by the above if? in which case isVariableDefined should // be replaced by is locally defined test. if(parentScope != null && parentScope.isVariableDefined(name)) { - logger.warn("Variable "+name+" defined in scope "+hashCode() - + " shadows variable of same name in scope " - + parentScope.hashCode()); + Warnings.warn("Variable " + name + ", defined on line " + CompilerUtils.getLine(src) + + ", shadows variable of same name on line " + parentScope.getDefinitionLine(name)); } if(global && this != rootScope) { - throw new CompilationException("Global "+name+" can only be declared in the root scope of a program."); + throw new CompilationException("Global " + name + " can only be declared in the root scope of a program."); } - boolean added = variables.add(name); + variables.put(name, src); varTypes.put(name, type); - if(!added) throw new CompilationException("Could not add variable "+name+" to scope."); - - if(global) { - added = globals.add(name); - if(!added) throw new CompilationException("Could not add global variable "+name+" to scope."); + + if (global) { + globals.put(name, src); } } @@ -161,22 +162,47 @@ * Does pretty much the same as addVariable() except it doesn't throw * an exception if the variable is defined in a parent scope */ - public void addInternalVariable(String name, String type) throws CompilationException { - logger.debug("Adding internal variable " + name + " of type " + type + " to scope " + hashCode()); + public void addInternalVariable(String name, String type, XmlObject src) throws CompilationException { + if (logger.isDebugEnabled()) { + logger.debug("Adding internal variable " + name + " of type " + type + " to scope " + hashCode()); + } if(isVariableLocallyDefined(name)) { - throw new CompilationException("Variable " + name + " is already defined."); + throw new CompilationException("Variable " + name + ", on line " + + CompilerUtils.getLine(src) + ", was already defined on line " + getDefinitionLine(name)); } - boolean added = variables.add(name); + variables.put(name, src); varTypes.put(name, type); - if(!added) throw new CompilationException("Could not add variable "+name+" to scope."); } + + public String getDefinitionLine(String name) { + XmlObject src = rootScope.getGlobalSrc(name); + if (src == null) { + src = getSrcRecursive(name); + } + if (src != null) { + return CompilerUtils.getLine(src); + } + else { + return "unknown"; + } + } public boolean isVariableDefined(String name) { if(rootScope.isGlobalDefined(name)) return true; return isVariableDefinedRecursive(name); } + + private XmlObject getSrcRecursive(String name) { + XmlObject src = variables.get(name); + if (src == null) { + if (enclosureType != ENCLOSURE_PROCEDURE && parentScope != null) { + src = parentScope.getSrcRecursive(name); + } + } + return src; + } private boolean isVariableDefinedRecursive(String name) { if(isVariableLocallyDefined(name)) return true; @@ -185,8 +211,12 @@ } public boolean isGlobalDefined(String name) { - return globals.contains(name); + return globals.containsKey(name); } + + public XmlObject getGlobalSrc(String name) { + return globals.get(name); + } /** note that this will return variable types for variables in containing scopes even if they are not accessible. non-null @@ -207,7 +237,7 @@ public boolean isVariableWriteable(String name, boolean partialWriter) { if(isVariableLocallyDefined(name)) return true; if(parentScope != null && parentScope.isVariableWriteable(name, true) && enclosureType == ENCLOSURE_CONDITION) { - logger.warn("Warning: variable " + name + " might have multiple writers"); + Warnings.warn("Variable " + name + ", defined on line " + getDefinitionLine(name) + ", might have multiple writers"); return true; } if(parentScope != null && parentScope.isVariableWriteable(name, partialWriter) && enclosureType == ENCLOSURE_ALL) return true; @@ -217,7 +247,7 @@ public boolean isVariableLocallyDefined(String name) { - return variables.contains(name); + return variables.containsKey(name); } /** List of templates to be executed in sequence after the present @@ -234,18 +264,22 @@ public void addWriter(String variableName, Object closeID, boolean partialWriter) throws CompilationException { if(!isVariableDefined(variableName)) { - throw new CompilationException("Variable "+variableName+" is not defined"); + throw new CompilationException("Variable " + variableName + " is not defined"); } if(isVariableLocallyDefined(variableName)) { StringTemplate ld = getLocalDeclaration(variableName); - if(ld != null) { - if(!partialWriter && ld.getAttribute("waitfor")!=null) { - throw new CompilationException("variable "+variableName+" has multiple writers."); + if (ld != null) { + if(!partialWriter && ld.getAttribute("waitfor") != null) { + throw new CompilationException("variable " + variableName + ", defined on line " + + getDefinitionLine(variableName) + ", has multiple writers."); } ld.setAttribute("waitfor", closeID); - } else { - logger.debug("Variable "+variableName+" is local but has no template."); + } + else { + if (logger.isDebugEnabled()) { + logger.debug("Variable " + variableName + " is local but has no template."); + } } outputs.add(variableName); if (!inhibitClosing.contains(variableName)) { @@ -276,10 +310,13 @@ if(!statementList.contains(closeID)) { statementList.add(closeID); } - logger.debug("added "+closeID+" to variable "+variableName+" in scope "+hashCode()); + if (logger.isDebugEnabled()) { + logger.debug("added " + closeID + " to variable " + + variableName + " in scope " + hashCode()); + } } else { isVariableWriteable(variableName, partialWriter); - throw new CompilationException("variable "+variableName+" is not writeable in this scope"); + throw new CompilationException("variable " + variableName + " is not writeable in this scope"); } } } @@ -306,15 +343,19 @@ @SuppressWarnings("unchecked") List list = (List) decls; for (StringTemplate declST : list) { - logger.debug("looking at declaration "+declST); + if (logger.isDebugEnabled()) { + logger.debug("looking at declaration " + declST); + } try { if(declST.getAttribute("name").equals(name)) { - logger.debug("thats the declaration for "+name); + if (logger.isDebugEnabled()) { + logger.debug("thats the declaration for " + name); + } return declST; } - } catch(java.util.NoSuchElementException nse) { - logger.debug - ("it so definitely wasn't in that one, we got an exception."); + } + catch (java.util.NoSuchElementException nse) { + logger.debug("it so definitely wasn't in that one, we got an exception."); // TODO this is not a nice use of exceptions... } } @@ -387,7 +428,7 @@ public List getCleanups() { List cleanups = new ArrayList(); - for (String var : variables) { + for (String var : variables.keySet()) { String type = getVariableType(var); if (!org.griphyn.vdl.type.Types.isPrimitive(type)) { cleanups.add(var); Added: trunk/src/org/griphyn/vdl/engine/Warnings.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/Warnings.java (rev 0) +++ trunk/src/org/griphyn/vdl/engine/Warnings.java 2012-11-11 01:53:55 UTC (rev 6026) @@ -0,0 +1,41 @@ +//---------------------------------------------------------------------- +//This code is developed as part of the Java CoG Kit project +//The terms of the license can be found at http://www.cogkit.org/license +//This message may not be removed or altered. +//---------------------------------------------------------------------- + +/* + * Created on Nov 10, 2012 + */ +package org.griphyn.vdl.engine; + +import java.util.HashSet; +import java.util.Set; + +import org.apache.log4j.Logger; +import org.apache.xmlbeans.XmlObject; +import org.w3c.dom.Node; + +public class Warnings { + public static final Logger logger = Logger.getLogger(Warnings.class); + + private static Set warnings = new HashSet(); + + public static void warn(XmlObject obj, String msg) { + if (!warnings.contains(msg)) { + warnings.add(msg); + msg = "Warning: " + msg + ", at " + CompilerUtils.getLine(obj.getDomNode()); + logger.info(msg); + System.err.println(msg); + } + } + + public static void warn(String msg) { + if (!warnings.contains(msg)) { + warnings.add(msg); + msg = "Warning: " + msg; + logger.info(msg); + System.err.println(msg); + } + } +} From wilde at ci.uchicago.edu Sun Nov 11 14:39:25 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Sun, 11 Nov 2012 14:39:25 -0600 (CST) Subject: [Swift-commit] r6027 - trunk/bin Message-ID: <20121111203925.A2FAA9CCC8@svn.ci.uchicago.edu> Author: wilde Date: 2012-11-11 14:39:24 -0600 (Sun, 11 Nov 2012) New Revision: 6027 Added: trunk/bin/slurm-qsub-emulator Modified: trunk/bin/slurm-qsub Log: Make slurm-qsub call a swift-modified copy of the SLURM qsub emulator command, which sets --exclusive on all requests to supress node sharing. This should be changed to allow the user to request shared or exclusive. Exclusive is a reasonable default. Modified: trunk/bin/slurm-qsub =================================================================== --- trunk/bin/slurm-qsub 2012-11-11 01:53:55 UTC (rev 6026) +++ trunk/bin/slurm-qsub 2012-11-11 20:39:24 UTC (rev 6027) @@ -3,7 +3,7 @@ now=$(date "+%Y.%m%d.%H:%M:%S") echo $now $0: $* >> $HOME/pbslog -slurmqsub=$( which qsub ) +slurmqsub=$( which slurm-qsub-emulator ) script=$1.slurm echo '#! /bin/sh' >$script # echo 'export PBS_NODELIST=SLURM_NODELIST'' >>$script # FIXME: convert SLURM_NODELIST to PBS_NODEFILE here... Added: trunk/bin/slurm-qsub-emulator =================================================================== --- trunk/bin/slurm-qsub-emulator (rev 0) +++ trunk/bin/slurm-qsub-emulator 2012-11-11 20:39:24 UTC (rev 6027) @@ -0,0 +1,432 @@ +#! /usr/bin/perl -w +############################################################################### +# +# qsub - submit a batch job in familar pbs format. +# +# +############################################################################### +# Copyright (C) 2007 The Regents of the University of California. +# Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER). +# Written by Danny Auble . +# CODE-OCEC-09-009. All rights reserved. +# +# This file is part of SLURM, a resource management program. +# For details, see . +# Please also read the included file: DISCLAIMER. +# +# SLURM is free software; you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# In addition, as a special exception, the copyright holders give permission +# to link the code of portions of this program with the OpenSSL library under +# certain conditions as described in each individual source file, and +# distribute linked combinations including the two. You must obey the GNU +# General Public License in all respects for all of the code used other than +# OpenSSL. If you modify file(s) with this exception, you may extend this +# exception to your version of the file(s), but you are not obligated to do +# so. If you do not wish to do so, delete this exception statement from your +# version. If you delete this exception statement from all source files in +# the program, then also delete it here. +# +# SLURM is distributed in the hope that it will be useful, but WITHOUT ANY +# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more +# details. +# +# You should have received a copy of the GNU General Public License along +# with SLURM; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +############################################################################### + +use strict; +use FindBin; +use Getopt::Long 2.24 qw(:config no_ignore_case require_order); +use lib qw(/software/slurm-2.4-el6-x86_64/lib64/perl5); +use autouse 'Pod::Usage' => qw(pod2usage); +use Slurm ':all'; +use Switch; + +my ($start_time, + $account, + # $checkpoint_interval, + $directive_prefix, + $err_path, + $interactive, + $hold, +# $join, +# $keep, + $resource_list, + $mail_options, + $mail_user_list, + $job_name, + $out_path, + $priority, + $destination, +# $rerunable, +# $script_path, +# $running_user_list, +# $variable_list, +# $all_env, + $additional_attributes, +# $no_std, + $help, + $man); + +# my $sbatch = "${FindBin::Bin}/sbatch"; +# my $salloc = "${FindBin::Bin}/salloc"; +# my $srun = "${FindBin::Bin}/srun"; + +my $sbatch = "sbatch"; +my $salloc = "salloc"; +my $srun = "$srun"; + +GetOptions('a=s' => \$start_time, + 'A=s' => \$account, +# 'c=i' => \$checkpoint_interval, + 'C=s' => \$directive_prefix, + 'e=s' => \$err_path, + 'h' => \$hold, + 'I' => \$interactive, + 'j:s' => sub { warn "option -j is the default, " . + "stdout/stderr go into the same file\n" }, +# 'k=s' => \$keep, + 'l=s' => \$resource_list, + 'm=s' => \$mail_options, + 'M=s' => \$mail_user_list, + 'N=s' => \$job_name, + 'o=s' => \$out_path, + 'p=i' => \$priority, + 'q=s' => \$destination, +# 'r=s' => \$rerunable, +# 'S=s' => \$script_path, +# 'u=s' => \$running_user_list, + 'v=s' => sub { warn "option -v is not supported, " . + "since the current environment " . + "is exported by default\n" }, + 'V' => sub { warn "option -V is not necessary, " . + "since the current environment " . + "is exported by default\n" }, + 'W' => \$additional_attributes, +# 'z' => \$no_std, + 'help|?' => \$help, + 'man' => \$man, + ) + or pod2usage(2); + +# Display usage if necessary +pod2usage(0) if $help; +if ($man) { + if ($< == 0) { # Cannot invoke perldoc as root + my $id = eval { getpwnam("nobody") }; + $id = eval { getpwnam("nouser") } unless defined $id; + $id = -2 unless defined $id; + $< = $id; + } + $> = $<; # Disengage setuid + $ENV{PATH} = "/bin:/usr/bin"; # Untaint PATH + delete @ENV{'IFS', 'CDPATH', 'ENV', 'BASH_ENV'}; + if ($0 =~ /^([-\/\w\.]+)$/) { $0 = $1; } # Untaint $0 + else { die "Illegal characters were found in \$0 ($0)\n"; } + pod2usage(-exitstatus => 0, -verbose => 2); +} + +# Use sole remaining argument as jobIds +my $script; +if ($ARGV[0]) { + foreach (@ARGV) { + $script .= "$_ "; + } +} else { + pod2usage(2); +} +my %res_opts; +my %node_opts; + +if($resource_list) { + %res_opts = %{parse_resource_list($resource_list)}; + +# while((my $key, my $val) = each(%res_opts)) { +# print "$key = "; +# if($val) { +# print "$val\n"; +# } else { +# print "\n"; +# } +# } + + if($res_opts{nodes}) { + %node_opts = %{parse_node_opts($res_opts{nodes})}; + } +} + +my $command; + +if($interactive) { + $command = "$salloc"; + +} else { + $command = "$sbatch"; + + $command .= " -D $directive_prefix" if $directive_prefix; + $command .= " -e $err_path" if $err_path; + $command .= " -o $out_path" if $out_path; +} + +$command .= " -N$node_opts{node_cnt}" if $node_opts{node_cnt}; +$command .= " -n$node_opts{task_cnt}" if $node_opts{task_cnt}; +$command .= " -w$node_opts{hostlist}" if $node_opts{hostlist}; + +if($res_opts{walltime}) { + $command .= " -t$res_opts{walltime}"; +} elsif($res_opts{cput}) { + $command .= " -t$res_opts{cput}"; +} elsif($res_opts{pcput}) { + $command .= " -t$res_opts{pcput}"; +} + +$command .= " --tmp=$res_opts{file}" if $res_opts{file}; +$command .= " --mem=$res_opts{mem}" if $res_opts{mem}; +$command .= " --nice=$res_opts{nice}" if $res_opts{nice}; +# Cray-specific options +$command .= " -n$res_opts{mppwidth}" if $res_opts{mppwidth}; +$command .= " -w$res_opts{mppnodes}" if $res_opts{mppnodes}; +$command .= " --cpus-per-task=$res_opts{mppdepth}" if $res_opts{mppdepth}; +$command .= " --ntasks-per-node=$res_opts{mppnppn}" if $res_opts{mppnppn}; + +$command .= " --begin=$start_time" if $start_time; +$command .= " --account=$account" if $account; +$command .= " -H" if $hold; + +if($mail_options) { + $command .= " --mail-type=FAIL" if $mail_options =~ /a/; + $command .= " --mail-type=BEGIN" if $mail_options =~ /b/; + $command .= " --mail-type=END" if $mail_options =~ /e/; +} +$command .= " --mail-user=$mail_user_list" if $mail_user_list; +$command .= " -J $job_name" if $job_name; +$command .= " --nice=$priority" if $priority; +$command .= " -p $destination" if $destination; +$command .= " -C $additional_attributes" if $additional_attributes; + +$command .= " --exclusive"; # MWilde: added for midway, which schedules by core. + +$command .= " $script"; + +system("(echo; echo $command; echo) >> $HOME/pbslog"); + +system($command); + + +sub parse_resource_list { + my ($rl) = @_; + my %opt = ('arch' => "", + 'cput' => "", + 'file' => "", + 'host' => "", + 'mem' => "", + 'nice' => "", + 'nodes' => "", + 'opsys' => "", + 'other' => "", + 'pcput' => "", + 'pmem' => "", + 'pvmem' => "", + 'software' => "", + 'vmem' => "", + # Cray-specific resources + 'mppwidth' => "", + 'mppdepth' => "", + 'mppnppn' => "", + 'mppmem' => "", + 'mppnodes' => "", + 'walltime' => "" + ); + my @keys = keys(%opt); + + foreach my $key (@keys) { + #print "$rl\n"; + ($opt{$key}) = $rl =~ m/$key=([\w:\+=+]+)/; + + } + if($opt{cput}) { + $opt{cput} = get_minutes($opt{cput}); + } + + if($opt{mppmem}) { + $opt{mem} = convert_mb_format($opt{mppmem}); + } elsif($opt{mem}) { + $opt{mem} = convert_mb_format($opt{mem}); + } + + if($opt{file}) { + $opt{file} = convert_mb_format($opt{file}); + } + + return \%opt; +} + +sub parse_node_opts { + my ($node_string) = @_; + my %opt = ('node_cnt' => 0, + 'hostlist' => "", + 'task_cnt' => 0 + ); + while($node_string =~ /ppn=(\d+)/g) { + $opt{task_cnt} += $1; + } + + my $hl = Slurm::Hostlist::create(""); + + my @parts = split(/\+/, $node_string); + foreach my $part (@parts) { + my @sub_parts = split(/:/, $part); + foreach my $sub_part (@sub_parts) { + if($sub_part =~ /ppn=(\d+)/) { + next; + } elsif($sub_part =~ /^(\d+)/) { + $opt{node_cnt} += $1; + } else { + if(!Slurm::Hostlist::push($hl, $sub_part)) { + print "problem pushing host $sub_part onto hostlist\n"; + } + } + } + } + + $opt{hostlist} = Slurm::Hostlist::ranged_string($hl); + + my $hl_cnt = Slurm::Hostlist::count($hl); + $opt{node_cnt} = $hl_cnt if $hl_cnt > $opt{node_cnt}; + + # we always want at least one here + if(!$opt{node_cnt}) { + + $opt{node_cnt} = 1; + } + + # figure out the amount of tasks based of the node cnt and the amount + # of ppn's in the request + if($opt{task_cnt}) { + $opt{task_cnt} *= $opt{node_cnt}; + } + + return \%opt; +} + +sub get_minutes { + my ($duration) = @_; + $duration = 0 unless $duration; + my $minutes = 0; + + # Convert [[HH:]MM:]SS to duration in minutes + if ($duration =~ /^(?:(\d+):)?(\d*):(\d+)$/) { + my ($hh, $mm, $ss) = ($1 || 0, $2 || 0, $3); + $minutes += 1 if $ss > 0; + $minutes += $mm; + $minutes += $hh * 60; + } elsif ($duration =~ /^(\d+)$/) { # Convert number in minutes to seconds + my $mod = $duration % 60; + $minutes = int($duration / 60); + $minutes++ if $mod; + } else { # Unsupported format + die("Invalid time limit specified ($duration)\n"); + } + + return $minutes; +} + +sub convert_mb_format { + my ($value) = @_; + my ($amount, $suffix) = $value =~ /(\d+)($|[KMGT])/i; + return if !$amount; + $suffix = lc($suffix); + + if (!$suffix) { + $amount /= 1048576; + } elsif ($suffix eq "k") { + $amount /= 1024; + } elsif ($suffix eq "m") { + #do nothing this is what we want. + } elsif ($suffix eq "g") { + $amount *= 1024; + } elsif ($suffix eq "t") { + $amount *= 1048576; + } else { + print "don't know what to do with suffix $suffix\n"; + return; + } + + $amount .= "M"; + + return $amount; +} +############################################################################## + +__END__ + +=head1 NAME + +B - submit a batch job in a familiar pbs format + +=head1 SYNOPSIS + +qsub [-a date_time] + [-A account_string] + [-b secs] + [-C directive_prefix] + [-e path] + [-I] + [-l resource_list] + [-m mail_options] [-M user_list] + [-N name] + [-o path] + [-p priority] + [-q destination] + [-W additional_attributes] + [-h] + [script] + +=head1 DESCRIPTION + +The B command displays information about nodes. It is aimed to be feature-compatible with PBS' qsub. + +=head1 OPTIONS + +=over 4 + +=item B<-a> + +Display information for all nodes. This is the default if no node name is specified. + +=item B<-I> + +Interactive execution. + +=item B<-j> join + +It is not necessary (currently also not possible) since stderr/stdout are always joined. + +=item B<-v> [variable_list] + +Exporting single variables via -v is not supported, since the entire login environment +is exported by the default. + +=item B<-V> + +The -V option to export the current environment is not required since it is done by default. + +=item B<-?> | B<--help> + +brief help message + +=item B<--man> + +full documentation + +=back + +=cut + Property changes on: trunk/bin/slurm-qsub-emulator ___________________________________________________________________ Added: svn:executable + * From swift at ci.uchicago.edu Sun Nov 11 20:00:06 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Sun, 11 Nov 2012 20:00:06 -0600 (CST) Subject: [Swift-commit] cog r3508 Message-ID: <20121112020006.74EA18D0009E@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3508 | hategan | 2012-11-11 19:58:20 -0600 (Sun, 11 Nov 2012) | 1 line removed old commented out lines ------------------------------------------------------------------------ Index: modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/ProxyIOProvider.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/ProxyIOProvider.java (revision 3507) +++ modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/ProxyIOProvider.java (working copy) @@ -214,15 +214,11 @@ private static class CReadBuffer extends ReadBuffer { - // private Exception error; - // private BlockingQueue queue; - // private boolean seenLast; private int crt; private LinkedList alloc; protected CReadBuffer(Buffers buffers, ReadBufferCallback cb) { super(buffers, cb, -1); - // queue = new LinkedBlockingQueue(); alloc = new LinkedList(); } @@ -266,18 +262,6 @@ } } - /* - private static class Entry { - public final boolean last; - public final ByteBuffer buf; - - public Entry(boolean last, ByteBuffer buf) { - this.last = last; - this.buf = buf; - } - } - */ - private static class Reader implements IOReader, Callback { private CustomGetFileCmd cmd; private ReadIOCallback cb; From swift at ci.uchicago.edu Sun Nov 11 20:05:05 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Sun, 11 Nov 2012 20:05:05 -0600 (CST) Subject: [Swift-commit] cog r3509 Message-ID: <20121112020505.70E6F8D0009E@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3509 | hategan | 2012-11-11 20:00:57 -0600 (Sun, 11 Nov 2012) | 1 line make sure empty sub-lists in SortedJobSet are removed since the iterator assumes that ------------------------------------------------------------------------ Index: modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/SortedJobSet.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/SortedJobSet.java (revision 3508) +++ modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/SortedJobSet.java (working copy) @@ -103,6 +103,12 @@ public void remove(Job job) { LinkedList l = sm.get(job.getMaxWallTime()); l.remove(job); + // the iterator assumes that a if a list for a key exists + // then it must have at least one element, so make sure + // that empty lists are not kept + if (l.isEmpty()) { + sm.remove(job.getMaxWallTime()); + } size--; } @@ -131,8 +137,7 @@ return null; } - Job removeOne(TimeInterval key, List jobs, int cpus) - { + private Job removeOne(TimeInterval key, List jobs, int cpus) { Job result = null; for (Iterator it = jobs.iterator(); it.hasNext(); ) { Job job = it.next(); From davidk at ci.uchicago.edu Tue Nov 13 13:17:59 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Tue, 13 Nov 2012 13:17:59 -0600 (CST) Subject: [Swift-commit] r6028 - in SwiftApps/SciColSim: . benchmarks benchmarks/conf Message-ID: <20121113191759.351269CC9B@svn.ci.uchicago.edu> Author: davidk Date: 2012-11-13 13:17:59 -0600 (Tue, 13 Nov 2012) New Revision: 6028 Added: SwiftApps/SciColSim/benchmarks/ SwiftApps/SciColSim/benchmarks/benchmark.sh SwiftApps/SciColSim/benchmarks/benchmark.swift SwiftApps/SciColSim/benchmarks/conf/ SwiftApps/SciColSim/benchmarks/conf/uc3.cf SwiftApps/SciColSim/benchmarks/conf/uc3.xml SwiftApps/SciColSim/benchmarks/plot.gp SwiftApps/SciColSim/benchmarks/plot.sh SwiftApps/SciColSim/benchmarks/start-benchmark.sh Modified: SwiftApps/SciColSim/trimgraph.sh Log: Initial commit of some SciColSim benchmark scripts Added: SwiftApps/SciColSim/benchmarks/benchmark.sh =================================================================== --- SwiftApps/SciColSim/benchmarks/benchmark.sh (rev 0) +++ SwiftApps/SciColSim/benchmarks/benchmark.sh 2012-11-13 19:17:59 UTC (rev 6028) @@ -0,0 +1,8 @@ +#!/bin/bash + +# Create run directory +cd /home/davidk/SciColSim + +# Run testgraph +echo -n "$1 " +./testgraph.py $1 |grep time | awk '{print $5}' Property changes on: SwiftApps/SciColSim/benchmarks/benchmark.sh ___________________________________________________________________ Added: svn:executable + * Added: SwiftApps/SciColSim/benchmarks/benchmark.swift =================================================================== --- SwiftApps/SciColSim/benchmarks/benchmark.swift (rev 0) +++ SwiftApps/SciColSim/benchmarks/benchmark.swift 2012-11-13 19:17:59 UTC (rev 6028) @@ -0,0 +1,32 @@ +type file; + +# Perform a single benchmark +app (file o, file e) benchmark (int target_innovation) +{ + benchmark target_innovation stdout=@o stderr=@e; +} + +# Create a plot from an array of result files +app (file png) plot (file in1[], file gpl) +{ + plot @filename(png); +} + +int target_innovation_start=100; +int target_innovation_stop=1000; +int step=100; + +# Run benchmark +#file graphs[] ; +file graph <"movie-graph.txt">; + +file results[]; +foreach ti,tidx in [target_innovation_start:target_innovation_stop:step] { + file out ; + file error ; + (out, error) = benchmark(ti); + results[tidx] = out; +} +file png <"results/plot.png">; +file gp <"plot.gp">; +png = plot(results, gp); Added: SwiftApps/SciColSim/benchmarks/conf/uc3.cf =================================================================== --- SwiftApps/SciColSim/benchmarks/conf/uc3.cf (rev 0) +++ SwiftApps/SciColSim/benchmarks/conf/uc3.cf 2012-11-13 19:17:59 UTC (rev 6028) @@ -0,0 +1,10 @@ +wrapperlog.always.transfer=false +sitedir.keep=false +execution.retries=0 +lazy.errors=true +status.mode=provider +use.provider.staging=false +provider.staging.pin.swiftfiles=false + +#app uc3 benchmark=$PWD/benchmark.sh +#app localhost plot=$PWD/plot.sh Added: SwiftApps/SciColSim/benchmarks/conf/uc3.xml =================================================================== --- SwiftApps/SciColSim/benchmarks/conf/uc3.xml (rev 0) +++ SwiftApps/SciColSim/benchmarks/conf/uc3.xml 2012-11-13 19:17:59 UTC (rev 6028) @@ -0,0 +1,20 @@ + + + + + + _WORK_ + 1 + 1 + 1000 + 10000 + + + + + + _WORK_ + 0 + + + Added: SwiftApps/SciColSim/benchmarks/plot.gp =================================================================== --- SwiftApps/SciColSim/benchmarks/plot.gp (rev 0) +++ SwiftApps/SciColSim/benchmarks/plot.gp 2012-11-13 19:17:59 UTC (rev 6028) @@ -0,0 +1,9 @@ +set terminal png enhanced size 1000,1000 +#set style line 1 linecolor rgb "blue" +set output "`echo $PLOT_OUTPUT`" +set nokey +set xlabel "Target Innovation" +set ylabel "Time in Seconds" +set title "`basename $PLOT_OUTPUT .png`" +plot "results.txt" using 1:2 with line + Added: SwiftApps/SciColSim/benchmarks/plot.sh =================================================================== --- SwiftApps/SciColSim/benchmarks/plot.sh (rev 0) +++ SwiftApps/SciColSim/benchmarks/plot.sh 2012-11-13 19:17:59 UTC (rev 6028) @@ -0,0 +1,6 @@ +#!/bin/bash + +export PLOT_OUTPUT=$1 +export PLOT_TITLE=$( basename $( dirname $1 )) +find . -name *.out -exec cat {} \; | sort -n > results.txt +gnuplot plot.gp Property changes on: SwiftApps/SciColSim/benchmarks/plot.sh ___________________________________________________________________ Added: svn:executable + * Added: SwiftApps/SciColSim/benchmarks/start-benchmark.sh =================================================================== --- SwiftApps/SciColSim/benchmarks/start-benchmark.sh (rev 0) +++ SwiftApps/SciColSim/benchmarks/start-benchmark.sh 2012-11-13 19:17:59 UTC (rev 6028) @@ -0,0 +1,41 @@ +#!/bin/bash + +# Process command line arguments +while [ $# -gt 0 ]; do + case $1 in + -p) paramfile=$2; shift 2;; + -s) execsite=$2; shift 2;; + *) echo $usage 1>&2 + exit 1;; + esac +done + +# Create run directory +rundir=$( echo run??? | sed -e 's/^.*run//' | awk '{ printf("run%03d\n", $1+1)}' ) + +if [ -d $rundir ]; +then + echo "$rundir already exists! exiting." >&2 + exit 2 +else + mkdir $rundir +fi + +cd $rundir + +# Copy needed files +cp ../benchmark.swift . +cp ../benchmark.sh . +cp ../../movie_graph.txt . +cp ../plot.sh . +cp ../plot.gp . + +# Run gensites +export WORK=$PWD/swiftwork +if [ ! "$USE_SCS" == 1 ]; then + cp ../conf/$execsite.cf cf + SWIFT_HOME=../../swift/bin ../../swift/bin/gensites -p ../conf/$execsite.cf ../conf/$execsite.xml > sites.xml +fi + +# Run swift +swift -sites.file sites.xml -tc.file tc.data benchmark.swift Property changes on: SwiftApps/SciColSim/benchmarks/start-benchmark.sh ___________________________________________________________________ Added: svn:executable + * Modified: SwiftApps/SciColSim/trimgraph.sh =================================================================== --- SwiftApps/SciColSim/trimgraph.sh 2012-11-11 20:39:24 UTC (rev 6027) +++ SwiftApps/SciColSim/trimgraph.sh 2012-11-13 19:17:59 UTC (rev 6028) @@ -9,7 +9,7 @@ NR == 1 { } (NR > 1) && ($1 < n) && ($2 < n) && (rand() < fraction) { print $1, $2 } -' n=$vertices fraction=$fraction $tmp +' n=$vertices fraction=$fraction < big_graph.v01.txt >$tmp edges=$(wc -l <$tmp) From davidk at ci.uchicago.edu Wed Nov 14 00:32:29 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Wed, 14 Nov 2012 00:32:29 -0600 (CST) Subject: [Swift-commit] r6040 - SwiftApps/SciColSim/benchmarks Message-ID: <20121114063229.A40F59CC9B@svn.ci.uchicago.edu> Author: davidk Date: 2012-11-14 00:32:29 -0600 (Wed, 14 Nov 2012) New Revision: 6040 Modified: SwiftApps/SciColSim/benchmarks/benchmark.sh SwiftApps/SciColSim/benchmarks/benchmark.swift Log: Timeouts Modified: SwiftApps/SciColSim/benchmarks/benchmark.sh =================================================================== --- SwiftApps/SciColSim/benchmarks/benchmark.sh 2012-11-14 02:15:28 UTC (rev 6039) +++ SwiftApps/SciColSim/benchmarks/benchmark.sh 2012-11-14 06:32:29 UTC (rev 6040) @@ -1,8 +1,30 @@ -#!/bin/bash +#!/bin/bash -x +cleanup() +{ + kill -s 0 $a || echo -1 >> $OUTPUT + trap - ALRM + kill -ALRM $a 2>/dev/null + kill $! 2>/dev/null && + exit 0 +} + +watchit() +{ + trap "cleanup" ALRM + sleep $1 & wait + kill -ALRM $$ +} + # Create run directory +ORIGDIR=$PWD +OUTPUT=$ORIGDIR/$3 cd /home/davidk/SciColSim -# Run testgraph -echo -n "$1 " -./testgraph.py $1 |grep time | awk '{print $5}' +watchit $2& a=$! +trap "cleanup" ALRM INT +echo -n "$1 " > $OUTPUT +./testgraph.py $1 |grep time | awk '{print $5}' >> $OUTPUT & wait $!; RET=$? +kill -ALRM $a +wait $a +exit $RET Modified: SwiftApps/SciColSim/benchmarks/benchmark.swift =================================================================== --- SwiftApps/SciColSim/benchmarks/benchmark.swift 2012-11-14 02:15:28 UTC (rev 6039) +++ SwiftApps/SciColSim/benchmarks/benchmark.swift 2012-11-14 06:32:29 UTC (rev 6040) @@ -1,9 +1,9 @@ type file; # Perform a single benchmark -app (file o, file e) benchmark (int target_innovation) +app (file o, file e) benchmark (int target_innovation, int tl) { - benchmark target_innovation stdout=@o stderr=@e; + benchmark target_innovation tl @filename(o) stderr=@e; } # Create a plot from an array of result files @@ -12,9 +12,10 @@ plot @filename(png); } -int target_innovation_start=100; -int target_innovation_stop=1000; -int step=100; +int target_innovation_start=1; +int target_innovation_stop=10; +int step=1; +int time_limit=10; # Run benchmark #file graphs[] ; @@ -24,7 +25,7 @@ foreach ti,tidx in [target_innovation_start:target_innovation_stop:step] { file out ; file error ; - (out, error) = benchmark(ti); + (out, error) = benchmark(ti, time_limit); results[tidx] = out; } file png <"results/plot.png">; From davidk at ci.uchicago.edu Wed Nov 14 17:14:21 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Wed, 14 Nov 2012 17:14:21 -0600 (CST) Subject: [Swift-commit] r6041 - SwiftApps/SciColSim/benchmarks Message-ID: <20121114231421.71F499CC9B@svn.ci.uchicago.edu> Author: davidk Date: 2012-11-14 17:14:21 -0600 (Wed, 14 Nov 2012) New Revision: 6041 Modified: SwiftApps/SciColSim/benchmarks/benchmark.sh Log: Only record a value of -1 in case of timeouts Modified: SwiftApps/SciColSim/benchmarks/benchmark.sh =================================================================== --- SwiftApps/SciColSim/benchmarks/benchmark.sh 2012-11-14 06:32:29 UTC (rev 6040) +++ SwiftApps/SciColSim/benchmarks/benchmark.sh 2012-11-14 23:14:21 UTC (rev 6041) @@ -2,7 +2,7 @@ cleanup() { - kill -s 0 $a || echo -1 >> $OUTPUT + kill -s 0 $a && echo -1 >> $OUTPUT trap - ALRM kill -ALRM $a 2>/dev/null kill $! 2>/dev/null && From swift at ci.uchicago.edu Wed Nov 14 21:05:14 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Wed, 14 Nov 2012 21:05:14 -0600 (CST) Subject: [Swift-commit] cog r3510 Message-ID: <20121115030515.1DE018D0009E@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3510 | davidkelly999 | 2012-11-14 21:04:52 -0600 (Wed, 14 Nov 2012) | 3 lines By default, condor will allocate 1gb of memory per job. For jobs with larger memory requirements, the request_memory parameter must be defined. This can now be set in sites.xml. Also provide a mechanism similar to what exists for pbs for defining arbitrary provider parameters such as accounting groups, etc. ------------------------------------------------------------------------ Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/condor/CondorExecutor.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/condor/CondorExecutor.java (revision 3509) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/condor/CondorExecutor.java (working copy) @@ -45,6 +45,7 @@ boolean grid = false; Task task = getTask(); JobSpecification spec = getSpec(); + getSpec().unpackProviderAttributes(); String type = (String) spec.getAttribute("jobType"); if (logger.isDebugEnabled()) { logger.debug("Job type: " + type); @@ -110,6 +111,19 @@ } } wr.write('\n'); + + String request_memory = (String) spec.getAttribute("request_memory"); + if(request_memory != null) { + wr.write("request_memory = " + request_memory + '\n'); + } + + String resources = (String) spec.getAttribute("condor.resource_list"); + if (resources != null && resources.length() > 0) { + if (logger.isDebugEnabled()) + logger.debug("condor.resource_list: " + resources); + wr.write(resources + '\n'); + } + wr.write("notification = Never\n"); wr.write("leave_in_queue = TRUE\n"); wr.write("queue\n"); @@ -220,7 +234,7 @@ } } else { - logger.warn("Failed makr job " + jobid + logger.warn("Failed to mark job " + jobid + " as removable from queue: " + getOutput(p.getInputStream())); } From swift at ci.uchicago.edu Wed Nov 14 21:10:11 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Wed, 14 Nov 2012 21:10:11 -0600 (CST) Subject: [Swift-commit] cog r3511 Message-ID: <20121115031011.B1E6E8D0009E@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3511 | davidkelly999 | 2012-11-14 21:07:35 -0600 (Wed, 14 Nov 2012) | 3 lines By default, condor will allocate 1gb of memory per job. For jobs with larger memory requirements, the request_memory parameter must be defined. This can now be set in sites.xml. Also provide a mechanism similar to what exists for pbs for defining arbitrary provider parameters such as accounting groups, etc. ------------------------------------------------------------------------ Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/condor/CondorExecutor.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/condor/CondorExecutor.java (revision 3510) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/condor/CondorExecutor.java (working copy) @@ -45,6 +45,7 @@ boolean grid = false; Task task = getTask(); JobSpecification spec = getSpec(); + getSpec().unpackProviderAttributes(); String type = (String) spec.getAttribute("jobType"); if (logger.isDebugEnabled()) { logger.debug("Job type: " + type); @@ -110,6 +111,19 @@ } } wr.write('\n'); + + String request_memory = (String) spec.getAttribute("request_memory"); + if(request_memory != null) { + wr.write("request_memory = " + request_memory + '\n'); + } + + String resources = (String) spec.getAttribute("condor.resource_list"); + if (resources != null && resources.length() > 0) { + if (logger.isDebugEnabled()) + logger.debug("condor.resource_list: " + resources); + wr.write(resources + '\n'); + } + wr.write("notification = Never\n"); wr.write("leave_in_queue = TRUE\n"); wr.write("queue\n"); @@ -220,7 +234,7 @@ } } else { - logger.warn("Failed makr job " + jobid + logger.warn("Failed to mark job " + jobid + " as removable from queue: " + getOutput(p.getInputStream())); } From davidk at ci.uchicago.edu Wed Nov 14 21:14:45 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Wed, 14 Nov 2012 21:14:45 -0600 (CST) Subject: [Swift-commit] r6042 - branches/release-0.94/bin Message-ID: <20121115031445.ABA1B9CD01@svn.ci.uchicago.edu> Author: davidk Date: 2012-11-14 21:14:45 -0600 (Wed, 14 Nov 2012) New Revision: 6042 Modified: branches/release-0.94/bin/swift Log: Disable chxml script to avoid erronous error messages Modified: branches/release-0.94/bin/swift =================================================================== --- branches/release-0.94/bin/swift 2012-11-14 23:14:21 UTC (rev 6041) +++ branches/release-0.94/bin/swift 2012-11-15 03:14:45 UTC (rev 6042) @@ -63,11 +63,11 @@ # make sure sites.xml file is well-formed -$SWIFT_HOME/bin/chxml $CMDLINE -if [ "$?" != 0 ]; then - echo "Could not process input files!" - exit 1 -fi +#$SWIFT_HOME/bin/chxml $CMDLINE +#if [ "$?" != 0 ]; then +# echo "Could not process input files!" +# exit 1 +#fi ### SETUP OTHER ENV VARIABLES #### From davidk at ci.uchicago.edu Wed Nov 14 21:15:51 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Wed, 14 Nov 2012 21:15:51 -0600 (CST) Subject: [Swift-commit] r6043 - trunk/bin Message-ID: <20121115031551.B7D889CD01@svn.ci.uchicago.edu> Author: davidk Date: 2012-11-14 21:15:51 -0600 (Wed, 14 Nov 2012) New Revision: 6043 Modified: trunk/bin/swift Log: Disable chxml script to avoid erronous error messages Modified: trunk/bin/swift =================================================================== --- trunk/bin/swift 2012-11-15 03:14:45 UTC (rev 6042) +++ trunk/bin/swift 2012-11-15 03:15:51 UTC (rev 6043) @@ -63,11 +63,11 @@ # make sure sites.xml file is well-formed -$SWIFT_HOME/bin/chxml $CMDLINE -if [ "$?" != 0 ]; then - echo "Could not process input files!" - exit 1 -fi +#$SWIFT_HOME/bin/chxml $CMDLINE +#if [ "$?" != 0 ]; then +# echo "Could not process input files!" +# exit 1 +#fi ### SETUP OTHER ENV VARIABLES #### From davidk at ci.uchicago.edu Wed Nov 14 22:28:09 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Wed, 14 Nov 2012 22:28:09 -0600 (CST) Subject: [Swift-commit] r6044 - branches/release-0.94/src/org/griphyn/vdl/engine Message-ID: <20121115042809.7E2B19CD01@svn.ci.uchicago.edu> Author: davidk Date: 2012-11-14 22:28:09 -0600 (Wed, 14 Nov 2012) New Revision: 6044 Modified: branches/release-0.94/src/org/griphyn/vdl/engine/VariableScope.java Log: Move scope warnings to log instead of stdout Modified: branches/release-0.94/src/org/griphyn/vdl/engine/VariableScope.java =================================================================== --- branches/release-0.94/src/org/griphyn/vdl/engine/VariableScope.java 2012-11-15 03:15:51 UTC (rev 6043) +++ branches/release-0.94/src/org/griphyn/vdl/engine/VariableScope.java 2012-11-15 04:28:09 UTC (rev 6044) @@ -138,9 +138,11 @@ // by the above if? in which case isVariableDefined should // be replaced by is locally defined test. if(parentScope != null && parentScope.isVariableDefined(name)) { - logger.warn("Variable "+name+" defined in scope "+hashCode() - + " shadows variable of same name in scope " - + parentScope.hashCode()); + if(logger.isDebugEnabled()) { + logger.debug("Variable "+name+" defined in scope "+hashCode() + + " shadows variable of same name in scope " + + parentScope.hashCode()); + } } if(global && this != rootScope) { From davidk at ci.uchicago.edu Fri Nov 16 16:24:06 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Fri, 16 Nov 2012 16:24:06 -0600 (CST) Subject: [Swift-commit] r6045 - in SwiftApps/SciColSim/benchmarks: . conf Message-ID: <20121116222406.F10879CC9B@svn.ci.uchicago.edu> Author: davidk Date: 2012-11-16 16:24:06 -0600 (Fri, 16 Nov 2012) New Revision: 6045 Added: SwiftApps/SciColSim/benchmarks/conf/beagle.cf SwiftApps/SciColSim/benchmarks/conf/beagle.xml Modified: SwiftApps/SciColSim/benchmarks/benchmark.sh Log: Use coreutils timeout for handling time limits Configuration files for beagle Modified: SwiftApps/SciColSim/benchmarks/benchmark.sh =================================================================== --- SwiftApps/SciColSim/benchmarks/benchmark.sh 2012-11-15 04:28:09 UTC (rev 6044) +++ SwiftApps/SciColSim/benchmarks/benchmark.sh 2012-11-16 22:24:06 UTC (rev 6045) @@ -1,30 +1,19 @@ #!/bin/bash -x -cleanup() -{ - kill -s 0 $a && echo -1 >> $OUTPUT - trap - ALRM - kill -ALRM $a 2>/dev/null - kill $! 2>/dev/null && - exit 0 -} - -watchit() -{ - trap "cleanup" ALRM - sleep $1 & wait - kill -ALRM $$ -} - # Create run directory ORIGDIR=$PWD OUTPUT=$ORIGDIR/$3 -cd /home/davidk/SciColSim +cd $HOME/SciColSim -watchit $2& a=$! -trap "cleanup" ALRM INT -echo -n "$1 " > $OUTPUT -./testgraph.py $1 |grep time | awk '{print $5}' >> $OUTPUT & wait $!; RET=$? -kill -ALRM $a -wait $a -exit $RET +if [ -f "bestdb.txt" ]; then + rm -f bestdb.txt +fi + +export OMP_NUM_THREADS=1 +result=$( bin/timeout $2 ./graphsim 0 0 0 0 0 $1 40000 20 1 2 1 2. 0.01 1 0.3 2.3 0 0 0 0 0 m 1 | grep time | awk '{print $5}' ) + +if [ -n "$result" ]; then + echo $1 $result > $OUTPUT +else + echo "$1 -1" > $OUTPUT +fi Added: SwiftApps/SciColSim/benchmarks/conf/beagle.cf =================================================================== --- SwiftApps/SciColSim/benchmarks/conf/beagle.cf (rev 0) +++ SwiftApps/SciColSim/benchmarks/conf/beagle.cf 2012-11-16 22:24:06 UTC (rev 6045) @@ -0,0 +1,10 @@ +wrapperlog.always.transfer=true +sitedir.keep=true +execution.retries=5 +lazy.errors=true +status.mode=provider +use.provider.staging=false +provider.staging.pin.swiftfiles=false + +#app beagle benchmark=$PWD/benchmark.sh +#app localhost plot=$PWD/plot.sh Added: SwiftApps/SciColSim/benchmarks/conf/beagle.xml =================================================================== --- SwiftApps/SciColSim/benchmarks/conf/beagle.xml (rev 0) +++ SwiftApps/SciColSim/benchmarks/conf/beagle.xml 2012-11-16 22:24:06 UTC (rev 6045) @@ -0,0 +1,25 @@ + + + + + 0.09 + 10000 + + _WORK_ + + + + + 2 + pbs.aprun;pbs.mpp;depth=24 + 24:00:00 + 12.00 + 10000 + CI-MCB000119 + + + _WORK_ + + + + From davidk at ci.uchicago.edu Fri Nov 16 16:42:10 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Fri, 16 Nov 2012 16:42:10 -0600 (CST) Subject: [Swift-commit] r6046 - SwiftApps/SciColSim/benchmarks/conf Message-ID: <20121116224210.DAC799CC9B@svn.ci.uchicago.edu> Author: davidk Date: 2012-11-16 16:42:10 -0600 (Fri, 16 Nov 2012) New Revision: 6046 Added: SwiftApps/SciColSim/benchmarks/conf/midway.cf SwiftApps/SciColSim/benchmarks/conf/midway.xml Log: Configuration files for midway Added: SwiftApps/SciColSim/benchmarks/conf/midway.cf =================================================================== --- SwiftApps/SciColSim/benchmarks/conf/midway.cf (rev 0) +++ SwiftApps/SciColSim/benchmarks/conf/midway.cf 2012-11-16 22:42:10 UTC (rev 6046) @@ -0,0 +1,10 @@ +wrapperlog.always.transfer=false +sitedir.keep=false +execution.retries=0 +lazy.errors=true +status.mode=provider +use.provider.staging=false +provider.staging.pin.swiftfiles=false + +#app midway benchmark=$PWD/benchmark.sh +#app localhost plot=$PWD/plot.sh Added: SwiftApps/SciColSim/benchmarks/conf/midway.xml =================================================================== --- SwiftApps/SciColSim/benchmarks/conf/midway.xml (rev 0) +++ SwiftApps/SciColSim/benchmarks/conf/midway.xml 2012-11-16 22:42:10 UTC (rev 6046) @@ -0,0 +1,20 @@ + + + + + 1 + 01:00:00 + sandyb + 10000 + + /home/davidkelly999/work + + + + + + _WORK_ + 0 + + + From hategan at ci.uchicago.edu Sat Nov 17 00:35:07 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sat, 17 Nov 2012 00:35:07 -0600 (CST) Subject: [Swift-commit] r6047 - trunk/libexec Message-ID: <20121117063507.23AAE9CD01@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-17 00:35:04 -0600 (Sat, 17 Nov 2012) New Revision: 6047 Modified: trunk/libexec/_swiftwrap trunk/libexec/_swiftwrap.staging trunk/libexec/_swiftwrap.wrapperstaging Log: allow stdout and stderr to be the same file; make sure stdout and stderr are transfered when a scratch file is used; also make sure that stdout, stderr and the info file are transfered when using a scratch dir and failing Modified: trunk/libexec/_swiftwrap =================================================================== --- trunk/libexec/_swiftwrap 2012-11-16 22:42:10 UTC (rev 6046) +++ trunk/libexec/_swiftwrap 2012-11-17 06:35:04 UTC (rev 6047) @@ -55,6 +55,25 @@ [[ $DEBUG == 1 ]] && echo "$@" >& "$INFO" } +moveMiscFilesFromScratch() { + if [ "X$PROGRESSIVE_INFO" == "X" ] && [ "X$SCRATCH" != "X" ]; then + mkdir -p "$WFDIR/info/$JOBDIR" + mv "$INFODIR/${ID}-info" "$WFDIR/info/$JOBDIR/${ID}-info" + fi + + if [ "X$SCRATCH" != "X" ]; then + if [ -f "$DIR/$STDOUT" ] || [ -f "$DIR/$STDERR" ]; then + mkdir -p "$WFDIR/jobs/$JOBDIR/$ID" + fi + if [ -f "$DIR/$STDOUT" ]; then + mv "$DIR/$STDOUT" "$WFDIR/jobs/$JOBDIR/$ID/$STDOUT" + fi + if [ -f "$DIR/$STDERR" ]; then + mv "$DIR/$STDERR" "$WFDIR/jobs/$JOBDIR/$ID/$STDERR" + fi + fi +} + fail() { EC=$1 shift @@ -67,6 +86,11 @@ log $@ info + + closeinfo + + moveMiscFilesFromScratch + if [ "$STATUSMODE" = "files" ]; then exit 0 else @@ -145,7 +169,7 @@ for CMDARG in "${CMDARGS[@]}"; do echo -n "\"$CMDARG\" " >> run.sh done - echo "1>\"$STDOUT\" 2>\"$STDERR\"" >> run.sh + echo "1>>\"$STDOUT\" 2>>\"$STDERR\"" >> run.sh chmod +x run.sh } @@ -372,56 +396,56 @@ if [[ $PMI_RANK == "" || $PMI_RANK == 0 ]]; then -logstate "CREATE_INPUTDIR" -for D in $DIRS ; do - mkdir -p "$DIR/$D" 2>&1 >>"$INFO" - checkError 254 "Failed to create input directory $D" - log "Created output directory: $DIR/$D" -done - -logstate "LINK_INPUTS" -for L in $INF ; do - CDM_POLICY="DEFAULT" - if [[ $CDM_FILE != "" ]]; then - CDM_POLICY=$( cdm_lookup shared/cdm.pl $CDM_FILE $L ) - fi - if [[ $CDM_POLICY != "DEFAULT" && $CDM_POLICY != "EXTERNAL"* ]]; then - log "CDM_POLICY: $L -> $CDM_POLICY" - eval cdm_action $DIR "INPUT" $L $CDM_POLICY - continue - fi - if [ $COPYNOTLINK = 1 ]; then - cp "$WFDIR/shared/$L" "$DIR/$L" 2>&1 >& $INFO - checkError 254 "Failed to copy input file $L" - log "Copied input: $WFDIR/shared/$L to $DIR/$L" - else - [ -f $WFDIR/shared/$L ] - checkError 254 "Could not locate input file: $L" - ln -s "$WFDIR/shared/$L" "$DIR/$L" 2>&1 >& $INFO - checkError 254 "Failed to link input file $L" - log "Linked input: $WFDIR/shared/$L to $DIR/$L" - fi -done - -if [[ $CDM_FILE != "" ]]; then - logstate "LINK_CDM_OUTPUTS" - SKIPPED_OUTPUT=() - GATHER_OUTPUT=() - for L in $OUTF ; do - CDM_POLICY=$( cdm_lookup shared/cdm.pl $CDM_FILE $L ) - log "CDM_POLICY: $L -> $CDM_POLICY" - if [[ $CDM_POLICY != "DEFAULT" && - $CDM_POLICY != "BROADCAST"* ]]; then - eval cdm_action $DIR "OUTPUT" $L $CDM_POLICY - SKIPPED_OUTPUT=( $SKIPPED_OUTPUT $L ) + logstate "CREATE_INPUTDIR" + for D in $DIRS ; do + mkdir -p "$DIR/$D" 2>&1 >>"$INFO" + checkError 254 "Failed to create input directory $D" + log "Created output directory: $DIR/$D" + done + + logstate "LINK_INPUTS" + for L in $INF ; do + CDM_POLICY="DEFAULT" + if [[ $CDM_FILE != "" ]]; then + CDM_POLICY=$( cdm_lookup shared/cdm.pl $CDM_FILE $L ) fi - if [ $CDM_POLICY == "GATHER" ]; then - GATHER_OUTPUT=( $GATHER_OUTPUT $L ) - elif [ $CDM_POLICY == "LOCAL" ]; then - CDM_LOCAL_OUTPUT=( $CDM_LOCAL_OUTPUT $L ) + if [[ $CDM_POLICY != "DEFAULT" && $CDM_POLICY != "EXTERNAL"* ]]; then + log "CDM_POLICY: $L -> $CDM_POLICY" + eval cdm_action $DIR "INPUT" $L $CDM_POLICY + continue fi + if [ $COPYNOTLINK = 1 ]; then + cp "$WFDIR/shared/$L" "$DIR/$L" 2>&1 >& $INFO + checkError 254 "Failed to copy input file $L" + log "Copied input: $WFDIR/shared/$L to $DIR/$L" + else + [ -f $WFDIR/shared/$L ] + checkError 254 "Could not locate input file: $L" + ln -s "$WFDIR/shared/$L" "$DIR/$L" 2>&1 >& $INFO + checkError 254 "Failed to link input file $L" + log "Linked input: $WFDIR/shared/$L to $DIR/$L" + fi done -fi + + if [[ $CDM_FILE != "" ]]; then + logstate "LINK_CDM_OUTPUTS" + SKIPPED_OUTPUT=() + GATHER_OUTPUT=() + for L in $OUTF ; do + CDM_POLICY=$( cdm_lookup shared/cdm.pl $CDM_FILE $L ) + log "CDM_POLICY: $L -> $CDM_POLICY" + if [[ $CDM_POLICY != "DEFAULT" && + $CDM_POLICY != "BROADCAST"* ]]; then + eval cdm_action $DIR "OUTPUT" $L $CDM_POLICY + SKIPPED_OUTPUT=( $SKIPPED_OUTPUT $L ) + fi + if [ $CDM_POLICY == "GATHER" ]; then + GATHER_OUTPUT=( $GATHER_OUTPUT $L ) + elif [ $CDM_POLICY == "LOCAL" ]; then + CDM_LOCAL_OUTPUT=( $CDM_LOCAL_OUTPUT $L ) + fi + done + fi fi # PMI_RANK==0 @@ -447,12 +471,12 @@ if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then genScripts fi - "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" + "$EXEC" "${CMDARGS[@]}" 1>>"$STDOUT" 2>>"$STDERR" else if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then genScripts fi - "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" <"$STDIN" + "$EXEC" "${CMDARGS[@]}" 1>>"$STDOUT" 2>>"$STDERR" <"$STDIN" fi checkError $? "Application $EXEC failed with an exit code of $?" else @@ -466,9 +490,9 @@ mkdir -p $WFDIR/kickstart/$JOBDIR log "Using Kickstart ($KICKSTART)" if [ "$STDIN" == "" ]; then - "$KICKSTART" -H -o "$STDOUT" -e "$STDERR" "$EXEC" "$@" 1>kickstart.xml 2>"$STDERR" + "$KICKSTART" -H -o "$STDOUT" -e "$STDERR" "$EXEC" "$@" 1>kickstart.xml 2>>"$STDERR" else - "$KICKSTART" -H -o "$STDOUT" -i "$STDIN" -e "$STDERR" "$EXEC" "$@" 1>kickstart.xml 2>"$STDERR" + "$KICKSTART" -H -o "$STDOUT" -i "$STDIN" -e "$STDERR" "$EXEC" "$@" 1>kickstart.xml 2>>"$STDERR" fi export APPEXIT=$? mv -f kickstart.xml "$WFDIR/kickstart/$JOBDIR/$ID-kickstart.xml" 2>&1 >& "$INFO" @@ -490,41 +514,41 @@ if [[ $MPI_RANK == "" || $MPI_RANK == 0 ]]; then -MISSING= -for O in $OUTF ; do - if [ ! -f "$DIR/$O" ]; then - if [ "$MISSING" == "" ]; then - MISSING=$O - else - MISSING="$MISSING, $O" + MISSING= + for O in $OUTF ; do + if [ ! -f "$DIR/$O" ]; then + if [ "$MISSING" == "" ]; then + MISSING=$O + else + MISSING="$MISSING, $O" + fi fi + done + if [ "$MISSING" != "" ]; then + log $( find . ) + fail 254 "The following output files were not created by the application: $MISSING" fi -done -if [ "$MISSING" != "" ]; then - log $( find . ) - fail 254 "The following output files were not created by the application: $MISSING" -fi - -logstate "MOVING_OUTPUTS $OUTF" -for O in $OUTF ; do - if ! contains SKIPPED_OUTPUT $O ; then - mv "$DIR/$O" "$WFDIR/shared/$O" 2>&1 >& "$INFO" - checkError 254 "Failed to move output file $O to shared directory" + + logstate "MOVING_OUTPUTS $OUTF" + + for O in "$OUTF"; do + if ! contains SKIPPED_OUTPUT $O ; then + mv "$DIR/$O" "$WFDIR/shared/$O" 2>&1 >& "$INFO" + checkError 254 "Failed to move output file $O to shared directory" + fi + done + + cdm_local_output $CDM_LOCAL_OUTPUT + cdm_gather $GATHER_OUTPUT + + logstate "RM_JOBDIR" + rm -rf "$DIR" 2>&1 >& "$INFO" + checkError 254 "Failed to remove job directory $DIR" + + if [ "$STATUSMODE" = "files" ]; then + logstate "TOUCH_SUCCESS" + touch $WFDIR/status/${JOBDIR}/${ID}-success fi -done - -cdm_local_output $CDM_LOCAL_OUTPUT -cdm_gather $GATHER_OUTPUT - -logstate "RM_JOBDIR" -rm -rf "$DIR" 2>&1 >& "$INFO" -checkError 254 "Failed to remove job directory $DIR" - -if [ "$STATUSMODE" = "files" ]; then - logstate "TOUCH_SUCCESS" - touch $WFDIR/status/${JOBDIR}/${ID}-success -fi - else # Allow rank 0 to write output sleep 1 @@ -534,10 +558,7 @@ closeinfo -if [ "X$PROGRESSIVE_INFO" == "X" ] && [ "X$SCRATCH" != "X" ]; then - mkdir -p "$WFDIR/info/$JOBDIR" - mv "$INFODIR/${ID}-info" "$WFDIR/info/$JOBDIR/${ID}-info" -fi +moveMiscFilesFromScratch # ensure we exit with a 0 after a successful execution exit 0 Modified: trunk/libexec/_swiftwrap.staging =================================================================== --- trunk/libexec/_swiftwrap.staging 2012-11-16 22:42:10 UTC (rev 6046) +++ trunk/libexec/_swiftwrap.staging 2012-11-17 06:35:04 UTC (rev 6047) @@ -288,17 +288,17 @@ if [ "$STDIN" == "" ]; then if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then echo "#!/bin/bash" > run.sh - echo "\"$EXEC\" \"${CMDARGS[@]}\" 1>\"$STDOUT\" 2>\"$STDERR\"" >> run.sh + echo "\"$EXEC\" \"${CMDARGS[@]}\" 1>>\"$STDOUT\" 2>>\"$STDERR\"" >> run.sh chmod +x run.sh fi - "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" + "$EXEC" "${CMDARGS[@]}" 1>>"$STDOUT" 2>>"$STDERR" else if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then echo "#!/bin/bash" > run.sh - echo "\"$EXEC\" \"${CMDARGS[@]}\" 1>\"$STDOUT\" 2>\"$STDERR\" <\"$STDIN\"" >> run.sh + echo "\"$EXEC\" \"${CMDARGS[@]}\" 1>>\"$STDOUT\" 2>>\"$STDERR\" <\"$STDIN\"" >> run.sh chmod +x run.sh fi - "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" <"$STDIN" + "$EXEC" "${CMDARGS[@]}" 1>>"$STDOUT" 2>>"$STDERR" <"$STDIN" fi checkError $? "Application $EXEC failed with an exit code of $?" <$STDERR Modified: trunk/libexec/_swiftwrap.wrapperstaging =================================================================== --- trunk/libexec/_swiftwrap.wrapperstaging 2012-11-16 22:42:10 UTC (rev 6046) +++ trunk/libexec/_swiftwrap.wrapperstaging 2012-11-17 06:35:04 UTC (rev 6047) @@ -224,7 +224,7 @@ for CMDARG in "${CMDARGS[@]}"; do echo -n "\"$CMDARG\" " >> run.sh done - echo "1>\"$STDOUT\" 2>\"$STDERR\"" >> run.sh + echo "1>>\"$STDOUT\" 2>>\"$STDERR\"" >> run.sh chmod +x run.sh } @@ -501,12 +501,12 @@ if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then genScripts fi - "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" + "$EXEC" "${CMDARGS[@]}" 1>>"$STDOUT" 2>>"$STDERR" else if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then genScripts fi - "$EXEC" "${CMDARGS[@]}" 1>"$STDOUT" 2>"$STDERR" <"$STDIN" + "$EXEC" "${CMDARGS[@]}" 1>>"$STDOUT" 2>>"$STDERR" <"$STDIN" fi checkError $? "Application $EXEC failed with an exit code of $?" From hategan at ci.uchicago.edu Sat Nov 17 01:00:52 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sat, 17 Nov 2012 01:00:52 -0600 (CST) Subject: [Swift-commit] r6048 - trunk/libexec Message-ID: <20121117070052.56B589D043@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-17 01:00:51 -0600 (Sat, 17 Nov 2012) New Revision: 6048 Modified: trunk/libexec/vdl-int.k Log: remove trailing whitespace from stdout/stderr Modified: trunk/libexec/vdl-int.k =================================================================== --- trunk/libexec/vdl-int.k 2012-11-17 06:35:04 UTC (rev 6047) +++ trunk/libexec/vdl-int.k 2012-11-17 07:00:51 UTC (rev 6048) @@ -374,7 +374,7 @@ destfile=destfile) nl() " {name}: " - file:read(destfile) + strip(file:read(destfile)) ) "" ) From hategan at ci.uchicago.edu Sat Nov 17 01:02:41 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sat, 17 Nov 2012 01:02:41 -0600 (CST) Subject: [Swift-commit] r6049 - trunk/etc Message-ID: <20121117070241.53DC29D043@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-17 01:02:41 -0600 (Sat, 17 Nov 2012) New Revision: 6049 Modified: trunk/etc/error.properties Log: removed a couple of errors from the translator; the default messages look fine and those patterns take a long time to match Modified: trunk/etc/error.properties =================================================================== --- trunk/etc/error.properties 2012-11-17 07:00:51 UTC (rev 6048) +++ trunk/etc/error.properties 2012-11-17 07:02:41 UTC (rev 6049) @@ -15,27 +15,6 @@ You should check your tc.data and sites.xml files, and make sure that the name (\2) is not misspelled. ; -Exception in (\S*):.* -Arguments: \[(.*)\].*Host: (\S*).*Directory: (\S*).*stderr.txt:\s*(.*)\s*stdout.txt:\s*(.*)\s*----\s* -KickstartRecord: (\S*).* -Caused by:\s* -(.*) --> -Application "\1" failed (\8)\n -\tArguments: "\2"\n\tHost: \3\n\tDirectory: \4\n\tSTDERR: \5\n\tSTDOUT: \6\n -\tKickstartRecord: \7 -; - -Exception in (\S*):.* -Arguments: \[(.*)\].*Host: (\S*).*Directory: (\S*).*stderr.txt:\s*(.*)\s*stdout.txt:\s*(.*)\s*----\s* -Caused by:\s* -(.*) --> -Application "\1" failed (\7)\n -\tArguments: "\2"\n\tHost: \3\n\tDirectory: \4\n\tSTDERR: \5\n\tSTDOUT: \6\n -; - - The cache already contains (\S*)\..* -> Multiple mappings pointing to the same file (\1) detected. @@ -49,7 +28,7 @@ ; Exception in (\S*):.* -Arguments: \[(.*)\].*Host: (\S*).*Directory: (\S*).*stderr.txt:\s*(.*)\s*stdout.txt:\s*(.*)\s*----.* +Arguments: \[(.*)\].*Host: (\S*).*Directory: (\S*).*stderr.txt:\s*(.*)\s*stdout.txt:\s*(.*)\s.* Exception in getFile.* Unable to open file (\S*).* -> From hategan at ci.uchicago.edu Sat Nov 17 01:05:31 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sat, 17 Nov 2012 01:05:31 -0600 (CST) Subject: [Swift-commit] r6050 - trunk/src/org/griphyn/vdl/karajan/functions Message-ID: <20121117070531.21FA79D043@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-17 01:05:30 -0600 (Sat, 17 Nov 2012) New Revision: 6050 Modified: trunk/src/org/griphyn/vdl/karajan/functions/ProcessBulkErrors.java Log: show all messages in exception chain Modified: trunk/src/org/griphyn/vdl/karajan/functions/ProcessBulkErrors.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/functions/ProcessBulkErrors.java 2012-11-17 07:02:41 UTC (rev 6049) +++ trunk/src/org/griphyn/vdl/karajan/functions/ProcessBulkErrors.java 2012-11-17 07:05:30 UTC (rev 6050) @@ -140,11 +140,6 @@ } e = e.getCause(); } - if (logger.isDebugEnabled()) { - return sb.toString(); - } - else{ - return lastmsg; - } + return sb.toString(); } } From hategan at ci.uchicago.edu Sat Nov 17 01:06:39 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sat, 17 Nov 2012 01:06:39 -0600 (CST) Subject: [Swift-commit] r6051 - in trunk/src/org/griphyn/vdl: karajan/lib mapping Message-ID: <20121117070639.7F9A89D043@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-17 01:06:38 -0600 (Sat, 17 Nov 2012) New Revision: 6051 Modified: trunk/src/org/griphyn/vdl/karajan/lib/New.java trunk/src/org/griphyn/vdl/karajan/lib/Tracer.java trunk/src/org/griphyn/vdl/mapping/MappingParamSet.java trunk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java trunk/src/org/griphyn/vdl/mapping/RootDataNode.java Log: moved formatting of mapper parameters to MappingParamSet and a few more tweaks to tracing Modified: trunk/src/org/griphyn/vdl/karajan/lib/New.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/New.java 2012-11-17 07:05:30 UTC (rev 6050) +++ trunk/src/org/griphyn/vdl/karajan/lib/New.java 2012-11-17 07:06:38 UTC (rev 6051) @@ -179,7 +179,7 @@ } else { if (tracer.isEnabled()) { - tracer.trace(threadPrefix, dbgname + formatMPS(mps)); + tracer.trace(threadPrefix, dbgname + " " + mps); } } } @@ -194,36 +194,6 @@ } } - private String formatMPS(MappingParamSet mps) { - Object desc = mps.get(MappingParam.SWIFT_DESCRIPTOR); - if (desc == null) { - return ""; - } - StringBuilder sb = new StringBuilder(); - sb.append(" <"); - sb.append(desc); - sb.append("; "); - boolean first = true; - for (String name : mps.names()) { - if (name.indexOf('#') >= 0) { - // skip internal parameters - continue; - } - if (first) { - first = false; - } - else { - sb.append(", "); - } - sb.append(name); - sb.append(" = \""); - sb.append(Tracer.unwrapHandle(mps._get(name))); - sb.append("\""); - } - sb.append('>'); - return sb.toString(); - } - private String formatList(List value) { StringBuilder sb = new StringBuilder(); sb.append('['); Modified: trunk/src/org/griphyn/vdl/karajan/lib/Tracer.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/Tracer.java 2012-11-17 07:05:30 UTC (rev 6050) +++ trunk/src/org/griphyn/vdl/karajan/lib/Tracer.java 2012-11-17 07:06:38 UTC (rev 6051) @@ -132,7 +132,7 @@ public boolean isEnabled() { return enabled; } - + public void trace(VariableStack stack, Object msg) throws VariableNotFoundException { trace(threadName(stack), msg); } Modified: trunk/src/org/griphyn/vdl/mapping/MappingParamSet.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/MappingParamSet.java 2012-11-17 07:05:30 UTC (rev 6050) +++ trunk/src/org/griphyn/vdl/mapping/MappingParamSet.java 2012-11-17 07:06:38 UTC (rev 6051) @@ -7,6 +7,8 @@ import java.util.HashMap; import java.util.Map; +import org.griphyn.vdl.karajan.lib.Tracer; + public class MappingParamSet { private Map params; @@ -30,10 +32,6 @@ return params.get(p.getName()); } - public Object _get(String name) { - return params.get(name); - } - public boolean isPresent(MappingParam p) { return params.containsKey(p.getName()); } @@ -51,4 +49,30 @@ public Collection names() { return params.keySet(); } + + @Override + public String toString() { + Object desc = get(MappingParam.SWIFT_DESCRIPTOR); + if (desc == null) { + return ""; + } + StringBuilder sb = new StringBuilder(); + sb.append("<"); + sb.append(desc); + sb.append("; "); + sb.append("input = " + get(MappingParam.SWIFT_INPUT)); + for (String name : names()) { + if (name.indexOf('#') >= 0) { + // skip internal parameters + continue; + } + sb.append(", "); + sb.append(name); + sb.append(" = \""); + sb.append(Tracer.unwrapHandle(params.get(name))); + sb.append("\""); + } + sb.append('>'); + return sb.toString(); + } } Modified: trunk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java 2012-11-17 07:05:30 UTC (rev 6050) +++ trunk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java 2012-11-17 07:06:38 UTC (rev 6051) @@ -146,7 +146,7 @@ initialized = true; waitingMapperParam = null; if (tracer.isEnabled()) { - tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " INITIALIZED"); + tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " INITIALIZED " + params); } } } Modified: trunk/src/org/griphyn/vdl/mapping/RootDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/RootDataNode.java 2012-11-17 07:05:30 UTC (rev 6050) +++ trunk/src/org/griphyn/vdl/mapping/RootDataNode.java 2012-11-17 07:06:38 UTC (rev 6051) @@ -135,8 +135,9 @@ // Try to get the path in order to check that the // path is valid - we'll get an exception if not root.getField(p); - if (logger.isInfoEnabled()) { - logger.info("Found mapped data " + root + "." + p); + if (tracer.isEnabled()) { + tracer.trace(root.getThread(), root.getDeclarationLine(), + root.getDisplayableName() + " MAPPING " + p + ", " + mapper.map(p)); } } catch (InvalidPathException e) { @@ -247,7 +248,7 @@ initialized = true; waitingMapperParam = null; if (tracer.isEnabled()) { - tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " INITIALIZED"); + tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " INITIALIZED " + params); } } } From hategan at ci.uchicago.edu Sat Nov 17 01:27:25 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sat, 17 Nov 2012 01:27:25 -0600 (CST) Subject: [Swift-commit] r6052 - trunk/libexec Message-ID: <20121117072725.241B19CCA5@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-17 01:27:24 -0600 (Sat, 17 Nov 2012) New Revision: 6052 Modified: trunk/libexec/_swiftwrap Log: oops; wrong place to have quotes Modified: trunk/libexec/_swiftwrap =================================================================== --- trunk/libexec/_swiftwrap 2012-11-17 07:06:38 UTC (rev 6051) +++ trunk/libexec/_swiftwrap 2012-11-17 07:27:24 UTC (rev 6052) @@ -531,7 +531,7 @@ logstate "MOVING_OUTPUTS $OUTF" - for O in "$OUTF"; do + for O in $OUTF ; do if ! contains SKIPPED_OUTPUT $O ; then mv "$DIR/$O" "$WFDIR/shared/$O" 2>&1 >& "$INFO" checkError 254 "Failed to move output file $O to shared directory" From hategan at ci.uchicago.edu Sat Nov 17 02:10:08 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sat, 17 Nov 2012 02:10:08 -0600 (CST) Subject: [Swift-commit] r6053 - trunk/libexec Message-ID: <20121117081008.7394A9CCA5@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-17 02:10:08 -0600 (Sat, 17 Nov 2012) New Revision: 6053 Modified: trunk/libexec/_swiftwrap trunk/libexec/vdl-int.k Log: fixed transfer of stdout/stderr if they are specified as outputs of the app; should fix this to only transfer stderr since stdout may contain application data Modified: trunk/libexec/_swiftwrap =================================================================== --- trunk/libexec/_swiftwrap 2012-11-17 07:27:24 UTC (rev 6052) +++ trunk/libexec/_swiftwrap 2012-11-17 08:10:08 UTC (rev 6053) @@ -63,7 +63,7 @@ if [ "X$SCRATCH" != "X" ]; then if [ -f "$DIR/$STDOUT" ] || [ -f "$DIR/$STDERR" ]; then - mkdir -p "$WFDIR/jobs/$JOBDIR/$ID" + mkdir -p "$WFDIR/jobs/$JOBDIR/$ID" fi if [ -f "$DIR/$STDOUT" ]; then mv "$DIR/$STDOUT" "$WFDIR/jobs/$JOBDIR/$ID/$STDOUT" @@ -90,7 +90,7 @@ closeinfo moveMiscFilesFromScratch - + if [ "$STATUSMODE" = "files" ]; then exit 0 else @@ -530,7 +530,7 @@ fi logstate "MOVING_OUTPUTS $OUTF" - + for O in $OUTF ; do if ! contains SKIPPED_OUTPUT $O ; then mv "$DIR/$O" "$WFDIR/shared/$O" 2>&1 >& "$INFO" Modified: trunk/libexec/vdl-int.k =================================================================== --- trunk/libexec/vdl-int.k 2012-11-17 07:27:24 UTC (rev 6052) +++ trunk/libexec/vdl-int.k 2012-11-17 08:10:08 UTC (rev 6053) @@ -367,7 +367,7 @@ concat( for(f, list(list("stderr.txt", stderr), list("stdout.txt", stdout)) [name, file] := each(f) - destfile := "{jobid}-{file}" + destfile := concat("{jobid}-", basename(file)) try( sequential( task:transfer(srchost=rhost, srcdir=tmpdir, srcfile=file, From hategan at ci.uchicago.edu Sat Nov 17 03:38:55 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sat, 17 Nov 2012 03:38:55 -0600 (CST) Subject: [Swift-commit] r6054 - trunk/libexec Message-ID: <20121117093855.7ABC99CCA5@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-17 03:38:47 -0600 (Sat, 17 Nov 2012) New Revision: 6054 Modified: trunk/libexec/_swiftwrap Log: be more careful about symbolic links since in certain circumstances long enough chains of links can be created to crash stuff Modified: trunk/libexec/_swiftwrap =================================================================== --- trunk/libexec/_swiftwrap 2012-11-17 08:10:08 UTC (rev 6053) +++ trunk/libexec/_swiftwrap 2012-11-17 09:38:47 UTC (rev 6054) @@ -421,7 +421,15 @@ else [ -f $WFDIR/shared/$L ] checkError 254 "Could not locate input file: $L" - ln -s "$WFDIR/shared/$L" "$DIR/$L" 2>&1 >& $INFO + # if it's a link, be careful about creating + # long link chains because there is a rather low + # limit to how long that chain can be + if [ -L $WFDIR/shared/$L ]; then + # this might be of questionable portability + cp -P "$WFDIR/shared/$L" "$DIR/$L" 2>&1 >& $INFO + else + ln -s "$WFDIR/shared/$L" "$DIR/$L" 2>&1 >& $INFO + fi checkError 254 "Failed to link input file $L" log "Linked input: $WFDIR/shared/$L to $DIR/$L" fi From hategan at ci.uchicago.edu Sun Nov 18 01:40:26 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sun, 18 Nov 2012 01:40:26 -0600 (CST) Subject: [Swift-commit] r6055 - trunk/src/org/griphyn/vdl/mapping Message-ID: <20121118074026.5674E9CD1A@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-18 01:40:16 -0600 (Sun, 18 Nov 2012) New Revision: 6055 Modified: trunk/src/org/griphyn/vdl/mapping/MappingParamSet.java Log: don't put quotes around all mapping param values Modified: trunk/src/org/griphyn/vdl/mapping/MappingParamSet.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/MappingParamSet.java 2012-11-17 09:38:47 UTC (rev 6054) +++ trunk/src/org/griphyn/vdl/mapping/MappingParamSet.java 2012-11-18 07:40:16 UTC (rev 6055) @@ -68,9 +68,8 @@ } sb.append(", "); sb.append(name); - sb.append(" = \""); + sb.append(" = "); sb.append(Tracer.unwrapHandle(params.get(name))); - sb.append("\""); } sb.append('>'); return sb.toString(); From hategan at ci.uchicago.edu Sun Nov 18 01:41:40 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sun, 18 Nov 2012 01:41:40 -0600 (CST) Subject: [Swift-commit] r6056 - trunk/src/org/griphyn/vdl/karajan Message-ID: <20121118074140.1651C9CD1A@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-18 01:41:37 -0600 (Sun, 18 Nov 2012) New Revision: 6056 Modified: trunk/src/org/griphyn/vdl/karajan/VDL2ExecutionContext.java Log: dump a karajan level stack trace to the logs in the event of failure to help figure out where in, say, vdl-int the problem has occurred Modified: trunk/src/org/griphyn/vdl/karajan/VDL2ExecutionContext.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/VDL2ExecutionContext.java 2012-11-18 07:40:16 UTC (rev 6055) +++ trunk/src/org/griphyn/vdl/karajan/VDL2ExecutionContext.java 2012-11-18 07:41:37 UTC (rev 6056) @@ -23,6 +23,7 @@ import java.util.List; import org.apache.log4j.Logger; +import org.globus.cog.karajan.stack.Trace; import org.globus.cog.karajan.stack.VariableStack; import org.globus.cog.karajan.workflow.ElementTree; import org.globus.cog.karajan.workflow.ExecutionContext; @@ -45,7 +46,7 @@ protected void printFailure(ExecutionException e) { if (logger.isDebugEnabled()) { - logger.debug(e.getMessage(), e); + logger.debug("Karajan level error: " + getKarajanTrace(e)); } String msg = e.getMessage(); if (!"Execution completed with errors".equals(msg)) { @@ -74,7 +75,26 @@ } } - protected void setGlobals(VariableStack stack) { + private String getKarajanTrace(ExecutionException e) { + StringBuilder sb = new StringBuilder(); + while (e != null) { + sb.append(e.getMessage()); + if (e.getStack() != null) { + sb.append(" at\n"); + sb.append(Trace.get(e.getStack())); + } + if (e.getCause() instanceof ExecutionException) { + e = (ExecutionException) e.getCause(); + sb.append("\ncaused by: "); + } + else { + e = null; + } + } + return sb.toString(); + } + + protected void setGlobals(VariableStack stack) { super.setGlobals(stack); stack.setGlobal(RUN_ID, runID); stack.setGlobal(SCRIPT_NAME, scriptName); From hategan at ci.uchicago.edu Sun Nov 18 02:44:53 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sun, 18 Nov 2012 02:44:53 -0600 (CST) Subject: [Swift-commit] r6057 - trunk/src/org/griphyn/vdl/mapping/file Message-ID: <20121118084453.092A69CD0B@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-18 02:44:46 -0600 (Sun, 18 Nov 2012) New Revision: 6057 Modified: trunk/src/org/griphyn/vdl/mapping/file/FixedArrayFileMapper.java Log: fixed fixed_array_mapper Modified: trunk/src/org/griphyn/vdl/mapping/file/FixedArrayFileMapper.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/file/FixedArrayFileMapper.java 2012-11-18 07:41:37 UTC (rev 6056) +++ trunk/src/org/griphyn/vdl/mapping/file/FixedArrayFileMapper.java 2012-11-18 08:44:46 UTC (rev 6057) @@ -18,23 +18,31 @@ package org.griphyn.vdl.mapping.file; import java.util.ArrayList; +import java.util.Arrays; import java.util.Collection; +import java.util.Iterator; import java.util.List; +import java.util.Map; +import java.util.Set; import java.util.StringTokenizer; +import java.util.TreeSet; import org.griphyn.vdl.mapping.AbsFile; import org.griphyn.vdl.mapping.AbstractMapper; +import org.griphyn.vdl.mapping.DSHandle; import org.griphyn.vdl.mapping.InvalidMappingParameterException; import org.griphyn.vdl.mapping.MappingParam; import org.griphyn.vdl.mapping.MappingParamSet; import org.griphyn.vdl.mapping.Path; import org.griphyn.vdl.mapping.PhysicalFormat; +import org.griphyn.vdl.type.Types; /** Maps a string (separated by space, comma or semicolon) of filenames to an array. */ public class FixedArrayFileMapper extends AbstractMapper { public static final MappingParam PARAM_FILES = new MappingParam("files"); + private List files; public FixedArrayFileMapper() { super(); @@ -42,26 +50,42 @@ public void setParams(MappingParamSet params) { super.setParams(params); - String cfiles = PARAM_FILES.getStringValue(this); - if (cfiles == null) { - throw new InvalidMappingParameterException("Missing required mapper parameter: " - + PARAM_FILES); + DSHandle dn = (DSHandle) PARAM_FILES.getRawValue(this); + if (dn == null) { + throw new InvalidMappingParameterException("Missing required mapper parameter: " + + PARAM_FILES); } - StringTokenizer st = new StringTokenizer(cfiles, " ,;"); - String[] files = new String[st.countTokens()]; - for (int i = 0; st.hasMoreTokens(); i++) { - files[i] = st.nextToken(); + if (Types.STRING.equals(dn.getType())) { + String cfiles = (String) dn.getValue(); + + StringTokenizer st = new StringTokenizer(cfiles, " ,;"); + String[] files = new String[st.countTokens()]; + for (int i = 0; st.hasMoreTokens(); i++) { + files[i] = st.nextToken(); + } + this.files = Arrays.asList(files); } - PARAM_FILES.setValue(this, files); + else if (dn.getType().isArray() && Types.STRING.equals(dn.getType().itemType())) { + files = new ArrayList(); + Map m = dn.getArrayValue(); + // must keep order + @SuppressWarnings("unchecked") + Set> s = new TreeSet>((Set>) m.keySet()); + Iterator i = s.iterator(); + while(i.hasNext()) { + Comparable nextKey = (Comparable) i.next(); + files.add((String) m.get(nextKey).getValue()); + } + } + else { + throw new InvalidMappingParameterException("Unrecognized value for " + + PARAM_FILES + " parameter: " + dn.getType() + ". Valid values are a string or an array of strings."); + } } - protected String[] getFiles() { - return (String[]) PARAM_FILES.getValue(this); - } - public Collection existing() { List l = new ArrayList(); - for (int i = 0; i < getFiles().length; i++) { + for (int i = 0; i < files.size(); i++) { l.add(Path.EMPTY_PATH.addLast(i, true)); } return l; @@ -75,7 +99,7 @@ Object o = path.getFirst(); if (o instanceof Integer) { int index = ((Integer) o).intValue(); - return new AbsFile(getFiles()[index]); + return new AbsFile(files.get(index)); } else { throw new IllegalArgumentException("The fixed array mapper can only be used with an int key array"); From hategan at ci.uchicago.edu Sun Nov 18 02:48:51 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sun, 18 Nov 2012 02:48:51 -0600 (CST) Subject: [Swift-commit] r6058 - trunk/src/org/griphyn/vdl/mapping Message-ID: <20121118084851.89F269CD0B@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-18 02:48:48 -0600 (Sun, 18 Nov 2012) New Revision: 6058 Modified: trunk/src/org/griphyn/vdl/mapping/RootDataNode.java Log: more tracing of existing fields and also mention whether no files are found Modified: trunk/src/org/griphyn/vdl/mapping/RootDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/RootDataNode.java 2012-11-18 08:44:46 UTC (rev 6057) +++ trunk/src/org/griphyn/vdl/mapping/RootDataNode.java 2012-11-18 08:48:48 UTC (rev 6058) @@ -155,13 +155,16 @@ } private static void addExisting(Mapper mapper, AbstractDataNode root) { + boolean any = false; for (Path p : mapper.existing()) { try { DSHandle field = root.getField(p); field.closeShallow(); - if (logger.isInfoEnabled()) { - logger.info("Found data " + root + "." + p); + if (tracer.isEnabled()) { + tracer.trace(root.getThread(), root.getDeclarationLine(), + root.getDisplayableName() + " MAPPING " + p + ", " + mapper.map(p)); } + any = true; } catch (InvalidPathException e) { throw new IllegalStateException("Structure of mapped data is " + @@ -169,6 +172,10 @@ } } root.closeDeep(); + if (!any && tracer.isEnabled()) { + tracer.trace(root.getThread(), root.getDeclarationLine(), + root.getDisplayableName() + " MAPPING no files found"); + } } public static void checkConsistency(DSHandle handle) { From hategan at ci.uchicago.edu Sun Nov 18 04:50:47 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sun, 18 Nov 2012 04:50:47 -0600 (CST) Subject: [Swift-commit] r6059 - in trunk/src/org/griphyn/vdl: karajan karajan/lib mapping Message-ID: <20121118105047.4CECC9CD1A@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-18 04:50:43 -0600 (Sun, 18 Nov 2012) New Revision: 6059 Modified: trunk/src/org/griphyn/vdl/karajan/VDL2ExecutionContext.java trunk/src/org/griphyn/vdl/karajan/lib/New.java trunk/src/org/griphyn/vdl/karajan/lib/SetFieldValue.java trunk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java trunk/src/org/griphyn/vdl/mapping/RootDataNode.java Log: added duplicate mapping checker Modified: trunk/src/org/griphyn/vdl/karajan/VDL2ExecutionContext.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/VDL2ExecutionContext.java 2012-11-18 08:48:48 UTC (rev 6058) +++ trunk/src/org/griphyn/vdl/karajan/VDL2ExecutionContext.java 2012-11-18 10:50:43 UTC (rev 6059) @@ -29,12 +29,14 @@ import org.globus.cog.karajan.workflow.ExecutionContext; import org.globus.cog.karajan.workflow.ExecutionException; import org.griphyn.vdl.karajan.functions.ProcessBulkErrors; +import org.griphyn.vdl.mapping.DuplicateMappingChecker; public class VDL2ExecutionContext extends ExecutionContext { public static final Logger logger = Logger.getLogger(VDL2ExecutionContext.class); public static final String RUN_ID = "vdl:runid"; public static final String SCRIPT_NAME = "vdl:scriptname"; + public static final String DM_CHECKER = "vdl:dpmchecker"; private String runID; private final String scriptName; @@ -98,6 +100,7 @@ super.setGlobals(stack); stack.setGlobal(RUN_ID, runID); stack.setGlobal(SCRIPT_NAME, scriptName); + stack.setGlobal(DM_CHECKER, new DuplicateMappingChecker()); } public String getRunID() { Modified: trunk/src/org/griphyn/vdl/karajan/lib/New.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/New.java 2012-11-18 08:48:48 UTC (rev 6058) +++ trunk/src/org/griphyn/vdl/karajan/lib/New.java 2012-11-18 10:50:43 UTC (rev 6059) @@ -29,8 +29,10 @@ import org.globus.cog.karajan.stack.VariableStack; import org.globus.cog.karajan.util.TypeUtil; import org.globus.cog.karajan.workflow.ExecutionException; +import org.griphyn.vdl.karajan.VDL2ExecutionContext; import org.griphyn.vdl.mapping.AbstractDataNode; import org.griphyn.vdl.mapping.DSHandle; +import org.griphyn.vdl.mapping.DuplicateMappingChecker; import org.griphyn.vdl.mapping.ExternalDataNode; import org.griphyn.vdl.mapping.MappingParam; import org.griphyn.vdl.mapping.MappingParamSet; @@ -120,7 +122,8 @@ } else if (type.isArray()) { // dealing with array variable - handle = new RootArrayDataNode(type); + handle = new RootArrayDataNode(type, + (DuplicateMappingChecker) stack.getGlobal(VDL2ExecutionContext.DM_CHECKER)); if (value != null) { if (value instanceof RootArrayDataNode) { if (tracer.isEnabled()) { @@ -169,7 +172,8 @@ handle = (DSHandle) value; } else { - handle = new RootDataNode(type); + handle = new RootDataNode(type, + (DuplicateMappingChecker) stack.getGlobal(VDL2ExecutionContext.DM_CHECKER)); handle.init(mps); if (value != null) { if (tracer.isEnabled()) { Modified: trunk/src/org/griphyn/vdl/karajan/lib/SetFieldValue.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/SetFieldValue.java 2012-11-18 08:48:48 UTC (rev 6058) +++ trunk/src/org/griphyn/vdl/karajan/lib/SetFieldValue.java 2012-11-18 10:50:43 UTC (rev 6059) @@ -28,13 +28,9 @@ import org.globus.cog.karajan.arguments.Arg; import org.globus.cog.karajan.stack.VariableNotFoundException; import org.globus.cog.karajan.stack.VariableStack; -import org.globus.cog.karajan.util.ThreadingContext; import org.globus.cog.karajan.workflow.ExecutionException; -import org.globus.cog.karajan.workflow.futures.Future; import org.globus.cog.karajan.workflow.futures.FutureFault; import org.globus.cog.karajan.workflow.futures.FutureNotYetAvailable; -import org.griphyn.vdl.karajan.DSHandleFutureWrapper; -import org.griphyn.vdl.karajan.FutureWrapper; import org.griphyn.vdl.karajan.Pair; import org.griphyn.vdl.karajan.PairIterator; import org.griphyn.vdl.karajan.WaitingThreadsMonitor; Modified: trunk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java 2012-11-18 08:48:48 UTC (rev 6058) +++ trunk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java 2012-11-18 10:50:43 UTC (rev 6059) @@ -34,6 +34,7 @@ private Mapper mapper; private MappingParamSet params; private AbstractDataNode waitingMapperParam; + private DuplicateMappingChecker dmc; private static final Tracer tracer = Tracer.getTracer("VARIABLE"); @@ -42,6 +43,11 @@ */ public RootArrayDataNode(Type type) { super(Field.Factory.createField(null, type), null, null); + } + + public RootArrayDataNode(Type type, DuplicateMappingChecker dmc) { + this(type); + this.dmc = dmc; } public void init(MappingParamSet params) { @@ -90,7 +96,7 @@ private void checkInputs() { try { - RootDataNode.checkInputs(params, mapper, this); + RootDataNode.checkInputs(params, mapper, this, dmc); } catch (DependentException e) { setValue(new MappingDependentException(this, e)); Modified: trunk/src/org/griphyn/vdl/mapping/RootDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/RootDataNode.java 2012-11-18 08:48:48 UTC (rev 6058) +++ trunk/src/org/griphyn/vdl/mapping/RootDataNode.java 2012-11-18 10:50:43 UTC (rev 6059) @@ -38,13 +38,19 @@ private Mapper mapper; private MappingParamSet params; private AbstractDataNode waitingMapperParam; + private DuplicateMappingChecker dmc; private static final Tracer tracer = Tracer.getTracer("VARIABLE"); - public RootDataNode(Type type) { + public RootDataNode(Type type, DuplicateMappingChecker dmc) { super(Field.Factory.createField(null, type)); + this.dmc = dmc; } + public RootDataNode(Type type) { + this(type, null); + } + public RootDataNode(Type type, Object value) { this(type); initialized(); @@ -99,7 +105,7 @@ private void checkInputs() { try { - checkInputs(params, mapper, this); + checkInputs(params, mapper, this, dmc); } catch (DependentException e) { setValue(new MappingDependentException(this, e)); @@ -113,10 +119,11 @@ } - static protected void checkInputs(MappingParamSet params, Mapper mapper, AbstractDataNode root) { + static protected void checkInputs(MappingParamSet params, Mapper mapper, AbstractDataNode root, + DuplicateMappingChecker dmc) { String input = (String) params.get(MappingParam.SWIFT_INPUT); if (input != null && Boolean.valueOf(input.trim()).booleanValue()) { - addExisting(mapper, root); + addExisting(mapper, root, dmc); checkConsistency(root); } else if (mapper.isStatic()) { @@ -131,10 +138,12 @@ logger.debug("mapper: " + mapper); } for (Path p : mapper.existing()) { + PhysicalFormat f = mapper.map(p); try { // Try to get the path in order to check that the // path is valid - we'll get an exception if not - root.getField(p); + DSHandle h = root.getField(p); + dmc.addWrite(f, h); if (tracer.isEnabled()) { tracer.trace(root.getThread(), root.getDeclarationLine(), root.getDisplayableName() + " MAPPING " + p + ", " + mapper.map(p)); @@ -154,12 +163,14 @@ } } - private static void addExisting(Mapper mapper, AbstractDataNode root) { + private static void addExisting(Mapper mapper, AbstractDataNode root, DuplicateMappingChecker dmc) { boolean any = false; for (Path p : mapper.existing()) { + PhysicalFormat f = mapper.map(p); try { DSHandle field = root.getField(p); field.closeShallow(); + dmc.addRead(f, field); if (tracer.isEnabled()) { tracer.trace(root.getThread(), root.getDeclarationLine(), root.getDisplayableName() + " MAPPING " + p + ", " + mapper.map(p)); From hategan at ci.uchicago.edu Sun Nov 18 05:07:46 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sun, 18 Nov 2012 05:07:46 -0600 (CST) Subject: [Swift-commit] r6060 - trunk/src/org/griphyn/vdl/mapping Message-ID: <20121118110746.EAD479CD1A@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-18 05:07:46 -0600 (Sun, 18 Nov 2012) New Revision: 6060 Added: trunk/src/org/griphyn/vdl/mapping/DuplicateMappingChecker.java Log: added the actual duplicate mapping check class Added: trunk/src/org/griphyn/vdl/mapping/DuplicateMappingChecker.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/DuplicateMappingChecker.java (rev 0) +++ trunk/src/org/griphyn/vdl/mapping/DuplicateMappingChecker.java 2012-11-18 11:07:46 UTC (rev 6060) @@ -0,0 +1,95 @@ +//---------------------------------------------------------------------- +//This code is developed as part of the Java CoG Kit project +//The terms of the license can be found at http://www.cogkit.org/license +//This message may not be removed or altered. +//---------------------------------------------------------------------- + +/* + * Created on Nov 18, 2012 + */ +package org.griphyn.vdl.mapping; + +import java.util.HashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +import org.apache.log4j.Logger; + +public class DuplicateMappingChecker { + public static final Logger logger = Logger.getLogger(DuplicateMappingChecker.class); + + private final Map map; + + public DuplicateMappingChecker() { + map = new HashMap(); + } + + private static class Entry { + private DSHandle write; + private List read; + + } + + private Entry getEntry(PhysicalFormat f) { + Entry e = map.get(f); + if (e == null) { + e = new Entry(); + map.put(f, e); + } + return e; + } + + public synchronized void addRead(PhysicalFormat f, DSHandle h) { + Entry e = getEntry(f); + if (e.write != null) { + warn("Duplicate mapping found:\n\t" + + getVarInfo(h) + " is used to read from " + f + "\n\t" + + getVarInfo(e.write) + " is used to write to " + f); + } + if (e.read == null) { + e.read = new LinkedList(); + } + e.read.add(h); + } + + public synchronized void addWrite(PhysicalFormat f, DSHandle h) { + Entry e = getEntry(f); + if (e.write != null) { + warn("Duplicate mapping found:\n\t" + + getVarInfo(h) + " and " + getVarInfo(e.write) + " are both used to write to " + f); + } + if (e.read != null) { + warn("Duplicate mapping found:\n\t" + + getVarInfo(e.write) + " is used to write to " + f + "\n\t" + + "The following variables(s) are also used to read from " + f + ":" + getVarInfos(e.read)); + } + e.write = h; + } + + private void warn(String s) { + if (logger.isInfoEnabled()) { + logger.info(s); + } + System.err.println(s); + } + + private String getVarInfos(List l) { + StringBuilder sb = new StringBuilder(); + for (DSHandle h : l) { + sb.append("\n\t\t"); + sb.append(getVarInfo(h)); + } + return sb.toString(); + } + + private String getVarInfo(DSHandle h) { + if (h instanceof AbstractDataNode) { + AbstractDataNode a = (AbstractDataNode) h; + return a.getDisplayableName() + " (line " + a.getDeclarationLine() + ")"; + } + else { + return String.valueOf(h); + } + } +} From swift at ci.uchicago.edu Sun Nov 18 16:55:55 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Sun, 18 Nov 2012 16:55:55 -0600 (CST) Subject: [Swift-commit] cog r3512 Message-ID: <20121118225555.8B7828D0007A@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3512 | hategan | 2012-11-18 16:54:36 -0600 (Sun, 18 Nov 2012) | 1 line small test to make sure paking is the same as perl's ------------------------------------------------------------------------ Index: modules/karajan/src/org/globus/cog/karajan/workflow/service/RequestReply.java =================================================================== --- modules/karajan/src/org/globus/cog/karajan/workflow/service/RequestReply.java (revision 3511) +++ modules/karajan/src/org/globus/cog/karajan/workflow/service/RequestReply.java (working copy) @@ -36,7 +36,7 @@ public static final int DEFAULT_TIMEOUT = 120 * 1000; private int timeout = DEFAULT_TIMEOUT; - + public static final int NOID = -1; private int id; private String outCmd; @@ -177,7 +177,7 @@ return inData; } - public byte[] getInData() { + public synchronized byte[] getInData() { if (inData == null) { return null; } @@ -246,11 +246,49 @@ } public static void main(String[] args) { - System.out.println(unpackLong(pack(1L))); - System.out.println(unpackLong(pack(10L))); - System.out.println(unpackLong(pack(1000000000L))); - System.out.println(unpackLong(pack(10000000000000L))); + test(1L); + test(10L); + test(1000000000L); + test(10000000000000L); + testPerl(1L); + testPerl(10L); + testPerl(1000000000L); + testPerl(10000000000000L); } + + private static void test(long l) { + System.out.print("Test " + l); + if (unpackLong(pack(l)) != l) { + throw new RuntimeException("Failed: " + l); + } + System.out.println(" OK"); + } + + private static void testPerl(long l) { + System.out.print("Test perl " + l); + if (unpackLong(packPerl(l)) != l) { + throw new RuntimeException("Failed perl: " + l); + } + System.out.println(" OK"); + } + + private static byte[] packPerl(long l) { + try { + Process p = Runtime.getRuntime().exec(new String[] + {"perl", "-e", "my $ts = " + l + "; print pack(\"VV\", ($ts & 0xffffffff), ($ts >> 32));"}); + int ec = p.waitFor(); + if (ec != 0) { + return new byte[8]; + } + byte b[] = new byte[8]; + p.getInputStream().read(b); + return b; + } + catch (Exception e) { + e.printStackTrace(); + return new byte[8]; + } + } public boolean getInDataAsBoolean(int index) { return getInDataAsInt(index) != 0; From swift at ci.uchicago.edu Sun Nov 18 17:00:39 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Sun, 18 Nov 2012 17:00:39 -0600 (CST) Subject: [Swift-commit] cog r3514 Message-ID: <20121118230039.781A78D0007A@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3514 | hategan | 2012-11-18 17:00:04 -0600 (Sun, 18 Nov 2012) | 1 line log stdout/stderr of bootstrap sub-process ------------------------------------------------------------------------ Index: modules/provider-coaster/src/org/globus/cog/abstraction/impl/execution/coaster/bootstrap/Bootstrap.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/impl/execution/coaster/bootstrap/Bootstrap.java (revision 3513) +++ modules/provider-coaster/src/org/globus/cog/abstraction/impl/execution/coaster/bootstrap/Bootstrap.java (working copy) @@ -212,6 +212,8 @@ } } logger.log("Exit code: " + ec); + logger.log("Stdout from process: " + out.toString()); + logger.log("Stderr from process: " + err.toString()); System.out.println(out.toString()); System.err.println(err.toString()); if (ec != 0) { From swift at ci.uchicago.edu Sun Nov 18 17:05:28 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Sun, 18 Nov 2012 17:05:28 -0600 (CST) Subject: [Swift-commit] cog r3515 Message-ID: <20121118230528.A484F8D0007A@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3515 | hategan | 2012-11-18 17:01:24 -0600 (Sun, 18 Nov 2012) | 1 line better channel names in logs; ensure unique ids for worker channels if two or more services are on the same jvm ------------------------------------------------------------------------ Index: modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/local/RegistrationHandler.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/local/RegistrationHandler.java (revision 3514) +++ modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/local/RegistrationHandler.java (working copy) @@ -48,7 +48,9 @@ options = Collections.emptyMap(); } - logger.debug("registering: " + id + " " + url); + if (logger.isDebugEnabled()) { + logger.debug("registering: " + id + " " + url); + } KarajanChannel channel = getChannel(); ChannelContext context = channel.getChannelContext(); Index: modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/local/LocalService.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/local/LocalService.java (revision 3514) +++ modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/local/LocalService.java (working copy) @@ -18,9 +18,7 @@ import org.apache.log4j.Logger; import org.globus.cog.abstraction.coaster.service.Registering; -import org.globus.cog.abstraction.impl.common.AbstractionFactory; import org.globus.cog.abstraction.impl.common.execution.JobException; -import org.globus.cog.abstraction.impl.common.task.ServiceContactImpl; import org.globus.cog.abstraction.impl.common.task.TaskSubmissionException; import org.globus.cog.abstraction.interfaces.Service; import org.globus.cog.abstraction.interfaces.Status; @@ -71,7 +69,7 @@ logger.info("Got connection"); try { ConnectionHandler handler = - new ConnectionHandler(this, sock, LocalRequestManager.INSTANCE); + new ConnectionHandler("service-" + sock.getPort(), this, sock, LocalRequestManager.INSTANCE); logger.info("Initialized connection handler"); handler.start(); logger.info("Connection handler started"); @@ -84,7 +82,7 @@ public void handleConnection(InputStream is, OutputStream os) { try { ConnectionHandler handler = - new ConnectionHandler(this, is, os, LocalRequestManager.INSTANCE); + new ConnectionHandler("service-pipe", this, is, os, LocalRequestManager.INSTANCE); handler.start(); } catch (Exception e) { @@ -93,7 +91,7 @@ } public PipedServerChannel newPipedServerChannel() { - return new PipedServerChannel(LocalRequestManager.INSTANCE, new ChannelContext(this)); + return new PipedServerChannel(LocalRequestManager.INSTANCE, new ChannelContext("spipe", this)); } public String waitForRegistration(Task t, String id) throws InterruptedException, Index: modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/CoasterService.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/CoasterService.java (revision 3514) +++ modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/CoasterService.java (working copy) @@ -128,7 +128,7 @@ } else { try { - ConnectionHandler handler = new ConnectionHandler(this, sock, + ConnectionHandler handler = new ConnectionHandler("cps-" + sock.getPort(), this, sock, COASTER_REQUEST_MANAGER); handler.start(); } @@ -194,7 +194,7 @@ PipedServerChannel psc = ServiceManager.getDefault().getLocalService().newPipedServerChannel(); PipedClientChannel pcc = - new PipedClientChannel(COASTER_REQUEST_MANAGER, new ChannelContext(), psc); + new PipedClientChannel(COASTER_REQUEST_MANAGER, new ChannelContext("cpipe"), psc); psc.setClientChannel(pcc); ChannelManager.getManager().registerChannel(pcc.getChannelContext().getChannelID(), pcc); ChannelManager.getManager().registerChannel(psc.getChannelContext().getChannelID(), psc); Index: modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/LocalTCPService.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/LocalTCPService.java (revision 3514) +++ modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/LocalTCPService.java (working copy) @@ -102,10 +102,13 @@ channel = ServerSocketChannel.open(); channel.configureBlocking(true); if(port == 0) { - PortRange portRange = PortRange.getTcpInstance(); - if(portRange != null && portRange.isEnabled()) { - port = portRange.getFreePort(port); - } + PortRange portRange = PortRange.getTcpInstance(); + if(portRange != null && portRange.isEnabled()) { + synchronized(portRange) { + port = portRange.getFreePort(port); + portRange.setUsed(port); + } + } } channel.socket().bind(new InetSocketAddress(port)); @@ -189,12 +192,18 @@ System.err.println("Irrecoverable channel exception: " + e.getMessage()); System.exit(2); } + + private static int idSeq = 1; + + private synchronized static int nextId() { + return idSeq++; + } private static class WorkerConnectionHandler extends ConnectionHandler { public WorkerConnectionHandler(Service service, Socket socket, RequestManager requestManager) throws IOException { super(socket, new WorkerChannel(socket, requestManager, - new ChannelContext(service)), requestManager); + new ChannelContext("worker-" + nextId(), service)), requestManager); } } Index: modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Node.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Node.java (revision 3514) +++ modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Node.java (working copy) @@ -85,6 +85,7 @@ } try { KarajanChannel channel = getChannel(); + channel.setLocalShutdown(); ChannelManager.getManager().reserveLongTerm(channel); ShutdownCommand cmd = new ShutdownCommand(); cmd.executeAsync(channel, this); Index: modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/BlockQueueProcessor.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/BlockQueueProcessor.java (revision 3514) +++ modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/BlockQueueProcessor.java (working copy) @@ -98,6 +98,12 @@ */ private static final NumberFormat SECONDS_3 = new DecimalFormat("0.000"); + + private static int sid; + + private synchronized static int nextSid() { + return sid++; + } public BlockQueueProcessor(Settings settings) { super("Block Queue Processor"); @@ -105,7 +111,7 @@ holding = new SortedJobSet(); blocks = new TreeMap(); tl = new HashMap>(); - id = DDF.format(new Date()); + id = DDF.format(new Date()) + nextSid(); incoming = new ArrayList(); metric = new OverallocatedJobDurationMetric(settings); queued = new SortedJobSet(metric); Index: modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Block.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Block.java (revision 3514) +++ modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/job/manager/Block.java (working copy) @@ -59,6 +59,10 @@ private long lastUsed; private static int sid; + + private synchronized static int nextSID() { + return sid++; + } private static final NumberFormat IDF = new DecimalFormat("000000"); @@ -72,7 +76,7 @@ } public Block(int workers, TimeInterval walltime, BlockQueueProcessor ap) { - this(ap.getBQPId() + "-" + IDF.format(sid++), workers, walltime, ap); + this(ap.getBQPId() + "-" + IDF.format(nextSID()), workers, walltime, ap); } public Block(String id, int workers, TimeInterval walltime, BlockQueueProcessor ap) { From davidk at ci.uchicago.edu Mon Nov 19 13:26:06 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Mon, 19 Nov 2012 13:26:06 -0600 (CST) Subject: [Swift-commit] r6061 - in SwiftApps/SciColSim: . benchmarks Message-ID: <20121119192606.84B189CD0B@svn.ci.uchicago.edu> Author: davidk Date: 2012-11-19 13:26:06 -0600 (Mon, 19 Nov 2012) New Revision: 6061 Modified: SwiftApps/SciColSim/Makefile SwiftApps/SciColSim/benchmarks/benchmark.sh SwiftApps/SciColSim/benchmarks/benchmark.swift SwiftApps/SciColSim/benchmarks/plot.gp SwiftApps/SciColSim/benchmarks/plot.sh SwiftApps/SciColSim/gengraphs.sh Log: Compile with profiling enabled Update benchmark scripts to record memory usage Various changes to the benchmarking scripts Modified: SwiftApps/SciColSim/Makefile =================================================================== --- SwiftApps/SciColSim/Makefile 2012-11-18 11:07:46 UTC (rev 6060) +++ SwiftApps/SciColSim/Makefile 2012-11-19 19:26:06 UTC (rev 6061) @@ -8,7 +8,7 @@ g++ -DP_OPENMP -static -O -fopenmp -I boost_1_47_0 -o openmp-optimizer optimizer.cpp graphsim: graphsim.cpp - g++ -DP_OPENMP -static -O -fopenmp -I boost_1_47_0 -o graphsim graphsim.cpp + g++ -DP_OPENMP -static -pg -O -fopenmp -I boost_1_47_0 -o graphsim graphsim.cpp clean: @rm -rvf openmp-optimizer Modified: SwiftApps/SciColSim/benchmarks/benchmark.sh =================================================================== --- SwiftApps/SciColSim/benchmarks/benchmark.sh 2012-11-18 11:07:46 UTC (rev 6060) +++ SwiftApps/SciColSim/benchmarks/benchmark.sh 2012-11-19 19:26:06 UTC (rev 6061) @@ -2,18 +2,31 @@ # Create run directory ORIGDIR=$PWD -OUTPUT=$ORIGDIR/$3 +TIME_OUTPUT=$ORIGDIR/$3 +MEM_OUTPUT=$ORIGDIR/$4 + cd $HOME/SciColSim +# Clean any previous run if [ -f "bestdb.txt" ]; then rm -f bestdb.txt fi +# Run programs export OMP_NUM_THREADS=1 -result=$( bin/timeout $2 ./graphsim 0 0 0 0 0 $1 40000 20 1 2 1 2. 0.01 1 0.3 2.3 0 0 0 0 0 m 1 | grep time | awk '{print $5}' ) +export WORK=$ORIGDIR +tmpfile=$( mktemp -p . ) +tmpfile2=$( mktemp -p . ) +/usr/bin/time --format='Memory: %R' -a -o $tmpfile bin/timeout $2 ./graphsim 0 0 0 0 0 $1 40000 20 1 2 1 2. 0.01 1 0.3 2.3 0 0 0 0 0 m 1 > $tmpfile2 +memusage=$( grep Memory $tmpfile | awk '{print $2}' ) +timing=$( grep time $tmpfile2 | awk '{print $5}' ) +rm $tmpfile +rm $tmpfile2 -if [ -n "$result" ]; then - echo $1 $result > $OUTPUT +# Write results +echo $1 $memusage > $MEM_OUTPUT +if [ -n "$timing" ]; then + echo $1 $timing > $TIME_OUTPUT else - echo "$1 -1" > $OUTPUT + echo "$1 -1" > $TIME_OUTPUT fi Modified: SwiftApps/SciColSim/benchmarks/benchmark.swift =================================================================== --- SwiftApps/SciColSim/benchmarks/benchmark.swift 2012-11-18 11:07:46 UTC (rev 6060) +++ SwiftApps/SciColSim/benchmarks/benchmark.swift 2012-11-19 19:26:06 UTC (rev 6061) @@ -1,33 +1,38 @@ type file; # Perform a single benchmark -app (file o, file e) benchmark (int target_innovation, int tl) +app (file timef, file memf, file errorf) benchmark (int target_innovation, int tl) { - benchmark target_innovation tl @filename(o) stderr=@e; + benchmark target_innovation tl @filename(timef) @filename(memf) stderr=@errorf; } # Create a plot from an array of result files -app (file png) plot (file in1[], file gpl) +app (file png) plot (file in1[], file in2[], file gpl) { plot @filename(png); } int target_innovation_start=1; -int target_innovation_stop=10; -int step=1; -int time_limit=10; +int target_innovation_stop=12000; +int step=1000; +int time_limit=86300; # Run benchmark #file graphs[] ; file graph <"movie-graph.txt">; -file results[]; +file timeresults[]; +file memresults[]; + foreach ti,tidx in [target_innovation_start:target_innovation_stop:step] { - file out ; + file times ; + file mem ; file error ; - (out, error) = benchmark(ti, time_limit); - results[tidx] = out; + (times, mem, error) = benchmark(ti, time_limit); + timeresults[tidx] = times; + memresults[tidx] = mem; } + file png <"results/plot.png">; file gp <"plot.gp">; -png = plot(results, gp); +png = plot(memresults, timeresults, gp); Modified: SwiftApps/SciColSim/benchmarks/plot.gp =================================================================== --- SwiftApps/SciColSim/benchmarks/plot.gp 2012-11-18 11:07:46 UTC (rev 6060) +++ SwiftApps/SciColSim/benchmarks/plot.gp 2012-11-19 19:26:06 UTC (rev 6061) @@ -5,5 +5,4 @@ set xlabel "Target Innovation" set ylabel "Time in Seconds" set title "`basename $PLOT_OUTPUT .png`" -plot "results.txt" using 1:2 with line - +plot "results.txt" using 1:2 with linespoints Modified: SwiftApps/SciColSim/benchmarks/plot.sh =================================================================== --- SwiftApps/SciColSim/benchmarks/plot.sh 2012-11-18 11:07:46 UTC (rev 6060) +++ SwiftApps/SciColSim/benchmarks/plot.sh 2012-11-19 19:26:06 UTC (rev 6061) @@ -2,5 +2,5 @@ export PLOT_OUTPUT=$1 export PLOT_TITLE=$( basename $( dirname $1 )) -find . -name *.out -exec cat {} \; | sort -n > results.txt +find . -name *.out -exec cat {} \; | sort -n | grep -v -- "-1" > results.txt gnuplot plot.gp Modified: SwiftApps/SciColSim/gengraphs.sh =================================================================== --- SwiftApps/SciColSim/gengraphs.sh 2012-11-18 11:07:46 UTC (rev 6060) +++ SwiftApps/SciColSim/gengraphs.sh 2012-11-19 19:26:06 UTC (rev 6061) @@ -6,4 +6,16 @@ ./trimgraph.sh $s 1.0 > graph/$s.100 ./trimgraph.sh $s 0.1 > graph/$s.10 ./trimgraph.sh $s 0.5 > graph/$s.50 -done \ No newline at end of file +done + +# Runtime, ram, science results (single floating pointer number called loss) for each param +# Params = nodes, edges, and ti +# As edges/nodes increase, time increases +# 1000 osg jobs between 10 seconds and 24 hours +# Example: iterate over ti in units of 100, graph sizes should be +# Automate, return eof when timed out, set max run time +# Ability to see gprof output +# Once with control structure he needs, then one run with large size to show hockey stick +# Generate a table, then generate a graph +# Email uc3 and midway account information + From davidk at ci.uchicago.edu Mon Nov 19 14:16:26 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Mon, 19 Nov 2012 14:16:26 -0600 (CST) Subject: [Swift-commit] r6062 - in SwiftApps/SciColSim/benchmarks: . conf Message-ID: <20121119201626.C7A449CD0B@svn.ci.uchicago.edu> Author: davidk Date: 2012-11-19 14:16:26 -0600 (Mon, 19 Nov 2012) New Revision: 6062 Added: SwiftApps/SciColSim/benchmarks/conf/raven.cf SwiftApps/SciColSim/benchmarks/conf/raven.xml Modified: SwiftApps/SciColSim/benchmarks/benchmark.sh SwiftApps/SciColSim/benchmarks/benchmark.swift SwiftApps/SciColSim/benchmarks/start-benchmark.sh Log: Pass start, stop, step, and timeout values from command line Modified: SwiftApps/SciColSim/benchmarks/benchmark.sh =================================================================== --- SwiftApps/SciColSim/benchmarks/benchmark.sh 2012-11-19 19:26:06 UTC (rev 6061) +++ SwiftApps/SciColSim/benchmarks/benchmark.sh 2012-11-19 20:16:26 UTC (rev 6062) @@ -5,7 +5,7 @@ TIME_OUTPUT=$ORIGDIR/$3 MEM_OUTPUT=$ORIGDIR/$4 -cd $HOME/SciColSim +cd $6 # Clean any previous run if [ -f "bestdb.txt" ]; then Modified: SwiftApps/SciColSim/benchmarks/benchmark.swift =================================================================== --- SwiftApps/SciColSim/benchmarks/benchmark.swift 2012-11-19 19:26:06 UTC (rev 6061) +++ SwiftApps/SciColSim/benchmarks/benchmark.swift 2012-11-19 20:16:26 UTC (rev 6062) @@ -1,9 +1,9 @@ type file; # Perform a single benchmark -app (file timef, file memf, file errorf) benchmark (int target_innovation, int tl) +app (file timef, file memf, file errorf) benchmark (int target_innovation, int tl, string scidir) { - benchmark target_innovation tl @filename(timef) @filename(memf) stderr=@errorf; + benchmark target_innovation tl @filename(timef) @filename(memf) scidir stderr=@errorf; } # Create a plot from an array of result files @@ -12,15 +12,14 @@ plot @filename(png); } -int target_innovation_start=1; -int target_innovation_stop=12000; -int step=1000; -int time_limit=86300; +int target_innovation_start=@toInt(@arg("start")); +int target_innovation_stop=@toInt(@arg("stop")); +int step=@toInt(@arg("step")); +int time_limit=@toInt(@arg("limit", "86300")); +string scicoldir=@arg("scicoldir"); # Run benchmark -#file graphs[] ; file graph <"movie-graph.txt">; - file timeresults[]; file memresults[]; Added: SwiftApps/SciColSim/benchmarks/conf/raven.cf =================================================================== --- SwiftApps/SciColSim/benchmarks/conf/raven.cf (rev 0) +++ SwiftApps/SciColSim/benchmarks/conf/raven.cf 2012-11-19 20:16:26 UTC (rev 6062) @@ -0,0 +1,10 @@ +wrapperlog.always.transfer=true +sitedir.keep=true +execution.retries=5 +lazy.errors=true +status.mode=provider +use.provider.staging=false +provider.staging.pin.swiftfiles=false + +#app raven benchmark=$PWD/benchmark.sh +#app localhost plot=$PWD/plot.sh Added: SwiftApps/SciColSim/benchmarks/conf/raven.xml =================================================================== --- SwiftApps/SciColSim/benchmarks/conf/raven.xml (rev 0) +++ SwiftApps/SciColSim/benchmarks/conf/raven.xml 2012-11-19 20:16:26 UTC (rev 6062) @@ -0,0 +1,24 @@ + + + + + CI-SES000031 + 1 + pbs.aprun;pbs.mpp;depth=24 + 24:00:00 + long + 60.99 + 10000 + + _WORK_ + + + + + 0.09 + 10000 + + _WORK_ + + + Modified: SwiftApps/SciColSim/benchmarks/start-benchmark.sh =================================================================== --- SwiftApps/SciColSim/benchmarks/start-benchmark.sh 2012-11-19 19:26:06 UTC (rev 6061) +++ SwiftApps/SciColSim/benchmarks/start-benchmark.sh 2012-11-19 20:16:26 UTC (rev 6062) @@ -1,22 +1,40 @@ #!/bin/bash +# Report error and exit +crash() +{ + MSG=$1 + echo ${MSG} >&2 + exit 1 +} + +# Provide usage info +usage="Usage: $0 -site sitename -start startval -stop stopval -step stepval [-limit limitval]" + # Process command line arguments while [ $# -gt 0 ]; do case $1 in - -p) paramfile=$2; shift 2;; - -s) execsite=$2; shift 2;; + -site) execsite=$2; shift 2;; + -start) start=$2; shift 2;; + -stop) stop=$2; shift 2;; + -step) step=$2; shift 2;; + -limit) limit=$2; shift 2;; *) echo $usage 1>&2 exit 1;; esac done +if [ -z "$execsite" ] || [ -z "$start" ] || [ -z "$stop" ] || [ -z "$step" ]; then + crash "$usage" +fi + + # Create run directory rundir=$( echo run??? | sed -e 's/^.*run//' | awk '{ printf("run%03d\n", $1+1)}' ) if [ -d $rundir ]; then - echo "$rundir already exists! exiting." >&2 - exit 2 + crash "$rundir already exists! exiting." >&2 else mkdir $rundir fi @@ -38,4 +56,9 @@ fi # Run swift -swift -sites.file sites.xml -tc.file tc.data benchmark.swift +scicoldir=$PWD/.. +command="swift -sites.file sites.xml -tc.file tc.data benchmark.swift -scicoldir=$scicoldir -start=$start -stop=$stop -step=$step" +if [ -n "$limit" ]; then + command="$command -limit=$limit" +fi +$( echo $command ) From davidk at ci.uchicago.edu Mon Nov 19 16:05:06 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Mon, 19 Nov 2012 16:05:06 -0600 (CST) Subject: [Swift-commit] r6063 - SwiftApps/SciColSim/benchmarks Message-ID: <20121119220507.00B489CD0B@svn.ci.uchicago.edu> Author: davidk Date: 2012-11-19 16:05:06 -0600 (Mon, 19 Nov 2012) New Revision: 6063 Modified: SwiftApps/SciColSim/benchmarks/benchmark.sh SwiftApps/SciColSim/benchmarks/benchmark.swift Log: Fix parameter mismatch Modified: SwiftApps/SciColSim/benchmarks/benchmark.sh =================================================================== --- SwiftApps/SciColSim/benchmarks/benchmark.sh 2012-11-19 20:16:26 UTC (rev 6062) +++ SwiftApps/SciColSim/benchmarks/benchmark.sh 2012-11-19 22:05:06 UTC (rev 6063) @@ -5,7 +5,7 @@ TIME_OUTPUT=$ORIGDIR/$3 MEM_OUTPUT=$ORIGDIR/$4 -cd $6 +cd $5 # Clean any previous run if [ -f "bestdb.txt" ]; then Modified: SwiftApps/SciColSim/benchmarks/benchmark.swift =================================================================== --- SwiftApps/SciColSim/benchmarks/benchmark.swift 2012-11-19 20:16:26 UTC (rev 6062) +++ SwiftApps/SciColSim/benchmarks/benchmark.swift 2012-11-19 22:05:06 UTC (rev 6063) @@ -27,7 +27,7 @@ file times ; file mem ; file error ; - (times, mem, error) = benchmark(ti, time_limit); + (times, mem, error) = benchmark(ti, time_limit, scicoldir); timeresults[tidx] = times; memresults[tidx] = mem; } From hategan at ci.uchicago.edu Tue Nov 20 04:20:34 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Tue, 20 Nov 2012 04:20:34 -0600 (CST) Subject: [Swift-commit] r6064 - trunk/etc Message-ID: <20121120102034.B8F229CCC8@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-20 04:19:49 -0600 (Tue, 20 Nov 2012) New Revision: 6064 Modified: trunk/etc/swift.properties Log: added tracing property to properties file Modified: trunk/etc/swift.properties =================================================================== --- trunk/etc/swift.properties 2012-11-19 22:05:06 UTC (rev 6063) +++ trunk/etc/swift.properties 2012-11-20 10:19:49 UTC (rev 6064) @@ -44,6 +44,16 @@ lazy.errors=false # +# Enables tracing of procedure invocations, assignments, +# iteration constructs, as well as certain dataflow events +# such as data intialization and waiting. This is done at +# a slight decrease in performance. Traces will be available +# in the log file. +# + +tracing.enabled=true + +# # What algorithm to use for caching of remote files. LRU (as in what # files to purge) is the only implementation right now. One can set # a target size (in bytes) for a host by using the swift:storagesize From hategan at ci.uchicago.edu Tue Nov 20 05:10:56 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Tue, 20 Nov 2012 05:10:56 -0600 (CST) Subject: [Swift-commit] r6065 - trunk/src/org/griphyn/vdl/mapping Message-ID: <20121120111056.B07969CCC8@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-20 05:10:54 -0600 (Tue, 20 Nov 2012) New Revision: 6065 Modified: trunk/src/org/griphyn/vdl/mapping/RootDataNode.java Log: changed duplicate mapping check; the previous scheme would only detect writes to files that already exist Modified: trunk/src/org/griphyn/vdl/mapping/RootDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/RootDataNode.java 2012-11-20 10:19:49 UTC (rev 6064) +++ trunk/src/org/griphyn/vdl/mapping/RootDataNode.java 2012-11-20 11:10:54 UTC (rev 6065) @@ -26,7 +26,6 @@ import org.globus.cog.karajan.workflow.futures.FutureListener; import org.globus.cog.karajan.workflow.futures.FutureNotYetAvailable; import org.griphyn.vdl.karajan.lib.Tracer; -import org.griphyn.vdl.karajan.lib.VDLFunction; import org.griphyn.vdl.type.Field; import org.griphyn.vdl.type.Type; @@ -123,8 +122,8 @@ DuplicateMappingChecker dmc) { String input = (String) params.get(MappingParam.SWIFT_INPUT); if (input != null && Boolean.valueOf(input.trim()).booleanValue()) { - addExisting(mapper, root, dmc); - checkConsistency(root); + addExisting(mapper, root); + checkConsistency(root, true, mapper, dmc); } else if (mapper.isStatic()) { if (root.isClosed()) { @@ -138,12 +137,10 @@ logger.debug("mapper: " + mapper); } for (Path p : mapper.existing()) { - PhysicalFormat f = mapper.map(p); try { // Try to get the path in order to check that the // path is valid - we'll get an exception if not DSHandle h = root.getField(p); - dmc.addWrite(f, h); if (tracer.isEnabled()) { tracer.trace(root.getThread(), root.getDeclarationLine(), root.getDisplayableName() + " MAPPING " + p + ", " + mapper.map(p)); @@ -159,18 +156,16 @@ if (root.isArray()) { root.closeArraySizes(); } - checkConsistency(root); + checkConsistency(root, false, mapper, dmc); } } - private static void addExisting(Mapper mapper, AbstractDataNode root, DuplicateMappingChecker dmc) { + private static void addExisting(Mapper mapper, AbstractDataNode root) { boolean any = false; for (Path p : mapper.existing()) { - PhysicalFormat f = mapper.map(p); try { DSHandle field = root.getField(p); field.closeShallow(); - dmc.addRead(f, field); if (tracer.isEnabled()) { tracer.trace(root.getThread(), root.getDeclarationLine(), root.getDisplayableName() + " MAPPING " + p + ", " + mapper.map(p)); @@ -189,12 +184,12 @@ } } - public static void checkConsistency(DSHandle handle) { + public static void checkConsistency(DSHandle handle, boolean input, Mapper mapper, DuplicateMappingChecker dmc) { if (handle.getType().isArray()) { // any number of indices is ok try { for (DSHandle dh : handle.getFields(Path.CHILDREN)) { - checkConsistency(dh); + checkConsistency(dh, input, mapper, dmc); } } catch (HandleOpenException e) { @@ -209,10 +204,20 @@ else { // all fields must be present Type type = handle.getType(); + if (!type.isPrimitive() && !type.isComposite()) { + // mapped. Feed the DMC. + PhysicalFormat f = mapper.map(handle.getPathFromRoot()); + if (input) { + dmc.addRead(f, handle); + } + else { + dmc.addWrite(f, handle); + } + } for (String fieldName : type.getFieldNames()) { Path fieldPath = Path.parse(fieldName); try { - checkConsistency(handle.getField(fieldPath)); + checkConsistency(handle.getField(fieldPath), input, mapper, dmc); } catch (InvalidPathException e) { throw new RuntimeException("Data set initialization failed for " + handle @@ -266,6 +271,9 @@ initialized = true; waitingMapperParam = null; if (tracer.isEnabled()) { + if ("sphOut".equals(getDisplayableName())) { + System.out.println(); + } tracer.trace(getThread(), getDeclarationLine(), getDisplayableName() + " INITIALIZED " + params); } } From davidk at ci.uchicago.edu Tue Nov 20 09:47:30 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Tue, 20 Nov 2012 09:47:30 -0600 (CST) Subject: [Swift-commit] r6066 - SwiftApps/SciColSim/benchmarks Message-ID: <20121120154730.582CE9CD05@svn.ci.uchicago.edu> Author: davidk Date: 2012-11-20 09:47:29 -0600 (Tue, 20 Nov 2012) New Revision: 6066 Added: SwiftApps/SciColSim/benchmarks/README Log: README for benchmark scripts Added: SwiftApps/SciColSim/benchmarks/README =================================================================== --- SwiftApps/SciColSim/benchmarks/README (rev 0) +++ SwiftApps/SciColSim/benchmarks/README 2012-11-20 15:47:29 UTC (rev 6066) @@ -0,0 +1,57 @@ +SciColSim Benchmark Scripts +=========================== + +Prerequisites +------------- +Please ensure that Swift is in your $PATH. This script +has been tested on the following machines: + +beagle +midway +pads +raven +uc3 + +How to Run +---------- +The start-benchmark.sh script is used to start a run. Each successful +invocation of this script will create a new run directory with the name +of run. + +The options you pass to this script will determine how the script is run. + +Usage: start-benchmark.sh -site sitename -start startval -stop stopval -step stepval [-limit limitval] + +Valid options for sitename are beagle, midway, pads, raven, and uc3. +The start value represents the minimum value for target innovation. +The stop value represents the maximum value for target innovation. +The step value determines the frequency of tasks to run between start and stop values +Limit represents a maximum time limit, in seconds. If the task exceeds this limit, the program will end and return a result of -1 + +Example: + +----- +./start-benchmark.sh -site beagle -start 1000 -stop 2000 -step 100 -limit 3600 +----- + +This will create 10 tasks on Beagle with target innovation values of 1000, 1100, +1200, 1300, 1400, 1500, 1600, 1700, 1800, 1900 and 2000. The tasks will run for a +maximum of 1 hour. + +Time logs +--------- +As tasks complete, a log file will be created for each task. The file name for these +time logs have the format of times/times-.out. The format of these files are: + + + +Errors +------ +The errors directory contains the standard error output of the benchmark.sh script. This +can be useful while debugging. + +Plot +---- +Plots get generated by a gnuplot script called plot.gp. The resulting plot file called be called +results/plot.png. + From davidk at ci.uchicago.edu Tue Nov 20 12:28:36 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Tue, 20 Nov 2012 12:28:36 -0600 (CST) Subject: [Swift-commit] r6067 - SwiftApps/SciColSim/benchmarks Message-ID: <20121120182836.4F3099CD05@svn.ci.uchicago.edu> Author: davidk Date: 2012-11-20 12:28:36 -0600 (Tue, 20 Nov 2012) New Revision: 6067 Modified: SwiftApps/SciColSim/benchmarks/start-benchmark.sh Log: Fix scicoldir path Modified: SwiftApps/SciColSim/benchmarks/start-benchmark.sh =================================================================== --- SwiftApps/SciColSim/benchmarks/start-benchmark.sh 2012-11-20 15:47:29 UTC (rev 6066) +++ SwiftApps/SciColSim/benchmarks/start-benchmark.sh 2012-11-20 18:28:36 UTC (rev 6067) @@ -56,7 +56,7 @@ fi # Run swift -scicoldir=$PWD/.. +scicoldir=$( dirname $( dirname $PWD ) ) command="swift -sites.file sites.xml -tc.file tc.data benchmark.swift -scicoldir=$scicoldir -start=$start -stop=$stop -step=$step" if [ -n "$limit" ]; then command="$command -limit=$limit" From hategan at ci.uchicago.edu Tue Nov 20 15:29:29 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Tue, 20 Nov 2012 15:29:29 -0600 (CST) Subject: [Swift-commit] r6068 - trunk/src/org/griphyn/vdl/karajan Message-ID: <20121120212929.4D42F9CD05@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-20 15:29:29 -0600 (Tue, 20 Nov 2012) New Revision: 6068 Modified: trunk/src/org/griphyn/vdl/karajan/Loader.java Log: print stack trace for NPEs when compiling Modified: trunk/src/org/griphyn/vdl/karajan/Loader.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/Loader.java 2012-11-20 18:28:36 UTC (rev 6067) +++ trunk/src/org/griphyn/vdl/karajan/Loader.java 2012-11-20 21:29:29 UTC (rev 6068) @@ -357,6 +357,9 @@ // if we leave a kml file around, then a subsequent // re-run will skip recompiling and cause a different // error message for the user + if (e instanceof NullPointerException) { + e.printStackTrace(); + } kml.delete(); throw new CompilationException( "Failed to convert .swiftx to .kml for " + project, e); From hategan at ci.uchicago.edu Tue Nov 20 16:11:26 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Tue, 20 Nov 2012 16:11:26 -0600 (CST) Subject: [Swift-commit] r6069 - trunk/src/org/griphyn/vdl/engine Message-ID: <20121120221126.E89339CCC8@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-20 16:11:26 -0600 (Tue, 20 Nov 2012) New Revision: 6069 Modified: trunk/src/org/griphyn/vdl/engine/CompilerUtils.java trunk/src/org/griphyn/vdl/engine/Karajan.java trunk/src/org/griphyn/vdl/engine/VariableScope.java Log: fixed NPE with shadowing warnings and improved warning messages to distinguish between "variable", "parameter" etc. Modified: trunk/src/org/griphyn/vdl/engine/CompilerUtils.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/CompilerUtils.java 2012-11-20 21:29:29 UTC (rev 6068) +++ trunk/src/org/griphyn/vdl/engine/CompilerUtils.java 2012-11-20 22:11:26 UTC (rev 6069) @@ -22,7 +22,13 @@ return getLine(n.getParentNode()); } else { - return src.getNodeValue(); + String loc = src.getNodeValue(); + if (loc == null || loc.length() == 0) { + return "unknown"; + } + else { + return loc.substring(loc.indexOf(' ') + 1); + } } } @@ -30,12 +36,6 @@ if (src == null) { return null; } - String loc = src.getDomNode().getAttributes().getNamedItem("src").getNodeValue(); - if (loc == null) { - return null; - } - else { - return loc.substring(loc.indexOf(' ') + 1); - } + return getLine(src.getDomNode()); } } Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/Karajan.java 2012-11-20 21:29:29 UTC (rev 6068) +++ trunk/src/org/griphyn/vdl/engine/Karajan.java 2012-11-20 22:11:26 UTC (rev 6069) @@ -355,7 +355,7 @@ procST.setAttribute("arguments", paramST); String type = normalize(param.getType().getLocalPart()); checkIsTypeDefined(type); - innerScope.addVariable(param.getName(), type, param); + innerScope.addVariable(param.getName(), type, "Return parameter", param); } for (int i = 0; i < proc.sizeOfInputArray(); i++) { FormalParameter param = proc.getInputArray(i); @@ -367,7 +367,7 @@ procST.setAttribute("arguments", paramST); String type = normalize(param.getType().getLocalPart()); checkIsTypeDefined(type); - outerScope.addVariable(param.getName(), type, param); + outerScope.addVariable(param.getName(), type, "Parameter", param); } Binding bind; @@ -441,9 +441,8 @@ } public void variableForSymbol(Variable var, VariableScope scope) throws CompilationException { - checkIsTypeDefined(var.getType().getLocalPart()); - scope.addVariable(var.getName(), var.getType().getLocalPart(), var.getIsGlobal(), var); + scope.addVariable(var.getName(), var.getType().getLocalPart(), "Variable", var.getIsGlobal(), var); } public void variable(Variable var, VariableScope scope) throws CompilationException { @@ -498,7 +497,7 @@ scope.bodyTemplate.setAttribute("declarations",variableDeclarationST); StringTemplate paramValueST=expressionToKarajan(param.getAbstractExpression(),scope); String paramValueType = datatype(paramValueST); - scope.addVariable(parameterVariableName, paramValueType, param); + scope.addVariable(parameterVariableName, paramValueType, "Variable", param); variableDeclarationST.setAttribute("type", paramValueType); StringTemplate variableReferenceST = template("id"); variableReferenceST.setAttribute("var",parameterVariableName); @@ -898,7 +897,7 @@ VariableScope loopScope = new VariableScope(this, scope, VariableScope.ENCLOSURE_LOOP); VariableScope innerScope = new VariableScope(this, loopScope, VariableScope.ENCLOSURE_LOOP); - loopScope.addVariable(iterate.getVar(), "int", iterate); + loopScope.addVariable(iterate.getVar(), "int", "Iteration variable", iterate); StringTemplate iterateST = template("iterate"); iterateST.setAttribute("line", getLine(iterate)); @@ -938,11 +937,11 @@ if (itemType == null) { throw new CompilationException("You can iterate through an array structure only"); } - innerScope.addVariable(foreach.getVar(), itemType, foreach); + innerScope.addVariable(foreach.getVar(), itemType, "Iteration variable", foreach); foreachST.setAttribute("indexVar", foreach.getIndexVar()); foreachST.setAttribute("indexVarType", keyType); if(foreach.getIndexVar() != null) { - innerScope.addVariable(foreach.getIndexVar(), keyType, foreach); + innerScope.addVariable(foreach.getIndexVar(), keyType, "Iteration variable", foreach); } innerScope.bodyTemplate = foreachST; Modified: trunk/src/org/griphyn/vdl/engine/VariableScope.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/VariableScope.java 2012-11-20 21:29:29 UTC (rev 6068) +++ trunk/src/org/griphyn/vdl/engine/VariableScope.java 2012-11-20 22:11:26 UTC (rev 6069) @@ -120,15 +120,15 @@ declaration already exists. Perhaps error in same scope and warning if it shadows an outer scope? */ - public void addVariable(String name, String type, XmlObject src) throws CompilationException { - addVariable(name, type, false, src); + public void addVariable(String name, String type, String context, XmlObject src) throws CompilationException { + addVariable(name, type, context, false, src); } public void inhibitClosing(String name) { inhibitClosing.add(name); } - public void addVariable(String name, String type, boolean global, XmlObject src) throws CompilationException { + public void addVariable(String name, String type, String context, boolean global, XmlObject src) throws CompilationException { if (logger.isDebugEnabled()) { logger.debug("Adding variable " + name + " of type " + type + " to scope " + hashCode()); } @@ -142,7 +142,7 @@ // by the above if? in which case isVariableDefined should // be replaced by is locally defined test. if(parentScope != null && parentScope.isVariableDefined(name)) { - Warnings.warn("Variable " + name + ", defined on line " + CompilerUtils.getLine(src) + Warnings.warn(context + " " + name + ", on line " + CompilerUtils.getLine(src) + ", shadows variable of same name on line " + parentScope.getDefinitionLine(name)); } From davidk at ci.uchicago.edu Wed Nov 21 11:01:03 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Wed, 21 Nov 2012 11:01:03 -0600 (CST) Subject: [Swift-commit] r6070 - branches Message-ID: <20121121170103.E212B9CFD6@svn.ci.uchicago.edu> Author: davidk Date: 2012-11-21 11:01:03 -0600 (Wed, 21 Nov 2012) New Revision: 6070 Removed: branches/release-0.94/ Log: From davidk at ci.uchicago.edu Wed Nov 21 11:02:39 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Wed, 21 Nov 2012 11:02:39 -0600 (CST) Subject: [Swift-commit] r6071 - branches Message-ID: <20121121170239.1676C9CFD6@svn.ci.uchicago.edu> Author: davidk Date: 2012-11-21 11:02:38 -0600 (Wed, 21 Nov 2012) New Revision: 6071 Added: branches/release-0.94/ Log: Copying trunk to 0.94 From davidk at ci.uchicago.edu Wed Nov 21 14:59:44 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Wed, 21 Nov 2012 14:59:44 -0600 (CST) Subject: [Swift-commit] r6072 - in trunk/tests: . release release/Lingua release/Lingua/EN release/Lingua/EN/Numbers Message-ID: <20121121205944.076609CCC8@svn.ci.uchicago.edu> Author: davidk Date: 2012-11-21 14:59:43 -0600 (Wed, 21 Nov 2012) New Revision: 6072 Added: trunk/tests/release/ trunk/tests/release/Lingua/ trunk/tests/release/Lingua/EN/ trunk/tests/release/Lingua/EN/Numbers/ trunk/tests/release/Lingua/EN/Numbers/Ordinate.pm trunk/tests/release/nightly-loop.sh trunk/tests/release/report.pl trunk/tests/release/style1col.css Log: Start of a release test group Added: trunk/tests/release/Lingua/EN/Numbers/Ordinate.pm =================================================================== --- trunk/tests/release/Lingua/EN/Numbers/Ordinate.pm (rev 0) +++ trunk/tests/release/Lingua/EN/Numbers/Ordinate.pm 2012-11-21 20:59:43 UTC (rev 6072) @@ -0,0 +1,175 @@ + +require 5; +package Lingua::EN::Numbers::Ordinate; +use strict; +# Time-stamp: "2004-12-29 19:06:20 AST" +use vars qw(@ISA @EXPORT @EXPORT_OK $VERSION); +require Exporter; + at ISA = ('Exporter'); + at EXPORT = ('ordinate'); + at EXPORT_OK = ('ordsuf', 'th'); +$VERSION = "1.02"; + +########################################################################### + +=head1 NAME + +Lingua::EN::Numbers::Ordinate -- go from cardinal number (3) to ordinal ("3rd") + +=head1 SYNOPSIS + + use Lingua::EN::Numbers::Ordinate; + print ordinate(4), "\n"; + # prints 4th + print ordinate(-342), "\n"; + # prints -342nd + + # Example of actual use: + ... + for(my $i = 0; $i < @records; $i++) { + unless(is_valid($record[$i]) { + warn "The ", ordinate($i), " record is invalid!\n"; + next; + } + ... + } + +=head1 DESCRIPTION + +There are two kinds of numbers in English -- cardinals (1, 2, 3...), and +ordinals (1st, 2nd, 3rd...). This library provides functions for giving +the ordinal form of a number, given its cardinal value. + +=head1 FUNCTIONS + +=over + +=item ordinate(SCALAR) + +Returns a string consisting of that scalar's string form, plus the +appropriate ordinal suffix. Example: C returns "23rd". + +As a special case, C and C return "0th", +not "th". + +This function is exported by default. + +=item th(SCALAR) + +Merely an alias for C, but not exported by default. + +=item ordsuf(SCALAR) + +Returns just the appropriate ordinal suffix for the given scalar +numeric value. This is what C uses to actually do its +work. For example, C is "rd". + +Not exported by default. + +=back + +The above functions are all prototyped to take a scalar value, +so C is the same as C. + +=head1 CAVEATS + +* Note that this library knows only about numbers, not number-words. +C might just as well be C +or C -- you'll get the fallthru case of the input +string plus "th". + +* As is unavoidable, C returns "174th" (because ordinate +sees the value 174). Similarly, C returns +"1000000000000th". Returning "trillionth" would be nice, but that's an +awfully atypical case. + +* Note that this library's algorithm (as well as the basic concept +and implementation of ordinal numbers) is totally language specific. + +To pick a trivial example, consider that in French, 1 ordinates +as "1ier", whereas 41 ordinates as "41ieme". + +=head1 STILL NOT SATISFIED? + +Bored of this...? + + use Lingua::EN::Numbers::Ordinate qw(ordinate th); + ... + print th($n), " entry processed...\n"; + ... + +Try this bit of lunacy: + + { + my $th_object; + sub _th () { $th_object } + + package Lingua::EN::Numbers::Ordinate::Overloader; + my $x; # Gotta have something to bless. + $th_object = bless \$x; # Define the object now, which _th returns + use Carp (); + use Lingua::EN::Numbers::Ordinate (); + sub overordinate { + Carp::croak "_th should be used only as postfix!" unless $_[2]; + Lingua::EN::Numbers::Ordinate::ordinate($_[1]); + } + use overload '&' => \&overordinate; + } + +Then you get to do: + + print 3 & _th, "\n"; + # prints "3rd" + + print 1 + 2 & _th, "\n"; + # prints "3rd" too! + # Because of the precedence of & ! + + print _th & 3, "\n"; + # dies with: "th should be used only as postfix!" + +Kooky, isn't it? For more delightful deleria like this, see +Damian Conway's I from Manning Press. + +Kinda makes you like C, doesn't it? + +=head1 COPYRIGHT + +Copyright (c) 2000 Sean M. Burke. All rights reserved. + +This library is free software; you can redistribute it and/or +modify it under the same terms as Perl itself. + +=head1 AUTHOR + +Sean M. Burke C + +=cut + +########################################################################### + +sub ordsuf ($) { + return 'th' if not(defined($_[0])) or not( 0 + $_[0] ); + # 'th' for undef, 0, or anything non-number. + my $n = abs($_[0]); # Throw away the sign. + return 'th' unless $n == int($n); # Best possible, I guess. + $n %= 100; + return 'th' if $n == 11 or $n == 12 or $n == 13; + $n %= 10; + return 'st' if $n == 1; + return 'nd' if $n == 2; + return 'rd' if $n == 3; + return 'th'; +} + +sub ordinate ($) { + my $i = $_[0] || 0; + return $i . ordsuf($i); +} + +*th = \&ordinate; # correctly copies the prototype, too. + +########################################################################### +1; + +__END__ Added: trunk/tests/release/nightly-loop.sh =================================================================== --- trunk/tests/release/nightly-loop.sh (rev 0) +++ trunk/tests/release/nightly-loop.sh 2012-11-21 20:59:43 UTC (rev 6072) @@ -0,0 +1,23 @@ +#!/bin/bash + +numtests=$1 + +if [ -z "$numtests" ]; then + echo "Usage: $0 " + exit 1 +fi + +origdir=$PWD +basedir=$( cd ../../../../..; echo $PWD; cd $origdir ) +testdir=$( cd ..; echo $PWD; cd $origdir ) +cd $testdir + +for i in $( seq -w 001 $numtests ); do + ./suite.sh -t -o $basedir groups/group-all-local.sh + outputdir=$( ls -1rtd $basedir/run-????-??-?? | tail -1 ) + mv $outputdir $outputdir.loop.$i +done + +cd $origdir +./report.pl $basedir > $basedir/release.html +echo Results can be found at $basedir/release.html Property changes on: trunk/tests/release/nightly-loop.sh ___________________________________________________________________ Added: svn:executable + * Added: trunk/tests/release/report.pl =================================================================== --- trunk/tests/release/report.pl (rev 0) +++ trunk/tests/release/report.pl 2012-11-21 20:59:43 UTC (rev 6072) @@ -0,0 +1,38 @@ +#!/usr/bin/perl +use strict; +use Lingua::EN::Numbers::Ordinate; +use File::Basename; + +my $basedir=shift; + +print <<'END'; +Content-type: text/html + + + +Swift Release Testing + + + +END + +my @table_data=(); +my $count=1; +my @months = ("Zero", "January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December"); + +foreach my $test_result(<$basedir/run*/*.html>) { + my $datestring = basename($test_result, ".html"); + my $iteration = (split('\.', basename(dirname($test_result))))[-1]; + (my $ignore, my $year, my $month, my $day) = split('-', $datestring); + $day =~ s/^0*//; + $table_data[$count] .= "$months[$month] " . ordinate($day) . ", $year - iteration $iteration"; + $count++; +} + +# Print table +print "
\n"; +print "$table_data[0]\n"; +foreach my $td(@table_data[1..$#table_data]) { + print "$td\n"; +} +print "
\n"; Property changes on: trunk/tests/release/report.pl ___________________________________________________________________ Added: svn:executable + * Added: trunk/tests/release/style1col.css =================================================================== --- trunk/tests/release/style1col.css (rev 0) +++ trunk/tests/release/style1col.css 2012-11-21 20:59:43 UTC (rev 6072) @@ -0,0 +1,310 @@ +#header { + background-color: #172B17; + /*background-image: url(../images/graphics/h_backgrnd.gif);*/ + background-image: url(../images/graphics/h_backgrnd2.gif); + background-repeat: repeat-x; + width: 100%; + background-position: center; +} +body { + margin: 0px auto; + font-family: Verdana, Arial, Helvetica, sans-serif; + background: #fff no-repeat center top; +} + +body#home a.home, +body#downloads a.downloads, +body#about a.about, +body#docs a.docs, +body#apps a.apps, +body#support a.support{ + background-color: #666; + color:#FFFFFF; +} + + +a:link { + font-family: Verdana, Arial, Helvetica, sans-serif; + font-size: 12px; + font-weight: normal; + color: #3366cc; + text-decoration: none; +} +a:visited { + font-family: Verdana, Arial, Helvetica, sans-serif; + font-size: 12px; + font-weight: normal; + color: #3366cc; + text-decoration: none; +} +a:hover { + font-family: Verdana, Arial, Helvetica, sans-serif; + font-size: 12px; + font-weight: normal; + color: #0099cc; + text-decoration: underline; +} + +pre, pre *, div.dp-highlighter * { + font-family: Consolas, Andale Mono, monospaced; +} + +/************* MENU ************/ + +#nav { +font-family: Arial, Helvetica, sans-serif; + width: 100%; + font-size: 12px; + font-weight: normal; + margin: 0px auto; + padding: 0px 0px 0px 0px; + background-color: #000000; + border-bottom: #ccc 2px solid; +} + +#nav ul { +width: 100%s; + margin: 0px; + padding: 0px; + list-style: none; + text-align: center; + letter-spacing: 1px; +} + +#nav li { + display: inline; +} + +#nav a { +font-size: 12px; + padding-right: 6px; + padding-left: 6px; + text-transform: uppercase; + text-decoration: none; + color: #FFFFFF; +} + +#nav a:hover { + background-color: #666; +} +#page { + width: 850px; + margin: 0px 0px 40px 0px; + padding: 20px 0px 0px 0px; + border-top: thin #000 solid; + color: #333; +} +.image { + background-color: #eee; + margin:10px auto; + padding:3px; + border: 1px solid #ccc; + width: 419px; + +} +.highlight{ + border: 1px #ccc solid; + padding:5px; + background-color: #FCFAFA; + font-size: 11px; + margin-left:10px; +} + +pre.programlisting { + padding: 10pt; + background: #FCFAFA; + border: 1px #ccc solid; + white-space: normal; +} + +pre.shell, pre.screen { + padding: 10pt; + background: #FCFAFA; + border: 1px #ccc solid; + white-space: pre; +} + +.mailinglists { width: 85%; margin: 0 0 0 35px; font-size:10px; } +.mailinglists td { padding: 5px 5px; line-height: 16px;} +.mailinglists th { font-size: 12px; text-align: left; color:#000; padding: 7px; letter-spacing: 1px; font-family:"Trebuchet MS", Arial, sans-serif;} + +#content { + float: left; + width: 100%; + padding: 0px 10px 70px 5px; + margin-bottom: 20px; + margin-top: 10px; +} + + +#content p{ + font-family: Verdana, Arial, Helvetica, sans-serif; + font-size: 12px; + line-height: 17px; + color: #333333; + padding: 5px 30px 0px 30px; + + margin: 0px 0px 8px 0px; +} + +#content ul{ + font-family: Verdana, Arial, Helvetica, sans-serif; + font-size: 12px; + line-height: 17px; +} + +#content li{ + font-family: Verdana, Arial, Helvetica, sans-serif; + font-size: 12px; + line-height: 17px; + list-style:circle; + margin: 10px 10px 10px 35px; +} + +h1 { + margin: 0px; + font-size: 16px; + font-weight: bold; + color: #172B17; + letter-spacing: 1px; + padding-top: 0px; + padding-right: 5px; + padding-bottom: 5px; + padding-left: 10px; + font-family:"Trebuchet MS", Arial, sans-serif; + +} +h2 { + margin: 0px; + font-size: 14px; + font-weight: bold; + color: #CC0000; + letter-spacing: 1px; + padding-top: 5px; + padding-right: 5px; + padding-bottom: 5px; + padding-left: 20px; + font-family:"Trebuchet MS", Arial, sans-serif; + +} +h3 { + margin: 0px; + font-size: 13px; + font-weight: bold; + color: #333; + letter-spacing: 1px; + padding-top: 5px; + padding-right: 5px; + padding-bottom: 5px; + padding-left: 20px; + font-family:"Trebuchet MS", Arial, sans-serif; + +} +h4 { + margin: 0px; + font-size: 12px; + font-weight: bold; + color: #666; + letter-spacing: 1px; + padding-top: 5px; + padding-right: 5px; + padding-bottom: 5px; + padding-left: 20px; + font-family:"Trebuchet MS", Arial, sans-serif; + text-transform: uppercase; + +} +h5 { + margin: 0px; + font-size: 12px; + font-weight: bold; + color: #666; + letter-spacing: 1px; + padding-top: 5px; + padding-right: 5px; + padding-bottom: 5px; + padding-left: 20px; + font-family:"Trebuchet MS", Arial, sans-serif; + text-transform: uppercase; + +} +#container { + background-color: #FFFFFF; + font-size: 10px; + font-weight: normal; + border: 1px #ccc solid; + width: 850px; + background-repeat: no-repeat; + background-position: center; + margin-top: 3px; + margin-right: auto; + margin-bottom: 0px; + margin-left: auto; +} +/************* FOOTER ************/ + +#footer { + clear: both; + width: 100%; + margin: 0px; + border-top: 1px solid #ddd; + border-bottom: 25px #FFF solid; + background-color: #FFF; + text-align: center; + padding: 0px; + color: #999; +} + +#footer a:link { + font-size: 9px; + font-weight: normal; + color: #3366cc; + text-decoration: none; +} +#footer a:visited { + font-size: 9px; + color: #3366cc; + text-decoration: none; +} + #footer a:hover { + font-size: 9px; + color: #0099cc; + text-decoration: underline; +} + +/********* DOCBOOK STUFF ********/ + +.article { + margin: 3px 20px; + font-size: 10px; +} + +.article p { + margin: 2px 25px; +} + +/* Syntax highlighting stuff */ + +div.dp-highlighter { + background: #FCFAFA; + border: 1px #ccc solid; + white-space: normal; + margin: 10px; +} + +span.comment { + color: #505050; +} + +span.keyword { + color: #202080; + font-weight: bold; +} + +span.string { + color: #209020; +} + +span.function { + color: #800040; +} From davidk at ci.uchicago.edu Wed Nov 21 15:01:21 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Wed, 21 Nov 2012 15:01:21 -0600 (CST) Subject: [Swift-commit] r6073 - in branches/release-0.94/tests: . release release/Lingua release/Lingua/EN release/Lingua/EN/Numbers Message-ID: <20121121210121.449359CCC8@svn.ci.uchicago.edu> Author: davidk Date: 2012-11-21 15:01:21 -0600 (Wed, 21 Nov 2012) New Revision: 6073 Added: branches/release-0.94/tests/release/ branches/release-0.94/tests/release/Lingua/ branches/release-0.94/tests/release/Lingua/EN/ branches/release-0.94/tests/release/Lingua/EN/Numbers/ branches/release-0.94/tests/release/Lingua/EN/Numbers/Ordinate.pm branches/release-0.94/tests/release/nightly-loop.sh branches/release-0.94/tests/release/report.pl branches/release-0.94/tests/release/style1col.css Log: Start of a release test group Added: branches/release-0.94/tests/release/Lingua/EN/Numbers/Ordinate.pm =================================================================== --- branches/release-0.94/tests/release/Lingua/EN/Numbers/Ordinate.pm (rev 0) +++ branches/release-0.94/tests/release/Lingua/EN/Numbers/Ordinate.pm 2012-11-21 21:01:21 UTC (rev 6073) @@ -0,0 +1,175 @@ + +require 5; +package Lingua::EN::Numbers::Ordinate; +use strict; +# Time-stamp: "2004-12-29 19:06:20 AST" +use vars qw(@ISA @EXPORT @EXPORT_OK $VERSION); +require Exporter; + at ISA = ('Exporter'); + at EXPORT = ('ordinate'); + at EXPORT_OK = ('ordsuf', 'th'); +$VERSION = "1.02"; + +########################################################################### + +=head1 NAME + +Lingua::EN::Numbers::Ordinate -- go from cardinal number (3) to ordinal ("3rd") + +=head1 SYNOPSIS + + use Lingua::EN::Numbers::Ordinate; + print ordinate(4), "\n"; + # prints 4th + print ordinate(-342), "\n"; + # prints -342nd + + # Example of actual use: + ... + for(my $i = 0; $i < @records; $i++) { + unless(is_valid($record[$i]) { + warn "The ", ordinate($i), " record is invalid!\n"; + next; + } + ... + } + +=head1 DESCRIPTION + +There are two kinds of numbers in English -- cardinals (1, 2, 3...), and +ordinals (1st, 2nd, 3rd...). This library provides functions for giving +the ordinal form of a number, given its cardinal value. + +=head1 FUNCTIONS + +=over + +=item ordinate(SCALAR) + +Returns a string consisting of that scalar's string form, plus the +appropriate ordinal suffix. Example: C returns "23rd". + +As a special case, C and C return "0th", +not "th". + +This function is exported by default. + +=item th(SCALAR) + +Merely an alias for C, but not exported by default. + +=item ordsuf(SCALAR) + +Returns just the appropriate ordinal suffix for the given scalar +numeric value. This is what C uses to actually do its +work. For example, C is "rd". + +Not exported by default. + +=back + +The above functions are all prototyped to take a scalar value, +so C is the same as C. + +=head1 CAVEATS + +* Note that this library knows only about numbers, not number-words. +C might just as well be C +or C -- you'll get the fallthru case of the input +string plus "th". + +* As is unavoidable, C returns "174th" (because ordinate +sees the value 174). Similarly, C returns +"1000000000000th". Returning "trillionth" would be nice, but that's an +awfully atypical case. + +* Note that this library's algorithm (as well as the basic concept +and implementation of ordinal numbers) is totally language specific. + +To pick a trivial example, consider that in French, 1 ordinates +as "1ier", whereas 41 ordinates as "41ieme". + +=head1 STILL NOT SATISFIED? + +Bored of this...? + + use Lingua::EN::Numbers::Ordinate qw(ordinate th); + ... + print th($n), " entry processed...\n"; + ... + +Try this bit of lunacy: + + { + my $th_object; + sub _th () { $th_object } + + package Lingua::EN::Numbers::Ordinate::Overloader; + my $x; # Gotta have something to bless. + $th_object = bless \$x; # Define the object now, which _th returns + use Carp (); + use Lingua::EN::Numbers::Ordinate (); + sub overordinate { + Carp::croak "_th should be used only as postfix!" unless $_[2]; + Lingua::EN::Numbers::Ordinate::ordinate($_[1]); + } + use overload '&' => \&overordinate; + } + +Then you get to do: + + print 3 & _th, "\n"; + # prints "3rd" + + print 1 + 2 & _th, "\n"; + # prints "3rd" too! + # Because of the precedence of & ! + + print _th & 3, "\n"; + # dies with: "th should be used only as postfix!" + +Kooky, isn't it? For more delightful deleria like this, see +Damian Conway's I from Manning Press. + +Kinda makes you like C, doesn't it? + +=head1 COPYRIGHT + +Copyright (c) 2000 Sean M. Burke. All rights reserved. + +This library is free software; you can redistribute it and/or +modify it under the same terms as Perl itself. + +=head1 AUTHOR + +Sean M. Burke C + +=cut + +########################################################################### + +sub ordsuf ($) { + return 'th' if not(defined($_[0])) or not( 0 + $_[0] ); + # 'th' for undef, 0, or anything non-number. + my $n = abs($_[0]); # Throw away the sign. + return 'th' unless $n == int($n); # Best possible, I guess. + $n %= 100; + return 'th' if $n == 11 or $n == 12 or $n == 13; + $n %= 10; + return 'st' if $n == 1; + return 'nd' if $n == 2; + return 'rd' if $n == 3; + return 'th'; +} + +sub ordinate ($) { + my $i = $_[0] || 0; + return $i . ordsuf($i); +} + +*th = \&ordinate; # correctly copies the prototype, too. + +########################################################################### +1; + +__END__ Added: branches/release-0.94/tests/release/nightly-loop.sh =================================================================== --- branches/release-0.94/tests/release/nightly-loop.sh (rev 0) +++ branches/release-0.94/tests/release/nightly-loop.sh 2012-11-21 21:01:21 UTC (rev 6073) @@ -0,0 +1,23 @@ +#!/bin/bash + +numtests=$1 + +if [ -z "$numtests" ]; then + echo "Usage: $0 " + exit 1 +fi + +origdir=$PWD +basedir=$( cd ../../../../..; echo $PWD; cd $origdir ) +testdir=$( cd ..; echo $PWD; cd $origdir ) +cd $testdir + +for i in $( seq -w 001 $numtests ); do + ./suite.sh -t -o $basedir groups/group-all-local.sh + outputdir=$( ls -1rtd $basedir/run-????-??-?? | tail -1 ) + mv $outputdir $outputdir.loop.$i +done + +cd $origdir +./report.pl $basedir > $basedir/release.html +echo Results can be found at $basedir/release.html Property changes on: branches/release-0.94/tests/release/nightly-loop.sh ___________________________________________________________________ Added: svn:executable + * Added: branches/release-0.94/tests/release/report.pl =================================================================== --- branches/release-0.94/tests/release/report.pl (rev 0) +++ branches/release-0.94/tests/release/report.pl 2012-11-21 21:01:21 UTC (rev 6073) @@ -0,0 +1,38 @@ +#!/usr/bin/perl +use strict; +use Lingua::EN::Numbers::Ordinate; +use File::Basename; + +my $basedir=shift; + +print <<'END'; +Content-type: text/html + + + +Swift Release Testing + + + +END + +my @table_data=(); +my $count=1; +my @months = ("Zero", "January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December"); + +foreach my $test_result(<$basedir/run*/*.html>) { + my $datestring = basename($test_result, ".html"); + my $iteration = (split('\.', basename(dirname($test_result))))[-1]; + (my $ignore, my $year, my $month, my $day) = split('-', $datestring); + $day =~ s/^0*//; + $table_data[$count] .= "$months[$month] " . ordinate($day) . ", $year - iteration $iteration"; + $count++; +} + +# Print table +print "
\n"; +print "$table_data[0]\n"; +foreach my $td(@table_data[1..$#table_data]) { + print "$td\n"; +} +print "
\n"; Property changes on: branches/release-0.94/tests/release/report.pl ___________________________________________________________________ Added: svn:executable + * Added: branches/release-0.94/tests/release/style1col.css =================================================================== --- branches/release-0.94/tests/release/style1col.css (rev 0) +++ branches/release-0.94/tests/release/style1col.css 2012-11-21 21:01:21 UTC (rev 6073) @@ -0,0 +1,310 @@ +#header { + background-color: #172B17; + /*background-image: url(../images/graphics/h_backgrnd.gif);*/ + background-image: url(../images/graphics/h_backgrnd2.gif); + background-repeat: repeat-x; + width: 100%; + background-position: center; +} +body { + margin: 0px auto; + font-family: Verdana, Arial, Helvetica, sans-serif; + background: #fff no-repeat center top; +} + +body#home a.home, +body#downloads a.downloads, +body#about a.about, +body#docs a.docs, +body#apps a.apps, +body#support a.support{ + background-color: #666; + color:#FFFFFF; +} + + +a:link { + font-family: Verdana, Arial, Helvetica, sans-serif; + font-size: 12px; + font-weight: normal; + color: #3366cc; + text-decoration: none; +} +a:visited { + font-family: Verdana, Arial, Helvetica, sans-serif; + font-size: 12px; + font-weight: normal; + color: #3366cc; + text-decoration: none; +} +a:hover { + font-family: Verdana, Arial, Helvetica, sans-serif; + font-size: 12px; + font-weight: normal; + color: #0099cc; + text-decoration: underline; +} + +pre, pre *, div.dp-highlighter * { + font-family: Consolas, Andale Mono, monospaced; +} + +/************* MENU ************/ + +#nav { +font-family: Arial, Helvetica, sans-serif; + width: 100%; + font-size: 12px; + font-weight: normal; + margin: 0px auto; + padding: 0px 0px 0px 0px; + background-color: #000000; + border-bottom: #ccc 2px solid; +} + +#nav ul { +width: 100%s; + margin: 0px; + padding: 0px; + list-style: none; + text-align: center; + letter-spacing: 1px; +} + +#nav li { + display: inline; +} + +#nav a { +font-size: 12px; + padding-right: 6px; + padding-left: 6px; + text-transform: uppercase; + text-decoration: none; + color: #FFFFFF; +} + +#nav a:hover { + background-color: #666; +} +#page { + width: 850px; + margin: 0px 0px 40px 0px; + padding: 20px 0px 0px 0px; + border-top: thin #000 solid; + color: #333; +} +.image { + background-color: #eee; + margin:10px auto; + padding:3px; + border: 1px solid #ccc; + width: 419px; + +} +.highlight{ + border: 1px #ccc solid; + padding:5px; + background-color: #FCFAFA; + font-size: 11px; + margin-left:10px; +} + +pre.programlisting { + padding: 10pt; + background: #FCFAFA; + border: 1px #ccc solid; + white-space: normal; +} + +pre.shell, pre.screen { + padding: 10pt; + background: #FCFAFA; + border: 1px #ccc solid; + white-space: pre; +} + +.mailinglists { width: 85%; margin: 0 0 0 35px; font-size:10px; } +.mailinglists td { padding: 5px 5px; line-height: 16px;} +.mailinglists th { font-size: 12px; text-align: left; color:#000; padding: 7px; letter-spacing: 1px; font-family:"Trebuchet MS", Arial, sans-serif;} + +#content { + float: left; + width: 100%; + padding: 0px 10px 70px 5px; + margin-bottom: 20px; + margin-top: 10px; +} + + +#content p{ + font-family: Verdana, Arial, Helvetica, sans-serif; + font-size: 12px; + line-height: 17px; + color: #333333; + padding: 5px 30px 0px 30px; + + margin: 0px 0px 8px 0px; +} + +#content ul{ + font-family: Verdana, Arial, Helvetica, sans-serif; + font-size: 12px; + line-height: 17px; +} + +#content li{ + font-family: Verdana, Arial, Helvetica, sans-serif; + font-size: 12px; + line-height: 17px; + list-style:circle; + margin: 10px 10px 10px 35px; +} + +h1 { + margin: 0px; + font-size: 16px; + font-weight: bold; + color: #172B17; + letter-spacing: 1px; + padding-top: 0px; + padding-right: 5px; + padding-bottom: 5px; + padding-left: 10px; + font-family:"Trebuchet MS", Arial, sans-serif; + +} +h2 { + margin: 0px; + font-size: 14px; + font-weight: bold; + color: #CC0000; + letter-spacing: 1px; + padding-top: 5px; + padding-right: 5px; + padding-bottom: 5px; + padding-left: 20px; + font-family:"Trebuchet MS", Arial, sans-serif; + +} +h3 { + margin: 0px; + font-size: 13px; + font-weight: bold; + color: #333; + letter-spacing: 1px; + padding-top: 5px; + padding-right: 5px; + padding-bottom: 5px; + padding-left: 20px; + font-family:"Trebuchet MS", Arial, sans-serif; + +} +h4 { + margin: 0px; + font-size: 12px; + font-weight: bold; + color: #666; + letter-spacing: 1px; + padding-top: 5px; + padding-right: 5px; + padding-bottom: 5px; + padding-left: 20px; + font-family:"Trebuchet MS", Arial, sans-serif; + text-transform: uppercase; + +} +h5 { + margin: 0px; + font-size: 12px; + font-weight: bold; + color: #666; + letter-spacing: 1px; + padding-top: 5px; + padding-right: 5px; + padding-bottom: 5px; + padding-left: 20px; + font-family:"Trebuchet MS", Arial, sans-serif; + text-transform: uppercase; + +} +#container { + background-color: #FFFFFF; + font-size: 10px; + font-weight: normal; + border: 1px #ccc solid; + width: 850px; + background-repeat: no-repeat; + background-position: center; + margin-top: 3px; + margin-right: auto; + margin-bottom: 0px; + margin-left: auto; +} +/************* FOOTER ************/ + +#footer { + clear: both; + width: 100%; + margin: 0px; + border-top: 1px solid #ddd; + border-bottom: 25px #FFF solid; + background-color: #FFF; + text-align: center; + padding: 0px; + color: #999; +} + +#footer a:link { + font-size: 9px; + font-weight: normal; + color: #3366cc; + text-decoration: none; +} +#footer a:visited { + font-size: 9px; + color: #3366cc; + text-decoration: none; +} + #footer a:hover { + font-size: 9px; + color: #0099cc; + text-decoration: underline; +} + +/********* DOCBOOK STUFF ********/ + +.article { + margin: 3px 20px; + font-size: 10px; +} + +.article p { + margin: 2px 25px; +} + +/* Syntax highlighting stuff */ + +div.dp-highlighter { + background: #FCFAFA; + border: 1px #ccc solid; + white-space: normal; + margin: 10px; +} + +span.comment { + color: #505050; +} + +span.keyword { + color: #202080; + font-weight: bold; +} + +span.string { + color: #209020; +} + +span.function { + color: #800040; +} From wilde at ci.uchicago.edu Wed Nov 21 17:38:43 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Wed, 21 Nov 2012 17:38:43 -0600 (CST) Subject: [Swift-commit] r6074 - in trunk: . bin Message-ID: <20121121233843.5FB7E9CFD6@svn.ci.uchicago.edu> Author: wilde Date: 2012-11-21 17:38:43 -0600 (Wed, 21 Nov 2012) New Revision: 6074 Modified: trunk/bin/slurm-qsub trunk/bin/slurm-qsub-emulator trunk/build.xml Log: Corrections to SLURM support: add local copy of qsub emulator to bin/ in build.xml; correct error in pbslog file naming; merge -l options on qsub line into one arg, as qsub emulator only handles one correctly; and remove #PBS lines from submit script, as these seem to be getting picked up by sbatch, causing contradictory settings. Modified: trunk/bin/slurm-qsub =================================================================== --- trunk/bin/slurm-qsub 2012-11-21 21:01:21 UTC (rev 6073) +++ trunk/bin/slurm-qsub 2012-11-21 23:38:43 UTC (rev 6074) @@ -7,7 +7,7 @@ script=$1.slurm echo '#! /bin/sh' >$script # echo 'export PBS_NODELIST=SLURM_NODELIST'' >>$script # FIXME: convert SLURM_NODELIST to PBS_NODEFILE here... -cat $1 >>$script +cat $1 | grep -v '^#PBS' >>$script chmod +x $script srunargs=$(awk <$1 ' @@ -29,7 +29,7 @@ # printf("-S %s -N %s -m %s -l nodes=%s:ppn=%s -l walltime=%s -q %s -o %s -e %s", # shell, jobname, m, nodes, ppn, walltime, queue, sofile, sefile); - printf("-N %s -m %s -l nodes=%s:ppn=%s -l walltime=%s -q %s -o %s -e %s", + printf("-N %s -m %s -l nodes=%s:ppn=%s,walltime=%s -q %s -o %s -e %s", jobname, m, nodes, ppn, walltime, queue, sofile, sefile); }') Modified: trunk/bin/slurm-qsub-emulator =================================================================== --- trunk/bin/slurm-qsub-emulator 2012-11-21 21:01:21 UTC (rev 6073) +++ trunk/bin/slurm-qsub-emulator 2012-11-21 23:38:43 UTC (rev 6074) @@ -81,7 +81,6 @@ my $sbatch = "sbatch"; my $salloc = "salloc"; -my $srun = "$srun"; GetOptions('a=s' => \$start_time, 'A=s' => \$account, @@ -215,7 +214,7 @@ $command .= " $script"; -system("(echo; echo $command; echo) >> $HOME/pbslog"); +system("(echo; echo $command; echo) >> \$HOME/pbslog"); system($command); Modified: trunk/build.xml =================================================================== --- trunk/build.xml 2012-11-21 21:01:21 UTC (rev 6073) +++ trunk/build.xml 2012-11-21 23:38:43 UTC (rev 6074) @@ -113,6 +113,7 @@ + From wilde at ci.uchicago.edu Wed Nov 21 20:15:51 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Wed, 21 Nov 2012 20:15:51 -0600 (CST) Subject: [Swift-commit] r6075 - trunk/libexec Message-ID: <20121122021551.8B6599CCC8@svn.ci.uchicago.edu> Author: wilde Date: 2012-11-21 20:15:51 -0600 (Wed, 21 Nov 2012) New Revision: 6075 Modified: trunk/libexec/vdl-int-staging.k Log: Fix - I think - for syntax error in vdl-int-staging.k. Modified: trunk/libexec/vdl-int-staging.k =================================================================== --- trunk/libexec/vdl-int-staging.k 2012-11-21 23:38:43 UTC (rev 6074) +++ trunk/libexec/vdl-int-staging.k 2012-11-22 02:15:51 UTC (rev 6075) @@ -182,7 +182,7 @@ "Exception in {tr}:", nl(), maybe(" Arguments: {arguments}", nl()), " Host: {rhost}", nl(), - " Directory: {tmpdir}", nl(), + " Directory: {tmpdir}", nl() ) exception ) From hategan at ci.uchicago.edu Fri Nov 23 02:44:26 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 23 Nov 2012 02:44:26 -0600 (CST) Subject: [Swift-commit] r6076 - trunk/resources Message-ID: <20121123084426.85E559CCB6@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-23 02:44:21 -0600 (Fri, 23 Nov 2012) New Revision: 6076 Modified: trunk/resources/Karajan.stg Log: update karajan output template to support reference counting for variable wriging; optimized range creation Modified: trunk/resources/Karajan.stg =================================================================== --- trunk/resources/Karajan.stg 2012-11-22 02:15:51 UTC (rev 6075) +++ trunk/resources/Karajan.stg 2012-11-23 08:44:21 UTC (rev 6076) @@ -57,7 +57,7 @@ >> -procedure(name,outputs,inputs,arguments,optargs,binding,declarations,statements,config,line,cleanups) ::= << +procedure(name, outputs, inputs, arguments, optargs, binding, declarations, statements, config, line, initWaitCounts, cleanups, closes) ::= << $optargs:default_arg();separator="\n"$ $inputs:vdl_log_input();separator="\n"$ @@ -65,21 +65,27 @@ $if(binding)$ $vdl_execute(outputs=outputs,inputs=inputs,application=binding.application, name=name,line=line)$ $else$ - $compound(outputs=outputs,inputs=inputs,declarations=declarations,statements=statements,name=name,cleanups=cleanups)$ + $compound(outputs=outputs, inputs=inputs, declarations=declarations, statements=statements, name=name, initWaitCounts=initWaitCounts, cleanups=cleanups)$ $endif$ >> -compound(outputs,inputs,declarations,statements,config,name,cleanups) ::= << +compound(outputs, inputs, declarations, statements, config, name, initWaitCounts, cleanups) ::= << +$if(initWaitCounts)$ +$initWaitCounts;separator="\n"$ +$endif$ $declarations;separator="\n"$ $if(statements)$ $parallel(statements=statements)$ $endif$ + $outputs:vdl_closedataset();separator="\n"$ + $cleanups:vdl_cleandataset();separator="\n"$ + >> proc_args(args) ::= << @@ -225,26 +231,29 @@ >> -foreach(var,in,indexVar,indexVarType,declarations,statements,line,selfClose,cleanups,trace) ::= << - - $in$ - - - +foreach(var, in, indexVar, indexVarType, declarations, statements, line, refs, selfClose, cleanups, trace) ::= << + +$! The iterator !$ + + $in$ + +$! Body !$ + + + $if(indexVar)$ - - - + + + + $endif$ - - - $declarations;separator="\n"$ - $if(statements)$ - $parallel(statements=statements)$ - $cleanups:vdl_cleandataset();separator="\n"$ - $endif$ - + + $declarations;separator="\n"$ +$if(statements)$ + $parallel(statements=statements)$ + $cleanups:vdl_cleandataset();separator="\n"$ +$endif$ + >> @@ -296,13 +305,16 @@ >> -callUserDefined(func, outputs, inputs, line, serialize) ::= << +callUserDefined(func, outputs, inputs, line, serialize, partialClose) ::= << <$func$ _traceline="$line$"> $if(!serialize)$$endif$ $outputs;separator="\n"$ $inputs;separator="\n"$ $if(!serialize)$$endif$ +$if(partialClose)$ +$partialClose$ +$endif$ >> call_arg(bind, expr, datatype) ::= << @@ -321,19 +333,19 @@ >> -variable(name,type,expr,mapping,nil,file,waitfor,datatype,isGlobal,line) ::= << +variable(name, type, expr, mapping, nil, file, waitCount, input, datatype, isGlobal, line) ::= << $if(isGlobal)$$else$$endif$ $if(mapping)$ - - $vdl_mapping(mapping=mapping,file=file,waitfor=waitfor)$ + + $vdl_mapping(mapping=mapping, file=file)$ $else$ $if(file)$ - - $vdl_mapping(mapping=mapping,file=file,waitfor=waitfor)$ + + $vdl_mapping(mapping=mapping, file=file)$ $else$ - + $endif$ $endif$ $if(isGlobal)$$else$$endif$ @@ -349,21 +361,15 @@ >> -vdl_mapping(mapping,file,waitfor) ::= << +vdl_mapping(mapping, file) ::= << $if(file)$ $if(file.params)$$file.params;separator="\n"$$endif$ - $if(!waitfor)$ - - $endif$ $else$ $mapping.params;separator="\n"$ - $if(!waitfor)$ - - $endif$ $endif$ >> @@ -372,21 +378,27 @@ $expr$ >> -assign(var,value,line) ::= << +assign(var, value, line, partialClose) ::= << $var$ $value$ + $if(partialClose)$ +$partialClose$ + $endif$ >> -append(array, value) ::= << +append(array, value, partialClose) ::= << $array$ $value$ + $if(partialClose)$ +$partialClose$ + $endif$ >> -callexpr(call, datatype, callID, prefix) ::= << +callexpr(call, datatype, prefix) ::= << @@ -408,25 +420,17 @@ >> -range(from,to,step,datatype) ::= << - - $from$ - $to$ - $if(step)$ - $step$ - $endif$ - - - swift#from - swift#to - $if(step)$ - swift#step - $endif$ - - - $range_log()$ - swift#rangeout - +range(from, to, step, datatype) ::= << + + + $from$ + $to$ +$if(step)$ + $step$ +$endif$ + + $range_log()$ + >> range_log() ::= << @@ -452,21 +456,25 @@ if(condition,vthen,velse,line,trace) ::= << - $condition$ - - - $vthen$ - - $if(velse)$ - - - $velse$ - - $endif$ + $condition$ + + + $vthen$ + +$if(velse)$ + + + $velse$ + +$endif$ + >> -sub_comp(declarations,statements,cleanups) ::= << +sub_comp(declarations, statements, cleanups, preClose) ::= << +$if(preClose)$ +$preClose;separator="\n"$ +$endif$ $declarations;separator="\n"$ $if(statements)$ $parallel(statements=statements)$ @@ -511,10 +519,14 @@ >> -partialclose(var, closeID) ::= << - +partialclose(var, count) ::= << + >> +setWaitCount(name, waitCount) ::= << + +>> + unitStart(type, outputs) ::= << >> From swift at ci.uchicago.edu Fri Nov 23 02:45:07 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Fri, 23 Nov 2012 02:45:07 -0600 (CST) Subject: [Swift-commit] cog r3518 Message-ID: <20121123084507.689868D0007A@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3518 | hategan | 2012-11-23 02:40:45 -0600 (Fri, 23 Nov 2012) | 1 line made PartialResume useable by derived classes ------------------------------------------------------------------------ Index: modules/karajan/src/org/globus/cog/karajan/workflow/nodes/PartialArgumentsContainer.java =================================================================== --- modules/karajan/src/org/globus/cog/karajan/workflow/nodes/PartialArgumentsContainer.java (revision 3517) +++ modules/karajan/src/org/globus/cog/karajan/workflow/nodes/PartialArgumentsContainer.java (working copy) @@ -87,7 +87,7 @@ } } - private class PartialResume implements FutureListener { + public class PartialResume implements FutureListener { public void futureModified(Future f, VariableStack stack) { try { partialArgumentsEvaluated(stack); From hategan at ci.uchicago.edu Fri Nov 23 02:57:21 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 23 Nov 2012 02:57:21 -0600 (CST) Subject: [Swift-commit] r6078 - trunk/libexec Message-ID: <20121123085721.216C99CCB6@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-23 02:57:20 -0600 (Fri, 23 Nov 2012) New Revision: 6078 Modified: trunk/libexec/vdl-lib.xml Log: added setWaitCount to vdl lib Modified: trunk/libexec/vdl-lib.xml =================================================================== --- trunk/libexec/vdl-lib.xml 2012-11-23 08:56:28 UTC (rev 6077) +++ trunk/libexec/vdl-lib.xml 2012-11-23 08:57:20 UTC (rev 6078) @@ -81,6 +81,7 @@ + From hategan at ci.uchicago.edu Fri Nov 23 03:00:47 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 23 Nov 2012 03:00:47 -0600 (CST) Subject: [Swift-commit] r6079 - trunk/tests/language-behaviour/arrays Message-ID: <20121123090047.C05709CCB6@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-23 03:00:46 -0600 (Fri, 23 Nov 2012) New Revision: 6079 Added: trunk/tests/language-behaviour/arrays/201-closing-if-in-foreach.swift trunk/tests/language-behaviour/arrays/202-closing-if-in-if-1.swift trunk/tests/language-behaviour/arrays/203-closing-foreach-in-if.swift trunk/tests/language-behaviour/arrays/205-closing-if-in-if-2.swift trunk/tests/language-behaviour/arrays/206-closing-one-writer.swift trunk/tests/language-behaviour/arrays/207-closing-no-writers.swift trunk/tests/language-behaviour/arrays/208-closing-self-closing.swift trunk/tests/language-behaviour/arrays/209-closing-return-array.swift trunk/tests/language-behaviour/arrays/bug-696.swift Log: added various tests for new reference counting code Added: trunk/tests/language-behaviour/arrays/201-closing-if-in-foreach.swift =================================================================== --- trunk/tests/language-behaviour/arrays/201-closing-if-in-foreach.swift (rev 0) +++ trunk/tests/language-behaviour/arrays/201-closing-if-in-foreach.swift 2012-11-23 09:00:46 UTC (rev 6079) @@ -0,0 +1,20 @@ +(int r) f(int[] b) { + foreach i in b { + trace(i); + } + r = 1; +} + +int a[]; + +int[] r; + +foreach i in [0:10] { + if (true) { + a[i] = 1; + } + else { + a[i] = 2; + } + r[i] = f(a); +} Added: trunk/tests/language-behaviour/arrays/202-closing-if-in-if-1.swift =================================================================== --- trunk/tests/language-behaviour/arrays/202-closing-if-in-if-1.swift (rev 0) +++ trunk/tests/language-behaviour/arrays/202-closing-if-in-if-1.swift 2012-11-23 09:00:46 UTC (rev 6079) @@ -0,0 +1,18 @@ +(int r) f(int[] b) { + foreach i in b { + trace(i); + } + r = 1; +} + + +int[] a; +int r; + +if (true) { + if (false) { + a[0] = 1; + } + r = f(a); +} + Added: trunk/tests/language-behaviour/arrays/203-closing-foreach-in-if.swift =================================================================== --- trunk/tests/language-behaviour/arrays/203-closing-foreach-in-if.swift (rev 0) +++ trunk/tests/language-behaviour/arrays/203-closing-foreach-in-if.swift 2012-11-23 09:00:46 UTC (rev 6079) @@ -0,0 +1,20 @@ +(int r) f(int[] b) { + foreach i in b { + trace(i); + } + r = 1; +} + + +int[] a; +int r; + +if (true) { + foreach j in [0:10] { + if (true) { + a[j] = 1; + } + } + r = f(a); +} + Added: trunk/tests/language-behaviour/arrays/205-closing-if-in-if-2.swift =================================================================== --- trunk/tests/language-behaviour/arrays/205-closing-if-in-if-2.swift (rev 0) +++ trunk/tests/language-behaviour/arrays/205-closing-if-in-if-2.swift 2012-11-23 09:00:46 UTC (rev 6079) @@ -0,0 +1,21 @@ +(int r) f(int[] b) { + foreach i in b { + trace(i); + } + r = 1; +} + + +int[] a; +int r; + +if (true) { + if (true) { + a[0] = 1; + } + r = f(a); +} +else { + a[1] = 1; +} + Added: trunk/tests/language-behaviour/arrays/206-closing-one-writer.swift =================================================================== --- trunk/tests/language-behaviour/arrays/206-closing-one-writer.swift (rev 0) +++ trunk/tests/language-behaviour/arrays/206-closing-one-writer.swift 2012-11-23 09:00:46 UTC (rev 6079) @@ -0,0 +1,6 @@ +int[] a; +a[0] = 1; + +foreach i in a { + trace(i); +} \ No newline at end of file Added: trunk/tests/language-behaviour/arrays/207-closing-no-writers.swift =================================================================== --- trunk/tests/language-behaviour/arrays/207-closing-no-writers.swift (rev 0) +++ trunk/tests/language-behaviour/arrays/207-closing-no-writers.swift 2012-11-23 09:00:46 UTC (rev 6079) @@ -0,0 +1,5 @@ +int[] a; + +foreach i in a { + trace(i); +} \ No newline at end of file Added: trunk/tests/language-behaviour/arrays/208-closing-self-closing.swift =================================================================== --- trunk/tests/language-behaviour/arrays/208-closing-self-closing.swift (rev 0) +++ trunk/tests/language-behaviour/arrays/208-closing-self-closing.swift 2012-11-23 09:00:46 UTC (rev 6079) @@ -0,0 +1,15 @@ +int[] a; +a[3] = 1; +int[][] b; + +foreach v, k in a { + if (k < 100) { + a[k + 10] = 1; + } + trace(k); +} + +foreach i in [0:2] { + a[i] = 1; + b[0][i] = 2; +} Added: trunk/tests/language-behaviour/arrays/209-closing-return-array.swift =================================================================== --- trunk/tests/language-behaviour/arrays/209-closing-return-array.swift (rev 0) +++ trunk/tests/language-behaviour/arrays/209-closing-return-array.swift 2012-11-23 09:00:46 UTC (rev 6079) @@ -0,0 +1,14 @@ +int[] a; + +(int[] r) f() { + foreach i in [0:9] { + r[i] = i; + r[i + 10] = i + 10; + } +} + +a = f(); + +foreach v in a { + trace(v); +} \ No newline at end of file Added: trunk/tests/language-behaviour/arrays/bug-696.swift =================================================================== --- trunk/tests/language-behaviour/arrays/bug-696.swift (rev 0) +++ trunk/tests/language-behaviour/arrays/bug-696.swift 2012-11-23 09:00:46 UTC (rev 6079) @@ -0,0 +1,25 @@ +main() +{ + foreach i in [0:0] { + int ia[]; + if(true) { + foreach j in [0:1] { + if (j==0) { + ia[j] = 100; + } + else { + ia[j] = 101; + } + } + sub(ia); + } + } +} + +sub(int iap[]) +{ + tracef("in sub: entered\n"); + tracef("in sub: iap=%q\n",iap); +} + +main(); From hategan at ci.uchicago.edu Fri Nov 23 02:56:37 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 23 Nov 2012 02:56:37 -0600 (CST) Subject: [Swift-commit] r6077 - in trunk/src/org/griphyn/vdl: engine karajan karajan/lib mapping Message-ID: <20121123085637.037099CCB6@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-23 02:56:28 -0600 (Fri, 23 Nov 2012) New Revision: 6077 Added: trunk/src/org/griphyn/vdl/karajan/lib/SetWaitCount.java Modified: trunk/src/org/griphyn/vdl/engine/CompilerUtils.java trunk/src/org/griphyn/vdl/engine/Karajan.java trunk/src/org/griphyn/vdl/engine/VariableScope.java trunk/src/org/griphyn/vdl/karajan/Loader.java trunk/src/org/griphyn/vdl/karajan/lib/New.java trunk/src/org/griphyn/vdl/karajan/lib/PartialCloseDataset.java trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java trunk/src/org/griphyn/vdl/mapping/AbstractDataNode.java trunk/src/org/griphyn/vdl/mapping/DSHandle.java trunk/src/org/griphyn/vdl/mapping/MappingParam.java trunk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java trunk/src/org/griphyn/vdl/mapping/RootDataNode.java Log: revamped variable closing stuff using reference counting and improved static analysis Modified: trunk/src/org/griphyn/vdl/engine/CompilerUtils.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/CompilerUtils.java 2012-11-23 08:44:21 UTC (rev 6076) +++ trunk/src/org/griphyn/vdl/engine/CompilerUtils.java 2012-11-23 08:56:28 UTC (rev 6077) @@ -14,8 +14,8 @@ public class CompilerUtils { public static String getLine(Node n) { - if (n == null) { - return "line unknown"; + if (n == null || n.getAttributes() == null) { + return "unknown"; } Node src = n.getAttributes().getNamedItem("src"); if (src == null) { @@ -38,4 +38,8 @@ } return getLine(src.getDomNode()); } + + public static String info(XmlObject src) { + return src.getDomNode().getLocalName() + ", line " + getLine(src); + } } Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/Karajan.java 2012-11-23 08:44:21 UTC (rev 6076) +++ trunk/src/org/griphyn/vdl/engine/Karajan.java 2012-11-23 08:56:28 UTC (rev 6077) @@ -17,6 +17,8 @@ package org.griphyn.vdl.engine; +import static org.griphyn.vdl.engine.CompilerUtils.getLine; + import java.io.File; import java.io.FileInputStream; import java.io.FileOutputStream; @@ -58,14 +60,15 @@ import org.globus.swift.language.TypesDocument.Types.Type; import org.globus.swift.language.Variable.Mapping; import org.globus.swift.language.Variable.Mapping.Param; +import org.griphyn.vdl.engine.VariableScope.EnclosureType; +import org.griphyn.vdl.engine.VariableScope.WriteType; import org.griphyn.vdl.karajan.CompilationException; import org.griphyn.vdl.karajan.Loader; import org.griphyn.vdl.toolkit.VDLt2VDLx; +import org.griphyn.vdl.type.NoSuchTypeException; import org.safehaus.uuid.UUIDGenerator; import org.w3c.dom.Node; -import static org.griphyn.vdl.engine.CompilerUtils.*; - public class Karajan { public static final Logger logger = Logger.getLogger(Karajan.class); @@ -79,7 +82,6 @@ Map typesMap = new HashMap(); List variables = new ArrayList(); - Set usedVariables = new HashSet(); public static final String TEMPLATE_FILE_NAME = "Karajan.stg"; public static final String TEMPLATE_FILE_NAME_NO_PROVENANCE = "Karajan-no-provenance.stg"; @@ -92,7 +94,6 @@ /** an arbitrary statement identifier. Start at some high number to aid visual distinction in logs, but the actual value doesn't matter. */ - int callID = 88000; StringTemplateGroup m_templates; @@ -126,8 +127,9 @@ ProgramDocument programDoc; try { programDoc = parseProgramXML(in); - } catch(Exception e) { - throw new CompilationException("Unable to parse intermediate XML",e); + } + catch(Exception e) { + throw new CompilationException("Unable to parse intermediate XML", e); } Program prog = programDoc.getProgram(); @@ -293,7 +295,7 @@ public StringTemplate program(Program prog) throws CompilationException { - VariableScope scope = new VariableScope(this, null); + VariableScope scope = new VariableScope(this, null, prog); scope.bodyTemplate = template("program"); scope.bodyTemplate.setAttribute("buildversion",Loader.buildVersion); @@ -310,37 +312,17 @@ for (Program program : importList) statements(program, scope); - checkUninitializedVariables(); generateInternedConstants(scope.bodyTemplate); + scope.analyzeWriters(); scope.bodyTemplate.setAttribute("cleanups", scope.getCleanups()); return scope.bodyTemplate; } - - private void checkUninitializedVariables() throws CompilationException { - for (StringTemplate var : variables) { - String name = (String) var.getAttribute("name"); - if (var.getAttribute("waitfor") == null) { - if (usedVariables.contains(name)) { - if (org.griphyn.vdl.type.Types.isPrimitive((String) var.getAttribute("type"))) { - throw new CompilationException("Uninitalized variable: " + name); - } - } - else { - logger.info("Unused variable " + name); - } - } - } - } - private void setVariableUsed(String s) { - usedVariables.add(s); - } - public void procedure(Procedure proc, VariableScope containingScope) throws CompilationException { - VariableScope outerScope = new VariableScope(this, containingScope, VariableScope.ENCLOSURE_PROCEDURE); - VariableScope innerScope = new VariableScope(this, outerScope, VariableScope.ENCLOSURE_NONE); + VariableScope outerScope = new VariableScope(this, containingScope, EnclosureType.PROCEDURE, proc); + VariableScope innerScope = new VariableScope(this, outerScope, EnclosureType.NONE, proc); StringTemplate procST = template("procedure"); containingScope.bodyTemplate.setAttribute("procedures", procST); procST.setAttribute("line", getLine(proc)); @@ -349,25 +331,14 @@ FormalParameter param = proc.getOutputArray(i); StringTemplate paramST = parameter(param, innerScope); procST.setAttribute("outputs", paramST); - if (!param.isNil()) - procST.setAttribute("optargs", paramST); - else - procST.setAttribute("arguments", paramST); - String type = normalize(param.getType().getLocalPart()); - checkIsTypeDefined(type); - innerScope.addVariable(param.getName(), type, "Return parameter", param); + addArg(procST, param, paramST, true, innerScope); } for (int i = 0; i < proc.sizeOfInputArray(); i++) { FormalParameter param = proc.getInputArray(i); StringTemplate paramST = parameter(param, outerScope); procST.setAttribute("inputs", paramST); - if (!param.isNil()) - procST.setAttribute("optargs", paramST); - else - procST.setAttribute("arguments", paramST); - String type = normalize(param.getType().getLocalPart()); - checkIsTypeDefined(type); - outerScope.addVariable(param.getName(), type, "Parameter", param); + addArg(procST, param, paramST, false, outerScope); + outerScope.addWriter(param.getName(), WriteType.FULL, proc, procST); } Binding bind; @@ -375,7 +346,7 @@ binding(bind, procST, innerScope); } else { - VariableScope compoundScope = new VariableScope(this, innerScope); + VariableScope compoundScope = new VariableScope(this, innerScope, proc); compoundScope.bodyTemplate = procST; statementsForSymbols(proc, compoundScope); statements(proc, compoundScope); @@ -383,6 +354,26 @@ } } + private void addArg(StringTemplate procST, FormalParameter param, + StringTemplate paramST, boolean returnArg, VariableScope scope) throws CompilationException { + if (!param.isNil()) { + procST.setAttribute("optargs", paramST); + } + else { + procST.setAttribute("arguments", paramST); + } + String type = normalize(param.getType().getLocalPart()); + checkIsTypeDefined(type); + scope.addVariable(param.getName(), type, returnArg ? "Return value" : "Parameter", false, param); + + if (returnArg) { + StringTemplate initWaitCountST = template("setWaitCount"); + initWaitCountST.setAttribute("name", param.getName()); + procST.setAttribute("initWaitCounts", initWaitCountST); + } + } + + /** * Convert to a case-insensitive representation by * pre-pending a '_' to upper case letters. If the @@ -435,8 +426,9 @@ typeST.setAttribute("name", normalize(param.getType().getLocalPart())); typeST.setAttribute("namespace", param.getType().getNamespaceURI()); paramST.setAttribute("type", typeST); - if(!param.isNil()) + if(!param.isNil()) { paramST.setAttribute("default",expressionToKarajan(param.getAbstractExpression(), scope)); + } return paramST; } @@ -452,9 +444,16 @@ variableST.setAttribute("isGlobal", Boolean.valueOf(var.getIsGlobal())); variableST.setAttribute("line", getLine(var)); variables.add(variableST); + + + /* + * possibly an input; mark it as such here, and if + * writers are detected, remove the input attribute (that is + * done in VariableScope). + */ + variableST.setAttribute("input", "true"); - if(!var.isNil()) { - + if (!var.isNil()) { if (var.getFile() != null) { StringTemplate fileST = new StringTemplate("file"); fileST.setAttribute("name", escapeQuotes(var.getFile().getName())); @@ -462,8 +461,6 @@ variableST.setAttribute("file", fileST); } - - Mapping mapping = var.getMapping(); if (mapping != null) { @@ -471,56 +468,14 @@ mappingST.setAttribute("descriptor", mapping.getDescriptor()); for (int i = 0; i < mapping.sizeOfParamArray(); i++) { Param param = mapping.getParamArray(i); - StringTemplate paramST = template("vdl_parameter"); - paramST.setAttribute("name", param.getName()); - Node expressionDOM = param.getAbstractExpression().getDomNode(); - String namespaceURI = expressionDOM.getNamespaceURI(); - String localName = expressionDOM.getLocalName(); - QName expressionQName = new QName(namespaceURI, localName); - if(expressionQName.equals(VARIABLE_REFERENCE_EXPR)) { - paramST.setAttribute("expr",expressionToKarajan(param.getAbstractExpression(),scope)); - } else { - String parameterVariableName="swift#mapper#"+(internedIDCounter++); - // make template for variable declaration (need to compute type of this variable too?) - StringTemplate variableDeclarationST = template("variable"); - variableDeclarationST.setAttribute("waitfor",""); - // TODO factorise this and other code in variable()? - StringTemplate pmappingST = new StringTemplate("mapping"); - pmappingST.setAttribute("descriptor", "concurrent_mapper"); - StringTemplate pparamST = template("vdl_parameter"); - pparamST.setAttribute("name", "prefix"); - pparamST.setAttribute("expr", parameterVariableName + "-" + UUIDGenerator.getInstance().generateRandomBasedUUID().toString()); - pmappingST.setAttribute("params", pparamST); - //variableDeclarationST.setAttribute("mapping", pmappingST); - variableDeclarationST.setAttribute("nil", Boolean.TRUE); - variableDeclarationST.setAttribute("name", parameterVariableName); - scope.bodyTemplate.setAttribute("declarations",variableDeclarationST); - StringTemplate paramValueST=expressionToKarajan(param.getAbstractExpression(),scope); - String paramValueType = datatype(paramValueST); - scope.addVariable(parameterVariableName, paramValueType, "Variable", param); - variableDeclarationST.setAttribute("type", paramValueType); - StringTemplate variableReferenceST = template("id"); - variableReferenceST.setAttribute("var",parameterVariableName); - StringTemplate variableAssignmentST = template("assign"); - variableAssignmentST.setAttribute("var",variableReferenceST); - variableAssignmentST.setAttribute("value",paramValueST); - scope.appendStatement(variableAssignmentST); - if (param.getAbstractExpression().getDomNode().getNodeName().equals("stringConstant")) { - StringTemplate valueST = template("sConst"); - valueST.setAttribute("innervalue", param.getAbstractExpression().getDomNode().getFirstChild().getNodeValue()); - paramST.setAttribute("expr",valueST); - } - else { - paramST.setAttribute("expr",variableReferenceST); - } - } - mappingST.setAttribute("params", paramST); + mappingST.setAttribute("params", mappingParameter(param, scope)); } variableST.setAttribute("mapping", mappingST); } - } else { - // add temporary mapping info - if (!org.griphyn.vdl.type.Types.isPrimitive(var.getType().getLocalPart())) { + } + else { + // add temporary mapping info in not primitive or array of primitive + if (!isPrimitiveOrArrayOfPrimitive(var.getType().getLocalPart())) { StringTemplate mappingST = new StringTemplate("mapping"); mappingST.setAttribute("descriptor", "concurrent_mapper"); StringTemplate paramST = template("vdl_parameter"); @@ -531,21 +486,79 @@ variableST.setAttribute("mapping", mappingST); variableST.setAttribute("nil", Boolean.TRUE); } - } + } scope.bodyTemplate.setAttribute("declarations", variableST); } + private StringTemplate mappingParameter(Param param, VariableScope scope) throws CompilationException { + StringTemplate paramST = template("vdl_parameter"); + paramST.setAttribute("name", param.getName()); + Node expressionDOM = param.getAbstractExpression().getDomNode(); + String namespaceURI = expressionDOM.getNamespaceURI(); + String localName = expressionDOM.getLocalName(); + QName expressionQName = new QName(namespaceURI, localName); + if (expressionQName.equals(VARIABLE_REFERENCE_EXPR)) { + paramST.setAttribute("expr", expressionToKarajan(param.getAbstractExpression(), scope)); + } + else { + String parameterVariableName="swift#mapper#"+(internedIDCounter++); + // make template for variable declaration (need to compute type of this variable too?) + StringTemplate variableDeclarationST = template("variable"); + // TODO factorise this and other code in variable()? + StringTemplate pmappingST = new StringTemplate("mapping"); + pmappingST.setAttribute("descriptor", "concurrent_mapper"); + StringTemplate pparamST = template("vdl_parameter"); + pparamST.setAttribute("name", "prefix"); + pparamST.setAttribute("expr", parameterVariableName + "-" + + UUIDGenerator.getInstance().generateRandomBasedUUID().toString()); + pmappingST.setAttribute("params", pparamST); + variableDeclarationST.setAttribute("nil", Boolean.TRUE); + variableDeclarationST.setAttribute("name", parameterVariableName); + scope.bodyTemplate.setAttribute("declarations", variableDeclarationST); + StringTemplate paramValueST=expressionToKarajan(param.getAbstractExpression(),scope); + String paramValueType = datatype(paramValueST); + scope.addVariable(parameterVariableName, paramValueType, "Variable", param); + variableDeclarationST.setAttribute("type", paramValueType); + StringTemplate variableReferenceST = template("id"); + variableReferenceST.setAttribute("var",parameterVariableName); + StringTemplate variableAssignmentST = template("assign"); + variableAssignmentST.setAttribute("var",variableReferenceST); + variableAssignmentST.setAttribute("value",paramValueST); + scope.appendStatement(variableAssignmentST); + if (param.getAbstractExpression().getDomNode().getNodeName().equals("stringConstant")) { + StringTemplate valueST = template("sConst"); + valueST.setAttribute("innervalue", param.getAbstractExpression().getDomNode().getFirstChild().getNodeValue()); + paramST.setAttribute("expr",valueST); + } + else { + paramST.setAttribute("expr",variableReferenceST); + } + } + return paramST; + } + void checkIsTypeDefined(String type) throws CompilationException { if (!org.griphyn.vdl.type.Types.isValidType(type, typesMap.keySet())) { throw new CompilationException("Type " + type + " is not defined."); } } + + private boolean isPrimitiveOrArrayOfPrimitive(String type) { + org.griphyn.vdl.type.Type t; + try { + t = org.griphyn.vdl.type.Types.getType(type); + return t.isPrimitive() || (t.isArray() && t.itemType().isPrimitive()); + } + catch (NoSuchTypeException e) { + return false; + } + } public void assign(Assign assign, VariableScope scope) throws CompilationException { try { StringTemplate assignST = template("assign"); - StringTemplate varST = expressionToKarajan(assign.getAbstractExpressionArray(0),scope); + StringTemplate varST = expressionToKarajan(assign.getAbstractExpressionArray(0), scope, true); StringTemplate valueST = expressionToKarajan(assign.getAbstractExpressionArray(1),scope); if (! (datatype(varST).equals(datatype(valueST)) || datatype(valueST).equals("java"))) @@ -555,7 +568,7 @@ assignST.setAttribute("value", valueST); assignST.setAttribute("line", getLine(assign)); String rootvar = abstractExpressionToRootVariable(assign.getAbstractExpressionArray(0)); - scope.addWriter(rootvar, new Integer(callID++), rootVariableIsPartial(assign.getAbstractExpressionArray(0))); + scope.addWriter(rootvar, getRootVariableWriteType(assign.getAbstractExpressionArray(0)), assign, assignST); scope.appendStatement(assignST); } catch(CompilationException re) { throw new CompilationException("Compile error in assignment at "+assign.getSrc()+": "+re.getMessage(),re); @@ -580,7 +593,7 @@ appendST.setAttribute("value", value); String rootvar = abstractExpressionToRootVariable(append.getAbstractExpressionArray(0)); // an append is always a partial write - scope.addWriter(rootvar, new Integer(callID++), true); + scope.addWriter(rootvar, WriteType.PARTIAL, append, appendST); scope.appendStatement(appendST); } catch(CompilationException re) { throw new CompilationException("Compile error in assignment at "+append.getSrc()+": "+re.getMessage(),re); @@ -797,7 +810,7 @@ ActualParameter output = call.getOutputArray(i); StringTemplate argST = actualParameter(output, scope); callST.setAttribute("outputs", argST); - addWriterToScope(scope, call.getOutputArray(i).getAbstractExpression()); + addWriterToScope(scope, call.getOutputArray(i).getAbstractExpression(), call, callST); } } if (keywordArgsOutput) { @@ -816,12 +829,12 @@ throw new CompilationException("Wrong type for output parameter number " + i + ", expected " + formalType + ", got " + actualType); - addWriterToScope(scope, call.getOutputArray(i).getAbstractExpression()); + addWriterToScope(scope, call.getOutputArray(i).getAbstractExpression(), call, callST); } } else { /* Positional arguments */ for (int i = 0; i < call.sizeOfOutputArray(); i++) { ActualParameter output = call.getOutputArray(i); - StringTemplate argST = actualParameter(output, scope); + StringTemplate argST = actualParameter(output, scope, true); callST.setAttribute("outputs", argST); FormalArgumentSignature formalArg =proceduresMap.get(procName).getOutputArray(i); @@ -833,7 +846,7 @@ throw new CompilationException("Wrong type for parameter number " + i + ", expected " + formalType + ", got " + actualType); - addWriterToScope(scope, call.getOutputArray(i).getAbstractExpression()); + addWriterToScope(scope, call.getOutputArray(i).getAbstractExpression(), call, callST); } } @@ -844,8 +857,9 @@ callST.setAttribute("serialize", Boolean.TRUE); } return callST; - } catch(CompilationException ce) { - throw new CompilationException("Compile error in procedure invocation at "+call.getSrc()+": "+ce.getMessage(),ce); + } + catch (CompilationException ce) { + throw new CompilationException("Compile error in procedure invocation at " + call.getSrc(), ce); } } @@ -883,19 +897,19 @@ return true; } - private void addWriterToScope(VariableScope scope, XmlObject var) throws CompilationException { + private void addWriterToScope(VariableScope scope, XmlObject var, XmlObject src, StringTemplate out) throws CompilationException { String rootvar = abstractExpressionToRootVariable(var); - boolean partial = rootVariableIsPartial(var); - if (!partial) { + WriteType writeType = getRootVariableWriteType(var); + if (writeType == WriteType.FULL) { // don't close variables that are already closed by the function itself scope.inhibitClosing(rootvar); } - scope.addWriter(rootvar, new Integer(callID++), partial); + scope.addWriter(rootvar, writeType, src, out); } public void iterateStat(Iterate iterate, VariableScope scope) throws CompilationException { - VariableScope loopScope = new VariableScope(this, scope, VariableScope.ENCLOSURE_LOOP); - VariableScope innerScope = new VariableScope(this, loopScope, VariableScope.ENCLOSURE_LOOP); + VariableScope loopScope = new VariableScope(this, scope, EnclosureType.LOOP, iterate); + VariableScope innerScope = new VariableScope(this, loopScope, EnclosureType.LOOP, iterate); loopScope.addVariable(iterate.getVar(), "int", "Iteration variable", iterate); @@ -904,6 +918,8 @@ iterateST.setAttribute("var", iterate.getVar()); innerScope.bodyTemplate = iterateST; + + loopScope.addWriter(iterate.getVar(), WriteType.FULL, iterate, iterateST); statementsForSymbols(iterate.getBody(), innerScope); statements(iterate.getBody(), innerScope); @@ -912,16 +928,13 @@ StringTemplate condST = expressionToKarajan(cond, innerScope); iterateST.setAttribute("cond", condST); - Object statementID = new Integer(callID++); - for (String v : innerScope.getVariables()) - scope.addWriter(v, statementID, true); scope.appendStatement(iterateST); iterateST.setAttribute("cleanups", innerScope.getCleanups()); } public void foreachStat(Foreach foreach, VariableScope scope) throws CompilationException { try { - VariableScope innerScope = new VariableScope(this, scope, VariableScope.ENCLOSURE_LOOP); + VariableScope innerScope = new VariableScope(this, scope, EnclosureType.LOOP, foreach); StringTemplate foreachST = template("foreach"); foreachST.setAttribute("var", foreach.getVar()); @@ -930,6 +943,10 @@ XmlObject in = foreach.getIn().getAbstractExpression(); StringTemplate inST = expressionToKarajan(in, scope); foreachST.setAttribute("in", inST); + + if ("id".equals(inST.getName())) { + innerScope.setForeachSourceVar((String) inST.getAttribute("var"), foreach); + } String inType = datatype(inST); String itemType = org.griphyn.vdl.type.Types.getArrayInnerItemTypeName(inType); @@ -938,30 +955,26 @@ throw new CompilationException("You can iterate through an array structure only"); } innerScope.addVariable(foreach.getVar(), itemType, "Iteration variable", foreach); + innerScope.addWriter(foreach.getVar(), WriteType.FULL, foreach, foreachST); foreachST.setAttribute("indexVar", foreach.getIndexVar()); foreachST.setAttribute("indexVarType", keyType); - if(foreach.getIndexVar() != null) { + if (foreach.getIndexVar() != null) { innerScope.addVariable(foreach.getIndexVar(), keyType, "Iteration variable", foreach); + innerScope.addWriter(foreach.getIndexVar(), WriteType.FULL, foreach, foreachST); } innerScope.bodyTemplate = foreachST; statementsForSymbols(foreach.getBody(), innerScope); statements(foreach.getBody(), innerScope); - - String inVar = (String) inST.getAttribute("var"); - Object statementID = new Integer(callID++); - for (String v : innerScope.getVariables()) { - scope.addWriter(v, statementID, true); - if (v.equals(inVar) && !innerScope.isVariableLocallyDefined(v)) - foreachST.setAttribute("selfClose", "true"); - } + scope.appendStatement(foreachST); Collection cleanups = innerScope.getCleanups(); cleanups.remove(foreach.getVar()); foreachST.setAttribute("cleanups", cleanups); - } catch(CompilationException re) { - throw new CompilationException("Compile error in foreach statement at "+foreach.getSrc()+": "+re.getMessage(),re); + } + catch (CompilationException re) { + throw new CompilationException("Compile error in foreach statement at " + foreach.getSrc(), re); } } @@ -977,7 +990,8 @@ Then thenstat = ifstat.getThen(); Else elsestat = ifstat.getElse(); - VariableScope innerThenScope = new VariableScope(this, scope, VariableScope.ENCLOSURE_CONDITION); + VariableScope innerThenScope = new VariableScope(this, scope, + EnclosureType.CONDITION, thenstat); innerThenScope.bodyTemplate = template("sub_comp"); ifST.setAttribute("vthen", innerThenScope.bodyTemplate); @@ -985,31 +999,32 @@ statements(thenstat, innerThenScope); innerThenScope.bodyTemplate.setAttribute("cleanups", innerThenScope.getCleanups()); - Object statementID = new Integer(callID++); - - for (String v : innerThenScope.getVariables()) - scope.addWriter(v, statementID, true); - if (elsestat != null) { - - VariableScope innerElseScope = new VariableScope(this, scope); + VariableScope innerElseScope = new VariableScope(this, scope, + EnclosureType.CONDITION, elsestat); innerElseScope.bodyTemplate = template("sub_comp"); + innerElseScope.setThen(innerThenScope); ifST.setAttribute("velse", innerElseScope.bodyTemplate); statementsForSymbols(elsestat, innerElseScope); statements(elsestat, innerElseScope); - - for (String v : innerElseScope.getVariables()) - scope.addWriter(v, statementID, true); innerElseScope.bodyTemplate.setAttribute("cleanups", innerElseScope.getCleanups()); } + else if (innerThenScope.hasPartialClosing()) { + // must do matching partial closing somewhere, so need + // else branch even though not specified in the swift source + VariableScope innerElseScope = new VariableScope(this, scope, + EnclosureType.CONDITION, elsestat); + innerElseScope.bodyTemplate = template("sub_comp"); + innerElseScope.setThen(innerThenScope); + ifST.setAttribute("velse", innerElseScope.bodyTemplate); + } scope.appendStatement(ifST); } public void switchStat(Switch switchstat, VariableScope scope) throws CompilationException { StringTemplate switchST = template("switch"); - Object statementID = new Integer(callID++); scope.bodyTemplate.setAttribute("statements", switchST); StringTemplate conditionST = expressionToKarajan(switchstat.getAbstractExpression(), scope); switchST.setAttribute("condition", conditionST.toString()); @@ -1020,24 +1035,19 @@ for (int i=0; i< switchstat.sizeOfCaseArray(); i++) { Case casestat = switchstat.getCaseArray(i); - VariableScope caseScope = new VariableScope(this, scope); + VariableScope caseScope = new VariableScope(this, scope, casestat); caseScope.bodyTemplate = new StringTemplate("case"); switchST.setAttribute("cases", caseScope.bodyTemplate); caseStat(casestat, caseScope); - - for (String v : caseScope.getVariables()) - scope.addWriter(v, statementID, true); } Default defaultstat = switchstat.getDefault(); if (defaultstat != null) { - VariableScope defaultScope = new VariableScope(this, scope); + VariableScope defaultScope = new VariableScope(this, scope, defaultstat); defaultScope.bodyTemplate = template("sub_comp"); switchST.setAttribute("sdefault", defaultScope.bodyTemplate); statementsForSymbols(defaultstat, defaultScope); statements(defaultstat, defaultScope); - for (String v : defaultScope.getVariables()) - scope.addWriter(v, statementID, true); } } @@ -1047,10 +1057,14 @@ statementsForSymbols(casestat.getStatements(), scope); statements(casestat.getStatements(), scope); } + + public StringTemplate actualParameter(ActualParameter arg, VariableScope scope) throws CompilationException { + return actualParameter(arg, scope, false); + } - public StringTemplate actualParameter(ActualParameter arg, VariableScope scope) throws CompilationException { + public StringTemplate actualParameter(ActualParameter arg, VariableScope scope, boolean lvalue) throws CompilationException { StringTemplate argST = template("call_arg"); - StringTemplate expST = expressionToKarajan(arg.getAbstractExpression(), scope); + StringTemplate expST = expressionToKarajan(arg.getAbstractExpression(), scope, lvalue); argST.setAttribute("bind", arg.getBind()); argST.setAttribute("expr", expST); argST.setAttribute("datatype", datatype(expST)); @@ -1188,11 +1202,15 @@ static final QName RANGE_EXPR = new QName(SWIFTSCRIPT_NS, "range"); static final QName FUNCTION_EXPR = new QName(SWIFTSCRIPT_NS, "function"); static final QName CALL_EXPR = new QName(SWIFTSCRIPT_NS, "call"); + + public StringTemplate expressionToKarajan(XmlObject expression, VariableScope scope) throws CompilationException { + return expressionToKarajan(expression, scope, false); + } /** converts an XML intermediate form expression into a * Karajan expression. */ - public StringTemplate expressionToKarajan(XmlObject expression, VariableScope scope) throws CompilationException + public StringTemplate expressionToKarajan(XmlObject expression, VariableScope scope, boolean lvalue) throws CompilationException { Node expressionDOM = expression.getDomNode(); String namespaceURI = expressionDOM.getNamespaceURI(); @@ -1322,10 +1340,12 @@ XmlString xmlString = (XmlString) expression; String s = xmlString.getStringValue(); if(!scope.isVariableDefined(s)) { - throw new CompilationException("Variable " + s + " is undefined."); + throw new CompilationException("Variable " + s + " was not declared in this scope."); } - - setVariableUsed(s); + + if (!lvalue) { + scope.addReader(s, false, expression); + } StringTemplate st = template("id"); st.setAttribute("var", s); String actualType; @@ -1336,7 +1356,7 @@ } else if (expressionQName.equals(ARRAY_SUBSCRIPT_EXPR)) { BinaryOperator op = (BinaryOperator) expression; StringTemplate arrayST = expressionToKarajan(op.getAbstractExpressionArray(1), scope); - StringTemplate parentST = expressionToKarajan(op.getAbstractExpressionArray(0), scope); + StringTemplate parentST = expressionToKarajan(op.getAbstractExpressionArray(0), scope, true); String indexType = datatype(arrayST); String declaredIndexType = org.griphyn.vdl.type.Types.getArrayOuterIndexTypeName(datatype(parentST)); @@ -1359,6 +1379,8 @@ + indexType + ") does not match the declared index type (" + declaredIndexType + ")"); } + scope.addReader((String) parentST.getAttribute("var"), true, expression); + StringTemplate newst = template("extractarrayelement"); newst.setAttribute("arraychild", arrayST); newst.setAttribute("parent", parentST); @@ -1367,7 +1389,7 @@ return newst; } else if (expressionQName.equals(STRUCTURE_MEMBER_EXPR)) { StructureMember sm = (StructureMember) expression; - StringTemplate parentST = expressionToKarajan(sm.getAbstractExpression(), scope); + StringTemplate parentST = expressionToKarajan(sm.getAbstractExpression(), scope, true); String parentType = datatype(parentST); @@ -1414,6 +1436,8 @@ else { newst = template("extractstructelement"); } + + scope.addReader((String) parentST.getAttribute("var"), true, expression); newst.setAttribute("parent", parentST); newst.setAttribute("memberchild", sm.getMemberName()); newst.setAttribute("datatype", actualType); @@ -1492,7 +1516,7 @@ } else if (expressionQName.equals(CALL_EXPR)) { Call c = (Call) expression; c.addNewOutput(); - VariableScope subscope = new VariableScope(this, scope); + VariableScope subscope = new VariableScope(this, scope, c); VariableReferenceDocument ref = VariableReferenceDocument.Factory.newInstance(); ref.setVariableReference("swift#callintermediate"); c.getOutputArray(0).set(ref); @@ -1517,7 +1541,6 @@ call.setAttribute("datatype", type); call.setAttribute("call", call(c, subscope, true)); - call.setAttribute("callID", new Integer(callID++)); call.setAttribute("prefix", UUIDGenerator.getInstance().generateRandomBasedUUID().toString()); return call; } else { @@ -1585,17 +1608,17 @@ } } - public boolean rootVariableIsPartial(XmlObject expression) { + public WriteType getRootVariableWriteType(XmlObject expression) { Node expressionDOM = expression.getDomNode(); String namespaceURI = expressionDOM.getNamespaceURI(); String localName = expressionDOM.getLocalName(); QName expressionQName = new QName(namespaceURI, localName); if (expressionQName.equals(VARIABLE_REFERENCE_EXPR)) { - return false; + return WriteType.FULL; } else if (expressionQName.equals(ARRAY_SUBSCRIPT_EXPR)) { - return true; + return WriteType.PARTIAL; } else if (expressionQName.equals(STRUCTURE_MEMBER_EXPR)) { - return true; + return WriteType.PARTIAL; } else { throw new RuntimeException("Could not find root for abstract expression."); } Modified: trunk/src/org/griphyn/vdl/engine/VariableScope.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/VariableScope.java 2012-11-23 08:44:21 UTC (rev 6076) +++ trunk/src/org/griphyn/vdl/engine/VariableScope.java 2012-11-23 08:56:28 UTC (rev 6077) @@ -18,12 +18,16 @@ package org.griphyn.vdl.engine; import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; import java.util.HashMap; import java.util.HashSet; import java.util.Iterator; +import java.util.LinkedList; import java.util.List; import java.util.Map; import java.util.Set; +import java.util.Stack; import org.antlr.stringtemplate.StringTemplate; import org.apache.log4j.Logger; @@ -32,25 +36,26 @@ public class VariableScope { - - /** permit array up-assignment, but not entire variables */ - public final static int ENCLOSURE_LOOP = 301923; - - /** permit all upwards assignments */ - public static final int ENCLOSURE_ALL = 301924; - - /** permit no upwards assignments */ - public static final int ENCLOSURE_NONE = 301925; - - /** permit no access to the containing scope except for finding - global variables */ - public static final int ENCLOSURE_PROCEDURE = 301926; + public enum EnclosureType { + /** permit array up-assignment, but not entire variables */ + LOOP, + /** permit all upwards assignments */ + ALL, + /** permit no upwards assignments */ + NONE, + /** permit no access to the containing scope except for finding + global variables */ + PROCEDURE, + /** Override ENCLOSURE_LOOP to allow assignment inside a loop + * based on some condition + */ + CONDITION; + } + + public enum WriteType { + FULL, PARTIAL + } - /** Override ENCLOSURE_LOOP to allow assignment inside a loop - * based on some condition - */ - public static final int ENCLOSURE_CONDITION = 301927; - public static final Logger logger = Logger.getLogger(VariableScope.class); /** need this for the program as a whole. probably should factor @@ -64,17 +69,114 @@ this will be the same as parentScope or this. */ VariableScope rootScope; - int enclosureType; + private EnclosureType enclosureType; /** The string template in which we will store statements outputted into this scope. */ public StringTemplate bodyTemplate; private List outputs = new ArrayList(); - private Set inhibitClosing = new HashSet(); + private Set inhibitClosing; + private List children = new LinkedList(); + private XmlObject src; + + /** List of templates to be executed in sequence after the present + in-preparation statement is outputted. */ + List presentStatementPostStatements = new ArrayList(); + + /** Set of variables (String token names) that are declared in + this scope - not variables in parent scopes, though. */ + Map variables; + + /** + * Usage (such as writing or reading) in this scope + */ + Map variableUsage; + + /** + * If this is the scope of an 'else', then store a reference + * to the corresponding 'then' + */ + private VariableScope thenScope; + + private static class Variable { + public final String name, type; + public final XmlObject src; + public final boolean isGlobal; + private XmlObject foreachSrc; + + public Variable(String name, String type, boolean isGlobal, XmlObject src) { + this.name = name; + this.type = type; + this.src = src; + this.isGlobal = isGlobal; + } + } + + /** Stores information about a variable that is referred to in this + scope. Should probably get used for dataset marking eventually. */ + private static class VariableUsage { + private final String name; + private XmlObject fullWritingLoc, foreachSourceVar; + private List partialWritingLoc; + private List fullReadingLoc; + private int referenceCount; + + public VariableUsage(String name) { + this.name = name; + } + + public boolean addWritingStatement(XmlObject loc, WriteType writeType) throws CompilationException { + if (writeType == WriteType.PARTIAL) { + if (partialWritingLoc == null) { + partialWritingLoc = new LinkedList(); + } + if (!partialWritingLoc.contains(loc)) { + partialWritingLoc.add(loc); + return true; + } + else { + return false; + } + } + else { + if (fullWritingLoc != null) { + throw new CompilationException("Variable " + name + " is written to in two different locations:\n\t1. " + + CompilerUtils.info(fullWritingLoc) + "\n\t2. " + CompilerUtils.info(loc)); + } + else { + fullWritingLoc = loc; + return true; + } + } + } - public VariableScope(Karajan c, VariableScope parent) { - this(c, parent, ENCLOSURE_ALL); + public boolean hasWriters() { + return fullWritingLoc != null || (partialWritingLoc != null && partialWritingLoc.size() > 0); + } + + public void addFullReadingStatement(XmlObject where) { + if (fullReadingLoc == null) { + fullReadingLoc = new LinkedList(); + } + fullReadingLoc.add(where); + } + + public boolean hasReaders() { + return fullReadingLoc != null && fullReadingLoc.size() > 0; + } + + public XmlObject getForeachSourceVar() { + return foreachSourceVar; + } + + public void setForeachSourceVar(XmlObject foreachSourceVar) { + this.foreachSourceVar = foreachSourceVar; + } + } + + public VariableScope(Karajan c, VariableScope parent, XmlObject src) { + this(c, parent, EnclosureType.ALL, src); } /** Creates a new variable scope. @@ -84,7 +186,7 @@ @param a specifies how assignments to variables made in enclosing scopes will be handled. */ - public VariableScope(Karajan c, VariableScope parent, int a) { + public VariableScope(Karajan c, VariableScope parent, EnclosureType a, XmlObject src) { if (logger.isDebugEnabled()) { if (parentScope != null) { logger.debug("New scope " + hashCode() + " with parent scope " + parentScope.hashCode()); @@ -96,24 +198,30 @@ compiler = c; parentScope = parent; enclosureType = a; - if(parentScope == null) { + if (parentScope == null) { rootScope = this; - } else { + } + else { rootScope = parentScope.getRootScope(); + parentScope.addChild(this); } + this.src = src; } - /** Set of variables (String token names) that are declared in - this scope - not variables in parent scopes, though. */ - Map variables = new HashMap(); - Map varTypes = new HashMap(); + public void addChild(VariableScope scope) { + children.add(scope); + } + + public List getChildren() { + return children; + } + + public void setThen(VariableScope thenScope) { + this.thenScope = thenScope; + updateBranchReferenceCounts(); + } - /** Set of variables (String token names) which are global and - declared in this scope (which must be a root scope). Variables - in this set must also appear in the variables set. */ - Map globals = new HashMap(); - /** Asserts that a named variable is declared in this scope. Might also eventually contain more information about the variable. Need to define behaviour here when the @@ -124,7 +232,81 @@ addVariable(name, type, context, false, src); } + private Collection getVariableUsageValues() { + if (variableUsage == null) { + return Collections.emptyList(); + } + else { + return variableUsage.values(); + } + } + + private Map getVariableUsage() { + if (variableUsage == null) { + variableUsage = new HashMap(); + } + return variableUsage; + } + + /** + * Get variable usage if it exists, otherwise return null + */ + protected VariableUsage getExistingUsage(String name) { + if (variableUsage == null) { + return null; + } + else { + return variableUsage.get(name); + } + } + + private Collection getUsedVariableNames() { + if (variableUsage == null) { + return Collections.emptyList(); + } + else { + return variableUsage.keySet(); + } + } + + private Collection getLocallyDeclaredVariableNames() { + if (variables == null) { + return Collections.emptyList(); + } + else { + return variables.keySet(); + } + } + + + /** + * Returns variable usage in this scope for the given name. + * If no previous usage exists, create it. This method + * guarantees a non-null return + */ + protected VariableUsage getUsageForUpdate(String name) { + Map usage = getVariableUsage(); + VariableUsage u = usage.get(name); + if (u == null) { + u = new VariableUsage(name); + usage.put(name, u); + } + return u; + } + + public void setForeachSourceVar(String name, XmlObject src) { + getUsageForUpdate(name).setForeachSourceVar(src); + } + + public boolean isForeachSourceVar(String name) { + VariableUsage u = getExistingUsage(name); + return u != null && u.getForeachSourceVar() != null; + } + public void inhibitClosing(String name) { + if (inhibitClosing == null) { + inhibitClosing = new HashSet(); + } inhibitClosing.add(name); } @@ -135,7 +317,7 @@ if(isVariableDefined(name)) { throw new CompilationException("Variable " + name + ", on line " - + CompilerUtils.getLine(src) + ", was already defined on line " + getDefinitionLine(name)); + + CompilerUtils.getLine(src) + ", was already defined on line " + getDeclarationLine(name)); } // TODO does this if() ever fire? or is it always taken @@ -143,22 +325,24 @@ // be replaced by is locally defined test. if(parentScope != null && parentScope.isVariableDefined(name)) { Warnings.warn(context + " " + name + ", on line " + CompilerUtils.getLine(src) - + ", shadows variable of same name on line " + parentScope.getDefinitionLine(name)); + + ", shadows variable of same name on line " + parentScope.getDeclarationLine(name)); } - if(global && this != rootScope) { + if (global && this != rootScope) { throw new CompilationException("Global " + name + " can only be declared in the root scope of a program."); } - - variables.put(name, src); - varTypes.put(name, type); - if (global) { - globals.put(name, src); - } + getVariablesMap().put(name, new Variable(name, type, global, src)); } - /** + private Map getVariablesMap() { + if (variables == null) { + variables = new HashMap(); + } + return variables; + } + + /** * Does pretty much the same as addVariable() except it doesn't throw * an exception if the variable is defined in a parent scope */ @@ -169,16 +353,18 @@ if(isVariableLocallyDefined(name)) { throw new CompilationException("Variable " + name + ", on line " - + CompilerUtils.getLine(src) + ", was already defined on line " + getDefinitionLine(name)); + + CompilerUtils.getLine(src) + ", was already defined on line " + getDeclarationLine(name)); } - variables.put(name, src); - varTypes.put(name, type); + getVariablesMap().put(name, new Variable(name, type, false, src)); } - public String getDefinitionLine(String name) { - XmlObject src = rootScope.getGlobalSrc(name); - if (src == null) { + public String getDeclarationLine(String name) { + XmlObject src; + if (rootScope.isGlobalDefined(name)) { + src = rootScope.getGlobalSrc(name); + } + else { src = getSrcRecursive(name); } if (src != null) { @@ -191,155 +377,609 @@ public boolean isVariableDefined(String name) { if(rootScope.isGlobalDefined(name)) return true; - return isVariableDefinedRecursive(name); + return isVariableVisible(name); } - private XmlObject getSrcRecursive(String name) { - XmlObject src = variables.get(name); - if (src == null) { - if (enclosureType != ENCLOSURE_PROCEDURE && parentScope != null) { - src = parentScope.getSrcRecursive(name); + /** + * Recursively find information about a variable visible in this scope. + * Also, look for globals. + */ + public Variable lookup(String name) { + if (variables != null) { + Variable var = variables.get(name); + if (var != null) { + return var; } } - return src; + if (enclosureType != EnclosureType.PROCEDURE && parentScope != null) { + return parentScope.lookup(name); + } + else { + // if the search fails, see if there is a global with this name + if (rootScope.variables != null) { + Variable v = rootScope.variables.get(name); + if (v != null && v.isGlobal) { + return v; + } + else { + return null; + } + } + else { + return null; + } + } } + + private XmlObject getSrcRecursive(String name) { + Variable var = lookup(name); + if (var == null) { + throw new IllegalArgumentException("Variable " + name + " is not visible in this scope"); + } + return var.src; + } - private boolean isVariableDefinedRecursive(String name) { - if(isVariableLocallyDefined(name)) return true; - if(enclosureType != ENCLOSURE_PROCEDURE && parentScope != null && parentScope.isVariableDefined(name)) return true; - return false; + private boolean isVariableVisible(String name) { + if (isVariableLocallyDefined(name)) { + return true; + } + if (enclosureType != EnclosureType.PROCEDURE && parentScope != null) { + return parentScope.isVariableVisible(name); + } + else { + return false; + } } + + private StringTemplate getExistingDeclaration(String name) { + if (isVariableLocallyDefined(name)) { + return getLocalDeclaration(name); + } + else if (parentScope != null) { + return parentScope.getExistingDeclaration(name); + } + else { + return null; + } + } public boolean isGlobalDefined(String name) { - return globals.containsKey(name); + Variable var = rootScope.lookup(name); + return var != null && var.isGlobal; } public XmlObject getGlobalSrc(String name) { - return globals.get(name); + Variable var = rootScope.lookup(name); + if (var != null && var.isGlobal) { + return var.src; + } + else { + throw new IllegalArgumentException("'" + name + "' is not a global variable"); + } } - /** note that this will return variable types for variables in - containing scopes even if they are not accessible. non-null - return value from this method should not be treated as an - indication that the variable is in any way accessible from - this scope. The isVariableDefined and isGlobalDefined methods - should be used to check that. */ public String getVariableType(String name) { - if(isVariableLocallyDefined(name)) - return varTypes.get(name).toString(); - if(parentScope != null && parentScope.isVariableDefined(name)) - return parentScope.getVariableType(name); - return null; + Variable var = lookup(name); + if (var != null) { + return var.type; + } + else { + throw new IllegalArgumentException("Variable " + name + " is not visible in this scope"); + } } -// TODO could also mark variable as written and catch duplicate assignments -// in the same scope here - public boolean isVariableWriteable(String name, boolean partialWriter) { - if(isVariableLocallyDefined(name)) return true; - if(parentScope != null && parentScope.isVariableWriteable(name, true) && enclosureType == ENCLOSURE_CONDITION) { - Warnings.warn("Variable " + name + ", defined on line " + getDefinitionLine(name) + ", might have multiple writers"); + public boolean isVariableWriteable(String name, WriteType writeType) { + if (isVariableLocallyDefined(name)) { return true; } - if(parentScope != null && parentScope.isVariableWriteable(name, partialWriter) && enclosureType == ENCLOSURE_ALL) return true; - if(parentScope != null && parentScope.isVariableWriteable(name, partialWriter) && enclosureType == ENCLOSURE_LOOP && partialWriter) return true; - return false; + if (parentScope == null) { + // if we are the root scope and the variable is not declared here, + // then there is no other place to look + return false; + } + switch (enclosureType) { + case CONDITION: + if (parentScope.isVariableWriteable(name, WriteType.FULL)) { + if (getTopmostLoopToDeclaration(name) != null) { + Warnings.warn("Variable " + name + ", defined on line " + + getDeclarationLine(name) + ", might have multiple conflicting writers"); + } + return true; + } + else { + return false; + } + case ALL: + return parentScope.isVariableWriteable(name, writeType); + case LOOP: + return parentScope.isVariableWriteable(name, writeType) && writeType == WriteType.PARTIAL; + default: + return false; + } } - public boolean isVariableLocallyDefined(String name) { - return variables.containsKey(name); + /** + * Assumes that the variable is defined in some parent scope. + * Returns the top-most loop between this scope and the declaration + * or null if no such loop is found + */ + private VariableScope getTopmostLoopToDeclaration(String name) { + if (isVariableLocallyDefined(name)) { + return null; + } + else if (enclosureType == EnclosureType.LOOP) { + VariableScope parentLoop = parentScope.getTopmostLoopToDeclaration(name); + if (parentLoop != null) { + return parentLoop; + } + else { + return this; + } + } + else { + return parentScope.getTopmostLoopToDeclaration(name); + } } + + /** + * This deals with cases like this: + * a[]; a[0] = 1; + * foreach v, k in a { + * if (c) { + * a[k + 1] = 1; + * } + * } + * ... + * foreach v, k in a { + * if (c) { + * a[k + 100] = 1; + * } + * } + * + * a[] can be considered closed when all of the following are true: + * 1. the write reference count is equal to the number of loops + * that both iterate on and write to a + * 2. none of those loops have any iterations running + * 3. there are no elements in a[] left to iterate on + * These three conditions guarantee that no more writing + * can happen to a[]. + * + * Unfortunately, it seems horribly complicated to deal with multiple + * loops that do this. Simple reference counting doesn't work, since + * any of the conditions above can become false at any point and it is + * hard to have an absolute time for all of the loops. + * + * So engineering compromise: this is allowed for a single loop (per + * array), and that's been shown to work (the self close scheme). + * If used in more than one array, throw an error. + * @throws CompilationException + */ + private void markInVariableUsedInsideLoop(String name) throws CompilationException { + if (isVariableLocallyDefined(name)) { + return; + } + switch (enclosureType) { + case PROCEDURE: + case NONE: + return; + case LOOP: + if (isForeachSourceVar(name)) { + // mark as selfClosing + Variable var = lookup(name); + if (var.foreachSrc != null) { + throw new CompilationException("Updating of an array (" + + name + ", line " + getDeclarationLine(name) + ") inside a " + + "foreach loop that iterates over it is limited to a single " + + "loop, but it is used in both " + CompilerUtils.info(var.foreachSrc) + + " and " + CompilerUtils.info(src)); + } + else { + setSelfClose(); + } + } + // also add partial close + + default: + if (parentScope != null) { + parentScope.markInVariableUsedInsideLoop(name); + } + } + } - /** List of templates to be executed in sequence after the present - in-preparation statement is outputted. */ - List presentStatementPostStatements = new ArrayList(); + private void setSelfClose() { + bodyTemplate.setAttribute("selfClose", "true"); + } - Map variableUsage = new HashMap(); + public boolean isVariableLocallyDefined(String name) { + return variables != null && variables.containsKey(name); + } /** indicates that the present in-preparation statement writes to the named variable. If the variable is declared in the local scope, register a closing statement; otherwise, record that this scope writes to the variable so that the scope-embedding code (such as the foreach compiler) can handle appropriately. */ - public void addWriter(String variableName, Object closeID, boolean partialWriter) throws CompilationException - { - if(!isVariableDefined(variableName)) { + public void addWriter(String variableName, WriteType writeType, + XmlObject where, StringTemplate out) throws CompilationException { + if (!isVariableDefined(variableName)) { throw new CompilationException("Variable " + variableName + " is not defined"); } + + if (logger.isDebugEnabled()) { + logger.debug("Writer " + variableName + " " + CompilerUtils.info(where)); + } - if(isVariableLocallyDefined(variableName)) { - StringTemplate ld = getLocalDeclaration(variableName); - if (ld != null) { - if(!partialWriter && ld.getAttribute("waitfor") != null) { - throw new CompilationException("variable " + variableName + ", defined on line " - + getDefinitionLine(variableName) + ", has multiple writers."); - } - ld.setAttribute("waitfor", closeID); - } - else { - if (logger.isDebugEnabled()) { - logger.debug("Variable " + variableName + " is local but has no template."); - } - } - outputs.add(variableName); - if (!inhibitClosing.contains(variableName)) { - StringTemplate postST = compiler.template("partialclose"); - postST.setAttribute("var", variableName); - postST.setAttribute("closeID", closeID); - - presentStatementPostStatements.add(postST); - } - } else { + if (isVariableWriteable(variableName, writeType)) { + setVariableWrittenToInThisScope(variableName, writeType, where, out); + } + else { + isVariableWriteable(variableName, writeType); + throw new CompilationException("variable " + variableName + " is not writeable in this scope"); + } + } + + public void partialClose(String name, XmlObject src, StringTemplate out) { + if (inhibitClosing == null || !inhibitClosing.contains(name)) { + StringTemplate decl = getExistingDeclaration(name); + if (decl != null) { + // a function return value will result + // in a missing declaration + // TODO fix that + incWaitCount(name); + + StringTemplate postST = compiler.template("partialclose"); + postST.setAttribute("var", name); + + out.setAttribute("partialClose", postST); + } + } + } + + public boolean hasPartialClosing() { + if (enclosureType != EnclosureType.CONDITION || thenScope != null) { + throw new IllegalStateException("hasPartialClosing should only be called in a 'then' condition scope"); + } + for (String name : getUsedVariableNames()) { + VariableUsage v = getExistingUsage(name); + if (v != null && v.referenceCount > 0) { + return true; + } + } + return false; + } + + private void setPreClose(String name, int count) { + setCount("preClose", name, count, + new RefCountSetter() { + @Override + public StringTemplate create(String name, int count) { + StringTemplate t = compiler.template("partialclose"); + t.setAttribute("var", name); + if (count != 1) { + t.setAttribute("count", count); + } + return t; + } + + @Override + public void set(StringTemplate t, int count) { + t.removeAttribute("count"); + if (count != 1) { + // 1 is the default count, so no need to have it explicitly + t.setAttribute("count", count); + } + } -// TODO now we have to walk up the scopes until either we find the -// variable or we find an upwards assignment prohibition or we run -// out of scopes + @Override + public boolean matches(StringTemplate t, String name) { + return t.getAttribute("var").equals(name); + } + }); + } + + private void setLoopReferenceCount(String name, int count) { + setCount("refs", name, count, new RefCountSetter() { + @Override + public SimpleRefCountRep create(String name, int count) { + return new SimpleRefCountRep(name, count); + } -// TODO so far this should find undeclared variables at compile time -// so perhaps worth making this into a separate patch if it actually -// works + @Override + public boolean matches(SimpleRefCountRep t, String name) { + return t.name.equals(name); + } - if(isVariableWriteable(variableName, partialWriter)) { - Variable variable = variableUsage.get(variableName); - if(variable == null) { - variable = new Variable(); - variableUsage.put(variableName, variable); - } + @Override + public void set(SimpleRefCountRep t, int count) { + t.count = count; + } + }); + } + + private static class SimpleRefCountRep { + public final String name; + public int count; + + public SimpleRefCountRep(String name, int count) { + this.name = name; + this.count = count; + } + + public String toString() { + return name + " " + count; + } + } + + private static interface RefCountSetter { + T create(String name, int count); + void set(T o, int count); + boolean matches(T t, String name); + } + + @SuppressWarnings("unchecked") + private void setCount(String attrName, String name, int count, RefCountSetter r) { + Object o = bodyTemplate.getAttribute(attrName); + List pcs; + T matching = null; + if (o instanceof List) { + pcs = (List) o; + Iterator i = pcs.iterator(); + while (i.hasNext()) { + T st = i.next(); + if (r.matches(st, name)) { + if (count == 0) { + i.remove(); + } + else { + matching = st; + break; + } + } + } + } + else if (o != null) { + T st = (T) o; + if (r.matches(st, name)) { + if (count == 0) { + bodyTemplate.removeAttribute("preClose"); + } + else { + matching = st; + } + } + } + if (count != 0 ) { + // add + if (matching == null) { + bodyTemplate.setAttribute(attrName, r.create(name, count)); + } + else { + r.set(matching, count); + } + } + } + + private void incWaitCountLocal(String name) { + StringTemplate decl = getLocalDeclaration(name); + Integer i = (Integer) decl.getAttribute("waitCount"); + if (i == null) { + i = 1; + } + else { + i = i + 1; + decl.removeAttribute("waitCount"); + } + decl.setAttribute("waitCount", i); + } - List statementList = variable.writingStatements; - if(!statementList.contains(closeID)) { - statementList.add(closeID); - } - if (logger.isDebugEnabled()) { - logger.debug("added " + closeID + " to variable " - + variableName + " in scope " + hashCode()); - } - } else { - isVariableWriteable(variableName, partialWriter); - throw new CompilationException("variable " + variableName + " is not writeable in this scope"); - } - } + private void incWaitCount(String name) { + if (this.isVariableLocallyDefined(name)) { + incWaitCountLocal(name); + } + else { + boolean propagateToParent = true; + if (enclosureType == EnclosureType.CONDITION) { + // due to the parsing order, the + // then scopes are done before the else scopes, + // so no need to update counts in subling else scopes + // if this is a then scope + if (thenScope == null) { + // we are then 'then' scope + incVariableReferenceCount(name); + } + else { + // else scope + int myCount = incVariableReferenceCount(name); + int thenCount = thenScope.getVariableReferenceCount(name); + if (myCount > thenCount) { + thenScope.setPreClose(name, myCount - thenCount); + this.setPreClose(name, 0); + } + else { + this.setPreClose(name, thenCount - myCount); + thenScope.setPreClose(name, 0); + // this has already been accounted for in the then branch + // so skip propagating it up + propagateToParent = false; + } + } + } + else if (enclosureType == EnclosureType.LOOP) { + // issue statements to dynamically increment wait + // count on each iteration + addLoopReferenceCount(name); + } + // keep going until we hit the declaration + if (propagateToParent && parentScope != null) { + parentScope.incWaitCount(name); + } + } + } + + /** + * This is only called by setThen(scope), so guaranteed to + * be an 'else' scope + */ + private void updateBranchReferenceCounts() { + for (VariableUsage v : thenScope.getVariableUsageValues()) { + if (v.referenceCount != 0) { + setPreClose(v.name, v.referenceCount); + } + } + } + + public void addReader(String name, boolean partial, XmlObject where) throws CompilationException { + setVariableReadInThisScope(name, partial, where); + if (!partial) { + if (logger.isDebugEnabled()) { + logger.debug(name + " fully read in " + CompilerUtils.info(where)); + } + } } + + private void setVariableWrittenToInThisScope(String name, WriteType writeType, XmlObject where, StringTemplate out) + throws CompilationException { + VariableUsage variable = getUsageForUpdate(name); + boolean added = variable.addWritingStatement(where, writeType); + if (added) { + partialClose(name, where, out); + setNonInput(name); + } + markInVariableUsedInsideLoop(name); + } + + private void setNonInput(String name) { + StringTemplate decl = getExistingDeclaration(name); + if (decl != null) { + decl.removeAttribute("input"); + } + } - /** looks up the template that declared the named variable in the + private void setVariableReadInThisScope(String name, boolean partial, XmlObject where) + throws CompilationException { + // only care about full reads, since they can only happen in + // a scope higher than the closing + if (partial) { + return; + } + getUsageForUpdate(name).addFullReadingStatement(where); + } + + private boolean isVariableWrittenToInThisScope(String name) { + VariableUsage v = getExistingUsage(name); + return v != null && v.hasWriters(); + } + + private int incVariableReferenceCount(String name) { + return ++getUsageForUpdate(name).referenceCount; + } + + private void addLoopReferenceCount(String name) { + int count = incVariableReferenceCount(name); + setLoopReferenceCount(name, count); + } + + private int getVariableReferenceCount(String name) { + VariableUsage v = getExistingUsage(name); + if (v == null) { + return 0; + } + else { + return v.referenceCount; + } + } + + private boolean isVariableFullyReadInThisScope(String name) { + VariableUsage v = getExistingUsage(name); + return v != null && v.hasReaders(); + } + + private XmlObject getFirstFullRead(String name) { + VariableUsage v = getExistingUsage(name); + if (v == null) { + return null; + } + if (v.fullReadingLoc == null) { + return null; + } + if (v.fullReadingLoc.isEmpty()) { + return null; + } + return v.fullReadingLoc.get(0); + } + + private XmlObject getFirstWrite(String name) { + VariableUsage v = getExistingUsage(name); + if (v == null) { + return null; + } + if (v.fullWritingLoc != null) { + return v.fullWritingLoc; + } + if (v.partialWritingLoc == null) { + return null; + } + if (v.partialWritingLoc.isEmpty()) { + return null; + } + return v.partialWritingLoc.get(0); + } + + private List getAllWriters(String name) { + VariableUsage v = getExistingUsage(name); + if (v == null) { + throw new IllegalArgumentException("Variable " + name + " is not written to in this scope"); + } + return v.partialWritingLoc; + } + + StringTemplate getLocalDeclaration(String name) { + StringTemplate st = getLocalDeclaration(name, "declarations"); + /* + * procedure scopes go like this: + * outerScope - contains all arguments - PROCEDURE + * innerScope - contains all returns - NONE + * compoundScope - this is where the proc template lives - ALL + * this prevents writing to arguments, since writing upwards of NONE is prohibited, + * but allows writing to return values + */ + + if (st == null && parentScope != null && parentScope.enclosureType == EnclosureType.PROCEDURE) { + if (children.size() != 1) { + throw new IllegalStateException("Procedure scope with more than one child found"); + } + return children.get(0).getLocalDeclaration(name, "initWaitCounts"); + } + else { + return st; + } + } + + /** looks up the template that declared the named variable in the present scope. If such a template does not exist, returns null. TODO this should probably merge into general variable structure rather then walking the template. */ - StringTemplate getLocalDeclaration(String name) { + StringTemplate getLocalDeclaration(String name, String type) { if (bodyTemplate == null) { return null; } - Object decls = bodyTemplate.getAttribute("declarations"); - if(decls == null) return null; + Object decls = bodyTemplate.getAttribute(type); + if (decls == null) { + return null; + } - if(decls instanceof StringTemplate) { + if (decls instanceof StringTemplate) { StringTemplate declST = (StringTemplate) decls; - if(declST.getAttribute("name").equals(name)) { - logger.debug("thats the declaration for "+name); + if (declST.getAttribute("name").equals(name)) { + if (logger.isDebugEnabled()) { + logger.debug("Found declaration for " + name); + } return declST; } - } else { // assume its a List of StringTemplate + } + else { // assume its a List of StringTemplate @SuppressWarnings("unchecked") List list = (List) decls; for (StringTemplate declST : list) { @@ -347,9 +987,9 @@ logger.debug("looking at declaration " + declST); } try { - if(declST.getAttribute("name").equals(name)) { + if (declST.getAttribute("name").equals(name)) { if (logger.isDebugEnabled()) { - logger.debug("thats the declaration for " + name); + logger.debug("Found declaration for " + name); } return declST; } @@ -361,7 +1001,9 @@ } } - logger.info("UH OH - couldn't find local definition for "+name); + if (logger.isInfoEnabled()) { + logger.info("Couldn't find local definition for " + name); + } return null; } @@ -393,7 +1035,7 @@ } presentStatementPostStatements.clear(); outputs.clear(); - inhibitClosing.clear(); + inhibitClosing = null; } private String join(List l) { @@ -407,20 +1049,6 @@ } return sb.toString(); } - - /** Stores information about a variable that is referred to in this - scope. Should probably get used for dataset marking eventually. */ - class Variable { - List writingStatements = new ArrayList(); - } - - Iterator getVariableIterator() { - return variableUsage.keySet().iterator(); - } - - Set getVariables() { - return variableUsage.keySet(); - } public VariableScope getRootScope() { return rootScope; @@ -428,7 +1056,7 @@ public List getCleanups() { List cleanups = new ArrayList(); - for (String var : variables.keySet()) { + for (String var : getLocallyDeclaredVariableNames()) { String type = getVariableType(var); if (!org.griphyn.vdl.type.Types.isPrimitive(type)) { cleanups.add(var); @@ -438,7 +1066,78 @@ } public String toString() { - return variables.toString(); + return CompilerUtils.info(src); } + + /** + * @throws CompilationException + */ + public void analyzeWriters() throws CompilationException { + // find variables that are read but not written to + + // keep a stack to account for shadowing + Map> access = new HashMap>(); + + analyzeWriters(access); + } + + private static class Usage { + public VariableScope lastWriter, lastReader; + } + + private void analyzeWriters(Map> access) throws CompilationException { + for (String name : getLocallyDeclaredVariableNames()) { + declare(access, name); + } + + // parent first, which ensures that the topmost writer + // is always first + for (Map.Entry> e : access.entrySet()) { + String name = e.getKey(); + Stack stack = e.getValue(); + + boolean fullyRead = isVariableFullyReadInThisScope(name); + + if (isVariableWrittenToInThisScope(name)) { + stack.peek().lastWriter = this; + } + + if (fullyRead && org.griphyn.vdl.type.Types.isPrimitive(getVariableType(name))) { + stack.peek().lastReader = this; + } + } + + for (VariableScope child : children) { + child.analyzeWriters(access); + } + + for (String name : getLocallyDeclaredVariableNames()) { + undeclare(access, name); + } + } + + private void declare(Map> access, String name) { + Stack s = access.get(name); + if (s == null) { + s = new Stack(); + access.put(name, s); + } + Usage u = new Usage(); + s.push(u); + } + + private void undeclare(Map> access, String name) throws CompilationException { + Stack s = access.get(name); + if (s == null || s.isEmpty()) { + throw new RuntimeException("Mistmatched undeclare for " + name + " in scope " + this); + } + Usage u = s.pop(); + if (s.isEmpty()) { + access.remove(name); + } + if (u.lastReader != null && u.lastWriter == null) { + throw new CompilationException("Uninitalized variable: " + name); + } + } } Modified: trunk/src/org/griphyn/vdl/karajan/Loader.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/Loader.java 2012-11-23 08:44:21 UTC (rev 6076) +++ trunk/src/org/griphyn/vdl/karajan/Loader.java 2012-11-23 08:56:28 UTC (rev 6077) @@ -248,9 +248,17 @@ private static String getMessages(Throwable e) { StringBuilder sb = new StringBuilder(); + String lastMessage = null; while (e != null) { - sb.append(":\n\t"); - sb.append(e.getMessage()); + String msg = e.getMessage(); + if (msg == null) { + msg = e.toString(); + } + sb.append("\n\t"); + if (lastMessage == null || !lastMessage.contains(msg)) { + sb.append(msg); + lastMessage = msg; + } e = e.getCause(); } return sb.toString(); @@ -336,7 +344,6 @@ if (recompile) { VDLt2VDLx.compile(new FileInputStream(swiftscript), new PrintStream(new FileOutputStream(xml))); - try { FileOutputStream f = new FileOutputStream(kml); Karajan.compile(xml.getAbsolutePath(), new PrintStream(f), provenanceEnabled); Modified: trunk/src/org/griphyn/vdl/karajan/lib/New.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/New.java 2012-11-23 08:44:21 UTC (rev 6076) +++ trunk/src/org/griphyn/vdl/karajan/lib/New.java 2012-11-23 08:56:28 UTC (rev 6077) @@ -51,11 +51,12 @@ public static final Arg OA_MAPPING = new Arg.Optional("mapping", null); public static final Arg OA_VALUE = new Arg.Optional("value", null); public static final Arg OA_DBGNAME = new Arg.Optional("dbgname", null); - public static final Arg OA_WAITFOR = new Arg.Optional("waitfor", null); + public static final Arg OA_WAITCOUNT = new Arg.Optional("waitcount", null); + public static final Arg OA_INPUT = new Arg.Optional("input", Boolean.FALSE); static { setArguments(New.class, - new Arg[] { OA_TYPE, OA_MAPPING, OA_VALUE, OA_DBGNAME, OA_WAITFOR}); + new Arg[] { OA_TYPE, OA_MAPPING, OA_VALUE, OA_DBGNAME, OA_WAITCOUNT, OA_INPUT}); } private Tracer tracer; @@ -73,7 +74,8 @@ Map mapping = (Map) OA_MAPPING.getValue(stack); String dbgname = TypeUtil.toString(OA_DBGNAME.getValue(stack)); - String waitfor = (String) OA_WAITFOR.getValue(stack); + String waitCount = (String) OA_WAITCOUNT.getValue(stack); + boolean input = TypeUtil.toBoolean(OA_INPUT.getValue(stack)); String line = (String) getProperty("_defline"); MappingParamSet mps = new MappingParamSet(); @@ -83,6 +85,10 @@ mps.set(MappingParam.SWIFT_DBGNAME, dbgname); } + if (input) { + mps.set(MappingParam.SWIFT_INPUT, true); + } + if (line != null) { mps.set(MappingParam.SWIFT_LINE, line); } @@ -91,9 +97,15 @@ mps.set(MappingParam.SWIFT_RESTARTID, threadPrefix + ":" + dbgname); - if (waitfor != null) { - mps.set(MappingParam.SWIFT_WAITFOR, waitfor); + // input means never written to, but read at least once + int initialWriteRefCount; + boolean noWriters = input; + if (waitCount != null) { + initialWriteRefCount = Integer.parseInt(waitCount); } + else { + initialWriteRefCount = 0; + } if (typename == null && value == null) { throw new ExecutionException("You must specify either a type or a value"); @@ -157,7 +169,7 @@ } handle.closeShallow(); } - else { + else { if (tracer.isEnabled()) { tracer.trace(threadPrefix, dbgname); } @@ -191,6 +203,7 @@ if (AbstractDataNode.provenance && logger.isDebugEnabled()) { logger.debug("NEW id=" + handle.getIdentifier()); } + handle.setWriteRefCount(initialWriteRefCount); return handle; } catch (Exception e) { Modified: trunk/src/org/griphyn/vdl/karajan/lib/PartialCloseDataset.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/PartialCloseDataset.java 2012-11-23 08:44:21 UTC (rev 6076) +++ trunk/src/org/griphyn/vdl/karajan/lib/PartialCloseDataset.java 2012-11-23 08:56:28 UTC (rev 6077) @@ -20,92 +20,32 @@ import org.apache.log4j.Logger; import org.globus.cog.karajan.arguments.Arg; import org.globus.cog.karajan.stack.VariableStack; +import org.globus.cog.karajan.util.TypeUtil; import org.globus.cog.karajan.workflow.ExecutionException; import org.griphyn.vdl.mapping.DSHandle; -import org.griphyn.vdl.mapping.MappingParam; -import java.util.StringTokenizer; -import java.util.List; -import java.util.Map; -import java.util.HashMap; -import java.util.ArrayList; - public class PartialCloseDataset extends VDLFunction { public static final Logger logger = Logger.getLogger(CloseDataset.class); - public static final Arg OA_CLOSE_ID = new Arg.Optional("closeID", null); + public static final Arg OA_COUNT = new Arg.Optional("count", 1); static { - setArguments(PartialCloseDataset.class, new Arg[] { PA_VAR, OA_CLOSE_ID }); + setArguments(PartialCloseDataset.class, new Arg[] { PA_VAR, OA_COUNT }); } - - /** Map from DSHandles (as keys) to lists of what we have seen - already. TODO this may end up growing too much when a program - has lots of objects. Consider alternative ways of doing this. */ - static Map> pendingDatasets = - new HashMap>(); - public Object function(VariableStack stack) throws ExecutionException { - boolean hasUnseenToken = false; DSHandle var = (DSHandle) PA_VAR.getValue(stack); - String closeID = (String) OA_CLOSE_ID.getValue(stack); if (logger.isDebugEnabled()) { - logger.debug("Partially closing " + var + - " hash: " + var.hashCode() + - " for statement " + closeID); + logger.debug("Partially closing " + var); } - if(var.isClosed()) { + if (var.isClosed()) { logger.debug("variable already closed - skipping partial close processing"); return null; } - - synchronized(pendingDatasets) { - - List c = pendingDatasets.get(var); - if (c == null) { - c = new ArrayList(); - pendingDatasets.put(var, c); - } - - c.add(closeID); - if (logger.isDebugEnabled()) { - logger.debug("Adding token "+closeID+" with hash "+closeID.hashCode()); - } - - String needToWaitFor = var.getParam(MappingParam.SWIFT_WAITFOR); - logger.debug("need to wait for " + needToWaitFor); - StringTokenizer stok = new StringTokenizer(needToWaitFor, " "); - while(stok.hasMoreTokens()) { - String s = stok.nextToken(); - // do we have this token in our list of already seen - // statement IDs? - if(! c.contains(s)) { - // then we have a required element that we have not - // seen yet, so... - hasUnseenToken = true; - if (logger.isDebugEnabled()) { - logger.debug("Container does not contain token " + s); - } - } else { - if (logger.isDebugEnabled()) { - logger.debug("Container does contain token " + s); - } - } - } - } - if (logger.isDebugEnabled()) { - logger.debug("hasUnseenToken = "+hasUnseenToken); - } - if(!hasUnseenToken) { - if(logger.isInfoEnabled()) { - logger.info("All partial closes for " + var + - " have happened. Closing fully."); - } - var.closeDeep(); - pendingDatasets.remove(var); - } + + int count = TypeUtil.toInt(OA_COUNT.getValue(stack)); + var.updateWriteRefCount(-count); return null; } } Added: trunk/src/org/griphyn/vdl/karajan/lib/SetWaitCount.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/SetWaitCount.java (rev 0) +++ trunk/src/org/griphyn/vdl/karajan/lib/SetWaitCount.java 2012-11-23 08:56:28 UTC (rev 6077) @@ -0,0 +1,47 @@ +/* + * Copyright 2012 University of Chicago + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package org.griphyn.vdl.karajan.lib; + +import org.apache.log4j.Logger; +import org.globus.cog.karajan.arguments.Arg; +import org.globus.cog.karajan.stack.VariableStack; +import org.globus.cog.karajan.util.TypeUtil; +import org.globus.cog.karajan.workflow.ExecutionException; +import org.griphyn.vdl.mapping.DSHandle; + +public class SetWaitCount extends VDLFunction { + public static final Logger logger = Logger.getLogger(CloseDataset.class); + + public static final Arg OA_COUNT = new Arg.Optional("count", 1); + + static { + setArguments(SetWaitCount.class, new Arg[] { PA_VAR, OA_COUNT }); + } + + public Object function(VariableStack stack) throws ExecutionException { + DSHandle var = (DSHandle) PA_VAR.getValue(stack); + + if (var.isClosed()) { + throw new ExecutionException("Attempted to set a wait count for a closed variable " + var); + } + + int count = TypeUtil.toInt(OA_COUNT.getValue(stack)); + var.setWriteRefCount(count); + return null; + } +} Modified: trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java 2012-11-23 08:44:21 UTC (rev 6076) +++ trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java 2012-11-23 08:56:28 UTC (rev 6077) @@ -21,6 +21,7 @@ import java.util.ArrayList; import java.util.Iterator; import java.util.List; +import java.util.StringTokenizer; import org.apache.log4j.Logger; import org.globus.cog.karajan.arguments.Arg; @@ -43,17 +44,17 @@ import org.griphyn.vdl.util.VDL2Config; public class ThrottledParallelFor extends AbstractParallelIterator { - public static final Logger logger = Logger - .getLogger(ThrottledParallelFor.class); + public static final Logger logger = Logger.getLogger(ThrottledParallelFor.class); public static final int DEFAULT_MAX_THREADS = 1024; public static final Arg A_NAME = new Arg.Positional("name"); public static final Arg A_IN = new Arg.Positional("in"); - public static final Arg A_SELF_CLOSE = new Arg.Optional("selfclose", Boolean.FALSE); + public static final Arg O_SELFCLOSE = new Arg.Optional("selfclose", Boolean.FALSE); + public static final Arg O_REFS = new Arg.Optional("refs", null); static { - setArguments(ThrottledParallelFor.class, new Arg[] { A_NAME, A_IN, A_SELF_CLOSE }); + setArguments(ThrottledParallelFor.class, new Arg[] { A_NAME, A_IN, O_SELFCLOSE, O_REFS }); } public static final String THREAD_COUNT = "#threadcount"; @@ -61,7 +62,36 @@ private int maxThreadCount = -1; private Tracer forTracer, iterationTracer; private String kvar, vvar; + private List srefs; + private static class StaticRefCount { + public final String name; + public final int count; + + public StaticRefCount(String name, int count) { + this.name = name; + this.count = count; + } + } + + private static class RefCount { + public final DSHandle var; + public final int count; + + public RefCount(DSHandle var, int count) { + this.var = var; + this.count = count; + } + + public void inc() { + + } + + public void dec() { + + } + } + @Override protected void initializeStatic() { super.initializeStatic(); @@ -69,11 +99,44 @@ iterationTracer = Tracer.getTracer(this, "ITERATION"); kvar = (String) getProperty("_kvar"); vvar = (String) getProperty("_vvar"); + srefs = buildStaticRefs(); } + private List buildStaticRefs() { + String refs = (String) O_REFS.getStatic(this); + if (refs == null) { + return null; + } + List l = new ArrayList(); + String name = null; + boolean flip = true; + StringTokenizer st = new StringTokenizer(refs); + while (st.hasMoreTokens()) { + if (flip) { + name = st.nextToken(); + } + else { + int count = Integer.parseInt(st.nextToken()); + l.add(new StaticRefCount(name, count)); + } + flip = !flip; + } + return l; + } + + private List buildRefs(VariableStack stack) throws VariableNotFoundException { + if (srefs == null) { + return null; + } + List l = new ArrayList(srefs.size()); + for (StaticRefCount s : srefs) { + l.add(new RefCount((DSHandle) stack.getVar(s.name), s.count)); + } + return l; + } + protected void partialArgumentsEvaluated(VariableStack stack) throws ExecutionException { - stack.setVar("selfclose", A_SELF_CLOSE.getValue(stack)); if (forTracer.isEnabled()) { forTracer.trace(ThreadingContext.get(stack).toString()); } @@ -89,8 +152,7 @@ stack.setVar(VAR, var); setChildFailed(stack, false); stack.setCaller(this); - initThreadCount(stack, TypeUtil.toBoolean(stack.currentFrame().getVar("selfclose")), i); - stack.currentFrame().deleteVar("selfclose"); + initThreadCount(stack, TypeUtil.toBoolean(O_SELFCLOSE.getStatic(this)), i); citerate(stack, var, i); } else { @@ -110,35 +172,18 @@ int j = 0; try { for (; j < available && i.hasNext(); j++) { - VariableStack copy = stack.copy(); - copy.enter(); - ThreadingContext ntc = ThreadingContext.get(copy).split(i.current()); - ThreadingContext.set(copy, ntc); - setIndex(copy, getArgCount()); - setArgsDone(copy); - Object value = i.next(); - if (iterationTracer.isEnabled()) { - iterationTracer.trace(ntc.toString(), unwrap(value)); - } - copy.setVar(var.getName(), value); - startElement(getArgCount(), copy); + startIteration(tc, var, i.current(), i.next(), stack); } } finally { tc.add(j); } while (i.hasNext()) { - Object value = tc.tryIncrement(); - VariableStack copy = stack.copy(); - copy.enter(); - ThreadingContext.set(copy, ThreadingContext.get(copy).split( - i.current())); - setIndex(copy, getArgCount()); - setArgsDone(copy); - copy.setVar(var.getName(), value); - startElement(getArgCount(), copy); + startIteration(tc, var, i.current(), tc.tryIncrement(), stack); } + decRefs(tc.rc); + int left; synchronized(tc) { // can only have closed and running = 0 in one place @@ -157,7 +202,38 @@ } } - private Object unwrap(Object value) { + private void startIteration(ThreadCount tc, Identifier var, int id, Object value, + VariableStack stack) throws ExecutionException { + incRefs(tc.rc); + VariableStack copy = stack.copy(); + copy.enter(); + ThreadingContext ntc = ThreadingContext.get(copy).split(id); + ThreadingContext.set(copy, ntc); + setIndex(copy, 2); + if (iterationTracer.isEnabled()) { + iterationTracer.trace(ntc.toString(), unwrap(value)); + } + copy.setVar(var.getName(), value); + startElement(1, copy); + } + + private void decRefs(List rcs) throws ExecutionException { + if (rcs != null) { + for (RefCount rc : rcs) { + rc.var.updateWriteRefCount(-rc.count); + } + } + } + + private void incRefs(List rcs) throws ExecutionException { + if (rcs != null) { + for (RefCount rc : rcs) { + rc.var.updateWriteRefCount(rc.count); + } + } + } + + private Object unwrap(Object value) { if (value instanceof Pair) { Pair p = (Pair) value; if (kvar != null) { @@ -171,6 +247,28 @@ return "!"; } } + + @Override + public void completed(VariableStack stack) throws ExecutionException { + int index = preIncIndex(stack) - 1; + if (index == 1) { + // iterator + stack.currentFrame().deleteVar(QUOTED); + processArguments(stack); + try { + partialArgumentsEvaluated(stack); + } + catch (FutureFault e) { + e.getFuture().addModificationAction(new PartialResume(), stack); + } + } + else if (index == elementCount()) { + iterationCompleted(stack); + } + else { + startElement(index, stack); + } + } public void failed(VariableStack stack, ExecutionException e) throws ExecutionException { if (!testAndSetChildFailed(stack)) { @@ -191,16 +289,21 @@ synchronized(tc) { closed = tc.isClosed(); running = tc.decrement(); - iteratorHasValues = !tc.selfClose || tc.iteratorHasValues(); + iteratorHasValues = tc.iteratorHasValues(); } + boolean done = false; if (running == 0) { - if (closed || !iteratorHasValues) { + if (closed) { complete(stack); } + if (tc.selfClose && !iteratorHasValues) { + decRefs(tc.rc); + complete(stack); + } } } - private void initThreadCount(VariableStack stack, boolean selfClose, KarajanIterator i) { + private void initThreadCount(VariableStack stack, boolean selfClose, KarajanIterator i) throws VariableNotFoundException { if (maxThreadCount < 0) { try { maxThreadCount = TypeUtil.toInt(VDL2Config.getConfig() @@ -210,7 +313,7 @@ maxThreadCount = DEFAULT_MAX_THREADS; } } - stack.setVar(THREAD_COUNT, new ThreadCount(maxThreadCount, selfClose, i)); + stack.setVar(THREAD_COUNT, new ThreadCount(maxThreadCount, selfClose, i, buildRefs(stack))); } private ThreadCount getThreadCount(VariableStack stack) @@ -224,20 +327,27 @@ } private static class ThreadCount implements FutureIterator { - private int maxThreadCount; + public boolean selfClose; + private int maxThreadCount; private int crt; - private boolean selfClose, closed; + private boolean closed; private List listeners; private KarajanIterator i; + private final List rc; - public ThreadCount(int maxThreadCount, boolean selfClose, KarajanIterator i) { + public ThreadCount(int maxThreadCount, boolean selfClose, KarajanIterator i, List rc) { this.maxThreadCount = maxThreadCount; - this.selfClose = selfClose; this.i = i; crt = 0; + this.selfClose = selfClose; + this.rc = rc; } - public boolean iteratorHasValues() { + public boolean raiseWaiting() { + return false; + } + + public boolean iteratorHasValues() { try { return i.hasNext(); } @@ -245,10 +355,6 @@ return false; } } - - public boolean isSelfClose() { - return selfClose; - } public synchronized int available() { return maxThreadCount - crt; Modified: trunk/src/org/griphyn/vdl/mapping/AbstractDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/AbstractDataNode.java 2012-11-23 08:44:21 UTC (rev 6076) +++ trunk/src/org/griphyn/vdl/mapping/AbstractDataNode.java 2012-11-23 08:56:28 UTC (rev 6077) @@ -29,6 +29,7 @@ import java.util.Map; import org.apache.log4j.Logger; +import org.globus.cog.karajan.workflow.ExecutionException; import org.globus.cog.karajan.workflow.futures.Future; import org.globus.cog.karajan.workflow.futures.FutureNotYetAvailable; import org.griphyn.vdl.karajan.DSHandleFutureWrapper; @@ -84,6 +85,7 @@ private Path pathFromRoot; protected FutureWrapper wrapper; + private int writeRefCount; protected AbstractDataNode(Field field) { this.field = field; @@ -681,4 +683,30 @@ value = null; pathFromRoot = null; } + + @Override + public synchronized void setWriteRefCount(int count) { + this.writeRefCount = count; + } + + @Override + public synchronized int updateWriteRefCount(int delta) { + this.writeRefCount += delta; + + if (this.writeRefCount < 0) { + throw new IllegalArgumentException("Reference count mismatch for " + this + ". Count is " + this.writeRefCount); + } + + if (logger.isDebugEnabled()) { + logger.debug(this + " writeRefCount " + this.writeRefCount); + } + if (this.writeRefCount == 0) { + if(logger.isInfoEnabled()) { + logger.info("All partial closes for " + this + + " have happened. Closing fully."); + } + closeDeep(); + } + return this.writeRefCount; + } } Modified: trunk/src/org/griphyn/vdl/mapping/DSHandle.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/DSHandle.java 2012-11-23 08:44:21 UTC (rev 6076) +++ trunk/src/org/griphyn/vdl/mapping/DSHandle.java 2012-11-23 08:56:28 UTC (rev 6077) @@ -103,4 +103,8 @@ public String getIdentifyingString(); public boolean isRestartable(); + + public void setWriteRefCount(int count); + + public int updateWriteRefCount(int delta); } Modified: trunk/src/org/griphyn/vdl/mapping/MappingParam.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/MappingParam.java 2012-11-23 08:44:21 UTC (rev 6076) +++ trunk/src/org/griphyn/vdl/mapping/MappingParam.java 2012-11-23 08:56:28 UTC (rev 6077) @@ -29,7 +29,6 @@ public static final MappingParam SWIFT_LINE = new MappingParam("swift#line"); public static final MappingParam SWIFT_DBGNAME = new MappingParam("swift#dbgname"); public static final MappingParam SWIFT_RESTARTID = new MappingParam("swift#restartid"); - public static final MappingParam SWIFT_WAITFOR = new MappingParam("swift#waitfor"); public static final MappingParam SWIFT_BASEDIR = new MappingParam("swift#basedir"); public static final MappingParam SWIFT_DESCRIPTOR = new MappingParam("swift#descriptor"); public static final MappingParam SWIFT_INPUT = new MappingParam("swift#input"); Modified: trunk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java 2012-11-23 08:44:21 UTC (rev 6076) +++ trunk/src/org/griphyn/vdl/mapping/RootArrayDataNode.java 2012-11-23 08:56:28 UTC (rev 6077) @@ -77,7 +77,12 @@ String desc = (String) params.get(MappingParam.SWIFT_DESCRIPTOR); if (desc == null) { - initialized(); + initialized(); + Boolean input = (Boolean) params.get(MappingParam.SWIFT_INPUT); + if (input != null && input.booleanValue()) { + // Empty array. Clearly only in test cases. + closeDeep(); + } return; } try { Modified: trunk/src/org/griphyn/vdl/mapping/RootDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/RootDataNode.java 2012-11-23 08:44:21 UTC (rev 6076) +++ trunk/src/org/griphyn/vdl/mapping/RootDataNode.java 2012-11-23 08:56:28 UTC (rev 6077) @@ -120,8 +120,8 @@ static protected void checkInputs(MappingParamSet params, Mapper mapper, AbstractDataNode root, DuplicateMappingChecker dmc) { - String input = (String) params.get(MappingParam.SWIFT_INPUT); - if (input != null && Boolean.valueOf(input.trim()).booleanValue()) { + Boolean input = (Boolean) params.get(MappingParam.SWIFT_INPUT); + if (input != null && input.booleanValue()) { addExisting(mapper, root); checkConsistency(root, true, mapper, dmc); } From swift at ci.uchicago.edu Fri Nov 23 03:10:05 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Fri, 23 Nov 2012 03:10:05 -0600 (CST) Subject: [Swift-commit] cog r3520 Message-ID: <20121123091005.E914E8D0007A@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3520 | hategan | 2012-11-23 03:07:10 -0600 (Fri, 23 Nov 2012) | 1 line removed gt4 from dependencides list ------------------------------------------------------------------------ Index: modules/abstraction/dependencies.xml =================================================================== --- modules/abstraction/dependencies.xml (revision 3519) +++ modules/abstraction/dependencies.xml (working copy) @@ -11,9 +11,6 @@ --> - - - From hategan at ci.uchicago.edu Fri Nov 23 03:24:03 2012 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 23 Nov 2012 03:24:03 -0600 (CST) Subject: [Swift-commit] r6080 - trunk/src/org/griphyn/vdl/engine Message-ID: <20121123092403.6AF709CC9C@svn.ci.uchicago.edu> Author: hategan Date: 2012-11-23 03:24:02 -0600 (Fri, 23 Nov 2012) New Revision: 6080 Modified: trunk/src/org/griphyn/vdl/engine/VariableScope.java Log: oops, I meant PARTIAL Modified: trunk/src/org/griphyn/vdl/engine/VariableScope.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/VariableScope.java 2012-11-23 09:00:46 UTC (rev 6079) +++ trunk/src/org/griphyn/vdl/engine/VariableScope.java 2012-11-23 09:24:02 UTC (rev 6080) @@ -479,7 +479,7 @@ } switch (enclosureType) { case CONDITION: - if (parentScope.isVariableWriteable(name, WriteType.FULL)) { + if (parentScope.isVariableWriteable(name, WriteType.PARTIAL)) { if (getTopmostLoopToDeclaration(name) != null) { Warnings.warn("Variable " + name + ", defined on line " + getDeclarationLine(name) + ", might have multiple conflicting writers"); From davidk at ci.uchicago.edu Mon Nov 26 12:00:22 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Mon, 26 Nov 2012 12:00:22 -0600 (CST) Subject: [Swift-commit] r6081 - branches/release-0.94/libexec Message-ID: <20121126180022.E941E9CCB6@svn.ci.uchicago.edu> Author: davidk Date: 2012-11-26 12:00:22 -0600 (Mon, 26 Nov 2012) New Revision: 6081 Modified: branches/release-0.94/libexec/vdl-int-staging.k Log: Fix for syntax error in vdl-int-staging.k Modified: branches/release-0.94/libexec/vdl-int-staging.k =================================================================== --- branches/release-0.94/libexec/vdl-int-staging.k 2012-11-23 09:24:02 UTC (rev 6080) +++ branches/release-0.94/libexec/vdl-int-staging.k 2012-11-26 18:00:22 UTC (rev 6081) @@ -182,7 +182,7 @@ "Exception in {tr}:", nl(), maybe(" Arguments: {arguments}", nl()), " Host: {rhost}", nl(), - " Directory: {tmpdir}", nl(), + " Directory: {tmpdir}", nl() ) exception ) From swift at ci.uchicago.edu Mon Nov 26 14:05:19 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Mon, 26 Nov 2012 14:05:19 -0600 (CST) Subject: [Swift-commit] cog r3521 Message-ID: <20121126200519.869338D0008D@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3521 | davidkelly999 | 2012-11-26 14:01:04 -0600 (Mon, 26 Nov 2012) | 2 lines Avoid using duplicate ports with multiple coaster-service's running, bug #881 ------------------------------------------------------------------------ Index: modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/CoasterPersistentService.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/CoasterPersistentService.java (revision 3520) +++ modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/CoasterPersistentService.java (working copy) @@ -108,7 +108,7 @@ cred = new GlobusGSSCredentialImpl(gc, GSSCredential.INITIATE_AND_ACCEPT); } - int port = 1984; + int port = 0; if (ap.hasValue("port")) { if (ap.hasValue("portfile")) { throw new ArgumentParserException("-portfile (-S) and -port are mutually exclusive"); Index: modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/LocalTCPService.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/LocalTCPService.java (revision 3520) +++ modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/LocalTCPService.java (working copy) @@ -96,22 +96,41 @@ public ServiceContext getContext() { return context; } - - public void start() { + + public boolean bindPort() { try { - channel = ServerSocketChannel.open(); - channel.configureBlocking(true); - if(port == 0) { - PortRange portRange = PortRange.getTcpInstance(); - if(portRange != null && portRange.isEnabled()) { - synchronized(portRange) { - port = portRange.getFreePort(port); - portRange.setUsed(port); - } - } + channel.socket().bind(new InetSocketAddress(port)); + return true; + } + catch (Exception e) { + return false; + } + } + + public void start() throws IOException { + channel = ServerSocketChannel.open(); + channel.configureBlocking(true); + + /* When GLOBUS_TCP_PORT_RANGE is defined, find an acceptable port in that range */ + String globusTCPPortRange = System.getenv("GLOBUS_TCP_PORT_RANGE"); + String rangeValues[] = {"0", "0"}; + + if(globusTCPPortRange != null) { + rangeValues = globusTCPPortRange.split(","); + } + + port = Integer.valueOf(rangeValues[0]); + while(!bindPort()) { + port++; + if(port > Integer.valueOf(rangeValues[1])) { + String msg = "Unable to find an available port"; + if(globusTCPPortRange != null) + msg += " in the range of " + rangeValues[0] + " to " + rangeValues[1]; + throw new IOException(msg); } - channel.socket().bind(new InetSocketAddress(port)); - + } + + try { if (serverThread == null) { serverThread = new Thread(this); serverThread.setDaemon(true); From swift at ci.uchicago.edu Mon Nov 26 14:10:13 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Mon, 26 Nov 2012 14:10:13 -0600 (CST) Subject: [Swift-commit] cog r3522 Message-ID: <20121126201013.A43028D0008D@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3522 | davidkelly999 | 2012-11-26 14:09:53 -0600 (Mon, 26 Nov 2012) | 2 lines Avoid using duplicate ports with multiple coaster-service's running, bug #881 ------------------------------------------------------------------------ Index: modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/CoasterPersistentService.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/CoasterPersistentService.java (revision 3521) +++ modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/CoasterPersistentService.java (working copy) @@ -108,7 +108,7 @@ cred = new GlobusGSSCredentialImpl(gc, GSSCredential.INITIATE_AND_ACCEPT); } - int port = 1984; + int port = 0; if (ap.hasValue("port")) { if (ap.hasValue("portfile")) { throw new ArgumentParserException("-portfile (-S) and -port are mutually exclusive"); Index: modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/LocalTCPService.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/LocalTCPService.java (revision 3521) +++ modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/LocalTCPService.java (working copy) @@ -96,22 +96,41 @@ public ServiceContext getContext() { return context; } - - public void start() { + + public boolean bindPort() { try { - channel = ServerSocketChannel.open(); - channel.configureBlocking(true); - if(port == 0) { - PortRange portRange = PortRange.getTcpInstance(); - if(portRange != null && portRange.isEnabled()) { - synchronized(portRange) { - port = portRange.getFreePort(port); - portRange.setUsed(port); - } - } + channel.socket().bind(new InetSocketAddress(port)); + return true; + } + catch (Exception e) { + return false; + } + } + + public void start() throws IOException { + channel = ServerSocketChannel.open(); + channel.configureBlocking(true); + + /* When GLOBUS_TCP_PORT_RANGE is defined, find an acceptable port in that range */ + String globusTCPPortRange = System.getenv("GLOBUS_TCP_PORT_RANGE"); + String rangeValues[] = {"0", "0"}; + + if(globusTCPPortRange != null) { + rangeValues = globusTCPPortRange.split(","); + } + + port = Integer.valueOf(rangeValues[0]); + while(!bindPort()) { + port++; + if(port > Integer.valueOf(rangeValues[1])) { + String msg = "Unable to find an available port"; + if(globusTCPPortRange != null) + msg += " in the range of " + rangeValues[0] + " to " + rangeValues[1]; + throw new IOException(msg); } - channel.socket().bind(new InetSocketAddress(port)); - + } + + try { if (serverThread == null) { serverThread = new Thread(this); serverThread.setDaemon(true); From wilde at ci.uchicago.edu Tue Nov 27 09:59:26 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Tue, 27 Nov 2012 09:59:26 -0600 (CST) Subject: [Swift-commit] r6082 - trunk/bin Message-ID: <20121127155927.0EE30178884@svn.ci.uchicago.edu> Author: wilde Date: 2012-11-27 09:59:25 -0600 (Tue, 27 Nov 2012) New Revision: 6082 Modified: trunk/bin/slurm-qsub trunk/bin/slurm-qsub-emulator Log: Adjust slurm pbs emulation code to honor advres resource attribute to pass reservation IDs. Modified: trunk/bin/slurm-qsub =================================================================== --- trunk/bin/slurm-qsub 2012-11-26 18:00:22 UTC (rev 6081) +++ trunk/bin/slurm-qsub 2012-11-27 15:59:25 UTC (rev 6082) @@ -18,6 +18,8 @@ sub(/nodes=/,"",nodes); sub(/:.*/,"",nodes); sub(/^.*ppn=/,"",ppn); } /#PBS -l walltime/ { walltime = $3; sub(/walltime=/,"",walltime);} +#/#PBS -l advres/ { advres = "," $3; sub(/advres=/,"",advres); advres="," advres } +/#PBS -l advres/ { advres = "," $3; } /#PBS -q/ { queue = $3 } /#PBS -o/ { sofile = $3 } /#PBS -e/ { sefile = $3 } @@ -29,8 +31,8 @@ # printf("-S %s -N %s -m %s -l nodes=%s:ppn=%s -l walltime=%s -q %s -o %s -e %s", # shell, jobname, m, nodes, ppn, walltime, queue, sofile, sefile); - printf("-N %s -m %s -l nodes=%s:ppn=%s,walltime=%s -q %s -o %s -e %s", - jobname, m, nodes, ppn, walltime, queue, sofile, sefile); + printf("-N %s -m %s -l nodes=%s:ppn=%s,walltime=%s%s -q %s -o %s -e %s", + jobname, m, nodes, ppn, walltime, advres, queue, sofile, sefile); }') Modified: trunk/bin/slurm-qsub-emulator =================================================================== --- trunk/bin/slurm-qsub-emulator 2012-11-26 18:00:22 UTC (rev 6081) +++ trunk/bin/slurm-qsub-emulator 2012-11-27 15:59:25 UTC (rev 6082) @@ -189,6 +189,7 @@ $command .= " --tmp=$res_opts{file}" if $res_opts{file}; $command .= " --mem=$res_opts{mem}" if $res_opts{mem}; $command .= " --nice=$res_opts{nice}" if $res_opts{nice}; +$command .= " --reservation=$res_opts{advres}" if $res_opts{advres}; # Cray-specific options $command .= " -n$res_opts{mppwidth}" if $res_opts{mppwidth}; $command .= " -w$res_opts{mppnodes}" if $res_opts{mppnodes}; @@ -235,6 +236,7 @@ 'pvmem' => "", 'software' => "", 'vmem' => "", + 'advres' => "", # Cray-specific resources 'mppwidth' => "", 'mppdepth' => "", @@ -246,9 +248,9 @@ my @keys = keys(%opt); foreach my $key (@keys) { - #print "$rl\n"; - ($opt{$key}) = $rl =~ m/$key=([\w:\+=+]+)/; - + # print "key=$key rl=$rl\n"; + ($opt{$key}) = $rl =~ m/$key=([-\w:\+=+]+)/; + # print " set to $opt{$key}\n"; } if($opt{cput}) { $opt{cput} = get_minutes($opt{cput}); From wilde at ci.uchicago.edu Tue Nov 27 10:01:57 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Tue, 27 Nov 2012 10:01:57 -0600 (CST) Subject: [Swift-commit] r6083 - trunk/libexec Message-ID: <20121127160157.A8FE3178884@svn.ci.uchicago.edu> Author: wilde Date: 2012-11-27 10:01:57 -0600 (Tue, 27 Nov 2012) New Revision: 6083 Modified: trunk/libexec/_swiftwrap Log: Added time command around non-provider-staging non-kickstart app execution. Awlays on - not yet an option. Adds APP_RESOURCES tag to info file via a local file in the job dir. Modified: trunk/libexec/_swiftwrap =================================================================== --- trunk/libexec/_swiftwrap 2012-11-27 15:59:25 UTC (rev 6082) +++ trunk/libexec/_swiftwrap 2012-11-27 16:01:57 UTC (rev 6083) @@ -1,3 +1,4 @@ + #!/bin/bash # this script must be invoked inside of bash, not plain sh # note that this script modifies $IFS @@ -475,17 +476,31 @@ fail 254 "The executable $EXEC does not have the executable bit set" fi if [ "$KICKSTART" == "" ]; then + + TIMEARGS="-o swiftapp.resources -f APP_RESOURCES=real_time:%E,real_secs:%e,kernel_secs:%S,user_secs:%U,percent_cpu:%P,max_rss:%M,avg_rss:%t,avg_tot_vm:%K,avg_priv_data:%D,avg_priv_stack:%p,avg_shared_text:%X,page_size:%Z,major_pgfaults:%F,minor_pgfaults:%R,swaps:%W,invol_context_switches:%c,vol_waits%w,fs_reads:%I,fs_writes:%O,sock_recv:%r,sock_send:%s,signals:%k,exit_status:%x,command:%C" + + if [ -x /usr/bin/time ]; then + TIMECMD="/usr/bin/time" + elif [ -x $HOME/swift.time ]; then + TIMECMD="$HOME/swift.time" + else + TIMECMD="" + TIMEARGS="" + fi if [ "$STDIN" == "" ]; then if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then genScripts fi - "$EXEC" "${CMDARGS[@]}" 1>>"$STDOUT" 2>>"$STDERR" + eval $TIMECMD $TIMEARGS "$EXEC" "${CMDARGS[@]}" 1>>"$STDOUT" 2>>"$STDERR" else if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then genScripts fi - "$EXEC" "${CMDARGS[@]}" 1>>"$STDOUT" 2>>"$STDERR" <"$STDIN" + eval $TIMECMD $TIMEARGS "$EXEC" "${CMDARGS[@]}" 1>>"$STDOUT" 2>>"$STDERR" <"$STDIN" fi + if [ "_$TIMECMD" != _ ]; then + log "$(cat swiftapp.resources)" + fi checkError $? "Application $EXEC failed with an exit code of $?" else if [ ! -f "$KICKSTART" ]; then From wozniak at ci.uchicago.edu Tue Nov 27 11:22:25 2012 From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu) Date: Tue, 27 Nov 2012 11:22:25 -0600 (CST) Subject: [Swift-commit] r6084 - / Message-ID: <20121127172225.1A1BA178884@svn.ci.uchicago.edu> Author: wozniak Date: 2012-11-27 11:22:24 -0600 (Tue, 27 Nov 2012) New Revision: 6084 Added: SwiftTutorials/ Log: Tutorials From wozniak at ci.uchicago.edu Tue Nov 27 11:23:05 2012 From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu) Date: Tue, 27 Nov 2012 11:23:05 -0600 (CST) Subject: [Swift-commit] r6085 - SwiftTutorials Message-ID: <20121127172305.BA6FC178884@svn.ci.uchicago.edu> Author: wozniak Date: 2012-11-27 11:23:05 -0600 (Tue, 27 Nov 2012) New Revision: 6085 Added: SwiftTutorials/UofC_2012-11-27/ Log: Swift tutorial at UofC, 2012/11/27 From wozniak at ci.uchicago.edu Tue Nov 27 11:24:46 2012 From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu) Date: Tue, 27 Nov 2012 11:24:46 -0600 (CST) Subject: [Swift-commit] r6086 - in SwiftTutorials/UofC_2012-11-27: . bin data scripts Message-ID: <20121127172446.A62D4178884@svn.ci.uchicago.edu> Author: wozniak Date: 2012-11-27 11:24:46 -0600 (Tue, 27 Nov 2012) New Revision: 6086 Added: SwiftTutorials/UofC_2012-11-27/README SwiftTutorials/UofC_2012-11-27/bin/ SwiftTutorials/UofC_2012-11-27/bin/cleanup SwiftTutorials/UofC_2012-11-27/bin/filterinfo.sh SwiftTutorials/UofC_2012-11-27/bin/genapplist SwiftTutorials/UofC_2012-11-27/bin/getinfo.sh SwiftTutorials/UofC_2012-11-27/bin/myapp SwiftTutorials/UofC_2012-11-27/bin/runswift SwiftTutorials/UofC_2012-11-27/bin/setup.sh SwiftTutorials/UofC_2012-11-27/bin/suminfo.sh SwiftTutorials/UofC_2012-11-27/data/ SwiftTutorials/UofC_2012-11-27/data/data.txt SwiftTutorials/UofC_2012-11-27/scripts/ SwiftTutorials/UofC_2012-11-27/scripts/catsn.swift SwiftTutorials/UofC_2012-11-27/scripts/script01.swift SwiftTutorials/UofC_2012-11-27/scripts/script02.swift SwiftTutorials/UofC_2012-11-27/scripts/script03.swift SwiftTutorials/UofC_2012-11-27/scripts/script04.swift SwiftTutorials/UofC_2012-11-27/scripts/script05.swift Log: Import tutorial content Added: SwiftTutorials/UofC_2012-11-27/README =================================================================== --- SwiftTutorials/UofC_2012-11-27/README (rev 0) +++ SwiftTutorials/UofC_2012-11-27/README 2012-11-27 17:24:46 UTC (rev 6086) @@ -0,0 +1,6 @@ + +Introduction to Swift + + +Exercise 1 - Run a simple command under Swift + Added: SwiftTutorials/UofC_2012-11-27/bin/cleanup =================================================================== --- SwiftTutorials/UofC_2012-11-27/bin/cleanup (rev 0) +++ SwiftTutorials/UofC_2012-11-27/bin/cleanup 2012-11-27 17:24:46 UTC (rev 6086) @@ -0,0 +1,17 @@ +#! /bin/sh + +echo Removing: +echo +ls -ltd script??-* swiftwork _concurrent output *.kml *.swiftx swift.log applist config sites.xml pathlist 2>/dev/null +echo +echo Enter y to proceed: +read ok +if [ _$ok = _y -o _$ok = _Y -o _$ok = yes ]; then + echo + set -x + rm -rf script??-* swiftwork _concurrent output *.kml *.swiftx swift.log applist config sites.xml pathlist +else + echo + echo No files removed. +fi + Property changes on: SwiftTutorials/UofC_2012-11-27/bin/cleanup ___________________________________________________________________ Added: svn:executable + * Added: SwiftTutorials/UofC_2012-11-27/bin/filterinfo.sh =================================================================== --- SwiftTutorials/UofC_2012-11-27/bin/filterinfo.sh (rev 0) +++ SwiftTutorials/UofC_2012-11-27/bin/filterinfo.sh 2012-11-27 17:24:46 UTC (rev 6086) @@ -0,0 +1,2 @@ +#! /bin/sh +grep getinfo.sh $1 | wc -l >$2 Property changes on: SwiftTutorials/UofC_2012-11-27/bin/filterinfo.sh ___________________________________________________________________ Added: svn:executable + * Added: SwiftTutorials/UofC_2012-11-27/bin/genapplist =================================================================== --- SwiftTutorials/UofC_2012-11-27/bin/genapplist (rev 0) +++ SwiftTutorials/UofC_2012-11-27/bin/genapplist 2012-11-27 17:24:46 UTC (rev 6086) @@ -0,0 +1,7 @@ +cat <pathlist +rule .*_concurrent/.* DIRECT $PWD/tmp +rule .* DIRECT / +END + +cat <config +status.mode=provider +use.provider.staging=false +use.wrapper.staging=false +wrapperlog.always.transfer=true +execution.retries=0 +lazy.errors=false +provider.staging.pin.swiftfiles=false +sitedir.keep=true +file.gc.enabled=false +tcp.port.range=50000,51000 +END + +reservationID=${reservationID:-swift} + +cat <sites.xml + + + + + + + 0.03 + 10000 + + $PWD/swiftwork + + + + + + + + 12 + 12 + 1 + 1 + 1 + westmere + + slurm.resource_list=advres=$reservationID + + 5.99 + 10000 + + 100 + 100 + 18000 + 02:00:00 + + + $PWD/swiftwork + + + + + + + + 1 + 12 + 1 + 1 + 1 + single + westmere + + slurm.resource_list=advres=$reservationID + + 0.64 + 10000 + + 100 + 100 + 9000 + 01:00:00 + + + $PWD/swiftwork + + + + + +END + +genapplist >applist + +set -x +#swift -config config -tc.file applist -sites.file sites.xml -cdm.file pathlist $* +swift -config config -tc.file applist -sites.file sites.xml $* + + + + Property changes on: SwiftTutorials/UofC_2012-11-27/bin/runswift ___________________________________________________________________ Added: svn:executable + * Added: SwiftTutorials/UofC_2012-11-27/bin/setup.sh =================================================================== --- SwiftTutorials/UofC_2012-11-27/bin/setup.sh (rev 0) +++ SwiftTutorials/UofC_2012-11-27/bin/setup.sh 2012-11-27 17:24:46 UTC (rev 6086) @@ -0,0 +1 @@ +PATH=/home/wilde/swift/rev/swift-training/bin:$PWD/bin:$PATH Added: SwiftTutorials/UofC_2012-11-27/bin/suminfo.sh =================================================================== --- SwiftTutorials/UofC_2012-11-27/bin/suminfo.sh (rev 0) +++ SwiftTutorials/UofC_2012-11-27/bin/suminfo.sh 2012-11-27 17:24:46 UTC (rev 6086) @@ -0,0 +1,2 @@ +#! /bin/sh +awk '{ sum += $1 } END { print "Total processes observed: " sum }' $* Property changes on: SwiftTutorials/UofC_2012-11-27/bin/suminfo.sh ___________________________________________________________________ Added: svn:executable + * Added: SwiftTutorials/UofC_2012-11-27/data/data.txt =================================================================== --- SwiftTutorials/UofC_2012-11-27/data/data.txt (rev 0) +++ SwiftTutorials/UofC_2012-11-27/data/data.txt 2012-11-27 17:24:46 UTC (rev 6086) @@ -0,0 +1 @@ +Hi Swift! Added: SwiftTutorials/UofC_2012-11-27/scripts/catsn.swift =================================================================== --- SwiftTutorials/UofC_2012-11-27/scripts/catsn.swift (rev 0) +++ SwiftTutorials/UofC_2012-11-27/scripts/catsn.swift 2012-11-27 17:24:46 UTC (rev 6086) @@ -0,0 +1,12 @@ +type file; + +app (file o) cat (file i) +{ + cat @i stdout=@o; +} + +file out[]; +foreach j in [1:@toint(@arg("n","1"))] { + file data<"data.txt">; + out[j] = cat(data); +} Added: SwiftTutorials/UofC_2012-11-27/scripts/script01.swift =================================================================== --- SwiftTutorials/UofC_2012-11-27/scripts/script01.swift (rev 0) +++ SwiftTutorials/UofC_2012-11-27/scripts/script01.swift 2012-11-27 17:24:46 UTC (rev 6086) @@ -0,0 +1,9 @@ +type file; + +app (file o) getinfo () +{ + getinfo stdout=@o; +} + +file info <"output/info.txt">; +info = getinfo(); Added: SwiftTutorials/UofC_2012-11-27/scripts/script02.swift =================================================================== --- SwiftTutorials/UofC_2012-11-27/scripts/script02.swift (rev 0) +++ SwiftTutorials/UofC_2012-11-27/scripts/script02.swift 2012-11-27 17:24:46 UTC (rev 6086) @@ -0,0 +1,13 @@ +type file; + +app (file o) getinfo () +{ + getinfo stdout=@o; +} + +foreach i in [0:24] { + string fname = @strcat("output/info.",i,".txt"); + file info ; + info = getinfo(); +} + Added: SwiftTutorials/UofC_2012-11-27/scripts/script03.swift =================================================================== --- SwiftTutorials/UofC_2012-11-27/scripts/script03.swift (rev 0) +++ SwiftTutorials/UofC_2012-11-27/scripts/script03.swift 2012-11-27 17:24:46 UTC (rev 6086) @@ -0,0 +1,19 @@ +type file; + +app (file o) getinfo () +{ + getinfo stdout=@o; +} + +app (file o) filterinfo (file i) +{ + filterinfo @i @o; +} + +foreach i in [0:24] { + string fname = @strcat("output/info.",i,".txt"); + file info ; + info = getinfo(); + file filtered = filterinfo(info); +} + Added: SwiftTutorials/UofC_2012-11-27/scripts/script04.swift =================================================================== --- SwiftTutorials/UofC_2012-11-27/scripts/script04.swift (rev 0) +++ SwiftTutorials/UofC_2012-11-27/scripts/script04.swift 2012-11-27 17:24:46 UTC (rev 6086) @@ -0,0 +1,25 @@ +type file; + +app (file o) getinfo () +{ + getinfo stdout=@o; +} + +app (file o) filter (file i) +{ + filterinfo @i @o; +} + +app (file o) reduce (file i[]) + reduceinfo @filenames(i) +} + +file filtered[]; + +foreach i in [0:24] { + string fname = @strcat("info.",i,".txt"); + file info ; + info = getinfo(); + filtered[i] = filterinfo(info); +} + Added: SwiftTutorials/UofC_2012-11-27/scripts/script05.swift =================================================================== --- SwiftTutorials/UofC_2012-11-27/scripts/script05.swift (rev 0) +++ SwiftTutorials/UofC_2012-11-27/scripts/script05.swift 2012-11-27 17:24:46 UTC (rev 6086) @@ -0,0 +1,30 @@ +type file; + +app (file o) getinfo () +{ + getinfo stdout=@o; +} + +app (file o) filter (file i) +{ + filterinfo @i @o; +} + +app (file o) reduce (file i[]) +{ + suminfo @filenames(i) stdout=@o; +} + +file filtered[]; + +foreach i in [0:24] { + string fname = @strcat("info.",i,".txt"); + file info ; + info = getinfo(); + filtered[i] = filter(info); +} + +file sum<"sum.data">; + +sum = reduce(filtered); + From wilde at ci.uchicago.edu Tue Nov 27 11:36:20 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Tue, 27 Nov 2012 11:36:20 -0600 (CST) Subject: [Swift-commit] r6087 - SwiftTutorials/UofC_2012-11-27/bin Message-ID: <20121127173620.9D6C6178884@svn.ci.uchicago.edu> Author: wilde Date: 2012-11-27 11:36:20 -0600 (Tue, 27 Nov 2012) New Revision: 6087 Modified: SwiftTutorials/UofC_2012-11-27/bin/runswift Log: Comment out reservation for testing. Modified: SwiftTutorials/UofC_2012-11-27/bin/runswift =================================================================== --- SwiftTutorials/UofC_2012-11-27/bin/runswift 2012-11-27 17:24:46 UTC (rev 6086) +++ SwiftTutorials/UofC_2012-11-27/bin/runswift 2012-11-27 17:36:20 UTC (rev 6087) @@ -45,7 +45,7 @@ 1 westmere - slurm.resource_list=advres=$reservationID + 5.99 10000 From wilde at ci.uchicago.edu Tue Nov 27 11:46:44 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Tue, 27 Nov 2012 11:46:44 -0600 (CST) Subject: [Swift-commit] r6088 - SwiftTutorials/UofC_2012-11-27/scripts Message-ID: <20121127174644.EEF8E178884@svn.ci.uchicago.edu> Author: wilde Date: 2012-11-27 11:46:44 -0600 (Tue, 27 Nov 2012) New Revision: 6088 Modified: SwiftTutorials/UofC_2012-11-27/scripts/script04.swift SwiftTutorials/UofC_2012-11-27/scripts/script05.swift Log: adjust scripts 4 and 5. Modified: SwiftTutorials/UofC_2012-11-27/scripts/script04.swift =================================================================== --- SwiftTutorials/UofC_2012-11-27/scripts/script04.swift 2012-11-27 17:36:20 UTC (rev 6087) +++ SwiftTutorials/UofC_2012-11-27/scripts/script04.swift 2012-11-27 17:46:44 UTC (rev 6088) @@ -10,16 +10,16 @@ filterinfo @i @o; } -app (file o) reduce (file i[]) - reduceinfo @filenames(i) -} - file filtered[]; foreach i in [0:24] { - string fname = @strcat("info.",i,".txt"); + string fname = @strcat("output/info.",i,".txt"); file info ; info = getinfo(); - filtered[i] = filterinfo(info); + // replace: filtered[i] = filter(info); + // with: + string filfname = ??? + filfname = + filtered[i] = } Modified: SwiftTutorials/UofC_2012-11-27/scripts/script05.swift =================================================================== --- SwiftTutorials/UofC_2012-11-27/scripts/script05.swift 2012-11-27 17:36:20 UTC (rev 6087) +++ SwiftTutorials/UofC_2012-11-27/scripts/script05.swift 2012-11-27 17:46:44 UTC (rev 6088) @@ -18,13 +18,13 @@ file filtered[]; foreach i in [0:24] { - string fname = @strcat("info.",i,".txt"); + string fname = @strcat("output/info.",i,".txt"); file info ; info = getinfo(); filtered[i] = filter(info); } -file sum<"sum.data">; +file sum<"output/sum.data">; sum = reduce(filtered); From wozniak at ci.uchicago.edu Tue Nov 27 11:49:13 2012 From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu) Date: Tue, 27 Nov 2012 11:49:13 -0600 (CST) Subject: [Swift-commit] r6089 - SwiftTutorials/UofC_2012-11-27/bin Message-ID: <20121127174913.B8218178884@svn.ci.uchicago.edu> Author: wozniak Date: 2012-11-27 11:49:13 -0600 (Tue, 27 Nov 2012) New Revision: 6089 Modified: SwiftTutorials/UofC_2012-11-27/bin/cleanup Log: Minor fix Modified: SwiftTutorials/UofC_2012-11-27/bin/cleanup =================================================================== --- SwiftTutorials/UofC_2012-11-27/bin/cleanup 2012-11-27 17:46:44 UTC (rev 6088) +++ SwiftTutorials/UofC_2012-11-27/bin/cleanup 2012-11-27 17:49:13 UTC (rev 6089) @@ -6,7 +6,7 @@ echo echo Enter y to proceed: read ok -if [ _$ok = _y -o _$ok = _Y -o _$ok = yes ]; then +if [ _$ok = _y -o _$ok = _Y -o _$ok = _yes ]; then echo set -x rm -rf script??-* swiftwork _concurrent output *.kml *.swiftx swift.log applist config sites.xml pathlist From wozniak at ci.uchicago.edu Tue Nov 27 11:59:27 2012 From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu) Date: Tue, 27 Nov 2012 11:59:27 -0600 (CST) Subject: [Swift-commit] r6090 - SwiftTutorials/UofC_2012-11-27 Message-ID: <20121127175927.A7ACC178884@svn.ci.uchicago.edu> Author: wozniak Date: 2012-11-27 11:59:27 -0600 (Tue, 27 Nov 2012) New Revision: 6090 Modified: SwiftTutorials/UofC_2012-11-27/README Log: Initial notes for setup and script01 Modified: SwiftTutorials/UofC_2012-11-27/README =================================================================== --- SwiftTutorials/UofC_2012-11-27/README 2012-11-27 17:49:13 UTC (rev 6089) +++ SwiftTutorials/UofC_2012-11-27/README 2012-11-27 17:59:27 UTC (rev 6090) @@ -1,6 +1,44 @@ -Introduction to Swift +INTRODUCTION TO SWIFT +SETUP -Exercise 1 - Run a simple command under Swift +Obtain the tutorial package +Change directory into the created directory called "tutorial" + +Use bin/setup.sh to add tutorial programs to your PATH + +This directory contains: + +bin: script tools for the tutorial +scripts: Swift scripts for the tutorial + +An output directory will be created by the Swift scripts + +Run: + +source bin/setup.sh + +EXERCISE 1 - Run a simple command under Swift + +This script simply runs the getinfo command found in bin/getinfo.sh + +You can run this on the login node by typing: + +bin/getinfo.sh + +Inspect getinfo.sh to see what it does. + +The Swift syntax in scripts/script01.swift directs the output of +getinfo.sh to file output/info.txt + +Run the Swift script by typing: + +runswift scripts/script01.swift + + + +CLEAN UP + +Simply run: bin/cleanup and enter y to confirm From wozniak at ci.uchicago.edu Tue Nov 27 12:03:54 2012 From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu) Date: Tue, 27 Nov 2012 12:03:54 -0600 (CST) Subject: [Swift-commit] r6091 - SwiftTutorials/UofC_2012-11-27 Message-ID: <20121127180354.EBA55178884@svn.ci.uchicago.edu> Author: wozniak Date: 2012-11-27 12:03:54 -0600 (Tue, 27 Nov 2012) New Revision: 6091 Modified: SwiftTutorials/UofC_2012-11-27/README Log: Notes for exercises 2 and 3 Modified: SwiftTutorials/UofC_2012-11-27/README =================================================================== --- SwiftTutorials/UofC_2012-11-27/README 2012-11-27 17:59:27 UTC (rev 6090) +++ SwiftTutorials/UofC_2012-11-27/README 2012-11-27 18:03:54 UTC (rev 6091) @@ -7,8 +7,6 @@ Change directory into the created directory called "tutorial" -Use bin/setup.sh to add tutorial programs to your PATH - This directory contains: bin: script tools for the tutorial @@ -16,6 +14,9 @@ An output directory will be created by the Swift scripts + +Use bin/setup.sh to add tutorial programs to your PATH + Run: source bin/setup.sh @@ -37,8 +38,31 @@ runswift scripts/script01.swift +Check that the output is created in the output directory +EXERCISE 2 - Run parallel jobs +The Swift syntax in scripts/script02.swift runs the getinfo +script 25 times in parallel. Each job produces an output +file with a different name. + +runswift scripts/script02.swift + +Check that the output is created successfully + +EXERCISE 3 - Run parallel jobs with pipelines + +The Swift syntax in scripts/script02.swift runs the getinfo +script 25 times in parallel. Each job produces an output +file with a different name. Each output file is then read by the +filterinfo task, which produces output in _concurrent + +Inspect bin/filterinfo.sh to see what it does to the output of getinfo.sh + +runswift scripts/script03.swift + + + CLEAN UP Simply run: bin/cleanup and enter y to confirm From wozniak at ci.uchicago.edu Tue Nov 27 12:09:37 2012 From: wozniak at ci.uchicago.edu (wozniak at ci.uchicago.edu) Date: Tue, 27 Nov 2012 12:09:37 -0600 (CST) Subject: [Swift-commit] r6092 - SwiftTutorials/UofC_2012-11-27 Message-ID: <20121127180937.C71D1178884@svn.ci.uchicago.edu> Author: wozniak Date: 2012-11-27 12:09:37 -0600 (Tue, 27 Nov 2012) New Revision: 6092 Modified: SwiftTutorials/UofC_2012-11-27/README Log: Notes for exercises 4 and 5 Modified: SwiftTutorials/UofC_2012-11-27/README =================================================================== --- SwiftTutorials/UofC_2012-11-27/README 2012-11-27 18:03:54 UTC (rev 6091) +++ SwiftTutorials/UofC_2012-11-27/README 2012-11-27 18:09:37 UTC (rev 6092) @@ -61,8 +61,21 @@ runswift scripts/script03.swift +EXERCISE 4 - Name files in pipelines +In Exercise 3, the final outputs were named by Swift. +Exercise 4 is similar to 3 but allows you to name the final +outputs. Edit script04.swift to name the files in an appropriate +way. +EXERCISE 5 - Reduction + +Exercise 5 allows you to collect all of the filtered outputs +and summarize them into one file. This Swift script allows +suminfo.sh to read all the files and collect a statistic. + +Inspect bin/suminfo.sh to see what it will do to the files it reads. + CLEAN UP Simply run: bin/cleanup and enter y to confirm From swift at ci.uchicago.edu Tue Nov 27 12:35:15 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Tue, 27 Nov 2012 12:35:15 -0600 (CST) Subject: [Swift-commit] cog r3523 Message-ID: <20121127183515.916208D0008D@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3523 | davidkelly999 | 2012-11-27 12:32:11 -0600 (Tue, 27 Nov 2012) | 2 lines Only use port ranges from GLOBUS_TCP_PORT_RANGE if a port is not explicitly set, eg. from the -localport option of coaster-service ------------------------------------------------------------------------ Index: modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/LocalTCPService.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/LocalTCPService.java (revision 3522) +++ modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/LocalTCPService.java (working copy) @@ -113,9 +113,9 @@ /* When GLOBUS_TCP_PORT_RANGE is defined, find an acceptable port in that range */ String globusTCPPortRange = System.getenv("GLOBUS_TCP_PORT_RANGE"); - String rangeValues[] = {"0", "0"}; + String rangeValues[] = {String.valueOf(port), String.valueOf(port)}; - if(globusTCPPortRange != null) { + if(globusTCPPortRange != null && port == 0) { rangeValues = globusTCPPortRange.split(","); } From swift at ci.uchicago.edu Tue Nov 27 12:40:17 2012 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Tue, 27 Nov 2012 12:40:17 -0600 (CST) Subject: [Swift-commit] cog r3524 Message-ID: <20121127184017.91CC28D0008D@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3524 | davidkelly999 | 2012-11-27 12:36:23 -0600 (Tue, 27 Nov 2012) | 2 lines Fix for when coaster-service -localport is used ------------------------------------------------------------------------ Index: modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/LocalTCPService.java =================================================================== --- modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/LocalTCPService.java (revision 3523) +++ modules/provider-coaster/src/org/globus/cog/abstraction/coaster/service/LocalTCPService.java (working copy) @@ -113,9 +113,9 @@ /* When GLOBUS_TCP_PORT_RANGE is defined, find an acceptable port in that range */ String globusTCPPortRange = System.getenv("GLOBUS_TCP_PORT_RANGE"); - String rangeValues[] = {"0", "0"}; + String rangeValues[] = {String.valueOf(port), String.valueOf(port)}; - if(globusTCPPortRange != null) { + if(globusTCPPortRange != null && port == 0) { rangeValues = globusTCPPortRange.split(","); } From wilde at ci.uchicago.edu Tue Nov 27 12:46:27 2012 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Tue, 27 Nov 2012 12:46:27 -0600 (CST) Subject: [Swift-commit] r6093 - SwiftTutorials/UofC_2012-11-27/bin Message-ID: <20121127184627.ADCED178884@svn.ci.uchicago.edu> Author: wilde Date: 2012-11-27 12:46:27 -0600 (Tue, 27 Nov 2012) New Revision: 6093 Modified: SwiftTutorials/UofC_2012-11-27/bin/runswift Log: Enable reservation usage for cluster pool. Modified: SwiftTutorials/UofC_2012-11-27/bin/runswift =================================================================== --- SwiftTutorials/UofC_2012-11-27/bin/runswift 2012-11-27 18:09:37 UTC (rev 6092) +++ SwiftTutorials/UofC_2012-11-27/bin/runswift 2012-11-27 18:46:27 UTC (rev 6093) @@ -45,7 +45,7 @@ 1 westmere - + slurm.resource_list=advres=$reservationID 5.99 10000 From davidk at ci.uchicago.edu Wed Nov 28 13:12:32 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Wed, 28 Nov 2012 13:12:32 -0600 (CST) Subject: [Swift-commit] r6094 - in branches/release-0.94: bin libexec Message-ID: <20121128191232.DCF869CCEC@svn.ci.uchicago.edu> Author: davidk Date: 2012-11-28 13:12:31 -0600 (Wed, 28 Nov 2012) New Revision: 6094 Modified: branches/release-0.94/bin/slurm-qsub branches/release-0.94/bin/slurm-qsub-emulator branches/release-0.94/libexec/_swiftwrap Log: Fixes to slurm scripts and extra debug info in _swiftwrap Modified: branches/release-0.94/bin/slurm-qsub =================================================================== --- branches/release-0.94/bin/slurm-qsub 2012-11-27 18:46:27 UTC (rev 6093) +++ branches/release-0.94/bin/slurm-qsub 2012-11-28 19:12:31 UTC (rev 6094) @@ -7,7 +7,7 @@ script=$1.slurm echo '#! /bin/sh' >$script # echo 'export PBS_NODELIST=SLURM_NODELIST'' >>$script # FIXME: convert SLURM_NODELIST to PBS_NODEFILE here... -cat $1 >>$script +cat $1 | grep -v '^#PBS' >>$script chmod +x $script srunargs=$(awk <$1 ' @@ -18,6 +18,8 @@ sub(/nodes=/,"",nodes); sub(/:.*/,"",nodes); sub(/^.*ppn=/,"",ppn); } /#PBS -l walltime/ { walltime = $3; sub(/walltime=/,"",walltime);} +#/#PBS -l advres/ { advres = "," $3; sub(/advres=/,"",advres); advres="," advres } +/#PBS -l advres/ { advres = "," $3; } /#PBS -q/ { queue = $3 } /#PBS -o/ { sofile = $3 } /#PBS -e/ { sefile = $3 } @@ -29,8 +31,8 @@ # printf("-S %s -N %s -m %s -l nodes=%s:ppn=%s -l walltime=%s -q %s -o %s -e %s", # shell, jobname, m, nodes, ppn, walltime, queue, sofile, sefile); - printf("-N %s -m %s -l nodes=%s:ppn=%s -l walltime=%s -q %s -o %s -e %s", - jobname, m, nodes, ppn, walltime, queue, sofile, sefile); + printf("-N %s -m %s -l nodes=%s:ppn=%s,walltime=%s%s -q %s -o %s -e %s", + jobname, m, nodes, ppn, walltime, advres, queue, sofile, sefile); }') Modified: branches/release-0.94/bin/slurm-qsub-emulator =================================================================== --- branches/release-0.94/bin/slurm-qsub-emulator 2012-11-27 18:46:27 UTC (rev 6093) +++ branches/release-0.94/bin/slurm-qsub-emulator 2012-11-28 19:12:31 UTC (rev 6094) @@ -81,7 +81,6 @@ my $sbatch = "sbatch"; my $salloc = "salloc"; -my $srun = "$srun"; GetOptions('a=s' => \$start_time, 'A=s' => \$account, @@ -190,6 +189,7 @@ $command .= " --tmp=$res_opts{file}" if $res_opts{file}; $command .= " --mem=$res_opts{mem}" if $res_opts{mem}; $command .= " --nice=$res_opts{nice}" if $res_opts{nice}; +$command .= " --reservation=$res_opts{advres}" if $res_opts{advres}; # Cray-specific options $command .= " -n$res_opts{mppwidth}" if $res_opts{mppwidth}; $command .= " -w$res_opts{mppnodes}" if $res_opts{mppnodes}; @@ -215,7 +215,7 @@ $command .= " $script"; -system("(echo; echo $command; echo) >> $HOME/pbslog"); +system("(echo; echo $command; echo) >> \$HOME/pbslog"); system($command); @@ -236,6 +236,7 @@ 'pvmem' => "", 'software' => "", 'vmem' => "", + 'advres' => "", # Cray-specific resources 'mppwidth' => "", 'mppdepth' => "", @@ -247,9 +248,9 @@ my @keys = keys(%opt); foreach my $key (@keys) { - #print "$rl\n"; - ($opt{$key}) = $rl =~ m/$key=([\w:\+=+]+)/; - + # print "key=$key rl=$rl\n"; + ($opt{$key}) = $rl =~ m/$key=([-\w:\+=+]+)/; + # print " set to $opt{$key}\n"; } if($opt{cput}) { $opt{cput} = get_minutes($opt{cput}); Modified: branches/release-0.94/libexec/_swiftwrap =================================================================== --- branches/release-0.94/libexec/_swiftwrap 2012-11-27 18:46:27 UTC (rev 6093) +++ branches/release-0.94/libexec/_swiftwrap 2012-11-28 19:12:31 UTC (rev 6094) @@ -1,3 +1,4 @@ + #!/bin/bash # this script must be invoked inside of bash, not plain sh # note that this script modifies $IFS @@ -475,17 +476,31 @@ fail 254 "The executable $EXEC does not have the executable bit set" fi if [ "$KICKSTART" == "" ]; then + + TIMEARGS="-o swiftapp.resources -f APP_RESOURCES=real_time:%E,real_secs:%e,kernel_secs:%S,user_secs:%U,percent_cpu:%P,max_rss:%M,avg_rss:%t,avg_tot_vm:%K,avg_priv_data:%D,avg_priv_stack:%p,avg_shared_text:%X,page_size:%Z,major_pgfaults:%F,minor_pgfaults:%R,swaps:%W,invol_context_switches:%c,vol_waits%w,fs_reads:%I,fs_writes:%O,sock_recv:%r,sock_send:%s,signals:%k,exit_status:%x,command:%C" + + if [ -x /usr/bin/time ]; then + TIMECMD="/usr/bin/time" + elif [ -x $HOME/swift.time ]; then + TIMECMD="$HOME/swift.time" + else + TIMECMD="" + TIMEARGS="" + fi if [ "$STDIN" == "" ]; then if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then genScripts fi - "$EXEC" "${CMDARGS[@]}" 1>>"$STDOUT" 2>>"$STDERR" + eval $TIMECMD $TIMEARGS "$EXEC" "${CMDARGS[@]}" 1>>"$STDOUT" 2>>"$STDERR" else if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then genScripts fi - "$EXEC" "${CMDARGS[@]}" 1>>"$STDOUT" 2>>"$STDERR" <"$STDIN" + eval $TIMECMD $TIMEARGS "$EXEC" "${CMDARGS[@]}" 1>>"$STDOUT" 2>>"$STDERR" <"$STDIN" fi + if [ "_$TIMECMD" != _ ]; then + log "$(cat swiftapp.resources)" + fi checkError $? "Application $EXEC failed with an exit code of $?" else if [ ! -f "$KICKSTART" ]; then From davidk at ci.uchicago.edu Thu Nov 29 15:39:48 2012 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Thu, 29 Nov 2012 15:39:48 -0600 (CST) Subject: [Swift-commit] r6095 - branches/release-0.94/libexec Message-ID: <20121129213948.D5F929CC9F@svn.ci.uchicago.edu> Author: davidk Date: 2012-11-29 15:39:48 -0600 (Thu, 29 Nov 2012) New Revision: 6095 Modified: branches/release-0.94/libexec/_swiftwrap Log: Reverting changes to _swiftwrap in 0.94 to avoid issues with quotes Modified: branches/release-0.94/libexec/_swiftwrap =================================================================== --- branches/release-0.94/libexec/_swiftwrap 2012-11-28 19:12:31 UTC (rev 6094) +++ branches/release-0.94/libexec/_swiftwrap 2012-11-29 21:39:48 UTC (rev 6095) @@ -1,4 +1,3 @@ - #!/bin/bash # this script must be invoked inside of bash, not plain sh # note that this script modifies $IFS @@ -476,31 +475,17 @@ fail 254 "The executable $EXEC does not have the executable bit set" fi if [ "$KICKSTART" == "" ]; then - - TIMEARGS="-o swiftapp.resources -f APP_RESOURCES=real_time:%E,real_secs:%e,kernel_secs:%S,user_secs:%U,percent_cpu:%P,max_rss:%M,avg_rss:%t,avg_tot_vm:%K,avg_priv_data:%D,avg_priv_stack:%p,avg_shared_text:%X,page_size:%Z,major_pgfaults:%F,minor_pgfaults:%R,swaps:%W,invol_context_switches:%c,vol_waits%w,fs_reads:%I,fs_writes:%O,sock_recv:%r,sock_send:%s,signals:%k,exit_status:%x,command:%C" - - if [ -x /usr/bin/time ]; then - TIMECMD="/usr/bin/time" - elif [ -x $HOME/swift.time ]; then - TIMECMD="$HOME/swift.time" - else - TIMECMD="" - TIMEARGS="" - fi if [ "$STDIN" == "" ]; then if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then genScripts fi - eval $TIMECMD $TIMEARGS "$EXEC" "${CMDARGS[@]}" 1>>"$STDOUT" 2>>"$STDERR" + "$EXEC" "${CMDARGS[@]}" 1>>"$STDOUT" 2>>"$STDERR" else if [ "$SWIFT_GEN_SCRIPTS" != "" ]; then genScripts fi - eval $TIMECMD $TIMEARGS "$EXEC" "${CMDARGS[@]}" 1>>"$STDOUT" 2>>"$STDERR" <"$STDIN" + "$EXEC" "${CMDARGS[@]}" 1>>"$STDOUT" 2>>"$STDERR" <"$STDIN" fi - if [ "_$TIMECMD" != _ ]; then - log "$(cat swiftapp.resources)" - fi checkError $? "Application $EXEC failed with an exit code of $?" else if [ ! -f "$KICKSTART" ]; then From tmoore at ci.uchicago.edu Tue Nov 6 15:17:43 2012 From: tmoore at ci.uchicago.edu (tmoore at ci.uchicago.edu) Date: Tue, 06 Nov 2012 21:17:43 -0000 Subject: [Swift-commit] r6007 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121106211742.A7C709D10A@svn.ci.uchicago.edu> Author: tmoore Date: 2012-11-06 15:17:42 -0600 (Tue, 06 Nov 2012) New Revision: 6007 Added: SwiftApps/ParVis/HiRAMTools/makeyearly2yr.swift SwiftApps/ParVis/HiRAMTools/makeyearly_2yr.sh Modified: SwiftApps/ParVis/HiRAMTools/README SwiftApps/ParVis/HiRAMTools/combine_realization.sh SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh Log: updated HiRAMTools scripts to handle 2year configurations Modified: SwiftApps/ParVis/HiRAMTools/README =================================================================== --- SwiftApps/ParVis/HiRAMTools/README 2012-11-05 03:44:51 UTC (rev 6006) +++ SwiftApps/ParVis/HiRAMTools/README 2012-11-06 21:17:42 UTC (rev 6007) @@ -231,3 +231,17 @@ - Naming conventions for outdir now used in combine need to be added to makeyearly + + +FILE NAMING CONVENTIONS: + +- under the combine data sets, we'll use a manual soft link to the successfully + completed realization called "done". This is a manual setting to identify + which subdirectory is fully complete. + +- under annualize, the annualize script should point to the "done" directory + within a realization combine directory, but not recreate the "done" directory. + A "done" soft link will be created for the completed annualize directory + *current swift script does not yet conform to the above. + + Modified: SwiftApps/ParVis/HiRAMTools/combine_realization.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/combine_realization.sh 2012-11-05 03:44:51 UTC (rev 6006) +++ SwiftApps/ParVis/HiRAMTools/combine_realization.sh 2012-11-06 21:17:42 UTC (rev 6007) @@ -60,8 +60,7 @@ - - MTCScienceApps + SU_Climate default 2.56 10000 Added: SwiftApps/ParVis/HiRAMTools/makeyearly2yr.swift =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly2yr.swift (rev 0) +++ SwiftApps/ParVis/HiRAMTools/makeyearly2yr.swift 2012-11-06 21:17:42 UTC (rev 6007) @@ -0,0 +1,24 @@ + +type file; + +type ncrec { + string f1; +} + +string rundir=@arg("rundir","NO-OUTDIR-SPECIFIED"); +string ncfile=@arg("ncfile","ncfiles"); + +app (file sout, file serr) makeyearly (string dir, string f1) +{ + makeyearly_2yr f1 dir stdout=@sout stderr=@serr; +} + +ncrec nc[] = readData(ncfile); + +foreach n,i in nc { + file so ; + file se ; + (so,se) = makeyearly(rundir,n.f1); +} + + Added: SwiftApps/ParVis/HiRAMTools/makeyearly_2yr.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly_2yr.sh (rev 0) +++ SwiftApps/ParVis/HiRAMTools/makeyearly_2yr.sh 2012-11-06 21:17:42 UTC (rev 6007) @@ -0,0 +1,88 @@ +#! /bin/bash + +# Create a fast (local-filesystem) working dir + +if mkdir -p /scratch/wilde; then + scratch=/scratch/wilde +elif mkdir -p $HOME/scratch; then + scratch=$HOME/scratch +else + echo $0: cant create scratch dir + exit 11 +fi + +tmp=$(mktemp -d $scratch/makeyearly_XXXXXX) +startdir=$PWD +cd $tmp + +echo running in $PWD + +# the following is not needed for a two-year case +#qtrdec1=$1 # 1st quarter-decade dir +#qtrdec2=$2 # 2nd quarter-decade dir + +the2year=$1 +destbase=$2 # base dir for output years + +echo destbase=$destbase + +# Add cdo and nco tools to PATH and Intel compiler v11 libs to LD_LIBRARY_PATH + +#export PATH=/soft/apps/cdo-1.4.7/bin:/soft/apps/nco-4.0.9/bin:$PATH +export PATH=/soft/apps/nco-4.0.9/bin:$PATH +export LD_LIBRARY_PATH=/soft/apps/intel-fc-11.0.074/lib/intel64:/soft/apps/intel-cc-11.0.074/lib/intel64:/soft/apps/netcdf-3.6.3/lib:/usr/lib:/usr/X11R6/lib:/usr/local/lib + +fname=$(basename $the2year .nc) +firstyear=$(basename $(dirname $the2year) | sed -e 's/....$//') + +# process only the time-based files +echo $fname +if echo $fname | egrep -v '.nc'; then + + # Merge 2 quarter-decade files into one half-decade file + +# ncrcat $qtrdec1 $qtrdec2 5years.nc + + # Extract each year from the half-decade file +echo "processing ${fname}" + for (( y=$firstyear; y<$firstyear+2; y++ )); do + ncks -d time,"${y}-01-01 00:00:00","${y}-12-31 23:59:59" $the2year $y.nc + done + +fi + +# Copy the years from temporary to permanent filesystem + +for (( y=$firstyear; y<$firstyear+2; y++ )) + +do + destdir=$destbase/${y}0101 + mkdir -p $destdir + if echo $fname | egrep -v 'grid_spec|scalar'; then + dd if=$y.nc of=$destdir/$fname.nc bs=8M + else + dd if=$the2year of=$destdir/$fname.nc bs=8M + fi +done + +rm -rf $tmp + +# Delete when this works: + + # Old method: + # + # cdo mergetime $qtrdec1 $qtrdec2 ${fname}.nc + # cdo splityear ${fname}.nc ${fname}. + # Yields, eg, from atmos_month.tile1: + # atmos_month.tile1.1950.nc + # atmos_month.tile1.1951.nc + # atmos_month.tile1.1952.nc + # atmos_month.tile1.1953.nc + # atmos_month.tile1.1954.nc + + # New method: + # + # ncrcat input1.nc input2.nc 5years.nc + # ncks -d time,"$yyyy-01-01 00:00:00","$yyyy-12-31 23:59:59" 5years.nc year1.nc + + Property changes on: SwiftApps/ParVis/HiRAMTools/makeyearly_2yr.sh ___________________________________________________________________ Added: svn:executable + * Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh 2012-11-05 03:44:51 UTC (rev 6006) +++ SwiftApps/ParVis/HiRAMTools/makeyearly_realization.sh 2012-11-06 21:17:42 UTC (rev 6007) @@ -16,6 +16,12 @@ # Default assumes cdir is a rundir from combine script # New rundir will be appended to end of outsuffix +# Strip out the "done" directory in the outsuffix if present + +STRIPTEXT="\/done" + +outsuffix=${outsuffix//$STRIPTEXT/} + # generate the list of files to process: do every half-decade echo f1 f2 >ncfiles From tmoore at ci.uchicago.edu Thu Nov 8 08:41:26 2012 From: tmoore at ci.uchicago.edu (tmoore at ci.uchicago.edu) Date: Thu, 08 Nov 2012 14:41:26 -0000 Subject: [Swift-commit] r6009 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121108144124.D09D49CC9F@svn.ci.uchicago.edu> Author: tmoore Date: 2012-11-08 08:41:23 -0600 (Thu, 08 Nov 2012) New Revision: 6009 Added: SwiftApps/ParVis/HiRAMTools/makeyearly_realization_2yr.sh Log: added file not included in a previous commit - a version of makeyearly_realization script designed to work with the 2-year realization files Added: SwiftApps/ParVis/HiRAMTools/makeyearly_realization_2yr.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly_realization_2yr.sh (rev 0) +++ SwiftApps/ParVis/HiRAMTools/makeyearly_realization_2yr.sh 2012-11-08 14:41:23 UTC (rev 6009) @@ -0,0 +1,113 @@ +#! /bin/bash + +cdir=$1 # combined dir +outbase=$2 # base of outdir for equivalent yearly files + +bindir=$(dirname $0) +bindir=$(cd $bindir; pwd) + +echo bindir=$bindir + +# Enable one of the following patterns to name the output dir: + +outsuffix=$(basename $cdir) # cdir ends in rname, use rname +outsuffix=$(basename $(dirname $cdir)) # cdir ends in rname/rundir, use rname +outsuffix=$(basename $(dirname $cdir))/$(basename $cdir) # cdir ends in rname/rundir, use rname/rundir + # Default assumes cdir is a rundir from combine script + # New rundir will be appended to end of outsuffix + +STRIPPATH="\/done" +outsuffix=${outsuffix//$STRIPPATH/} +# generate the list of files to process: do every half-decade + +echo f1 >ncfiles +for y in $(seq 1950 2 1998); do + startyear=${y}0101; +# nextyear=$((y+1))0101 + +#this line does the following at sed +# ^\\(.*\\)/$startyear/\\(.*\\), searches for everything before startyear and saves it to 1 +# \\(.*\\) searches for everything after startyear and saves as 2 +# builds a new line with full path for first file, full path for second and targets the output directory +# find $cdir/$startyear -type f | fgrep .nc | sed -e "s,^\\(.*\\)/$startyear/\\(.*\\),\\1/$startyear/\\2 \\1/$nextyear/\\2 $outdir," + +#in the 2 year version, we only want the first file + find $cdir/$startyear -type f | fgrep .nc | sed -e "s,^\\(.*\\)/$startyear/\\(.*\\),\\1/$startyear/\\2 $outdir," + +done >>ncfiles +#now we have a list of the 2 year files and the outdir + +# Create new runNNN directory + +rundir=$( echo run??? | sed -e 's/^.*run//' | awk '{ printf("run%03d\n", $1+1)}' ) +mkdir $rundir + +# Generate Swift config files (cf, tc, sites.xml): + +cat >$rundir/cf <$rundir/sites.xml < + + + + .01 + 10000 + + $PWD/swiftwork + + + + + + + + SU_Climate + default + 2.56 + 10000 + 8 + + + 32 + 28000 + 00:20:00 + 100 + 100 + 1 + 1 + $PWD/$rundir/swiftwork + + + + +END + +cat >$rundir/tc <& swift.out + Property changes on: SwiftApps/ParVis/HiRAMTools/makeyearly_realization_2yr.sh ___________________________________________________________________ Added: svn:executable + * From tmoore at ci.uchicago.edu Thu Nov 8 16:52:30 2012 From: tmoore at ci.uchicago.edu (tmoore at ci.uchicago.edu) Date: Thu, 08 Nov 2012 22:52:30 -0000 Subject: [Swift-commit] r6010 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121108225229.733CF9CC9F@svn.ci.uchicago.edu> Author: tmoore Date: 2012-11-08 16:52:29 -0600 (Thu, 08 Nov 2012) New Revision: 6010 Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_2yr.sh Log: first attempt at repairing a bug in the 2year make yearly script. The script was not correctly excuding some files Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_2yr.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly_2yr.sh 2012-11-08 14:41:23 UTC (rev 6009) +++ SwiftApps/ParVis/HiRAMTools/makeyearly_2yr.sh 2012-11-08 22:52:29 UTC (rev 6010) @@ -37,7 +37,7 @@ # process only the time-based files echo $fname -if echo $fname | egrep -v '.nc'; then +if echo $fname | egrep -v 'grid_spec|scalar'; then # Merge 2 quarter-decade files into one half-decade file From tmoore at ci.uchicago.edu Tue Nov 13 13:37:31 2012 From: tmoore at ci.uchicago.edu (tmoore at ci.uchicago.edu) Date: Tue, 13 Nov 2012 19:37:31 -0000 Subject: [Swift-commit] r6029 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121113193731.191589CCA3@svn.ci.uchicago.edu> Author: tmoore Date: 2012-11-13 13:37:30 -0600 (Tue, 13 Nov 2012) New Revision: 6029 Added: SwiftApps/ParVis/HiRAMTools/realizationRealignScript.sh SwiftApps/ParVis/HiRAMTools/runRealign.sh Log: added some tools for working with 6 month cases Added: SwiftApps/ParVis/HiRAMTools/realizationRealignScript.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/realizationRealignScript.sh (rev 0) +++ SwiftApps/ParVis/HiRAMTools/realizationRealignScript.sh 2012-11-13 19:37:30 UTC (rev 6029) @@ -0,0 +1,156 @@ +#!/bin/sh + +#set defaults +YEAR="1950" +WORKDIR="" +TARGETDIR="" +FILENAME="" +GROUP="0" + +while getopts "hy:s:t:f:p:" opt; do + case $opt in + h) + echo "reorganize realization output v. 0.1 help" >&2 + echo "options:" >&2 + echo "-h - help" >&2 + echo "-y - set the year, e.g. 1950. Must be between 1900-2999" >&2 + echo "-s - original history file directory.(e.g. /intrepid-fs0/users/lzamboni/persistent/enstdIc2/history)" >&2 + echo "-t - target directory - similar to the above, but a writable directory.)" >&2 + echo "-f - filename without full path. (e.g. atmos_4xdaily.tile1.nc.0000)" >&2 + echo "-p - 6 month period. 0 Jan-Jun, 1 Jul-Dec" >&2 + exit 1 + ;; + \?) + echo "Invalid option : -$OPTARG" >&2 + exit 1 + ;; + y) YEAR="$OPTARG";; + s) WORKDIR="$OPTARG";; + t) TARGETDIR="$OPTARG";; + f) FILENAME="$OPTARG";; + p) GROUP="$OPTARG";; + esac +done + +# Now we need to test the input options for validity +#strip trailing slashes if present +WORKDIR=`echo "${WORKDIR}" | sed -e "s/\/*$//" ` +TARGETDIR=`echo "${TARGETDIR}" | sed -e "s/\/*$//" ` + +if [ ! -d "$WORKDIR" ]; then + echo "$WORKDIR does not appear to be a valid directory" + exit 1 +fi +if [ ! -d "$TARGETDIR" ]; then + echo "$TARGETDIR does not appear to be a valid directory" + exit 1 +fi + +#echo ${WORKDIR} +#echo ${TARGETDIR} + +#test for empty file name + +LEN1=$(echo ${#FILENAME}) +if [ "$LEN1" -lt "1" ]; then + echo "Invalid file name" + exit 1 +fi + +#GROUP should always be less than 2 +if [ "$GROUP" -gt "1" ]; then + echo "Group setting must be 0 or 1" + exit 1 +fi + +if [ "$GROUP" -lt "0" ]; then + echo "Group setting must be 0 or 1" + exit 1 +fi + +#now we test the year. + +if [ "$YEAR" -gt "2999" ]; then + echo "This script only accepts years from 1900-2999" + exit 1 +fi + +if [ "$YEAR" -lt "1900" ]; then + echo "This script only accepts years from 1900-2999" + exit 1 +fi + +#we have a valid year. + +PRIORYEAR=$(($YEAR-1)) +NEXTYEAR=$(($YEAR+1)) + +#cd to working dir +cd $WORKDIR + +#read a list of directories for our target year +DIRS=( ) +for aDir in * +do + if [[ "$aDir" == "$PRIORYEAR"* ]]; then + DIRS=("${DIRS[@]}" "${aDir}") + fi + if [[ "$aDir" == "$YEAR"* ]]; then + DIRS=("${DIRS[@]}" "${aDir}") + fi +done +#DIRS=`ls -d ${PRIORYEAR}*/` +#DIRS+=`ls -d ${YEAR}*/` + +for aDir in ${DIRS[*]} +do + echo "${aDir}" +done + + +TARGETFILE="$TARGETDIR/$YEAR" +if [ "$GROUP" = "0" ]; then + STARTDATE="$YEAR-1-1 0:00:0.1" + ENDDATE="$YEAR-7-1 0:00:0.0" + TARGETFILE="${TARGETFILE}0101" +else + STARTDATE="$YEAR-7-1 0:00:0.1" + ENDDATE="$NEXTYEAR-1-1 0:00:0.0" + TARGETFILE="${TARGETFILE}0701" +fi +echo $STARTDATE +echo $ENDDATE + +#build a list of files we'll use +SOURCEFILES="" + +for ASOURCEDIR in ${DIRS[*]} +do + #echo $ASOURCEDIR + SOURCEFILES="$SOURCEFILES $WORKDIR/$ASOURCEDIR/$FILENAME" +done + +#echo $SOURCEFILES +#echo $TARGETFILE + +if [ ! -d "$TARGETFILE" ]; then + mkdir $TARGETFILE +fi + +TARGETFILE="$TARGETFILE/$FILENAME" + +#echo $TARGETFILE + +#if the target file already exists, let's remove it. + +if [ -f "$TARGETFILE" ] +then + echo "File already exists! Removing" + rm -Rf $TARGETFILE +fi + +#at this point, we're ready to give things a shot at concatenation, or other testing. + +cmd="ncrcat -d time,'${STARTDATE}','${ENDDATE}' ${SOURCEFILES} ${TARGETFILE}" +echo $cmd +eval $cmd Property changes on: SwiftApps/ParVis/HiRAMTools/realizationRealignScript.sh ___________________________________________________________________ Added: svn:executable + * Added: SwiftApps/ParVis/HiRAMTools/runRealign.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/runRealign.sh (rev 0) +++ SwiftApps/ParVis/HiRAMTools/runRealign.sh 2012-11-13 19:37:30 UTC (rev 6029) @@ -0,0 +1,72 @@ +#!/bin/sh + +#SOURCEDIR is the directory where all of the history files are located +#SOURCEDIR=/intrepid-fs0/users/lzamboni/persistent/enstdIc2/history +SOURCEDIR=/intrepid-fs0/projects/SU_Climate/tlmoore/originals/enstdIc2/history/ + +#FILELISTDIR is a subdirectory in the SOURCEDIR that contains the filesnames +#FILELISTDIR=/intrepid-fs0/users/lzamboni/persistent/enstdIc2/history/19500101 +FILELISTDIR=/intrepid-fs0/projects/SU_Climate/tlmoore/originals/enstdIc2/history/19500101 + +STARTYEAR=1950 +ENDYEAR=2000 +SUBYEARS[0]=0 +SUBYEARS[1]=1 + +#TARGETDIR is the location new files will be created +TARGETDIR=/intrepid-fs0/projects/SU_Climate/tlmoore/processed/enstdIc2/history + + +#need to start with a list of files + +cd $FILELISTDIR +TARGETFILENAMES=( ) +for aFile in * +do +# echo $aFile + TARGETFILENAMES=("${TARGETFILENAMES[@]}" "${aFile}"); +done +##echo "${TARGETFILENAMES}" +#let's organize years +YEARS=( ) +while [ $((STARTYEAR)) -lt $((ENDYEAR)) ] +do + YEARS=("${YEARS[@]}" "${STARTYEAR}"); + STARTYEAR=$[$STARTYEAR+1] +done + +#validate starting info + +#for aYear in ${YEARS[@]} +#do +# echo $aYear +#done + +#for aName in ${TARGETFILENAMES[@]} +#do +# echo $aName +#done + +#for aSubyear in ${SUBYEARS[@]} +#do +# echo $aSubyear +#done + + +# Here we loop through the possibilities to create our files + +echo "starting" +for aYear in ${YEARS[@]} +do + #echo "Doing ${aYear}" + for aName in ${TARGETFILENAMES[@]} + do + #echo $aName + for aSubyear in ${SUBYEARS[@]} + do + echo "${aYear} ${aName} ${aSubyear}" + /home/tlmoore/Documents/realizationRealignScript.sh -y $aYear -s $SOURCEDIR -t $TARGETDIR -f $aName -p $aSubyear + done + done + +done Property changes on: SwiftApps/ParVis/HiRAMTools/runRealign.sh ___________________________________________________________________ Added: svn:executable + * From tmoore at ci.uchicago.edu Tue Nov 13 14:02:21 2012 From: tmoore at ci.uchicago.edu (tmoore at ci.uchicago.edu) Date: Tue, 13 Nov 2012 20:02:21 -0000 Subject: [Swift-commit] r6030 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121113200202.98EB19CCA3@svn.ci.uchicago.edu> Author: tmoore Date: 2012-11-13 14:02:02 -0600 (Tue, 13 Nov 2012) New Revision: 6030 Added: SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh Log: added script to create semiannual to annual cases Added: SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh (rev 0) +++ SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh 2012-11-13 20:02:02 UTC (rev 6030) @@ -0,0 +1,132 @@ +#! /bin/bash + +cdir=$1 # combined dir +outbase=$2 # base of outdir for equivalent yearly files + +bindir=$(dirname $0) +bindir=$(cd $bindir; pwd) + +echo bindir=$bindir + +# Enable one of the following patterns to name the output dir: + +outsuffix=$(basename $cdir) # cdir ends in rname, use rname +outsuffix=$(basename $(dirname $cdir)) # cdir ends in rname/rundir, use rname +outsuffix=$(basename $(dirname $cdir))/$(basename $cdir) # cdir ends in rname/rundir, use rname/rundir + # Default assumes cdir is a rundir from combine script + # New rundir will be appended to end of outsuffix + +STRIPPATH="\/done" +outsuffix=${outsuffix//$STRIPPATH/} +# generate the list of files to process: do every half-decade + +echo f1 >ncfiles +#we need to step through every year and create out target file list +for y in $(seq 1950 1 1998); do + + startyear=${y}0101; + +#All cases use 3 directories +#if 1950, then we have the following + if [[ "$y" == "1950"* ]]; then + timestep1=${y}0101; + timestep2=${y}0201; + timestep3=${y}0801; + else + PRIORYEAR=$(($y-1)) + timestep1=${PRIORYEAR}0801; + timestep2=${y}0201; + timestep3=${y}0801; + fi + +# nextyear=$((y+1))0101 + +#this line does the following at sed +# ^\\(.*\\)/$startyear/\\(.*\\), searches for everything before startyear and saves it to 1 +# \\(.*\\) searches for everything after startyear and saves as 2 +# builds a new line with full path for first file, full path for second and targets the output directory +# find $cdir/$startyear -type f | fgrep .nc | sed -e "s,^\\(.*\\)/$startyear/\\(.*\\),\\1/$startyear/\\2 \\1/$nextyear/\\2 $outdir," + +#in the 2 year version, we only want the first file + find $cdir/$timestep1 -type f | fgrep .nc | sed -e "s,^\\(.*\\)/$timestep1/\\(.*\\),\\1/$timestep1/\\2 \\1/$timestep2/\\2 \\1/$timestep3/\\2 $outdir," + +done >>ncfiles + +exit + +#now we have a list of the 2 year files and the outdir + +# Create new runNNN directory + +rundir=$( echo run??? | sed -e 's/^.*run//' | awk '{ printf("run%03d\n", $1+1)}' ) +mkdir $rundir + +# Generate Swift config files (cf, tc, sites.xml): + +cat >$rundir/cf <$rundir/sites.xml < + + + + .01 + 10000 + + $PWD/swiftwork + + + + + + + + SU_Climate + default + 2.56 + 10000 + 8 + + + 32 + 28000 + 00:20:00 + 100 + 100 + 1 + 1 + $PWD/$rundir/swiftwork + + + + +END + +cat >$rundir/tc <& swift.out + Property changes on: SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh ___________________________________________________________________ Added: svn:executable + * From tmoore at ci.uchicago.edu Tue Nov 13 15:12:57 2012 From: tmoore at ci.uchicago.edu (tmoore at ci.uchicago.edu) Date: Tue, 13 Nov 2012 21:12:57 -0000 Subject: [Swift-commit] r6031 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121113211256.BB3459CC9B@svn.ci.uchicago.edu> Author: tmoore Date: 2012-11-13 15:12:56 -0600 (Tue, 13 Nov 2012) New Revision: 6031 Added: SwiftApps/ParVis/HiRAMTools/makeyearly_semi.sh SwiftApps/ParVis/HiRAMTools/makeyearly_semi.swift Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh Log: Updated new semi-annual cases for testing Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh 2012-11-13 20:02:02 UTC (rev 6030) +++ SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh 2012-11-13 21:12:56 UTC (rev 6031) @@ -52,8 +52,8 @@ done >>ncfiles -exit + #now we have a list of the 2 year files and the outdir # Create new runNNN directory @@ -114,13 +114,13 @@ cat >$rundir/tc <& swift.out +swift -config cf -tc.file tc -sites.file sites.xml makeyearly_semi.swift -rundir=$outdir -ncfiles=ncfiles >& swift.out Added: SwiftApps/ParVis/HiRAMTools/makeyearly_semi.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly_semi.sh (rev 0) +++ SwiftApps/ParVis/HiRAMTools/makeyearly_semi.sh 2012-11-13 21:12:56 UTC (rev 6031) @@ -0,0 +1,84 @@ +#! /bin/bash + +# Create a fast (local-filesystem) working dir + +if mkdir -p /scratch/wilde; then + scratch=/scratch/wilde +elif mkdir -p $HOME/scratch; then + scratch=$HOME/scratch +else + echo $0: cant create scratch dir + exit 11 +fi + +tmp=$(mktemp -d $scratch/makeyearly_XXXXXX) +startdir=$PWD +cd $tmp + +echo running in $PWD + +qtrdec1=$1 # 1st quarter-decade dir +qtrdec2=$2 # 2nd quarter-decade dir +qtrdec3=$3 # 2nd quarter-decade dir +destbase=$4 # base dir for output years + +echo destbase=$destbase qtrdec1=$1 qtrdec2=$2 qtrdec3=$3 + +# Add cdo and nco tools to PATH and Intel compiler v11 libs to LD_LIBRARY_PATH + +#export PATH=/soft/apps/cdo-1.4.7/bin:/soft/apps/nco-4.0.9/bin:$PATH +export PATH=/soft/apps/nco-4.0.9/bin:$PATH +export LD_LIBRARY_PATH=/soft/apps/intel-fc-11.0.074/lib/intel64:/soft/apps/intel-cc-11.0.074/lib/intel64:/soft/apps/netcdf-3.6.3/lib:/usr/lib:/usr/X11R6/lib:/usr/local/lib + +fname=$(basename $qtrdec1 .nc) +#file 2 should always be the target year +firstyear=$(basename $(dirname $qtrdec2) | sed -e 's/....$//') + +# process only the time-based files + +if echo $fname | egrep -v 'grid_spec|scalar'; then + + # Merge 2 quarter-decade files into one half-decade file +#for simplicity at this point, keep the script the same + + ncrcat $qtrdec1 $qtrdec2 $qtrdec1 theyears.nc + + # Extract each year from the half-decade file + + ncks -d time,"${firstyear}-01-01 00:00:00","${firstyear}-12-31 23:59:59" theyears.nc $y.nc + + +fi + +# Copy the years from temporary to permanent filesystem + + + destdir=$destbase/${firstyear}0101 + mkdir -p $destdir + if echo $fname | egrep -v 'grid_spec|scalar'; then + dd if=$firstyear.nc of=$destdir/$fname.nc bs=8M + else + dd if=$qtrdec1 of=$destdir/$fname.nc bs=8M + fi + + +rm -rf $tmp + +# Delete when this works: + + # Old method: + # + # cdo mergetime $qtrdec1 $qtrdec2 ${fname}.nc + # cdo splityear ${fname}.nc ${fname}. + # Yields, eg, from atmos_month.tile1: + # atmos_month.tile1.1950.nc + # atmos_month.tile1.1951.nc + # atmos_month.tile1.1952.nc + # atmos_month.tile1.1953.nc + # atmos_month.tile1.1954.nc + + # New method: + # + # ncrcat input1.nc input2.nc 5years.nc + # ncks -d time,"$yyyy-01-01 00:00:00","$yyyy-12-31 23:59:59" 5years.nc year1.nc + Added: SwiftApps/ParVis/HiRAMTools/makeyearly_semi.swift =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly_semi.swift (rev 0) +++ SwiftApps/ParVis/HiRAMTools/makeyearly_semi.swift 2012-11-13 21:12:56 UTC (rev 6031) @@ -0,0 +1,24 @@ + +type file; + +type ncrec { + string f1; + string f2; + string f3 +} + +string rundir=@arg("rundir","NO-OUTDIR-SPECIFIED"); +string ncfile=@arg("ncfile","ncfiles"); + +app (file sout, file serr) makeyearly_semi (string dir, string f1, string f2, string f3) +{ + makeyearly_semi f1 f2 f3 dir stdout=@sout stderr=@serr; +} + +ncrec nc[] = readData(ncfile); + +foreach n,i in nc { + file so ; + file se ; + (so,se) = makeyearly_semi(rundir,n.f1,n.f2,,n.f3); +} From tmoore at ci.uchicago.edu Tue Nov 13 15:15:48 2012 From: tmoore at ci.uchicago.edu (tmoore at ci.uchicago.edu) Date: Tue, 13 Nov 2012 21:15:48 -0000 Subject: [Swift-commit] r6032 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121113211547.C05579CC9B@svn.ci.uchicago.edu> Author: tmoore Date: 2012-11-13 15:15:47 -0600 (Tue, 13 Nov 2012) New Revision: 6032 Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_semi.swift Log: fixed a bug in the swift script for semi-annual annualization Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_semi.swift =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly_semi.swift 2012-11-13 21:12:56 UTC (rev 6031) +++ SwiftApps/ParVis/HiRAMTools/makeyearly_semi.swift 2012-11-13 21:15:47 UTC (rev 6032) @@ -4,7 +4,7 @@ type ncrec { string f1; string f2; - string f3 + string f3; } string rundir=@arg("rundir","NO-OUTDIR-SPECIFIED"); From tmoore at ci.uchicago.edu Tue Nov 13 15:18:18 2012 From: tmoore at ci.uchicago.edu (tmoore at ci.uchicago.edu) Date: Tue, 13 Nov 2012 21:18:18 -0000 Subject: [Swift-commit] r6033 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121113211817.F09A59CC9B@svn.ci.uchicago.edu> Author: tmoore Date: 2012-11-13 15:18:17 -0600 (Tue, 13 Nov 2012) New Revision: 6033 Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_semi.swift Log: fixed a bug in the swift script for semi-annual annualization 2 Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_semi.swift =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly_semi.swift 2012-11-13 21:15:47 UTC (rev 6032) +++ SwiftApps/ParVis/HiRAMTools/makeyearly_semi.swift 2012-11-13 21:18:17 UTC (rev 6033) @@ -20,5 +20,5 @@ foreach n,i in nc { file so ; file se ; - (so,se) = makeyearly_semi(rundir,n.f1,n.f2,,n.f3); + (so,se) = makeyearly_semi(rundir,n.f1,n.f2,n.f3); } From tmoore at ci.uchicago.edu Tue Nov 13 15:21:26 2012 From: tmoore at ci.uchicago.edu (tmoore at ci.uchicago.edu) Date: Tue, 13 Nov 2012 21:21:26 -0000 Subject: [Swift-commit] r6034 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121113212125.889389CC9B@svn.ci.uchicago.edu> Author: tmoore Date: 2012-11-13 15:21:25 -0600 (Tue, 13 Nov 2012) New Revision: 6034 Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh Log: fixed a bug that the ncfiles header had too few header items Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh 2012-11-13 21:18:17 UTC (rev 6033) +++ SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh 2012-11-13 21:21:25 UTC (rev 6034) @@ -20,7 +20,7 @@ outsuffix=${outsuffix//$STRIPPATH/} # generate the list of files to process: do every half-decade -echo f1 >ncfiles +echo f1 f2 f3 >ncfiles #we need to step through every year and create out target file list for y in $(seq 1950 1 1998); do From tmoore at ci.uchicago.edu Tue Nov 13 17:37:09 2012 From: tmoore at ci.uchicago.edu (tmoore at ci.uchicago.edu) Date: Tue, 13 Nov 2012 23:37:09 -0000 Subject: [Swift-commit] r6035 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121113233708.C110A9CC9B@svn.ci.uchicago.edu> Author: tmoore Date: 2012-11-13 17:37:08 -0600 (Tue, 13 Nov 2012) New Revision: 6035 Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_semi.sh Log: fixed a problem with file lists Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_semi.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly_semi.sh 2012-11-13 21:21:25 UTC (rev 6034) +++ SwiftApps/ParVis/HiRAMTools/makeyearly_semi.sh 2012-11-13 23:37:08 UTC (rev 6035) @@ -30,7 +30,7 @@ export PATH=/soft/apps/nco-4.0.9/bin:$PATH export LD_LIBRARY_PATH=/soft/apps/intel-fc-11.0.074/lib/intel64:/soft/apps/intel-cc-11.0.074/lib/intel64:/soft/apps/netcdf-3.6.3/lib:/usr/lib:/usr/X11R6/lib:/usr/local/lib -fname=$(basename $qtrdec1 .nc) +fname=$(basename $qtrdec2 .nc) #file 2 should always be the target year firstyear=$(basename $(dirname $qtrdec2) | sed -e 's/....$//') @@ -41,7 +41,7 @@ # Merge 2 quarter-decade files into one half-decade file #for simplicity at this point, keep the script the same - ncrcat $qtrdec1 $qtrdec2 $qtrdec1 theyears.nc + ncrcat $qtrdec1 $qtrdec2 $qtrdec3 theyears.nc # Extract each year from the half-decade file From tmoore at ci.uchicago.edu Tue Nov 13 19:08:20 2012 From: tmoore at ci.uchicago.edu (tmoore at ci.uchicago.edu) Date: Wed, 14 Nov 2012 01:08:20 -0000 Subject: [Swift-commit] r6036 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121114010819.53BFD9CCA3@svn.ci.uchicago.edu> Author: tmoore Date: 2012-11-13 19:08:19 -0600 (Tue, 13 Nov 2012) New Revision: 6036 Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_semi.sh Log: fixed a problem with file lists #2 Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_semi.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly_semi.sh 2012-11-13 23:37:08 UTC (rev 6035) +++ SwiftApps/ParVis/HiRAMTools/makeyearly_semi.sh 2012-11-14 01:08:19 UTC (rev 6036) @@ -30,7 +30,7 @@ export PATH=/soft/apps/nco-4.0.9/bin:$PATH export LD_LIBRARY_PATH=/soft/apps/intel-fc-11.0.074/lib/intel64:/soft/apps/intel-cc-11.0.074/lib/intel64:/soft/apps/netcdf-3.6.3/lib:/usr/lib:/usr/X11R6/lib:/usr/local/lib -fname=$(basename $qtrdec2 .nc) +fname=$(basename $qtrdec1 .nc) #file 2 should always be the target year firstyear=$(basename $(dirname $qtrdec2) | sed -e 's/....$//') From tmoore at ci.uchicago.edu Tue Nov 13 19:56:24 2012 From: tmoore at ci.uchicago.edu (tmoore at ci.uchicago.edu) Date: Wed, 14 Nov 2012 01:56:24 -0000 Subject: [Swift-commit] r6037 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121114015623.B8FC09CC9B@svn.ci.uchicago.edu> Author: tmoore Date: 2012-11-13 19:56:23 -0600 (Tue, 13 Nov 2012) New Revision: 6037 Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_semi.sh Log: fixed date information and executable settings on HiRAMTools for semi-annual cases. Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_semi.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly_semi.sh 2012-11-14 01:08:19 UTC (rev 6036) +++ SwiftApps/ParVis/HiRAMTools/makeyearly_semi.sh 2012-11-14 01:56:23 UTC (rev 6037) @@ -45,7 +45,7 @@ # Extract each year from the half-decade file - ncks -d time,"${firstyear}-01-01 00:00:00","${firstyear}-12-31 23:59:59" theyears.nc $y.nc + ncks -d time,"${firstyear}-01-01 00:00:00","${firstyear}-12-31 23:59:59" theyears.nc $firstyear.nc fi Property changes on: SwiftApps/ParVis/HiRAMTools/makeyearly_semi.sh ___________________________________________________________________ Added: svn:executable + * From tmoore at ci.uchicago.edu Tue Nov 13 19:57:46 2012 From: tmoore at ci.uchicago.edu (tmoore at ci.uchicago.edu) Date: Wed, 14 Nov 2012 01:57:46 -0000 Subject: [Swift-commit] r6038 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121114015746.56E9C9CC9B@svn.ci.uchicago.edu> Author: tmoore Date: 2012-11-13 19:57:46 -0600 (Tue, 13 Nov 2012) New Revision: 6038 Modified: SwiftApps/ParVis/HiRAMTools/makeyearly.sh SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh Log: Updated executable settings for makeyearly Property changes on: SwiftApps/ParVis/HiRAMTools/makeyearly.sh ___________________________________________________________________ Added: svn:executable + * Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh 2012-11-14 01:56:23 UTC (rev 6037) +++ SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh 2012-11-14 01:57:46 UTC (rev 6038) @@ -22,7 +22,7 @@ echo f1 f2 f3 >ncfiles #we need to step through every year and create out target file list -for y in $(seq 1950 1 1998); do +for y in $(seq 1950 1 1951); do startyear=${y}0101; @@ -69,7 +69,7 @@ sitedir.keep=true execution.retries=10 lazy.errors=true -#status.mode=provider +status.mode=provider use.wrapper.staging=false END From tmoore at ci.uchicago.edu Tue Nov 13 20:15:29 2012 From: tmoore at ci.uchicago.edu (tmoore at ci.uchicago.edu) Date: Wed, 14 Nov 2012 02:15:29 -0000 Subject: [Swift-commit] r6039 - SwiftApps/ParVis/HiRAMTools Message-ID: <20121114021528.C0F0D9CC9B@svn.ci.uchicago.edu> Author: tmoore Date: 2012-11-13 20:15:28 -0600 (Tue, 13 Nov 2012) New Revision: 6039 Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh Log: updated for full length run Modified: SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh =================================================================== --- SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh 2012-11-14 01:57:46 UTC (rev 6038) +++ SwiftApps/ParVis/HiRAMTools/makeyearly_realization_semi.sh 2012-11-14 02:15:28 UTC (rev 6039) @@ -22,7 +22,7 @@ echo f1 f2 f3 >ncfiles #we need to step through every year and create out target file list -for y in $(seq 1950 1 1951); do +for y in $(seq 1950 1 1999); do startyear=${y}0101;