Open provenance model (OPM)

Open provenance model (OPM) - -should provenance record track compound procedures? at present its -heading in a direction where it only tracks app procedures, without -regard for inputs and outputs of compound procedures. -The turns datasets into having multiple producers - each level counts as -having produced it. OPM model has some discussion of that as -'alternate descriptions' in s3. - - -relation to OPM Open Provenance Model -http://twiki.ipaw.info/bin/view/Challenge/OPM -http://eprints.ecs.soton.ac.uk/14979/1/opm.pdf" -might be useful for the vocabulary at least - -

Input parameters Should also work on workflows which take an input parameter, so that we @@ -1760,7 +1740,56 @@

Open Provenance Model (OPM) +

OPM-defined terms and their relation to Swift + +OPM defines a number of terms. This section describes how those terms +relate to Swift. + + +artifact: This OPM term maps well onto the internal Swift representation +of DSHandles. Each DSHandle in a Swift run is an +OPM artifact, and each OPM artifact in a graph is a DSHandle. + +collection: OPM collections are a specific kind of artifact, containing +other artifacts. This corresponds with DSHandles for composite data types +(structs and arrays). OPM has collection accessors and collection +constructors which correspond to the [] and +. operators (for accessors) and various assignment +forms for constructors. + + +process: An OPM process corresponds to a number of Swift concepts (although +they are slowly converging in Swift to a single concept). Those concepts +are: procedure invocations, function calls, and operators. + + +agent: There are several entities which can act as an agent. At the +highest level where only Swift is involved, a run of the +swift commandline client is an agent which drives +everything. Some other components of Swift may be regarded as agents, +such as the client side wrapper script. For present OPM work, the +only agent will be the Swift command line client invocation. + + +account: For present OPM work, there will be one account per workflow run. +In future, different levels of granularity that could be expressed through +different accounts might include representing compound procedure calls as +processes vs representing atomic procedures calls explicitly. + + +OPM graph: there are two kinds of OPM graph that appear interesting and +straightforward to export: i) of entire provenance database (thus containing +multiple workflow runs); ii) of a single run + +

OPM links +Open Provenance Model at ipaw.info +

stuff TODO transcribe info from the pile of papers I wrote. esp query analysis From noreply at svn.ci.uchicago.edu Mon Mar 2 11:57:39 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 2 Mar 2009 11:57:39 -0600 (CST) Subject: [Swift-commit] r2633 - provenancedb Message-ID: <20090302175739.D6D26228187@www.ci.uchicago.edu> Author: benc Date: 2009-03-02 11:57:38 -0600 (Mon, 02 Mar 2009) New Revision: 2633 Added: provenancedb/drive-opm provenancedb/prov-to-opm.sh Log: A prototype of a log->OPM XML form converter using a variation of Luc Moreau's XML schema. Added: provenancedb/drive-opm =================================================================== --- provenancedb/drive-opm (rev 0) +++ provenancedb/drive-opm 2009-03-02 17:57:38 UTC (rev 2633) @@ -0,0 +1,13 @@ +#!/bin/bash + +# driver for log file -> OPM conversion + +filename=$1 + +export PATH=$LOGPATH:$PATH + +echo log to OPM conversion for $filename + +./prepare-for-import $filename +./prov-to-opm.sh $filename + Property changes on: provenancedb/drive-opm ___________________________________________________________________ Name: svn:executable + * Added: provenancedb/prov-to-opm.sh =================================================================== --- provenancedb/prov-to-opm.sh (rev 0) +++ provenancedb/prov-to-opm.sh 2009-03-02 17:57:38 UTC (rev 2633) @@ -0,0 +1,85 @@ +#!/bin/bash + +echo Generating OPM for a single run + +rm -f opm.xml + +# TODO make swift-opm-ns into a proper URI +echo "" > opm.xml + +echo "" >> opm.xml + +echo "" >> opm.xml + +while read time duration thread endstate app scratch; do + +echo " " +echo " " +echo " " +# TODO no value here - this is some URI into an ontology, which is don't +# really know how should be mapped from Swift +echo " " + +done < $LOGDIR/execute.global.event >> opm.xml + +echo "" >> opm.xml + +# TODO artifacts + +echo "" >> opm.xml + +# we need a list of all artifacts here. for now, take everything we can +# find in the tie-data-invocs and containment tables, uniquefied. +# This is probably the wrong thing to do. + +while read outer inner; do + echo $input + echo $output +done < $LOGDIR/tie-containers.txt > tmp-dshandles.txt + +while read t d dataset rest ; do + echo $dataset +done < $LOGDIR/tie-data-invocs.txt >> tmp-dshandles.txt + +cat tmp-dshandles.txt | sort | uniq > tmp-dshandles2.txt + +while read artifact ; do +echo " " +echo " " +echo " " +done < tmp-dshandles2.txt >> opm.xml + +echo "" >> opm.xml + + +echo "" >> opm.xml + +# other stuff can do this in any order, but here we must probably do it +# in two passes, one for each relation, in order to satisfy schema. +# but for now do it in a single pass... + +while read thread direction dataset variable rest; do + if [ "$direction" == "input" ] ; then + echo " " + echo " " + echo " " + echo " " + echo " " + echo " " + else + echo " " + echo " " + echo " " + echo " " + echo " " + echo " " + fi +done < $LOGDIR/tie-data-invocs.txt >> opm.xml + + + +echo "" >> opm.xml + +echo "" >> opm.xml +echo Finished generating OPM, in opm.xml + Property changes on: provenancedb/prov-to-opm.sh ___________________________________________________________________ Name: svn:executable + * From noreply at svn.ci.uchicago.edu Mon Mar 2 12:41:13 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 2 Mar 2009 12:41:13 -0600 (CST) Subject: [Swift-commit] r2634 - SwiftApps/SIDGrid/site_management Message-ID: <20090302184113.E4645228217@www.ci.uchicago.edu> Author: skenny Date: 2009-03-02 12:41:08 -0600 (Mon, 02 Mar 2009) New Revision: 2634 Modified: SwiftApps/SIDGrid/site_management/catmyapp Log: updated description Modified: SwiftApps/SIDGrid/site_management/catmyapp =================================================================== --- SwiftApps/SIDGrid/site_management/catmyapp 2009-03-02 17:57:38 UTC (rev 2633) +++ SwiftApps/SIDGrid/site_management/catmyapp 2009-03-02 18:41:08 UTC (rev 2634) @@ -1,5 +1,9 @@ #!/usr/bin/env python +# catmyapp is used to add a package of executables to +# Swift's transformation catalog (tc.data file) +# jan 2009 skenny at uchicago.edu + # looks for 2 files in the cwd: cmds.txt containing a list of the binaries to be added # and a tc.data file to add them to # given a list of sites and the location of the executables, catmyapp From noreply at svn.ci.uchicago.edu Tue Mar 3 13:05:24 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 3 Mar 2009 13:05:24 -0600 (CST) Subject: [Swift-commit] r2636 - SwiftApps/SIDGrid/swift/swift_scripts/afni Message-ID: <20090303190524.2CDB32281F3@www.ci.uchicago.edu> Author: andric Date: 2009-03-03 13:05:23 -0600 (Tue, 03 Mar 2009) New Revision: 2636 Added: SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIdeconvolve.swift Log: 3dDeconvolve for event related design Added: SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIdeconvolve.swift =================================================================== --- SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIdeconvolve.swift (rev 0) +++ SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIdeconvolve.swift 2009-03-03 19:05:23 UTC (rev 2636) @@ -0,0 +1,62 @@ +#---- pinche deconvolution +type file{} + +type AFNI_obj{ + file HEAD; + file BRIK; +} + +type stims{ + string labels[]; + string fnames[]; +} + +app (AFNI_obj deconvBucketResult, AFNI_obj irfResult, file deconvXsave) AFNI_pincheDeconvolution (AFNI_obj normalized, stims stimuli, file censor, file concat, file motionfile, file designmatrix, string outPrefix){ + AFNI_3dDeconvolve "-input" @filename(normalized.BRIK) "-polort" 3 "-tshift" "-num_stimts" 10 + "-stim_file" 1 stimuli.fnames[0] "-stim_label" 1 stimuli.labels[0] + "-stim_file" 2 stimuli.fnames[1] "-stim_label" 2 stimuli.labels[1] + "-stim_file" 3 stimuli.fnames[2] "-stim_label" 3 stimuli.labels[2] + "-stim_file" 4 stimuli.fnames[3] "-stim_label" 4 stimuli.labels[3] + "-stim_file" 5 stimuli.fnames[4] "-stim_base" 5 "-stim_label" 5 stimuli.labels[4] + "-stim_file" 6 stimuli.fnames[5] "-stim_base" 6 "-stim_label" 6 stimuli.labels[5] + "-stim_file" 7 stimuli.fnames[6] "-stim_base" 7 "-stim_label" 7 stimuli.labels[6] + "-stim_file" 8 stimuli.fnames[7] "-stim_base" 8 "-stim_label" 8 stimuli.labels[7] + "-stim_file" 9 stimuli.fnames[8] "-stim_base" 9 "-stim_label" 9 stimuli.labels[8] + "-stim_file" 10 stimuli.fnames[9] "-stim_base" 10 "-stim_label" 10 stimuli.labels[9] + "-stim_maxlag" 1 11 + "-stim_maxlag" 2 11 + "-stim_maxlag" 3 11 + "-stim_maxlag" 4 11 + "-censor" @filename(censor) + "-iresp" 1 @strcat(outPrefix,".IRF") + "-full_first" "-tout" "-nodmbase" "-xsave" "-nfirst" "0" "-concat" @filename(concat) "-bucket" + outPrefix; +} + + +string declarelist[] = ["PK2"]; +foreach subject in declarelist{ + ## map inputs + file designmatrix; + file motionfile; + file concat; + file censor; + AFNI_obj normalized; + + ## map outputs + string outPrefix = @strcat("DeconvResults_",subject); + file xsave; + AFNI_obj bucket; + AFNI_obj irf; + + #---- stimuli info + stims stimuli; + stimuli.labels = ["still_pa","move_pa","still_noise","move_noise","roll","pitch","yaw","dx","dy","dz"]; + stimuli.fnames = [@strcat(@filename(designmatrix),"[0]"), at strcat(@filename(designmatrix),"[1]"), + @strcat(@filename(designmatrix),"[2]"), at strcat(@filename(designmatrix),"[3]"), + @strcat(@filename(motionfile),"[1]"), at strcat(@filename(motionfile),"[2]"), + @strcat(@filename(motionfile),"[3]"), at strcat(@filename(motionfile),"[4]"), + @strcat(@filename(motionfile),"[5]"), at strcat(@filename(motionfile),"[6]")]; + + (bucket, irf, xsave) = AFNI_pincheDeconvolution(normalized, stimuli, censor, concat, motionfile, designmatrix, outPrefix); +} From noreply at svn.ci.uchicago.edu Tue Mar 3 13:12:15 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 3 Mar 2009 13:12:15 -0600 (CST) Subject: [Swift-commit] r2637 - SwiftApps/SIDGrid/swift/swift_scripts/afni Message-ID: <20090303191215.CB7B02281F6@www.ci.uchicago.edu> Author: andric Date: 2009-03-03 13:12:14 -0600 (Tue, 03 Mar 2009) New Revision: 2637 Added: SwiftApps/SIDGrid/swift/swift_scripts/afni/SNRvol2surf.swift Log: 3dVol2Surf for mapping surfaces from SUMA directory Added: SwiftApps/SIDGrid/swift/swift_scripts/afni/SNRvol2surf.swift =================================================================== --- SwiftApps/SIDGrid/swift/swift_scripts/afni/SNRvol2surf.swift (rev 0) +++ SwiftApps/SIDGrid/swift/swift_scripts/afni/SNRvol2surf.swift 2009-03-03 19:12:14 UTC (rev 2637) @@ -0,0 +1,37 @@ +#--- this is for the PK2 SNR surface projections +type file{} + +type AFNI_obj{ + file HEAD; + file BRIK; +} + + +app (file out_1d) AFNI_Vol2Surf (file spec_file, file smoothwm, file pial, string map, AFNI_obj surfvol, AFNI_obj grid_parent){ + AFNI_3dVol2Surf "-spec" @spec_file "-surf_A" @smoothwm "-surf_B" @pial + "-map_func" map "-f_steps" 15 "-f_index" "voxels" "-oob_index" "-1" "-oob_value" 0 "-no_headers" "-outcols_NSD_format" + "-sv" @surfvol.BRIK "-grid_parent" @grid_parent.BRIK "-out_1D" @filename(out_1d); +} + + +string declarelist[] = ["PK2"]; +string mapFunctions[] = ["ave"]; +string hemis[] = ["lh","rh"]; +string conditions[] = ["SNR"]; + +foreach subject in declarelist{ + foreach map in mapFunctions{ + foreach h in hemis{ + foreach cc in conditions{ + string baseDir = @strcat("/disks/ci-gpfs/fmri/cnari/swift/projects/andric/peakfitTESTS/",subject,"/"); + file spec_file; + file smoothwm; + file pial; + AFNI_obj surfvol; + AFNI_obj grid_parent; + file output1D; + output1D = AFNI_Vol2Surf(spec_file, smoothwm, pial, map, surfvol, grid_parent); + } + } + } +} From noreply at svn.ci.uchicago.edu Tue Mar 3 18:03:36 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 3 Mar 2009 18:03:36 -0600 (CST) Subject: [Swift-commit] r2638 - SwiftApps/SIDGrid/swift/swift_scripts/afni Message-ID: <20090304000336.2175A2281D8@www.ci.uchicago.edu> Author: andric Date: 2009-03-03 18:03:35 -0600 (Tue, 03 Mar 2009) New Revision: 2638 Added: SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIprocss.swift Log: Preprocessing workflow Added: SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIprocss.swift =================================================================== --- SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIprocss.swift (rev 0) +++ SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIprocss.swift 2009-03-04 00:03:35 UTC (rev 2638) @@ -0,0 +1,49 @@ +#---- doing volreg, despike, mean, percent sig change for PK2 +type file{} + +type AFNI_obj{ + file HEAD; + file BRIK; +}; + + +app (AFNI_obj volregResult, file resultVolregMotion) AFNI_volreg (string baseName, AFNI_obj inputTS, AFNI_obj referenceTS, string volregBaseString){ + AFNI_3dvolreg "-twopass" "-twodup" "-dfile" @strcat("mot_",baseName) "-base" @strcat(volregBaseString,"+orig[225]") "-prefix" @strcat("volreg.",baseName) @inputTS.BRIK; +} + +app (AFNI_obj despikeResult) AFNI_despike (string baseName, AFNI_obj volregdInputTS){ + AFNI_3dDespike "-prefix" @strcat("despiked.",baseName) @volregdInputTS.BRIK; +} + +app (AFNI_obj meanResult) AFNI_mean (AFNI_obj despikeResult, string baseName){ + AFNI_3dTstat "-mean" "-prefix" @strcat("mean.",baseName) @despikeResult.BRIK; +} + +app (AFNI_obj nrmlzPerChngResult) AFNI_normalizePerChng (AFNI_obj despikeResult, AFNI_obj meanResult, string baseName){ + AFNI_3dcalc "-a" @despikeResult.BRIK "-b" @meanResult.BRIK "-expr" @strcat( "((a-b)/b)*100" ) "-fscale" "-prefix" @strcat("nrmlzdPerChng.",baseName); +} + +(AFNI_obj despikeResult, AFNI_obj meanResult, AFNI_obj nrmlzPerChngResult) AFNI_PerChngTS (string baseName, AFNI_obj volregdInput){ + (despikeResult) = AFNI_despike(baseName, volregdInput); + (meanResult) = AFNI_mean(despikeResult, baseName); + (nrmlzPerChngResult) = AFNI_normalizePerChng(despikeResult, meanResult, baseName); +} + + +string declarelist[] = ["PK2"]; +foreach subject in declarelist{ + int runs[] = [1:8]; + foreach run in runs{ + string baseName = @strcat(subject,".run",run); + AFNI_obj inputTS; + AFNI_obj volregResult; + file resultVolregMotion; + AFNI_obj volregRefTS; + string volregBaseString = @strcat("ts.4"); + AFNI_obj despikeResult; + AFNI_obj meanResult; + AFNI_obj nrmlzPerChngResult; + (volregResult, resultVolregMotion) = AFNI_volreg(baseName, inputTS, volregRefTS, volregBaseString); + (despikeResult, meanResult, nrmlzPerChngResult) = AFNI_PerChngTS(baseName, volregResult); + } +} From noreply at svn.ci.uchicago.edu Wed Mar 4 14:41:01 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Wed, 4 Mar 2009 14:41:01 -0600 (CST) Subject: [Swift-commit] r2641 - SwiftApps/SIDGrid/swift/projects/andric/peakfit_pilots/PK2/preprocessing Message-ID: <20090304204101.C595C2281F3@www.ci.uchicago.edu> Author: andric Date: 2009-03-04 14:41:00 -0600 (Wed, 04 Mar 2009) New Revision: 2641 Modified: SwiftApps/SIDGrid/swift/projects/andric/peakfit_pilots/PK2/preprocessing/swift_execute.sh Log: new site Modified: SwiftApps/SIDGrid/swift/projects/andric/peakfit_pilots/PK2/preprocessing/swift_execute.sh =================================================================== --- SwiftApps/SIDGrid/swift/projects/andric/peakfit_pilots/PK2/preprocessing/swift_execute.sh 2009-03-04 19:55:06 UTC (rev 2640) +++ SwiftApps/SIDGrid/swift/projects/andric/peakfit_pilots/PK2/preprocessing/swift_execute.sh 2009-03-04 20:41:00 UTC (rev 2641) @@ -1,3 +1,3 @@ #!/bin/tcsh -swift AFNIdeconvolve.swift -sites.file /disks/gpfs/fmri/cnari/swift/config/sites_ucanl64.xml -tc.file /disks/gpfs/fmri/cnari/swift/config/tc.data -user="andric" +swift AFNIdeconvolve.swift -sites.file /disks/gpfs/fmri/cnari/swift/config/sites_bsd.xml -tc.file /disks/gpfs/fmri/cnari/swift/config/tc.data -user="andric" From noreply at svn.ci.uchicago.edu Wed Mar 4 15:28:32 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Wed, 4 Mar 2009 15:28:32 -0600 (CST) Subject: [Swift-commit] r2642 - SwiftApps/SIDGrid/swift/swift_scripts/afni Message-ID: <20090304212832.610882281D8@www.ci.uchicago.edu> Author: andric Date: 2009-03-04 15:28:31 -0600 (Wed, 04 Mar 2009) New Revision: 2642 Added: SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIsnr.swift Log: For doing SNR ratios with AFNI Added: SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIsnr.swift =================================================================== --- SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIsnr.swift (rev 0) +++ SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIsnr.swift 2009-03-04 21:28:31 UTC (rev 2642) @@ -0,0 +1,56 @@ +#---- for doing SNR on the PK2 afni data +type file{} + +type AFNI_obj{ +file HEAD; +file BRIK; +}; + + +app (AFNI_obj volregResult, file resultVolregMotion) AFNI_volreg (string baseName, AFNI_obj inputTS, AFNI_obj referenceTS, string volregBaseString){ + AFNI_3dvolreg "-twopass" "-twodup" "-dfile" @strcat("mot_",baseName) "-base" @strcat(volregBaseString,"+orig[225]") "-prefix" @strcat("volreg.",baseName) @inputTS.BRIK; +} + +app (AFNI_obj meanResult) AFNI_mean (AFNI_obj meanInput, string baseName){ + AFNI_3dTstat "-mean" "-prefix" @strcat("./mean.",baseName) @meanInput.BRIK; +} + +app (AFNI_obj stdevResult) AFNI_stdev (AFNI_obj stdevInput, string baseName){ + AFNI_3dTstat "-stdev" "-prefix" @strcat("./stdev.",baseName) @stdevInput.BRIK; +} + +app (AFNI_obj detrendResult) AFNI_detrend (AFNI_obj detrendInput, string baseName){ + AFNI_3dDetrend "-polort" "3" "-prefix" @strcat("detrend.",baseName) @detrendInput.BRIK; +} + +app (AFNI_obj ratioResult) AFNI_doratio (AFNI_obj stdevResult, AFNI_obj meanResult , string baseName){ + AFNI_3dcalc "-verbose" "-a" @meanResult.BRIK "-b" @stdevResult.BRIK "-expr" "(a/b)" "-prefix" @strcat("snr.",baseName); +} + +(AFNI_obj detrendResult, AFNI_obj stdevResult, AFNI_obj meanResult, AFNI_obj ratioResult) AFNI_snr (string baseName, AFNI_obj inputTS){ + (meanResult) = AFNI_mean(inputTS, baseName); + (detrendResult) = AFNI_detrend(inputTS, baseName); + (stdevResult) = AFNI_stdev(detrendResult, baseName); + (ratioResult) = AFNI_doratio(stdevResult, meanResult, baseName); +} + + +string declarelist[] = ["PK2"]; +foreach subject in declarelist{ + int runs[] = [1:8]; + foreach run in runs{ + AFNI_obj srun; + string baseName = @strcat(subject,".run",run); + AFNI_obj volregResult; + AFNI_obj meanResult; + AFNI_obj stdevResult; + AFNI_obj detrendResult; + AFNI_obj ratioResult; + // begin set up volreg parameters + file resultVolregMotion; + AFNI_obj volregRefTS; + string volregBaseString = @strcat("ts.4"); + (volregResult, resultVolregMotion) = AFNI_volreg(baseName, srun, volregRefTS, volregBaseString); + (detrendResult, stdevResult, meanResult, ratioResult) = AFNI_snr(baseName, volregResult); + } +} Property changes on: SwiftApps/SIDGrid/swift/swift_scripts/afni/AFNIsnr.swift ___________________________________________________________________ Name: svn:executable + * From noreply at svn.ci.uchicago.edu Fri Mar 6 01:22:42 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 6 Mar 2009 01:22:42 -0600 (CST) Subject: [Swift-commit] r2643 - SwiftApps/SIDGrid/swift/projects/andric/peakfit_pilots/PK2/PK2surfaces/surfaceData Message-ID: <20090306072242.282C42281F4@www.ci.uchicago.edu> Author: andric Date: 2009-03-06 01:22:41 -0600 (Fri, 06 Mar 2009) New Revision: 2643 Added: SwiftApps/SIDGrid/swift/projects/andric/peakfit_pilots/PK2/PK2surfaces/surfaceData/make_tables.py Log: Python script using the MySQLdb module to create a MySQL table Added: SwiftApps/SIDGrid/swift/projects/andric/peakfit_pilots/PK2/PK2surfaces/surfaceData/make_tables.py =================================================================== --- SwiftApps/SIDGrid/swift/projects/andric/peakfit_pilots/PK2/PK2surfaces/surfaceData/make_tables.py (rev 0) +++ SwiftApps/SIDGrid/swift/projects/andric/peakfit_pilots/PK2/PK2surfaces/surfaceData/make_tables.py 2009-03-06 07:22:41 UTC (rev 2643) @@ -0,0 +1,27 @@ +#!/usr/bin/python + +import os +import sys +import commands +import MySQLdb +import time + +try: + connection = MySQLdb.connect(read_default_file="~/.my.cnf",db="HEL") +except MySQLdb.Error, e: + print "Error %d: %s" % (e.args[0], e.args[1]) + sys.exit (1) + +cursor = connection.cursor() + +def create_new_table(): + print "que hora es?\n"+time.ctime() + try: + create_table_statement = 'CREATE TABLE peakFunc_datalh (subject VARCHAR(30), vertex INTEGER, roi INTEGER, snr_run1 FLOAT, snr_run2 FLOAT, snr_run3 FLOAT, snr_run4 FLOAT, snr_run5 FLOAT, snr_run6 FLOAT, snr_run7 FLOAT, snr_run8 FLOAT, stillpa0B FLOAT, stillpa0T FLOAT, stillpa1B FLOAT, stillpa1T FLOAT, stillpa2B FLOAT, stillpa2T FLOAT, stillpa3B FLOAT, stillpa3T FLOAT, stillpa4B FLOAT, stillpa4T FLOAT, stillpa5B FLOAT, stillpa5T FLOAT, stillpa6B FLOAT, stillpa6T FLOAT, stillpa7B FLOAT, stillpa7T FLOAT, stillpa8B FLOAT, stillpa8T FLOAT, stillpa9B FLOAT, stillpa9T FLOAT, stillpa10B FLOAT, stillpa10T FLOAT, stillpa11B FLOAT, stillpa11T FLOAT, movepa0B FLOAT, movepa0T FLOAT, movepa1B FLOAT, movepa1T FLOAT, movepa2B FLOAT, movepa2T FLOAT, movepa3B FLOAT, movepa3T FLOAT, movepa4B FLOAT, movepa4T FLOAT, movepa5B FLOAT, movepa5T FLOAT, movepa6B FLOAT, movepa6T FLOAT, movepa7B FLOAT, movepa7T FLOAT, movepa8B FLOAT, movepa8T FLOAT, movepa9B FLOAT, movepa9T FLOAT, movepa10B FLOAT , movepa10T FLOAT, movepa11B FLOAT, movepa11T FLOAT, stillnoise0B FLOAT, stillnoise0T FLOAT, stillnoise1B FLOAT, stillnoise1T FLOAT, stillnoise2B FLOAT, stillnoise2T FLOAT, stillnoise3B FLOAT, stillnoise3T FLOAT, stillnoise4B FLOAT, stillnoise4T FLOAT, stillnoise5B FLOAT, stillnoise5T FLOAT, stillnoise6B FLOAT, stillnoise6T FLOAT, stillnoise7B FLOAT, stillnoise7T FLOAT, stillnoise8B FLOAT, stillnoise8T FLOAT, stillnoise9B FLOAT, stillnoise9T FLOAT, stillnoise10B FLOAT, stillnoise10T FLOAT, stillnoise11B FLOAT, stillnoise11T FLOAT, movenoise0B FLOAT, movenoise0T FLOAT, movenoise1B FLOAT, movenoise1T FLOAT, movenoise2B FLOAT, movenoise2T FLOAT, movenoise3B FLOAT, movenoise3T FLOAT, movenoise4B FLOAT, movenoise4T FLOAT, movenoise5B FLOAT, movenoise5T FLOAT, movenoise6B FLOAT, movenoise6T FLOAT, movenoise7B FLOAT, movenoise7T FLOAT, movenoise8B FLOAT, movenoise8T FLOAT, movenoise9B FLOAT, movenoise9T FLOAT, movenoise10B FLOAT, movenoise10T FLOAT, movenoise11B FLOAT, movenoise11T FLOAT);' + cursor.execute(create_table_statement) + except MySQLdb.Error, e: + print "Error !!!!!" % (e.args[0], e.args[1]) + sys.exit (1) + + +create_new_table() From noreply at svn.ci.uchicago.edu Fri Mar 6 01:23:21 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 6 Mar 2009 01:23:21 -0600 (CST) Subject: [Swift-commit] r2644 - SwiftApps/SIDGrid/swift/projects/andric/peakfit_pilots/PK2/PK2surfaces/surfaceData Message-ID: <20090306072321.5124D2281F4@www.ci.uchicago.edu> Author: andric Date: 2009-03-06 01:23:20 -0600 (Fri, 06 Mar 2009) New Revision: 2644 Added: SwiftApps/SIDGrid/swift/projects/andric/peakfit_pilots/PK2/PK2surfaces/surfaceData/insertData.py Log: Python script using MySQLdb module to insert data into MySQL tables Added: SwiftApps/SIDGrid/swift/projects/andric/peakfit_pilots/PK2/PK2surfaces/surfaceData/insertData.py =================================================================== --- SwiftApps/SIDGrid/swift/projects/andric/peakfit_pilots/PK2/PK2surfaces/surfaceData/insertData.py (rev 0) +++ SwiftApps/SIDGrid/swift/projects/andric/peakfit_pilots/PK2/PK2surfaces/surfaceData/insertData.py 2009-03-06 07:23:20 UTC (rev 2644) @@ -0,0 +1,35 @@ +#!/usr/bin/python + +import commands +import os +import sys +import MySQLdb +import time + + +hemis = ['lh','rh'] + +try: + connection = MySQLdb.connect(read_default_file="~/.my.cnf",db="HEL") +except MySQLdb.Error, e: + print "Error %d: %s" % (e.args[0], e.args[1]) + sys.exit (1) + +cursor = connection.cursor() + +def insertdata_to_db(h): + print "que hora es?\n"+time.ctime() + try: + file = "'/disks/ci-gpfs/fmri/cnari/swift/projects/andric/peakfit_pilots/PK2/PK2surfaces/surfaceData/PK2_"+h+"_fordb.txt'" + print "File loading: "+file + insert_statement = "load data local infile "+file+" into table peakFunc_data"+h+" fields terminated by ' ';" + print "Insert statement: "+insert_statement + cursor.execute(insert_statement) + print "Number of rows inserted: %d" % cursor.rowcount + except MySQLdb.Error, e: + print "Error loading peakFunc_data"+h+" %d -->> %s" % (e.args[0], e.args[1]) + sys.exit (1) + + +for h in hemis: + insertdata_to_db(h) From noreply at svn.ci.uchicago.edu Fri Mar 6 02:10:01 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 6 Mar 2009 02:10:01 -0600 (CST) Subject: [Swift-commit] r2645 - SwiftApps/SIDGrid/swift/projects/andric/peakfit_pilots/PK2/PK2surfaces/surfaceData Message-ID: <20090306081002.0AD962281FF@www.ci.uchicago.edu> Author: andric Date: 2009-03-06 02:10:00 -0600 (Fri, 06 Mar 2009) New Revision: 2645 Modified: SwiftApps/SIDGrid/swift/projects/andric/peakfit_pilots/PK2/PK2surfaces/surfaceData/Vol2Surf.swift Log: Projecting the time series to surface Modified: SwiftApps/SIDGrid/swift/projects/andric/peakfit_pilots/PK2/PK2surfaces/surfaceData/Vol2Surf.swift =================================================================== --- SwiftApps/SIDGrid/swift/projects/andric/peakfit_pilots/PK2/PK2surfaces/surfaceData/Vol2Surf.swift 2009-03-06 07:23:20 UTC (rev 2644) +++ SwiftApps/SIDGrid/swift/projects/andric/peakfit_pilots/PK2/PK2surfaces/surfaceData/Vol2Surf.swift 2009-03-06 08:10:00 UTC (rev 2645) @@ -17,18 +17,18 @@ string declarelist[] = ["PK2"]; string mapFunctions[] = ["ave"]; string hemis[] = ["lh","rh"]; -string conditions[] = ["DeconvResults"]; +string conditions[] = ["TScat_Dtrnd.nrmlzdPerChng"]; foreach subject in declarelist{ foreach map in mapFunctions{ foreach h in hemis{ foreach cc in conditions{ - string baseDir = @strcat("/disks/ci-gpfs/fmri/cnari/swift/projects/andric/peakfitTESTS/",subject,"/"); + string baseDir = @strcat("/disks/ci-gpfs/fmri/cnari/swift/projects/andric/peakfit_pilots/",subject,"/"); file spec_file; file smoothwm; file pial; AFNI_obj surfvol; - AFNI_obj grid_parent; + AFNI_obj grid_parent; file output1D; output1D = AFNI_Vol2Surf(spec_file, smoothwm, pial, map, surfvol, grid_parent); } From noreply at svn.ci.uchicago.edu Fri Mar 6 07:39:27 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 6 Mar 2009 07:39:27 -0600 (CST) Subject: [Swift-commit] r2646 - trunk/docs Message-ID: <20090306133927.1FEFA228187@www.ci.uchicago.edu> Author: benc Date: 2009-03-06 07:39:26 -0600 (Fri, 06 Mar 2009) New Revision: 2646 Modified: trunk/docs/quickstartguide.xml trunk/docs/reallyquickstartguide.xml Log: Change use of elements for console interactions into elements. elements undergo magic syntax highlighting under the swiftsh_html.xsl style sheet that the quickstart guides were switched to in r2614 Modified: trunk/docs/quickstartguide.xml =================================================================== --- trunk/docs/quickstartguide.xml 2009-03-06 08:10:00 UTC (rev 2645) +++ trunk/docs/quickstartguide.xml 2009-03-06 13:39:26 UTC (rev 2646) @@ -96,10 +96,10 @@ class="file">swift-<version>.tar.gz) into a directory of your choice: - + > tar swift-<version>.tar.gz - + This will create a swift-<version> directory @@ -120,10 +120,10 @@ be a good idea to add this directory to your PATH environment variable: - + > export PATH=/path/to/swift/bin:$PATH - + If you are installing Swift on a machine with access to a graphical display, you can run the cog-setup tool to @@ -209,9 +209,9 @@ will be inherited from the shared configuration file. Properties are specified in the following format: - + name=value - + For details about the various properties Swift accepts, please take a look at the swift command, which takes the Swift workflow file name as an argument: - + > swift - + The Swift Modified: trunk/docs/reallyquickstartguide.xml =================================================================== --- trunk/docs/reallyquickstartguide.xml 2009-03-06 08:10:00 UTC (rev 2645) +++ trunk/docs/reallyquickstartguide.xml 2009-03-06 13:39:26 UTC (rev 2646) @@ -76,11 +76,11 @@ Use the example site catalog and transformation catalog (they are configured for local submission): - + cd swift-xyz/etc cp sites.xml.example sites.xml cp tc.data.example tc.data - + From noreply at svn.ci.uchicago.edu Sun Mar 8 17:11:04 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 8 Mar 2009 17:11:04 -0500 (CDT) Subject: [Swift-commit] r2647 - trunk Message-ID: <20090308221104.5731B2281F3@www.ci.uchicago.edu> Author: hategan Date: 2009-03-08 17:11:03 -0500 (Sun, 08 Mar 2009) New Revision: 2647 Modified: trunk/build.xml Log: 256M default max heap size (bug 109) Modified: trunk/build.xml =================================================================== --- trunk/build.xml 2009-03-06 13:39:26 UTC (rev 2646) +++ trunk/build.xml 2009-03-08 22:11:03 UTC (rev 2647) @@ -108,6 +108,10 @@ LOCALCLASSPATH=$SWIFT_HOME/etc$CPDELIM$SWIFT_HOME/libexec + HEAPMAX= + HEAPMAX=256M + + updateOptions "$SWIFT_HOME" "SWIFT_HOME" updateOptions "$SWIFT_HOME" "COG_INSTALL_PATH" From noreply at svn.ci.uchicago.edu Mon Mar 9 03:28:44 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 03:28:44 -0500 (CDT) Subject: [Swift-commit] r2648 - log-processing/libexec/log-processing Message-ID: <20090309082845.1547E2281D8@www.ci.uchicago.edu> Author: benc Date: 2009-03-09 03:28:43 -0500 (Mon, 09 Mar 2009) New Revision: 2648 Modified: log-processing/libexec/log-processing/sort-preserve Log: Change sort invocation to be more posixy Modified: log-processing/libexec/log-processing/sort-preserve =================================================================== --- log-processing/libexec/log-processing/sort-preserve 2009-03-08 22:11:03 UTC (rev 2647) +++ log-processing/libexec/log-processing/sort-preserve 2009-03-09 08:28:43 UTC (rev 2648) @@ -12,7 +12,7 @@ I=$(( $I + 1 )) done > sort-preserve.tmp -sort $@ -n +0f -2 < sort-preserve.tmp > sort-preserve2.tmp +sort $@ -n -k 1f,3 < sort-preserve.tmp > sort-preserve2.tmp sed 's/^$[^ ]*$ $[^ ]*$ $[^ ]*$$.*$/\1 \2\4/' < sort-preserve2.tmp From noreply at svn.ci.uchicago.edu Mon Mar 9 03:29:35 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 03:29:35 -0500 (CDT) Subject: [Swift-commit] r2649 - in log-processing: . tests tests/log-processing Message-ID: <20090309082935.4D77D2281D8@www.ci.uchicago.edu> Author: benc Date: 2009-03-09 03:29:34 -0500 (Mon, 09 Mar 2009) New Revision: 2649 Added: log-processing/tests/ log-processing/tests/log-processing/ log-processing/tests/log-processing/README Log: skeleton for log-processing tests Added: log-processing/tests/log-processing/README =================================================================== --- log-processing/tests/log-processing/README (rev 0) +++ log-processing/tests/log-processing/README 2009-03-09 08:29:34 UTC (rev 2649) @@ -0,0 +1,11 @@ +There are 109 executables in the log-processing libexec directory. + +One test for each would be useful. + +However, some of these executables are probably unused, in which case +removing them would be a better thing to do. + +For the text processing ones, a diff based approach would be good. For +the ones that output graphs or other hard-to-compare data, checking that +a non-trivial output file of some kind is made is probably the best thing +to do. From noreply at svn.ci.uchicago.edu Mon Mar 9 03:30:14 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 03:30:14 -0500 (CDT) Subject: [Swift-commit] r2650 - log-processing/libexec/log-processing Message-ID: <20090309083014.AC21C2281FB@www.ci.uchicago.edu> Author: benc Date: 2009-03-09 03:30:14 -0500 (Mon, 09 Mar 2009) New Revision: 2650 Modified: log-processing/libexec/log-processing/sort-preserve Log: fix timestamp ordering bug in sort-preserve when the timestamp fields are of different length (which occurs rarely in practice) Modified: log-processing/libexec/log-processing/sort-preserve =================================================================== --- log-processing/libexec/log-processing/sort-preserve 2009-03-09 08:29:34 UTC (rev 2649) +++ log-processing/libexec/log-processing/sort-preserve 2009-03-09 08:30:14 UTC (rev 2650) @@ -3,8 +3,18 @@ # sort name timestamp rest # by name, then by timestamp, then by order in file -# do this by numbering each line as a second field and sorting on that. +# This differs from sorting on the first two fields. In sort-preserve, where +# two lines have the same key (the same first two fields), then those two +# lines will always be output in the same order as they appear in the +# input file. This is not the case for POSIX sort. +# This by numbering each line as a third field and using that as a third +# key; and then removing that field after sorting. + +# TODO the temporary files should be made properly unique, and removed +# after use (or removed entirely, with this entire script made into a +# single pipe) + I=0 while read name timestamp rest; do @@ -12,7 +22,7 @@ I=$(( $I + 1 )) done > sort-preserve.tmp -sort $@ -n -k 1f,3 < sort-preserve.tmp > sort-preserve2.tmp +sort $@ -k 1f,1f -k 2n,2n -k 3n,3n < sort-preserve.tmp > sort-preserve2.tmp sed 's/^$[^ ]*$ $[^ ]*$ $[^ ]*$$.*$/\1 \2\4/' < sort-preserve2.tmp From noreply at svn.ci.uchicago.edu Mon Mar 9 03:31:41 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 03:31:41 -0500 (CDT) Subject: [Swift-commit] r2651 - in log-processing/tests/log-processing: . sort-preserve Message-ID: <20090309083141.DC7FD2281FB@www.ci.uchicago.edu> Author: benc Date: 2009-03-09 03:31:41 -0500 (Mon, 09 Mar 2009) New Revision: 2651 Added: log-processing/tests/log-processing/run log-processing/tests/log-processing/sort-preserve/ log-processing/tests/log-processing/sort-preserve/1.in log-processing/tests/log-processing/sort-preserve/1.out log-processing/tests/log-processing/sort-preserve/2.in log-processing/tests/log-processing/sort-preserve/2.out log-processing/tests/log-processing/sort-preserve/3.in log-processing/tests/log-processing/sort-preserve/3.out log-processing/tests/log-processing/sort-preserve/4.in log-processing/tests/log-processing/sort-preserve/4.out log-processing/tests/log-processing/sort-preserve/5.in log-processing/tests/log-processing/sort-preserve/5.out log-processing/tests/log-processing/sort-preserve/run Log: test and doc for sort-preserve Added: log-processing/tests/log-processing/run =================================================================== --- log-processing/tests/log-processing/run (rev 0) +++ log-processing/tests/log-processing/run 2009-03-09 08:31:41 UTC (rev 2651) @@ -0,0 +1,11 @@ +#!/bin/bash + +pushd sort-preserve +./run +if [ "$?" -ne "0" ]; then + echo "sort-preserve test failed" + exit 1 +fi + +popd + Property changes on: log-processing/tests/log-processing/run ___________________________________________________________________ Name: svn:executable + * Added: log-processing/tests/log-processing/sort-preserve/1.in =================================================================== --- log-processing/tests/log-processing/sort-preserve/1.in (rev 0) +++ log-processing/tests/log-processing/sort-preserve/1.in 2009-03-09 08:31:41 UTC (rev 2651) @@ -0,0 +1,11 @@ +foo 1 I +foo 2 II +foo 3 III +foo 4 IV +foo 5 V +foo 6 VI +foo 7 VII +foo 8 VIII +foo 9 IX +foo 10 X +foo 11 XI Added: log-processing/tests/log-processing/sort-preserve/1.out =================================================================== --- log-processing/tests/log-processing/sort-preserve/1.out (rev 0) +++ log-processing/tests/log-processing/sort-preserve/1.out 2009-03-09 08:31:41 UTC (rev 2651) @@ -0,0 +1,11 @@ +foo 1 I +foo 2 II +foo 3 III +foo 4 IV +foo 5 V +foo 6 VI +foo 7 VII +foo 8 VIII +foo 9 IX +foo 10 X +foo 11 XI Added: log-processing/tests/log-processing/sort-preserve/2.in =================================================================== --- log-processing/tests/log-processing/sort-preserve/2.in (rev 0) +++ log-processing/tests/log-processing/sort-preserve/2.in 2009-03-09 08:31:41 UTC (rev 2651) @@ -0,0 +1,11 @@ +foo 11 I +foo 10 II +foo 9 III +foo 8 IV +foo 7 V +foo 6 VI +foo 5 VII +foo 4 VIII +foo 3 IX +foo 2 X +foo 1 XI Added: log-processing/tests/log-processing/sort-preserve/2.out =================================================================== --- log-processing/tests/log-processing/sort-preserve/2.out (rev 0) +++ log-processing/tests/log-processing/sort-preserve/2.out 2009-03-09 08:31:41 UTC (rev 2651) @@ -0,0 +1,11 @@ +foo 1 XI +foo 2 X +foo 3 IX +foo 4 VIII +foo 5 VII +foo 6 VI +foo 7 V +foo 8 IV +foo 9 III +foo 10 II +foo 11 I Added: log-processing/tests/log-processing/sort-preserve/3.in =================================================================== --- log-processing/tests/log-processing/sort-preserve/3.in (rev 0) +++ log-processing/tests/log-processing/sort-preserve/3.in 2009-03-09 08:31:41 UTC (rev 2651) @@ -0,0 +1,11 @@ +bar 3 III +foo 1 I +foo 5 V +bar 10 X +bar 6 VI +foo 4 IV +foo 7 VII +foo 2 II +foo 9 IX +qux 8 VIII +fol 11 XI Added: log-processing/tests/log-processing/sort-preserve/3.out =================================================================== --- log-processing/tests/log-processing/sort-preserve/3.out (rev 0) +++ log-processing/tests/log-processing/sort-preserve/3.out 2009-03-09 08:31:41 UTC (rev 2651) @@ -0,0 +1,11 @@ +bar 3 III +bar 6 VI +bar 10 X +fol 11 XI +foo 1 I +foo 2 II +foo 4 IV +foo 5 V +foo 7 VII +foo 9 IX +qux 8 VIII Added: log-processing/tests/log-processing/sort-preserve/4.in =================================================================== --- log-processing/tests/log-processing/sort-preserve/4.in (rev 0) +++ log-processing/tests/log-processing/sort-preserve/4.in 2009-03-09 08:31:41 UTC (rev 2651) @@ -0,0 +1,16 @@ +bar 3 III +foo 1 I +foo 5 AAAAA +foo 5 CCCCC +foo 5 BBBBB +bar 10 X +bar 6 VI +foo 4 IV2 +foo 4 IV1 +foo 7 VII +foo 5 EEEEE +foo 5 DDDDD +foo 2 II +foo 9 IX +qux 8 VIII +fol 11 XI Added: log-processing/tests/log-processing/sort-preserve/4.out =================================================================== --- log-processing/tests/log-processing/sort-preserve/4.out (rev 0) +++ log-processing/tests/log-processing/sort-preserve/4.out 2009-03-09 08:31:41 UTC (rev 2651) @@ -0,0 +1,16 @@ +bar 3 III +bar 6 VI +bar 10 X +fol 11 XI +foo 1 I +foo 2 II +foo 4 IV2 +foo 4 IV1 +foo 5 AAAAA +foo 5 CCCCC +foo 5 BBBBB +foo 5 EEEEE +foo 5 DDDDD +foo 7 VII +foo 9 IX +qux 8 VIII Added: log-processing/tests/log-processing/sort-preserve/5.in =================================================================== --- log-processing/tests/log-processing/sort-preserve/5.in (rev 0) +++ log-processing/tests/log-processing/sort-preserve/5.in 2009-03-09 08:31:41 UTC (rev 2651) @@ -0,0 +1,16 @@ +bar 3 III +foo 1 I +foo 5.32789 AAAAA +foo 5.32787 CCCCC +foo 5.32787 BBBBB +bar 10 X +bar 6 VI +foo 4 IV2 +foo 4 IV1 +foo 7 VII +foo 5 EEEEE +foo 5 DDDDD +foo 2 II +foo 9 IX +qux 8 VIII +fol 11 XI Added: log-processing/tests/log-processing/sort-preserve/5.out =================================================================== --- log-processing/tests/log-processing/sort-preserve/5.out (rev 0) +++ log-processing/tests/log-processing/sort-preserve/5.out 2009-03-09 08:31:41 UTC (rev 2651) @@ -0,0 +1,16 @@ +bar 3 III +bar 6 VI +bar 10 X +fol 11 XI +foo 1 I +foo 2 II +foo 4 IV2 +foo 4 IV1 +foo 5 EEEEE +foo 5 DDDDD +foo 5.32787 CCCCC +foo 5.32787 BBBBB +foo 5.32789 AAAAA +foo 7 VII +foo 9 IX +qux 8 VIII Added: log-processing/tests/log-processing/sort-preserve/run =================================================================== --- log-processing/tests/log-processing/sort-preserve/run (rev 0) +++ log-processing/tests/log-processing/sort-preserve/run 2009-03-09 08:31:41 UTC (rev 2651) @@ -0,0 +1,16 @@ +#!/bin/bash + +rm -f *.tmp + +for a in 1 2 3 4 5 ; do + + cat ${a}.in | sort-preserve > ${a}.tmp + + diff -q ${a}.tmp ${a}.out + + if [ "$?" -ne "0" ]; then + echo "OUTPUT DIFFERS for test case $a in sort-preserve" + exit 1 + fi + +done Property changes on: log-processing/tests/log-processing/sort-preserve/run ___________________________________________________________________ Name: svn:executable + * From noreply at svn.ci.uchicago.edu Mon Mar 9 03:32:36 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 03:32:36 -0500 (CDT) Subject: [Swift-commit] r2652 - in log-processing/tests/log-processing: . logs Message-ID: <20090309083236.67C152281FB@www.ci.uchicago.edu> Author: benc Date: 2009-03-09 03:32:35 -0500 (Mon, 09 Mar 2009) New Revision: 2652 Added: log-processing/tests/log-processing/logs/ log-processing/tests/log-processing/logs/first-r2522.log Log: Add example log to be used in log-processing testing Added: log-processing/tests/log-processing/logs/first-r2522.log =================================================================== --- log-processing/tests/log-processing/logs/first-r2522.log (rev 0) +++ log-processing/tests/log-processing/logs/first-r2522.log 2009-03-09 08:32:35 UTC (rev 2652) @@ -0,0 +1,210 @@ +2009-03-08 20:28:37,424+0100 DEBUG Loader kmlversion is >040EAFAD-C2D7-4E4B-9879-899F8B0F9D67< +2009-03-08 20:28:37,425+0100 DEBUG Loader build version is >F86E8FF4-B906-4D6D-AC76-860FA86310B9< +2009-03-08 20:28:37,425+0100 INFO Loader first.swift: source file was compiled with a different version of Swift. Recompiling. +2009-03-08 20:28:38,145+0100 INFO Karajan Validation of XML intermediate file was successful +2009-03-08 20:28:38,153+0100 INFO VariableScope New scope 11922340 with no parent. +2009-03-08 20:28:38,173+0100 INFO VariableScope New scope 13706772 with no parent. +2009-03-08 20:28:38,173+0100 INFO VariableScope New scope 1766184 with no parent. +2009-03-08 20:28:38,175+0100 INFO VariableScope Adding variable t of type messagefile to scope 1766184 +2009-03-08 20:28:38,244+0100 INFO VariableScope Adding variable outfile of type messagefile to scope 11922340 +2009-03-08 20:28:38,248+0100 INFO VariableScope thats the declaration for outfile +2009-03-08 20:28:39,293+0100 INFO unknown Using sites file: /Users/benc/work/cog/modules/swift/dist/swift-svn/bin/../etc/sites.xml +2009-03-08 20:28:39,294+0100 INFO unknown Using tc.data: /Users/benc/work/cog/modules/swift/dist/swift-svn/bin/../etc/tc.data +2009-03-08 20:28:39,496+0100 INFO AbstractScheduler Setting resources to: {localhost=localhost} +2009-03-08 20:28:40,028+0100 INFO unknown Swift svn swift-r2522 (swift modified locally) cog-r2322 + +2009-03-08 20:28:40,032+0100 INFO unknown RUNID id=tag:benc at ci.uchicago.edu,2007:swift:run:20090308-2028-mfnmsxx2 +2009-03-08 20:28:40,063+0100 INFO AbstractDataNode closed org.griphyn.vdl.mapping.RootDataNode identifier tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000001 type string value=Hello, world! dataset=unnamed SwiftScript value (closed) +2009-03-08 20:28:40,063+0100 INFO AbstractDataNode ROOTPATH dataset=tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000001 path=$ +2009-03-08 20:28:40,063+0100 INFO AbstractDataNode VALUE dataset=tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000001 VALUE=Hello, world! +2009-03-08 20:28:40,063+0100 INFO AbstractDataNode closed org.griphyn.vdl.mapping.RootDataNode identifier tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000001 type string value=Hello, world! dataset=unnamed SwiftScript value (closed) +2009-03-08 20:28:40,063+0100 INFO AbstractDataNode ROOTPATH dataset=tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000001 path=$ +2009-03-08 20:28:40,063+0100 INFO AbstractDataNode VALUE dataset=tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000001 VALUE=Hello, world! +2009-03-08 20:28:40,110+0100 INFO New NEW id=tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000001 +2009-03-08 20:28:40,144+0100 INFO AbstractDataNode Found mapped data org.griphyn.vdl.mapping.RootDataNode identifier tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000002 type messagefile with no value at dataset=outfile (not closed).$ +2009-03-08 20:28:40,145+0100 INFO New NEW id=tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000002 +2009-03-08 20:28:40,187+0100 INFO greeting PROCEDURE thread=0 name=greeting +2009-03-08 20:28:40,188+0100 INFO vdl:parameterlog PARAM thread=0 direction=output variable=t provenanceid=tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000002 +2009-03-08 20:28:40,194+0100 INFO AbstractDataNode closed org.griphyn.vdl.mapping.RootDataNode identifier tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000003 type string value=hello.txt dataset=unnamed SwiftScript value (closed) +2009-03-08 20:28:40,194+0100 INFO AbstractDataNode ROOTPATH dataset=tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000003 path=$ +2009-03-08 20:28:40,194+0100 INFO AbstractDataNode VALUE dataset=tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000003 VALUE=hello.txt +2009-03-08 20:28:40,197+0100 INFO AbstractDataNode closed org.griphyn.vdl.mapping.RootDataNode identifier tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000003 type string value=hello.txt dataset=unnamed SwiftScript value (closed) +2009-03-08 20:28:40,197+0100 INFO AbstractDataNode ROOTPATH dataset=tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000003 path=$ +2009-03-08 20:28:40,197+0100 INFO AbstractDataNode VALUE dataset=tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000003 VALUE=hello.txt +2009-03-08 20:28:40,203+0100 INFO vdl:execute START thread=0 tr=echo +2009-03-08 20:28:40,217+0100 INFO WeightedHostScoreScheduler Sorted: [localhost:0.000(1.000):0/1 overload: 0] +2009-03-08 20:28:40,217+0100 DEBUG WeightedHostScoreScheduler Rand: 0.004711219396950295, sum: 1.0 +2009-03-08 20:28:40,217+0100 DEBUG WeightedHostScoreScheduler Next contact: localhost:0.000(1.000):0/1 overload: 0 +2009-03-08 20:28:40,220+0100 INFO vdl:initshareddir START host=localhost - Initializing shared directory +2009-03-08 20:28:40,222+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.000(1.000):1/1 overload: 0, -0.01) +2009-03-08 20:28:40,222+0100 DEBUG WeightedHostScoreScheduler Old score: 0.000, new score: -0.010 +2009-03-08 20:28:40,225+0100 INFO GlobalSubmitQueue No global submit throttle set. Using default (100) +2009-03-08 20:28:40,264+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520147) setting status to Submitting +2009-03-08 20:28:40,264+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520147) setting status to Submitted +2009-03-08 20:28:40,265+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520147) setting status to Active +2009-03-08 20:28:40,265+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520147) setting status to Completed +2009-03-08 20:28:40,265+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:-0.010(0.994):1/1 overload: -52, 0.01) +2009-03-08 20:28:40,266+0100 DEBUG WeightedHostScoreScheduler Old score: -0.010, new score: 0.000 +2009-03-08 20:28:40,266+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.000(1.000):1/1 overload: 0, 0.1) +2009-03-08 20:28:40,266+0100 DEBUG WeightedHostScoreScheduler Old score: 0.000, new score: 0.100 +2009-03-08 20:28:40,266+0100 INFO LateBindingScheduler Task(type=FILE_OPERATION, identity=urn:0-1-1236540520147) Completed. Waiting: 0, Running: 0. Heap size: 10M, Heap free: 1M, Max heap: 63M +2009-03-08 20:28:40,269+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.100(1.060):1/1 overload: 0, -0.2) +2009-03-08 20:28:40,269+0100 DEBUG WeightedHostScoreScheduler Old score: 0.100, new score: -0.100 +2009-03-08 20:28:40,284+0100 DEBUG TaskImpl Task(type=FILE_TRANSFER, identity=urn:0-1-1236540520150) setting status to Submitting +2009-03-08 20:28:40,284+0100 DEBUG TaskImpl Task(type=FILE_TRANSFER, identity=urn:0-1-1236540520150) setting status to Submitted +2009-03-08 20:28:40,284+0100 DEBUG TaskImpl Task(type=FILE_TRANSFER, identity=urn:0-1-1236540520150) setting status to Active +2009-03-08 20:28:40,285+0100 DEBUG TaskImpl Task(type=FILE_TRANSFER, identity=urn:0-1-1236540520150) setting status to Completed +2009-03-08 20:28:40,285+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:-0.100(0.943):1/1 overload: -39, 0.2) +2009-03-08 20:28:40,286+0100 DEBUG WeightedHostScoreScheduler Old score: -0.100, new score: 0.100 +2009-03-08 20:28:40,286+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.100(1.060):1/1 overload: 0, 0.1) +2009-03-08 20:28:40,286+0100 DEBUG WeightedHostScoreScheduler Old score: 0.100, new score: 0.200 +2009-03-08 20:28:40,286+0100 INFO LateBindingScheduler Task(type=FILE_TRANSFER, identity=urn:0-1-1236540520150) Completed. Waiting: 0, Running: 0. Heap size: 10M, Heap free: 1M, Max heap: 63M +2009-03-08 20:28:40,287+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.200(1.124):1/1 overload: 0, -0.2) +2009-03-08 20:28:40,287+0100 DEBUG WeightedHostScoreScheduler Old score: 0.200, new score: 0.000 +2009-03-08 20:28:40,287+0100 DEBUG TaskImpl Task(type=FILE_TRANSFER, identity=urn:0-1-1236540520153) setting status to Submitting +2009-03-08 20:28:40,287+0100 DEBUG TaskImpl Task(type=FILE_TRANSFER, identity=urn:0-1-1236540520153) setting status to Submitted +2009-03-08 20:28:40,287+0100 DEBUG TaskImpl Task(type=FILE_TRANSFER, identity=urn:0-1-1236540520153) setting status to Active +2009-03-08 20:28:40,288+0100 DEBUG TaskImpl Task(type=FILE_TRANSFER, identity=urn:0-1-1236540520153) setting status to Completed +2009-03-08 20:28:40,288+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.000(1.000):1/1 overload: 0, 0.2) +2009-03-08 20:28:40,288+0100 DEBUG WeightedHostScoreScheduler Old score: 0.000, new score: 0.200 +2009-03-08 20:28:40,289+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.200(1.124):1/1 overload: 0, 0.1) +2009-03-08 20:28:40,289+0100 DEBUG WeightedHostScoreScheduler Old score: 0.200, new score: 0.300 +2009-03-08 20:28:40,289+0100 INFO LateBindingScheduler Task(type=FILE_TRANSFER, identity=urn:0-1-1236540520153) Completed. Waiting: 0, Running: 0. Heap size: 10M, Heap free: 1M, Max heap: 63M +2009-03-08 20:28:40,290+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.300(1.192):1/1 overload: 0, -0.01) +2009-03-08 20:28:40,290+0100 DEBUG WeightedHostScoreScheduler Old score: 0.300, new score: 0.290 +2009-03-08 20:28:40,290+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520156) setting status to Submitting +2009-03-08 20:28:40,290+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520156) setting status to Submitted +2009-03-08 20:28:40,290+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520156) setting status to Active +2009-03-08 20:28:40,290+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520156) setting status to Completed +2009-03-08 20:28:40,290+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.290(1.185):1/1 overload: 0, 0.01) +2009-03-08 20:28:40,291+0100 DEBUG WeightedHostScoreScheduler Old score: 0.290, new score: 0.300 +2009-03-08 20:28:40,291+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.300(1.192):1/1 overload: 0, 0.1) +2009-03-08 20:28:40,291+0100 DEBUG WeightedHostScoreScheduler Old score: 0.300, new score: 0.400 +2009-03-08 20:28:40,291+0100 INFO LateBindingScheduler Task(type=FILE_OPERATION, identity=urn:0-1-1236540520156) Completed. Waiting: 0, Running: 0. Heap size: 10M, Heap free: 1M, Max heap: 63M +2009-03-08 20:28:40,292+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.400(1.264):1/1 overload: 0, -0.01) +2009-03-08 20:28:40,292+0100 DEBUG WeightedHostScoreScheduler Old score: 0.400, new score: 0.390 +2009-03-08 20:28:40,293+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520158) setting status to Submitting +2009-03-08 20:28:40,293+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520158) setting status to Submitted +2009-03-08 20:28:40,293+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520158) setting status to Active +2009-03-08 20:28:40,293+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520158) setting status to Completed +2009-03-08 20:28:40,293+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.390(1.256):1/1 overload: 0, 0.01) +2009-03-08 20:28:40,293+0100 DEBUG WeightedHostScoreScheduler Old score: 0.390, new score: 0.400 +2009-03-08 20:28:40,293+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.400(1.264):1/1 overload: 0, 0.1) +2009-03-08 20:28:40,294+0100 DEBUG WeightedHostScoreScheduler Old score: 0.400, new score: 0.500 +2009-03-08 20:28:40,294+0100 INFO LateBindingScheduler Task(type=FILE_OPERATION, identity=urn:0-1-1236540520158) Completed. Waiting: 0, Running: 0. Heap size: 10M, Heap free: 1M, Max heap: 63M +2009-03-08 20:28:40,294+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.500(1.339):1/1 overload: 0, -0.01) +2009-03-08 20:28:40,294+0100 DEBUG WeightedHostScoreScheduler Old score: 0.500, new score: 0.490 +2009-03-08 20:28:40,295+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520160) setting status to Submitting +2009-03-08 20:28:40,295+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520160) setting status to Submitted +2009-03-08 20:28:40,295+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520160) setting status to Active +2009-03-08 20:28:40,295+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520160) setting status to Completed +2009-03-08 20:28:40,296+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.490(1.332):1/1 overload: 0, 0.01) +2009-03-08 20:28:40,296+0100 DEBUG WeightedHostScoreScheduler Old score: 0.490, new score: 0.500 +2009-03-08 20:28:40,296+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.500(1.339):1/1 overload: 0, 0.1) +2009-03-08 20:28:40,296+0100 DEBUG WeightedHostScoreScheduler Old score: 0.500, new score: 0.600 +2009-03-08 20:28:40,296+0100 INFO LateBindingScheduler Task(type=FILE_OPERATION, identity=urn:0-1-1236540520160) Completed. Waiting: 0, Running: 0. Heap size: 10M, Heap free: 1M, Max heap: 63M +2009-03-08 20:28:40,299+0100 INFO vdl:initshareddir END host=localhost - Done initializing shared directory +2009-03-08 20:28:40,301+0100 DEBUG vdl:execute2 THREAD_ASSOCIATION jobid=echo-8f60cn7j thread=0-1 host=localhost replicationGroup=7f60cn7j +2009-03-08 20:28:40,308+0100 INFO vdl:createdirset START jobid=echo-8f60cn7j host=localhost - Initializing directory structure +2009-03-08 20:28:40,310+0100 INFO vdl:createdirs START path= dir=first-20090308-2028-mfnmsxx2/shared - Creating directory structure +2009-03-08 20:28:40,310+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.600(1.419):1/1 overload: 0, -0.01) +2009-03-08 20:28:40,310+0100 DEBUG WeightedHostScoreScheduler Old score: 0.600, new score: 0.590 +2009-03-08 20:28:40,310+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520162) setting status to Submitting +2009-03-08 20:28:40,311+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520162) setting status to Submitted +2009-03-08 20:28:40,311+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520162) setting status to Active +2009-03-08 20:28:40,311+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520162) setting status to Completed +2009-03-08 20:28:40,311+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.590(1.411):1/1 overload: 0, 0.01) +2009-03-08 20:28:40,311+0100 DEBUG WeightedHostScoreScheduler Old score: 0.590, new score: 0.600 +2009-03-08 20:28:40,311+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.600(1.419):1/1 overload: 0, 0.1) +2009-03-08 20:28:40,311+0100 DEBUG WeightedHostScoreScheduler Old score: 0.600, new score: 0.700 +2009-03-08 20:28:40,311+0100 INFO LateBindingScheduler Task(type=FILE_OPERATION, identity=urn:0-1-1236540520162) Completed. Waiting: 0, Running: 0. Heap size: 10M, Heap free: 1M, Max heap: 63M +2009-03-08 20:28:40,312+0100 INFO vdl:createdirset END jobid=echo-8f60cn7j - Done initializing directory structure +2009-03-08 20:28:40,312+0100 INFO vdl:dostagein START jobid=echo-8f60cn7j - Staging in files +2009-03-08 20:28:40,313+0100 INFO vdl:dostagein END jobid=echo-8f60cn7j - Staging in finished +2009-03-08 20:28:40,314+0100 DEBUG vdl:execute2 JOB_START jobid=echo-8f60cn7j tr=echo arguments=[Hello, world!] tmpdir=first-20090308-2028-mfnmsxx2/jobs/8/echo-8f60cn7j host=localhost +2009-03-08 20:28:40,328+0100 INFO Execute jobid=echo-8f60cn7j task=Task(type=JOB_SUBMISSION, identity=urn:0-1-1236540520164) +2009-03-08 20:28:40,328+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.700(1.503):1/1 overload: 0, -0.2) +2009-03-08 20:28:40,328+0100 DEBUG WeightedHostScoreScheduler Old score: 0.700, new score: 0.500 +2009-03-08 20:28:40,333+0100 DEBUG TaskImpl Task(type=JOB_SUBMISSION, identity=urn:0-1-1236540520164) setting status to Submitting +2009-03-08 20:28:40,333+0100 INFO JobSubmissionTaskHandler Submitting task Task(type=JOB_SUBMISSION, identity=urn:0-1-1236540520164) +2009-03-08 20:28:40,333+0100 DEBUG TaskImpl Task(type=JOB_SUBMISSION, identity=urn:0-1-1236540520164) setting status to Submitted +2009-03-08 20:28:40,334+0100 DEBUG WeightedHostScoreScheduler Submission time for Task(type=JOB_SUBMISSION, identity=urn:0-1-1236540520164): 0ms. Score delta: 0.002564102564102564 +2009-03-08 20:28:40,334+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.500(1.339):1/1 overload: 0, 0.002564102564102564) +2009-03-08 20:28:40,334+0100 DEBUG WeightedHostScoreScheduler Old score: 0.500, new score: 0.503 +2009-03-08 20:28:40,352+0100 DEBUG TaskImpl Task(type=JOB_SUBMISSION, identity=urn:0-1-1236540520164) setting status to Active +2009-03-08 20:28:40,517+0100 DEBUG TaskImpl Task(type=JOB_SUBMISSION, identity=urn:0-1-1236540520164) setting status to Completed +2009-03-08 20:28:40,518+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.503(1.341):1/1 overload: 0, 0.2) +2009-03-08 20:28:40,518+0100 DEBUG WeightedHostScoreScheduler Old score: 0.503, new score: 0.703 +2009-03-08 20:28:40,518+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.703(1.506):1/1 overload: 0, 0.1) +2009-03-08 20:28:40,519+0100 DEBUG WeightedHostScoreScheduler Old score: 0.703, new score: 0.803 +2009-03-08 20:28:40,519+0100 INFO LateBindingScheduler Task(type=JOB_SUBMISSION, identity=urn:0-1-1236540520164) Completed. Waiting: 0, Running: 0. Heap size: 10M, Heap free: 1M, Max heap: 63M +2009-03-08 20:28:40,522+0100 DEBUG vdl:checkjobstatus START jobid=echo-8f60cn7j +2009-03-08 20:28:40,523+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.803(1.595):1/1 overload: 0, -0.01) +2009-03-08 20:28:40,523+0100 DEBUG WeightedHostScoreScheduler Old score: 0.803, new score: 0.793 +2009-03-08 20:28:40,523+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520166) setting status to Submitting +2009-03-08 20:28:40,523+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520166) setting status to Submitted +2009-03-08 20:28:40,524+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520166) setting status to Active +2009-03-08 20:28:40,524+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1236540520166) setting status to Completed +2009-03-08 20:28:40,524+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.793(1.585):1/1 overload: 0, 0.01) +2009-03-08 20:28:40,525+0100 DEBUG WeightedHostScoreScheduler Old score: 0.793, new score: 0.803 +2009-03-08 20:28:40,525+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.803(1.595):1/1 overload: 0, 0.1) +2009-03-08 20:28:40,525+0100 DEBUG WeightedHostScoreScheduler Old score: 0.803, new score: 0.903 +2009-03-08 20:28:40,526+0100 INFO LateBindingScheduler Task(type=FILE_OPERATION, identity=urn:0-1-1236540520166) Completed. Waiting: 0, Running: 0. Heap size: 10M, Heap free: 1M, Max heap: 63M +2009-03-08 20:28:40,526+0100 INFO vdl:checkjobstatus SUCCESS jobid=echo-8f60cn7j - Success file found +2009-03-08 20:28:40,526+0100 DEBUG vdl:execute2 STAGING_OUT jobid=echo-8f60cn7j +2009-03-08 20:28:40,527+0100 INFO vdl:dostageout START jobid=echo-8f60cn7j - Staging out files +2009-03-08 20:28:40,533+0100 DEBUG vdl:dostageout FILE_STAGE_OUT_START srcname=hello.txt srcdir=first-20090308-2028-mfnmsxx2/shared/ srchost=localhost destdir= desthost=localhost provider=file +2009-03-08 20:28:40,533+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1-1236540520168) setting status to Submitting +2009-03-08 20:28:40,534+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1-1236540520168) setting status to Submitted +2009-03-08 20:28:40,534+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1-1236540520168) setting status to Active +2009-03-08 20:28:40,534+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1-1236540520168) setting status to Completed +2009-03-08 20:28:40,534+0100 INFO LateBindingScheduler Task(type=FILE_OPERATION, identity=urn:0-1-1-1236540520168) Completed. Waiting: 0, Running: 0. Heap size: 10M, Heap free: 1M, Max heap: 63M +2009-03-08 20:28:40,535+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.903(1.688):1/1 overload: 0, -0.2) +2009-03-08 20:28:40,535+0100 DEBUG WeightedHostScoreScheduler Old score: 0.903, new score: 0.703 +2009-03-08 20:28:40,536+0100 DEBUG TaskImpl Task(type=FILE_TRANSFER, identity=urn:0-1-1-1236540520171) setting status to Submitting +2009-03-08 20:28:40,536+0100 DEBUG TaskImpl Task(type=FILE_TRANSFER, identity=urn:0-1-1-1236540520171) setting status to Submitted +2009-03-08 20:28:40,536+0100 DEBUG TaskImpl Task(type=FILE_TRANSFER, identity=urn:0-1-1-1236540520171) setting status to Active +2009-03-08 20:28:40,537+0100 DEBUG TaskImpl Task(type=FILE_TRANSFER, identity=urn:0-1-1-1236540520171) setting status to Completed +2009-03-08 20:28:40,537+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.703(1.506):1/1 overload: 0, 0.2) +2009-03-08 20:28:40,538+0100 DEBUG WeightedHostScoreScheduler Old score: 0.703, new score: 0.903 +2009-03-08 20:28:40,538+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.903(1.688):1/1 overload: 0, 0.1) +2009-03-08 20:28:40,538+0100 DEBUG WeightedHostScoreScheduler Old score: 0.903, new score: 1.003 +2009-03-08 20:28:40,538+0100 INFO LateBindingScheduler Task(type=FILE_TRANSFER, identity=urn:0-1-1-1236540520171) Completed. Waiting: 0, Running: 0. Heap size: 10M, Heap free: 1M, Max heap: 63M +2009-03-08 20:28:40,539+0100 DEBUG vdl:dostageout FILE_STAGE_OUT_END srcname=hello.txt srcdir=first-20090308-2028-mfnmsxx2/shared/ srchost=localhost destdir= desthost=localhost provider=file +2009-03-08 20:28:40,539+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1-1236540520174) setting status to Submitting +2009-03-08 20:28:40,539+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1-1236540520174) setting status to Submitted +2009-03-08 20:28:40,540+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1-1236540520174) setting status to Active +2009-03-08 20:28:40,542+0100 DEBUG TaskImpl Task(type=FILE_OPERATION, identity=urn:0-1-1-1236540520174) setting status to Completed +2009-03-08 20:28:40,542+0100 INFO LateBindingScheduler Task(type=FILE_OPERATION, identity=urn:0-1-1-1236540520174) Completed. Waiting: 0, Running: 0. Heap size: 10M, Heap free: 1M, Max heap: 63M +2009-03-08 20:28:40,548+0100 INFO vdl:dostageout END jobid=echo-8f60cn7j - Staging out finished +2009-03-08 20:28:40,552+0100 DEBUG vdl:execute2 JOB_END jobid=echo-8f60cn7j +2009-03-08 20:28:40,553+0100 DEBUG WeightedHostScoreScheduler Releasing contact 2 +2009-03-08 20:28:40,554+0100 DEBUG WeightedHostScoreScheduler commitDelayedScore(localhost:1.003(1.786):0/1 overload: 0, 0.1 +2009-03-08 20:28:40,556+0100 INFO vdl:execute END_SUCCESS thread=0 tr=echo +2009-03-08 20:28:40,558+0100 INFO CloseDataset Closing org.griphyn.vdl.mapping.RootDataNode identifier tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000002 type messagefile with no value at dataset=outfile (not closed) +2009-03-08 20:28:40,558+0100 INFO AbstractDataNode closed org.griphyn.vdl.mapping.RootDataNode identifier tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000002 type messagefile with no value at dataset=outfile (closed) +2009-03-08 20:28:40,559+0100 INFO AbstractDataNode ROOTPATH dataset=tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000002 path=$ +2009-03-08 20:28:40,559+0100 INFO AbstractDataNode FILENAME dataset=tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000002 filename=file://localhost/hello.txt +2009-03-08 20:28:40,561+0100 INFO CloseDataset Partially closing org.griphyn.vdl.mapping.RootDataNode identifier tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000002 type messagefile with no value at dataset=outfile (closed) for statement 88000 +2009-03-08 20:28:40,561+0100 INFO CloseDataset var is org.griphyn.vdl.mapping.RootDataNode identifier tag:benc at ci.uchicago.edu,2008:swift:dataset:20090308-2028-yxq17tk9:720000000002 type messagefile with no value at dataset=outfile (closed) +2009-03-08 20:28:40,561+0100 INFO CloseDataset var hash is 142508 +2009-03-08 20:28:40,561+0100 INFO CloseDataset variable already closed - skipping partial close processing +2009-03-08 20:28:40,607+0100 INFO vdl:cleanups START cleanups=[[first-20090308-2028-mfnmsxx2, localhost]] +2009-03-08 20:28:40,608+0100 INFO vdl:cleanup START dir=first-20090308-2028-mfnmsxx2 host=localhost +2009-03-08 20:28:40,611+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:1.103(1.889):0/1 overload: 0, -0.2) +2009-03-08 20:28:40,611+0100 DEBUG WeightedHostScoreScheduler Old score: 1.103, new score: 0.903 +2009-03-08 20:28:40,611+0100 DEBUG TaskImpl Task(type=JOB_SUBMISSION, identity=urn:0-1-1236540520176) setting status to Submitting +2009-03-08 20:28:40,611+0100 INFO JobSubmissionTaskHandler Submitting task Task(type=JOB_SUBMISSION, identity=urn:0-1-1236540520176) +2009-03-08 20:28:40,611+0100 DEBUG TaskImpl Task(type=JOB_SUBMISSION, identity=urn:0-1-1236540520176) setting status to Submitted +2009-03-08 20:28:40,611+0100 DEBUG WeightedHostScoreScheduler Submission time for Task(type=JOB_SUBMISSION, identity=urn:0-1-1236540520176): 0ms. Score delta: 0.002564102564102564 +2009-03-08 20:28:40,611+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.903(1.688):0/1 overload: 0, 0.002564102564102564) +2009-03-08 20:28:40,612+0100 DEBUG WeightedHostScoreScheduler Old score: 0.903, new score: 0.905 +2009-03-08 20:28:40,612+0100 DEBUG TaskImpl Task(type=JOB_SUBMISSION, identity=urn:0-1-1236540520176) setting status to Active +2009-03-08 20:28:40,612+0100 DEBUG TaskImpl Task(type=JOB_SUBMISSION, identity=urn:0-1-1236540520176) setting status to Completed +2009-03-08 20:28:40,628+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:0.905(1.691):0/1 overload: 0, 0.2) +2009-03-08 20:28:40,628+0100 DEBUG WeightedHostScoreScheduler Old score: 0.905, new score: 1.105 +2009-03-08 20:28:40,632+0100 DEBUG WeightedHostScoreScheduler multiplyScore(localhost:1.105(1.892):0/1 overload: 0, 0.1) +2009-03-08 20:28:40,633+0100 DEBUG WeightedHostScoreScheduler Old score: 1.105, new score: 1.205 +2009-03-08 20:28:40,633+0100 INFO LateBindingScheduler Task(type=JOB_SUBMISSION, identity=urn:0-1-1236540520176) Completed. Waiting: 0, Running: 0. Heap size: 10M, Heap free: 1M, Max heap: 63M +2009-03-08 20:28:40,638+0100 INFO vdl:cleanup END dir=first-20090308-2028-mfnmsxx2 host=localhost +2009-03-08 20:28:40,644+0100 DEBUG Loader Swift finished with no errors From noreply at svn.ci.uchicago.edu Mon Mar 9 03:33:32 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 03:33:32 -0500 (CDT) Subject: [Swift-commit] r2653 - in log-processing/tests/log-processing: . make-targets Message-ID: <20090309083332.6B7BA2281FB@www.ci.uchicago.edu> Author: benc Date: 2009-03-09 03:33:32 -0500 (Mon, 09 Mar 2009) New Revision: 2653 Added: log-processing/tests/log-processing/make-targets/ log-processing/tests/log-processing/make-targets/execute.event.first-r2522.expected log-processing/tests/log-processing/make-targets/execute2.event.first-r2522.expected log-processing/tests/log-processing/make-targets/run Modified: log-processing/tests/log-processing/run Log: Test framework for checking makefile targets generated by swift-plot-log against expected output of sample log file Added: log-processing/tests/log-processing/make-targets/execute.event.first-r2522.expected =================================================================== --- log-processing/tests/log-processing/make-targets/execute.event.first-r2522.expected (rev 0) +++ log-processing/tests/log-processing/make-targets/execute.event.first-r2522.expected 2009-03-09 08:33:32 UTC (rev 2653) @@ -0,0 +1 @@ +1236540520.203 0.352999925613403 0 END_SUCCESS echo Added: log-processing/tests/log-processing/make-targets/execute2.event.first-r2522.expected =================================================================== --- log-processing/tests/log-processing/make-targets/execute2.event.first-r2522.expected (rev 0) +++ log-processing/tests/log-processing/make-targets/execute2.event.first-r2522.expected 2009-03-09 08:33:32 UTC (rev 2653) @@ -0,0 +1 @@ +1236540520.301 0.250999927520752 echo-8f60cn7j JOB_END 0-1 localhost 7f60cn7j Added: log-processing/tests/log-processing/make-targets/run =================================================================== --- log-processing/tests/log-processing/make-targets/run (rev 0) +++ log-processing/tests/log-processing/make-targets/run 2009-03-09 08:33:32 UTC (rev 2653) @@ -0,0 +1,23 @@ +#!/bin/bash + +TARGETS="execute.event execute2.event" + + +LOG=$(pwd)/../logs/first-r2522.log + +rm -f $TARGETS +for t in $TARGETS ; do + swift-plot-log $LOG $t + + l=$(basename $LOG .log) + + + diff -q $t $t.$l.expected + + if [ "$?" -ne "0" ]; then + echo "OUTPUT DIFFERS for log $l in target $t" + exit 1 + fi +done + + Property changes on: log-processing/tests/log-processing/make-targets/run ___________________________________________________________________ Name: svn:executable + * Modified: log-processing/tests/log-processing/run =================================================================== --- log-processing/tests/log-processing/run 2009-03-09 08:32:35 UTC (rev 2652) +++ log-processing/tests/log-processing/run 2009-03-09 08:33:32 UTC (rev 2653) @@ -1,11 +1,14 @@ #!/bin/bash -pushd sort-preserve -./run -if [ "$?" -ne "0" ]; then - echo "sort-preserve test failed" - exit 1 -fi +for d in sort-preserve make-targets; do -popd + pushd $d + ./run + if [ "$?" -ne "0" ]; then + echo "sort-preserve test failed" + exit 1 + fi + popd +done + From noreply at svn.ci.uchicago.edu Mon Mar 9 03:34:19 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 03:34:19 -0500 (CDT) Subject: [Swift-commit] r2654 - log-processing/tests/log-processing/make-targets Message-ID: <20090309083419.96DB62281FB@www.ci.uchicago.edu> Author: benc Date: 2009-03-09 03:34:19 -0500 (Mon, 09 Mar 2009) New Revision: 2654 Added: log-processing/tests/log-processing/make-targets/execute.global.event.first-r2522.expected Modified: log-processing/tests/log-processing/make-targets/run Log: test generation of execute.global.event Added: log-processing/tests/log-processing/make-targets/execute.global.event.first-r2522.expected =================================================================== --- log-processing/tests/log-processing/make-targets/execute.global.event.first-r2522.expected (rev 0) +++ log-processing/tests/log-processing/make-targets/execute.global.event.first-r2522.expected 2009-03-09 08:34:19 UTC (rev 2654) @@ -0,0 +1 @@ +1236540520.203 0.352999925613403 tag:benc at ci.uchicago.edu,2008:swiftlogs:execute:first-r2522:0 0 END_SUCCESS echo Modified: log-processing/tests/log-processing/make-targets/run =================================================================== --- log-processing/tests/log-processing/make-targets/run 2009-03-09 08:33:32 UTC (rev 2653) +++ log-processing/tests/log-processing/make-targets/run 2009-03-09 08:34:19 UTC (rev 2654) @@ -1,6 +1,6 @@ #!/bin/bash -TARGETS="execute.event execute2.event" +TARGETS="execute.event execute2.event execute.global.event" LOG=$(pwd)/../logs/first-r2522.log From noreply at svn.ci.uchicago.edu Mon Mar 9 03:35:08 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 03:35:08 -0500 (CDT) Subject: [Swift-commit] r2655 - log-processing/tests/log-processing/make-targets Message-ID: <20090309083508.9AE8D2281F6@www.ci.uchicago.edu> Author: benc Date: 2009-03-09 03:35:08 -0500 (Mon, 09 Mar 2009) New Revision: 2655 Added: log-processing/tests/log-processing/make-targets/workflow.event.first-r2522.expected Modified: log-processing/tests/log-processing/make-targets/run Log: Test generation of workflow.event Modified: log-processing/tests/log-processing/make-targets/run =================================================================== --- log-processing/tests/log-processing/make-targets/run 2009-03-09 08:34:19 UTC (rev 2654) +++ log-processing/tests/log-processing/make-targets/run 2009-03-09 08:35:08 UTC (rev 2655) @@ -1,6 +1,6 @@ #!/bin/bash -TARGETS="execute.event execute2.event execute.global.event" +TARGETS="execute.event execute2.event execute.global.event workflow.event" LOG=$(pwd)/../logs/first-r2522.log Added: log-processing/tests/log-processing/make-targets/workflow.event.first-r2522.expected =================================================================== --- log-processing/tests/log-processing/make-targets/workflow.event.first-r2522.expected (rev 0) +++ log-processing/tests/log-processing/make-targets/workflow.event.first-r2522.expected 2009-03-09 08:35:08 UTC (rev 2655) @@ -0,0 +1 @@ +1236540517.424 3.220 tag:benc at ci.uchicago.edu,2007:swift:run:20090308-2028-mfnmsxx2 /Users/benc/work/swift-svn.git/vdl2/log-processing/tests/log-processing/make-targets/../logs/first-r2522.log From noreply at svn.ci.uchicago.edu Mon Mar 9 05:14:27 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 05:14:27 -0500 (CDT) Subject: [Swift-commit] r2656 - nmi-build-test Message-ID: <20090309101427.4791F2281FB@www.ci.uchicago.edu> Author: benc Date: 2009-03-09 05:14:26 -0500 (Mon, 09 Mar 2009) New Revision: 2656 Added: nmi-build-test/log-processing Log: script to drive log-processing tests on NMI build/test Added: nmi-build-test/log-processing =================================================================== --- nmi-build-test/log-processing (rev 0) +++ nmi-build-test/log-processing 2009-03-09 10:14:26 UTC (rev 2656) @@ -0,0 +1,7 @@ +#!/bin/bash +cd log-processing/tests/log-processing + +echo running tests... +./run || exit 1 + +echo log-processing tests all finished Property changes on: nmi-build-test/log-processing ___________________________________________________________________ Name: svn:executable + * From noreply at svn.ci.uchicago.edu Mon Mar 9 05:28:25 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 05:28:25 -0500 (CDT) Subject: [Swift-commit] r2657 - nmi-build-test Message-ID: <20090309102825.58CCA22814E@www.ci.uchicago.edu> Author: benc Date: 2009-03-09 05:28:24 -0500 (Mon, 09 Mar 2009) New Revision: 2657 Modified: nmi-build-test/log-processing Log: need to put log processing stuff on path before running tests on NMI Modified: nmi-build-test/log-processing =================================================================== --- nmi-build-test/log-processing 2009-03-09 10:14:26 UTC (rev 2656) +++ nmi-build-test/log-processing 2009-03-09 10:28:24 UTC (rev 2657) @@ -1,6 +1,10 @@ #!/bin/bash + +export PATH=`pwd`/log-processing/bin:`pwd`/log-processing/libexec/log-processing:$PATH + cd log-processing/tests/log-processing + echo running tests... ./run || exit 1 From noreply at svn.ci.uchicago.edu Mon Mar 9 05:51:48 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 05:51:48 -0500 (CDT) Subject: [Swift-commit] r2658 - log-processing/tests/log-processing Message-ID: <20090309105148.61FE42281F6@www.ci.uchicago.edu> Author: benc Date: 2009-03-09 05:51:47 -0500 (Mon, 09 Mar 2009) New Revision: 2658 Modified: log-processing/tests/log-processing/run Log: fix error output when test fails Modified: log-processing/tests/log-processing/run =================================================================== --- log-processing/tests/log-processing/run 2009-03-09 10:28:24 UTC (rev 2657) +++ log-processing/tests/log-processing/run 2009-03-09 10:51:47 UTC (rev 2658) @@ -5,7 +5,7 @@ pushd $d ./run if [ "$?" -ne "0" ]; then - echo "sort-preserve test failed" + echo "log-processing test $d failed" exit 1 fi From noreply at svn.ci.uchicago.edu Mon Mar 9 07:22:28 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 07:22:28 -0500 (CDT) Subject: [Swift-commit] r2659 - log-processing/tests/log-processing/make-targets Message-ID: <20090309122228.279F622814E@www.ci.uchicago.edu> Author: benc Date: 2009-03-09 07:22:24 -0500 (Mon, 09 Mar 2009) New Revision: 2659 Modified: log-processing/tests/log-processing/make-targets/run Log: output diff when makefile test fails Modified: log-processing/tests/log-processing/make-targets/run =================================================================== --- log-processing/tests/log-processing/make-targets/run 2009-03-09 10:51:47 UTC (rev 2658) +++ log-processing/tests/log-processing/make-targets/run 2009-03-09 12:22:24 UTC (rev 2659) @@ -16,6 +16,7 @@ if [ "$?" -ne "0" ]; then echo "OUTPUT DIFFERS for log $l in target $t" + diff -u $t $t.$l.expected exit 1 fi done From noreply at svn.ci.uchicago.edu Mon Mar 9 07:30:36 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 07:30:36 -0500 (CDT) Subject: [Swift-commit] r2660 - log-processing/tests/log-processing/make-targets Message-ID: <20090309123036.2FB322281D8@www.ci.uchicago.edu> Author: benc Date: 2009-03-09 07:30:35 -0500 (Mon, 09 Mar 2009) New Revision: 2660 Removed: log-processing/tests/log-processing/make-targets/workflow.event.first-r2522.expected Modified: log-processing/tests/log-processing/make-targets/run Log: workflow.event contains a full path, so we cannot test using this mechanism - this commit removes that test case. Modified: log-processing/tests/log-processing/make-targets/run =================================================================== --- log-processing/tests/log-processing/make-targets/run 2009-03-09 12:22:24 UTC (rev 2659) +++ log-processing/tests/log-processing/make-targets/run 2009-03-09 12:30:35 UTC (rev 2660) @@ -1,6 +1,6 @@ #!/bin/bash -TARGETS="execute.event execute2.event execute.global.event workflow.event" +TARGETS="execute.event execute2.event execute.global.event" LOG=$(pwd)/../logs/first-r2522.log Deleted: log-processing/tests/log-processing/make-targets/workflow.event.first-r2522.expected =================================================================== --- log-processing/tests/log-processing/make-targets/workflow.event.first-r2522.expected 2009-03-09 12:22:24 UTC (rev 2659) +++ log-processing/tests/log-processing/make-targets/workflow.event.first-r2522.expected 2009-03-09 12:30:35 UTC (rev 2660) @@ -1 +0,0 @@ -1236540517.424 3.220 tag:benc at ci.uchicago.edu,2007:swift:run:20090308-2028-mfnmsxx2 /Users/benc/work/swift-svn.git/vdl2/log-processing/tests/log-processing/make-targets/../logs/first-r2522.log From noreply at svn.ci.uchicago.edu Mon Mar 9 09:45:20 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 09:45:20 -0500 (CDT) Subject: [Swift-commit] r2661 - provenancedb Message-ID: <20090309144520.2E0222281FB@www.ci.uchicago.edu> Author: benc Date: 2009-03-09 09:45:19 -0500 (Mon, 09 Mar 2009) New Revision: 2661 Modified: provenancedb/import-all provenancedb/prov-to-sql.sh Log: Fix paths for calling log-processing code, due to changes in log-processing module that have happened since provenancedb was last worked on. Modified: provenancedb/import-all =================================================================== --- provenancedb/import-all 2009-03-09 12:30:35 UTC (rev 2660) +++ provenancedb/import-all 2009-03-09 14:45:19 UTC (rev 2661) @@ -4,8 +4,8 @@ # of individual runs will clean other files. source paths.source +export PATH=$LOGDIR:$PATH - pushd $LOGDIR ./create-everylog-vs-versions-data $LOGREPO popd @@ -25,10 +25,12 @@ if [ $version -ge 1538 ]; then echo -n "Log: $filename ... " -EXISTING=$(psql -p 5435 -d $PROVDB -U benc --tuples-only -c "select count(*) from known_workflows where workflow_log_filename='$filename';") -# echo EXISTING is X${EXISTING}X +# TODO: this only works for psql, not for other DBs, so for now force it +# to be non-existing +#EXISTING=$(psql -p 5435 -d $PROVDB -U benc --tuples-only -c "select count(*) from known_workflows where workflow_log_filename='$filename';") +EXISTING=0 -if [ $EXISTING -eq 0 ]; then +if [ "$EXISTING" -eq "0" ]; then echo IMPORTING if grep --silent "DEBUG Loader Swift finished with no errors" $filename; then Modified: provenancedb/prov-to-sql.sh =================================================================== Property changes on: provenancedb/prov-to-sql.sh ___________________________________________________________________ Name: svn:executable + * From noreply at svn.ci.uchicago.edu Mon Mar 9 09:45:53 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 09:45:53 -0500 (CDT) Subject: [Swift-commit] r2662 - provenancedb Message-ID: <20090309144553.9BF572281FB@www.ci.uchicago.edu> Author: benc Date: 2009-03-09 09:45:53 -0500 (Mon, 09 Mar 2009) New Revision: 2662 Modified: provenancedb/import-all Log: make import-all only do sql, not xml Modified: provenancedb/import-all =================================================================== --- provenancedb/import-all 2009-03-09 14:45:19 UTC (rev 2661) +++ provenancedb/import-all 2009-03-09 14:45:53 UTC (rev 2662) @@ -39,7 +39,7 @@ echo ============= will import ============= ./prepare-for-import $filename ./import-run-to-sql $filename - ./import-run-to-xml $filename +# ./import-run-to-xml $filename $SQLCMD -c "INSERT INTO known_workflows (workflow_log_filename, version, importstatus) VALUES ('$filename','$version','imported');" else echo SKIP: Workflow does not report successful end. @@ -53,5 +53,5 @@ # now pull the XML data into eXist, in as few runs as possible to avoid # JVM startup. -ls `pwd`/*.log.xml | time xargs ~/work/eXist/bin/client.sh -m /db/prov -ouri=xmldb:exist:// -p +# ls `pwd`/*.log.xml | time xargs ~/work/eXist/bin/client.sh -m /db/prov -ouri=xmldb:exist:// -p From noreply at svn.ci.uchicago.edu Mon Mar 9 09:46:28 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 09:46:28 -0500 (CDT) Subject: [Swift-commit] r2663 - provenancedb Message-ID: <20090309144628.EA7862281FB@www.ci.uchicago.edu> Author: benc Date: 2009-03-09 09:46:28 -0500 (Mon, 09 Mar 2009) New Revision: 2663 Modified: provenancedb/prov-init.sql provenancedb/provenance.xml Log: Documentation on how to run own provenance database Modified: provenancedb/prov-init.sql =================================================================== --- provenancedb/prov-init.sql 2009-03-09 14:45:53 UTC (rev 2662) +++ provenancedb/prov-init.sql 2009-03-09 14:46:28 UTC (rev 2663) @@ -68,5 +68,7 @@ duration numeric ); +-- this does not work for sqlite; you'll get a syntax error but +-- ignore it grant all on dataset_containment, dataset_filenames, dataset_usage, executes_in_workflows, invocation_procedure_names, known_workflows, workflow_events to public; Modified: provenancedb/provenance.xml =================================================================== --- provenancedb/provenance.xml 2009-03-09 14:45:53 UTC (rev 2662) +++ provenancedb/provenance.xml 2009-03-09 14:46:28 UTC (rev 2663) @@ -140,6 +140,42 @@

Running your own provenance database +This section details running your own SQL-based provenance database on +servers of your own control. + +

Configuring your SQL database + +This section describes configuring the SQL scripts to use +sqlite, which is +appropriate for a single-user installation. + +Install sqlite3. On OS X with MacPorts, this command works: + +$ sudo port install sqlite3 + +Similar commands using apt or yum will +probably work under Linux. + + +Now create a paths.source file to define local +configuration. An example that I use on my laptop is present in +paths.source.soju. +The SQLCMD indicates which command to run for SQL +access. This is used by other scripts to access the database. The +LOGREPO and IDIR variables should +point to the directory under which you collect your Swift logs. + + +Now import your logs for the first time like this: + +$ ./import-all rebuild + + +

+ +

What this work does not address This work explicitly excludes a number of uses which traditionally From noreply at svn.ci.uchicago.edu Mon Mar 9 10:12:24 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 10:12:24 -0500 (CDT) Subject: [Swift-commit] r2664 - provenancedb Message-ID: <20090309151224.9D7EC2281F3@www.ci.uchicago.edu> Author: benc Date: 2009-03-09 10:12:24 -0500 (Mon, 09 Mar 2009) New Revision: 2664 Modified: provenancedb/provenance.xml Log: note on checking out SVN code in the own-database section Modified: provenancedb/provenance.xml =================================================================== --- provenancedb/provenance.xml 2009-03-09 14:46:28 UTC (rev 2663) +++ provenancedb/provenance.xml 2009-03-09 15:12:24 UTC (rev 2664) @@ -144,6 +144,19 @@ This section details running your own SQL-based provenance database on servers of your own control. +

Check out the latest SVN code + +Use the following two commands to check out the provenancedb +and log-processing modules: + + +svn co https://svn.ci.uchicago.edu/svn/vdl2/provenancedb +svn co https://svn.ci.uchicago.edu/svn/vdl2/log-processing + + +

+ +

Configuring your SQL database This section describes configuring the SQL scripts to use From noreply at svn.ci.uchicago.edu Mon Mar 9 10:23:16 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 10:23:16 -0500 (CDT) Subject: [Swift-commit] r2665 - in trunk: . etc libexec resources Message-ID: <20090309152317.32F3E2281F6@www.ci.uchicago.edu> Author: hategan Date: 2009-03-09 10:23:13 -0500 (Mon, 09 Mar 2009) New Revision: 2665 Modified: trunk/CHANGES.txt trunk/etc/swift.properties trunk/libexec/vdl-lib.xml trunk/resources/Karajan.stg Log: limit number of concurrent iterations in foreach Modified: trunk/CHANGES.txt =================================================================== --- trunk/CHANGES.txt 2009-03-09 15:12:24 UTC (rev 2664) +++ trunk/CHANGES.txt 2009-03-09 15:23:13 UTC (rev 2665) @@ -1,3 +1,9 @@ +(03/09/09) +*** Added a throttling parallelFor and changed the swift foreach underlying + implementation to it. The throttling parallelFor limits the number of + concurrent iterations allowed, which should allow swift to scale better + in certain cases. + (02/09/24) *** New configuration option wrapper.invocation.mode, specifiable either globally in the configuration file or per-site as a profile key, Modified: trunk/etc/swift.properties =================================================================== --- trunk/etc/swift.properties 2009-03-09 15:12:24 UTC (rev 2664) +++ trunk/etc/swift.properties 2009-03-09 15:23:13 UTC (rev 2665) @@ -302,3 +302,13 @@ # valid values: absolute, relative # wrapper.invocation.mode=absolute +# +# Limits the number of concurrent iterations that each foreach statement +# can have at one time. This conserves memory for swift programs that +# have large numbers of iterations (which would otherwise all be executed +# in parallel). +# +# Default: 1024 +# + +foreach.max.threads=1024 Modified: trunk/libexec/vdl-lib.xml =================================================================== --- trunk/libexec/vdl-lib.xml 2009-03-09 15:12:24 UTC (rev 2664) +++ trunk/libexec/vdl-lib.xml 2009-03-09 15:23:13 UTC (rev 2665) @@ -90,5 +90,7 @@ + + Modified: trunk/resources/Karajan.stg =================================================================== --- trunk/resources/Karajan.stg 2009-03-09 15:12:24 UTC (rev 2664) +++ trunk/resources/Karajan.stg 2009-03-09 15:23:13 UTC (rev 2665) @@ -200,7 +200,7 @@ >> foreach(var,in,indexVar,declarations,statements) ::= << - + $in$ @@ -214,7 +214,7 @@ $if(statements)$ $parallel(statements=statements)$ $endif$ - + >> call(func, outputs, inputs) ::= << From noreply at svn.ci.uchicago.edu Mon Mar 9 11:03:35 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 11:03:35 -0500 (CDT) Subject: [Swift-commit] r2666 - trunk/src/org/griphyn/vdl/engine Message-ID: <20090309160335.DF4BD22814E@www.ci.uchicago.edu> Author: hategan Date: 2009-03-09 11:03:35 -0500 (Mon, 09 Mar 2009) New Revision: 2666 Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java trunk/src/org/griphyn/vdl/engine/VariableScope.java Log: fixed nested procedure calls causing variable redefinition error Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/Karajan.java 2009-03-09 15:23:13 UTC (rev 2665) +++ trunk/src/org/griphyn/vdl/engine/Karajan.java 2009-03-09 16:03:35 UTC (rev 2666) @@ -1131,7 +1131,7 @@ StringTemplate call = template("callexpr"); String type = funcSignature.getOutputArray(0).getType(); - subscope.addVariable("swift#callintermediate", type); + subscope.addInternalVariable("swift#callintermediate", type); call.setAttribute("datatype", type); call.setAttribute("call", call(c, subscope, true)); Modified: trunk/src/org/griphyn/vdl/engine/VariableScope.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/VariableScope.java 2009-03-09 15:23:13 UTC (rev 2665) +++ trunk/src/org/griphyn/vdl/engine/VariableScope.java 2009-03-09 16:03:35 UTC (rev 2666) @@ -91,8 +91,24 @@ varTypes.put(name, type); if(!added) throw new CompilationException("Could not add variable "+name+" to scope."); } + + /** + * Does pretty much the same as addVariable() except it doesn't throw + * an exception if the variable is defined in a parent scope + */ + public void addInternalVariable(String name, String type) throws CompilationException { + logger.info("Adding internal variable " + name + " of type " + type + " to scope " + hashCode()); + + if(isVariableLocallyDefined(name)) { + throw new CompilationException("Variable " + name + " is already defined."); + } + boolean added = variables.add(name); + varTypes.put(name, type); + if(!added) throw new CompilationException("Could not add variable "+name+" to scope."); + } + public boolean isVariableDefined(String name) { if(isVariableLocallyDefined(name)) return true; if(parentScope != null && parentScope.isVariableDefined(name)) return true; From noreply at svn.ci.uchicago.edu Mon Mar 9 11:06:24 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 11:06:24 -0500 (CDT) Subject: [Swift-commit] r2667 - trunk/tests/language-behaviour Message-ID: <20090309160624.A85502281D8@www.ci.uchicago.edu> Author: hategan Date: 2009-03-09 11:06:24 -0500 (Mon, 09 Mar 2009) New Revision: 2667 Added: trunk/tests/language-behaviour/0026-function-param.swift Log: added test for nested procedure invocation Added: trunk/tests/language-behaviour/0026-function-param.swift =================================================================== --- trunk/tests/language-behaviour/0026-function-param.swift (rev 0) +++ trunk/tests/language-behaviour/0026-function-param.swift 2009-03-09 16:06:24 UTC (rev 2667) @@ -0,0 +1,7 @@ +(int r) add(int a, int b) { + r = a + b; +} + +trace(add(1, 1)); + +trace(add(add(1, 1), add(2, 2))); From noreply at svn.ci.uchicago.edu Mon Mar 9 11:16:39 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 11:16:39 -0500 (CDT) Subject: [Swift-commit] r2668 - provenancedb Message-ID: <20090309161639.2D6DC228223@www.ci.uchicago.edu> Author: benc Date: 2009-03-09 11:16:38 -0500 (Mon, 09 Mar 2009) New Revision: 2668 Modified: provenancedb/import-run-to-sql provenancedb/import-run-to-xml provenancedb/prepare-for-import Log: assume provenance executables are on path, and do not assume they are in pwd Modified: provenancedb/import-run-to-sql =================================================================== --- provenancedb/import-run-to-sql 2009-03-09 16:06:24 UTC (rev 2667) +++ provenancedb/import-run-to-sql 2009-03-09 16:16:38 UTC (rev 2668) @@ -9,5 +9,5 @@ # with kickstart records expected to be in the same directory as the # log file. -./prov-to-sql.sh $1 +prov-to-sql.sh $1 Modified: provenancedb/import-run-to-xml =================================================================== --- provenancedb/import-run-to-xml 2009-03-09 16:06:24 UTC (rev 2667) +++ provenancedb/import-run-to-xml 2009-03-09 16:16:38 UTC (rev 2668) @@ -11,7 +11,7 @@ export TF=`pwd`/provenance-$(basename $1).xml -./prov-to-xml.sh $1 > $TF +prov-to-xml.sh $1 > $TF # ~/work/eXist/bin/client.sh -m /db/prov -p $TF -ouri=xmldb:exist:// Modified: provenancedb/prepare-for-import =================================================================== --- provenancedb/prepare-for-import 2009-03-09 16:06:24 UTC (rev 2667) +++ provenancedb/prepare-for-import 2009-03-09 16:16:38 UTC (rev 2668) @@ -8,5 +8,5 @@ make LOG=$1 clean execute.global.event execute2.event workflow.event popd -./prepare-provenance-chart $1 +prepare-provenance-chart $1 From noreply at svn.ci.uchicago.edu Mon Mar 9 11:17:17 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 11:17:17 -0500 (CDT) Subject: [Swift-commit] r2669 - provenancedb Message-ID: <20090309161717.3C527228223@www.ci.uchicago.edu> Author: benc Date: 2009-03-09 11:17:16 -0500 (Mon, 09 Mar 2009) New Revision: 2669 Modified: provenancedb/import-all Log: fixed to import-all based on discussion with Luiz M. R. Gadelha Jr. Modified: provenancedb/import-all =================================================================== --- provenancedb/import-all 2009-03-09 16:16:38 UTC (rev 2668) +++ provenancedb/import-all 2009-03-09 16:17:16 UTC (rev 2669) @@ -1,13 +1,15 @@ #!/bin/bash +PROVDIR=$(pwd)/$(dirname $0)/ + # we need to keep this out of the log-proceesing dir because import # of individual runs will clean other files. source paths.source -export PATH=$LOGDIR:$PATH +export PATH=$LOGDIR:$PROVDIR:$PATH pushd $LOGDIR -./create-everylog-vs-versions-data $LOGREPO +create-everylog-vs-versions-data $LOGREPO popd @@ -18,7 +20,7 @@ echo first commandline param is $1 if [ "$1" == "rebuild" ]; then echo CLEANING DATABASE - $SQLCMD < prov-init.sql + $SQLCMD < $PROVDIR/prov-init.sql fi while read start version filename; do @@ -37,9 +39,9 @@ echo version $version in log file $filename echo ============= will import ============= - ./prepare-for-import $filename - ./import-run-to-sql $filename -# ./import-run-to-xml $filename + prepare-for-import $filename + import-run-to-sql $filename +# import-run-to-xml $filename $SQLCMD -c "INSERT INTO known_workflows (workflow_log_filename, version, importstatus) VALUES ('$filename','$version','imported');" else echo SKIP: Workflow does not report successful end. From noreply at svn.ci.uchicago.edu Mon Mar 9 11:26:38 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 11:26:38 -0500 (CDT) Subject: [Swift-commit] r2670 - provenancedb Message-ID: <20090309162638.CF25922814E@www.ci.uchicago.edu> Author: benc Date: 2009-03-09 11:26:38 -0500 (Mon, 09 Mar 2009) New Revision: 2670 Modified: provenancedb/import-all Log: build everylog version file in pwd rather than in the log-processing directory Modified: provenancedb/import-all =================================================================== --- provenancedb/import-all 2009-03-09 16:17:16 UTC (rev 2669) +++ provenancedb/import-all 2009-03-09 16:26:38 UTC (rev 2670) @@ -8,14 +8,11 @@ source paths.source export PATH=$LOGDIR:$PROVDIR:$PATH -pushd $LOGDIR create-everylog-vs-versions-data $LOGREPO -popd - # TODO better tmp handling that always using the same name in a shared # directory -cp $LOGDIR/everylog-vs-versions.data /tmp/ +cp everylog-vs-versions.data /tmp/ echo first commandline param is $1 if [ "$1" == "rebuild" ]; then From noreply at svn.ci.uchicago.edu Mon Mar 9 12:18:21 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 12:18:21 -0500 (CDT) Subject: [Swift-commit] r2671 - trunk/src/org/griphyn/vdl/karajan/lib Message-ID: <20090309171821.4C003228223@www.ci.uchicago.edu> Author: hategan Date: 2009-03-09 12:18:20 -0500 (Mon, 09 Mar 2009) New Revision: 2671 Added: trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java Log: forgot the essential part Added: trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java (rev 0) +++ trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java 2009-03-09 17:18:20 UTC (rev 2671) @@ -0,0 +1,222 @@ +// ---------------------------------------------------------------------- +//This code is developed as part of the Java CoG Kit project +//The terms of the license can be found at http://www.cogkit.org/license +//This message may not be removed or altered. +//---------------------------------------------------------------------- + +/* + * Created on Mar 21, 2005 + */ +package org.griphyn.vdl.karajan.lib; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Iterator; +import java.util.List; + +import org.apache.log4j.Logger; +import org.globus.cog.karajan.arguments.Arg; +import org.globus.cog.karajan.stack.VariableNotFoundException; +import org.globus.cog.karajan.stack.VariableStack; +import org.globus.cog.karajan.util.Identifier; +import org.globus.cog.karajan.util.KarajanIterator; +import org.globus.cog.karajan.util.ThreadingContext; +import org.globus.cog.karajan.util.TypeUtil; +import org.globus.cog.karajan.workflow.ExecutionException; +import org.globus.cog.karajan.workflow.events.Event; +import org.globus.cog.karajan.workflow.events.EventBus; +import org.globus.cog.karajan.workflow.events.EventListener; +import org.globus.cog.karajan.workflow.events.EventTargetPair; +import org.globus.cog.karajan.workflow.events.FutureNotificationEvent; +import org.globus.cog.karajan.workflow.futures.FutureEvaluationException; +import org.globus.cog.karajan.workflow.futures.FutureIterator; +import org.globus.cog.karajan.workflow.futures.FutureIteratorIncomplete; +import org.globus.cog.karajan.workflow.nodes.AbstractParallelIterator; +import org.griphyn.vdl.util.VDL2Config; + +public class ThrottledParallelFor extends AbstractParallelIterator { + public static final Logger logger = Logger + .getLogger(ThrottledParallelFor.class); + + public static final int DEFAULT_MAX_THREADS = 1024; + + public static final Arg A_NAME = new Arg.Positional("name"); + public static final Arg A_IN = new Arg.Positional("in"); + + static { + setArguments(ThrottledParallelFor.class, new Arg[] { A_NAME, A_IN }); + } + + public static final String THREAD_COUNT = "#threadcount"; + + private int maxThreadCount = -1; + + public void iterate(VariableStack stack, Identifier var, KarajanIterator i) + throws ExecutionException { + if (elementCount() > 0) { + if (logger.isDebugEnabled()) { + logger.debug("iterateParallel: " + stack.parentFrame()); + } + stack.setVar(VAR, var); + setChildFailed(stack, false); + stack.setVar(CALLER, this); + initializeChannelBuffers(stack); + initThreadCount(stack); + citerate(stack, var, i); + } + else { + complete(stack); + } + } + + protected void citerate(VariableStack stack, Identifier var, + KarajanIterator i) throws ExecutionException { + ThreadCount tc = getThreadCount(stack); + try { + while (i.hasNext()) { + Object value = tc.tryIncrement(i); + VariableStack copy = stack.copy(); + copy.enter(); + ThreadingContext.set(copy, ThreadingContext.get(copy).split( + i.current())); + setIndex(copy, getArgCount()); + setArgsDone(copy); + copy.setVar(var.getName(), value); + addChannelBuffers(copy); + startElement(getArgCount(), copy); + } + + int left = tc.decrement(); + if (left == 0) { + complete(stack); + } + } + catch (FutureIteratorIncomplete fii) { + stack.setVar(ITERATOR, i); + fii.getFutureIterator().addModificationAction( + this, + new FutureNotificationEvent(ITERATE, this, fii + .getFutureIterator(), stack)); + } + } + + protected void iterationCompleted(VariableStack stack) + throws ExecutionException { + closeBuffers(stack); + stack.leave(); + ThreadCount tc = getThreadCount(stack); + int running = tc.decrement(); + if (running == 0) { + complete(stack); + } + } + + private void initThreadCount(VariableStack stack) { + if (maxThreadCount < 0) { + try { + maxThreadCount = TypeUtil.toInt(VDL2Config.getConfig() + .getProperty("foreach.max.threads", String.valueOf(DEFAULT_MAX_THREADS))); + } + catch (IOException e) { + maxThreadCount = DEFAULT_MAX_THREADS; + } + } + stack.setVar(THREAD_COUNT, new ThreadCount(maxThreadCount)); + } + + private ThreadCount getThreadCount(VariableStack stack) + throws VariableNotFoundException { + return (ThreadCount) stack.getVar(THREAD_COUNT); + } + + private static class ThreadCount implements FutureIterator { + private int maxThreadCount; + private int crt; + private List listeners; + + public ThreadCount(int maxThreadCount) { + this.maxThreadCount = maxThreadCount; + crt = 1; + } + + public synchronized Object tryIncrement(KarajanIterator i) { + if (crt < maxThreadCount) { + Object o = i.next(); + crt++; + return o; + } + else { + throw new FutureIteratorIncomplete(this, this); + } + } + + public synchronized int decrement() { + crt--; + notifyListeners(); + return crt; + } + + private void notifyListeners() { + if (listeners != null) { + Iterator i = listeners.iterator(); + while (i.hasNext()) { + EventTargetPair etp = (EventTargetPair) i.next(); + i.remove(); + EventBus.post(etp.getTarget(), etp.getEvent()); + } + } + } + + public boolean hasAvailable() { + return false; + } + + public int count() { + return 0; + } + + public int current() { + return 0; + } + + public Object peek() { + return null; + } + + public boolean hasNext() { + return false; + } + + public Object next() { + return null; + } + + public void remove() { + } + + public synchronized void addModificationAction(EventListener target, + Event event) { + if (listeners == null) { + listeners = new ArrayList(); + } + listeners.add(new EventTargetPair(event, target)); + if (crt < maxThreadCount) { + notifyListeners(); + } + } + + public void close() { + } + + public void fail(FutureEvaluationException e) { + } + + public Object getValue() throws ExecutionException { + return null; + } + + public boolean isClosed() { + return false; + } + } +} \ No newline at end of file From noreply at svn.ci.uchicago.edu Mon Mar 9 15:04:18 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 15:04:18 -0500 (CDT) Subject: [Swift-commit] r2672 - trunk/src/org/griphyn/vdl/mapping Message-ID: <20090309200419.0882D228223@www.ci.uchicago.edu> Author: hategan Date: 2009-03-09 15:04:18 -0500 (Mon, 09 Mar 2009) New Revision: 2672 Modified: trunk/src/org/griphyn/vdl/mapping/AbstractDataNode.java Log: optimized some fairly inefficient code and reformatted a bit Modified: trunk/src/org/griphyn/vdl/mapping/AbstractDataNode.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/AbstractDataNode.java 2009-03-09 17:18:20 UTC (rev 2671) +++ trunk/src/org/griphyn/vdl/mapping/AbstractDataNode.java 2009-03-09 20:04:18 UTC (rev 2672) @@ -24,23 +24,28 @@ static final String DATASET_URI_PREFIX = "tag:benc at ci.uchicago.edu,2008:swift:dataset:"; - public static final Logger logger = Logger.getLogger(AbstractDataNode.class); - - public static final MappingParam PARAM_PREFIX = new MappingParam("prefix", null); + public static final Logger logger = Logger + .getLogger(AbstractDataNode.class); - /** Datasets are identified within a run by this sequence number and the - partial ID field. - The initial value is chosen to aid human recognition of sequence - numbers in the wild. There is no requirement that it start at this - (or any other) particular value. Note that this introduces a - maximum on the number of datasets which can be dealt with in any - run to be about 2^62. */ + public static final MappingParam PARAM_PREFIX = new MappingParam("prefix", + null); + + /** + * Datasets are identified within a run by this sequence number and the + * partial ID field. The initial value is chosen to aid human recognition of + * sequence numbers in the wild. There is no requirement that it start at + * this (or any other) particular value. Note that this introduces a maximum + * on the number of datasets which can be dealt with in any run to be about + * 2^62. + */ private static long datasetIDCounter = 720000000000l; - /** This is used to provide a (hopefully) globally unique identifier for - each time the datasetIDCounter is reset (whenever this class is - loaded, which will usually happen once per JVM). No meaning should be - inferred from this value - it exists purely for making unique URIs. */ + /** + * This is used to provide a (hopefully) globally unique identifier for each + * time the datasetIDCounter is reset (whenever this class is loaded, which + * will usually happen once per JVM). No meaning should be inferred from + * this value - it exists purely for making unique URIs. + */ private static final String datasetIDPartialID = Loader.getUUID(); private Field field; @@ -49,6 +54,7 @@ private boolean closed; private List listeners; final String identifierURI = makeIdentifierURIString(); + private Path pathFromRoot; protected AbstractDataNode(Field field) { this.field = field; @@ -89,8 +95,9 @@ return String.valueOf(n.intValue()); } catch (ClassCastException e) { - throw new RuntimeException("Internal type error. Value is not a Number for " - + getDisplayableName() + getPathFromRoot()); + throw new RuntimeException( + "Internal type error. Value is not a Number for " + + getDisplayableName() + getPathFromRoot()); } } else { @@ -100,38 +107,44 @@ return getIdentifyingString(); } - public String getIdentifyingString() { + StringBuffer sb = new StringBuffer(); + sb.append(this.getClass().getName()); - String prefix = this.getClass().getName(); + sb.append(" identifier "); + sb.append(this.getIdentifier()); - prefix = prefix + " identifier "+this.getIdentifier(); + sb.append(" type "); + sb.append(getType()); - prefix = prefix + " type "+getType(); - - if(value == null) { - prefix = prefix + " with no value at dataset="; - } else if (value instanceof Throwable) { - prefix = prefix + " containing throwable "+value.getClass(); - } else { - prefix = prefix + " value="+this.value.toString()+" dataset="; + if (value == null) { + sb.append(" with no value at dataset="); } + else if (value instanceof Throwable) { + sb.append(" containing throwable "); + sb.append(value.getClass()); + } + else { + sb.append(" value="); + sb.append(this.value.toString()); + sb.append(" dataset="); + } - prefix = prefix + getDisplayableName(); + sb.append(getDisplayableName()); if (!Path.EMPTY_PATH.equals(getPathFromRoot())) { - prefix = prefix + " path="+ getPathFromRoot().toString(); + sb.append(" path="); + sb.append(getPathFromRoot().toString()); } - if(closed) { - prefix = prefix + " (closed)"; + if (closed) { + sb.append(" (closed)"); } else { - prefix = prefix + " (not closed)"; + sb.append(" (not closed)"); } - return prefix; - + return sb.toString(); } protected String getDisplayableName() { @@ -163,13 +176,15 @@ } } - public Collection getFields(Path path) throws InvalidPathException, HandleOpenException { + public Collection getFields(Path path) throws InvalidPathException, + HandleOpenException { List fields = new ArrayList(); getFields(fields, path); return fields; } - private void getFields(List fields, Path path) throws InvalidPathException, HandleOpenException { + private void getFields(List fields, Path path) throws InvalidPathException, + HandleOpenException { if (path.isEmpty()) { fields.add(this); } @@ -187,7 +202,8 @@ } else { try { - ((AbstractDataNode) getField(path.getFirst())).getFields(fields, rest); + ((AbstractDataNode) getField(path.getFirst())).getFields( + fields, rest); } catch (NoSuchFieldException e) { throw new InvalidPathException(path, this); @@ -199,7 +215,8 @@ public void set(DSHandle handle) { // TODO check type if (closed) { - throw new IllegalArgumentException(this.getDisplayableName() + " is already assigned"); + throw new IllegalArgumentException(this.getDisplayableName() + + " is already assigned"); } if (getParent() == null) { /* @@ -220,7 +237,8 @@ } } - protected synchronized DSHandle getField(String name) throws NoSuchFieldException { + protected synchronized DSHandle getField(String name) + throws NoSuchFieldException { DSHandle handle = getHandle(name); if (handle == null) { if (closed) { @@ -246,10 +264,11 @@ } } - public DSHandle createDSHandle(String fieldName) throws NoSuchFieldException { + public DSHandle createDSHandle(String fieldName) + throws NoSuchFieldException { if (closed) { - throw new RuntimeException("Cannot write to closed handle: " + this + " (" + fieldName - + ")"); + throw new RuntimeException("Cannot write to closed handle: " + this + + " (" + fieldName + ")"); } AbstractDataNode child; @@ -264,15 +283,17 @@ synchronized (handles) { Object o = handles.put(fieldName, child); if (o != null) { - throw new RuntimeException("Trying to create a handle that already exists (" - + fieldName + ") in " + this); + throw new RuntimeException( + "Trying to create a handle that already exists (" + + fieldName + ") in " + this); } } return child; } protected Field getChildField(String fieldName) throws NoSuchFieldException { - return Field.Factory.createField(fieldName, field.getType().getField(fieldName).getType()); + return Field.Factory.createField(fieldName, field.getType().getField( + fieldName).getType()); } protected void checkDataException() { @@ -300,7 +321,8 @@ public Map getArrayValue() { checkDataException(); if (!field.getType().isArray()) { - throw new RuntimeException("getArrayValue called on a struct: " + this); + throw new RuntimeException("getArrayValue called on a struct: " + + this); } return handles; } @@ -312,7 +334,7 @@ public void setValue(Object value) { if (this.closed) { throw new IllegalArgumentException(this.getDisplayableName() - + " is closed with a value of "+this.value); + + " is closed with a value of " + this.value); } if (this.value != null) { throw new IllegalArgumentException(this.getDisplayableName() @@ -328,7 +350,8 @@ return list; } - public void getFringePaths(List list, Path parentPath) throws HandleOpenException { + public void getFringePaths(List list, Path parentPath) + throws HandleOpenException { checkMappingException(); if (getField().getType().getBaseType() != null) { list.add(Path.EMPTY_PATH); @@ -346,8 +369,10 @@ "Inconsistency between type declaration and handle for field '" + field.getName() + "'"); } - //[m] Hmm. Should there be a difference between field and mapper.getField()? - //Path fullPath = parentPath.addLast(mapper.getField().getName()); + // [m] Hmm. Should there be a difference between field and + // mapper.getField()? + // Path fullPath = + // parentPath.addLast(mapper.getField().getName()); Path fullPath = parentPath.addLast(field.getName()); if (!mapper.field.getType().isPrimitive() && !mapper.isArray()) { list.add(fullPath); @@ -360,53 +385,70 @@ } public synchronized void closeShallow() { + if (this.closed) { + return; + } this.closed = true; notifyListeners(); - logger.info("closed "+this.getIdentifyingString()); -// so because its closed, we can dump the contents + logger.info("closed " + this.getIdentifyingString()); + // so because its closed, we can dump the contents try { logContent(); - } catch(Exception e) { - logger.warn("Exception whilst logging dataset content for "+this,e); } -// TODO record retrospective provenance information for this dataset here -// we should do it at closing time because that's the point at which we -// know the dataset has its values (all the way down the tree) assigned. + catch (Exception e) { + logger.warn("Exception whilst logging dataset content for " + this, + e); + } + // TODO record retrospective provenance information for this dataset + // here + // we should do it at closing time because that's the point at which we + // know the dataset has its values (all the way down the tree) assigned. -// provenance-id for this dataset should have been assigned at creation time, -// though, so that we can refer to this dataset elsewhere before it is -// closed. + // provenance-id for this dataset should have been assigned at creation + // time, + // though, so that we can refer to this dataset elsewhere before it is + // closed. -// is this method the only one called to set this.closed? or do subclasses -// or other methods ever change it? + // is this method the only one called to set this.closed? or do + // subclasses + // or other methods ever change it? } public synchronized void logContent() { - if(this.getPathFromRoot() != null) { - logger.info("ROOTPATH dataset="+this.getIdentifier()+" path="+ - this.getPathFromRoot()); - if(this.getType().isPrimitive()) { - logger.info("VALUE dataset="+this.getIdentifier()+" VALUE="+this.toString()); + String identifier = this.getIdentifier(); + Path pathFromRoot = this.getPathFromRoot(); + if (this.getPathFromRoot() != null) { + logger.info("ROOTPATH dataset=" + identifier + " path=" + + pathFromRoot); + if (this.getType().isPrimitive()) { + logger.info("VALUE dataset=" + identifier + " VALUE=" + + this.toString()); } Mapper m; try { m = this.getMapper(); - } catch(VDL2FutureException fe) { + } + catch (VDL2FutureException fe) { m = null; // no mapping info if mapper isn't initialised yet } - if(m != null) { -// TODO proper type here -// Not sure catching exception here is really the right thing to do here -// anyway - should perhaps only be trying to map leafnodes? Mapping -// non-leaf stuff is giving wierd paths anyway + if (m != null) { + // TODO proper type here + // Not sure catching exception here is really the right thing to + // do here + // anyway - should perhaps only be trying to map leafnodes? + // Mapping + // non-leaf stuff is giving wierd paths anyway try { - Object path=m.map(this.getPathFromRoot()); - logger.info("FILENAME dataset="+this.getIdentifier()+" filename="+path) ; - } catch(Exception e) { - logger.info("dataset "+this.getIdentifier()+" exception while mapping path from root",e); + Object path = m.map(pathFromRoot); + logger.info("FILENAME dataset=" + identifier + " filename=" + + path); } + catch (Exception e) { + logger.info("dataset " + identifier + + " exception while mapping path from root", e); + } } } @@ -415,7 +457,8 @@ while (i.hasNext()) { Map.Entry e = (Map.Entry) i.next(); AbstractDataNode node = (AbstractDataNode) e.getValue(); - logger.info("CONTAINMENT parent="+this.getIdentifier()+" child="+node.getIdentifier()); + logger.info("CONTAINMENT parent=" + identifier + " child=" + + node.getIdentifier()); node.logContent(); } } @@ -440,16 +483,20 @@ } } - public Path getPathFromRoot() { - AbstractDataNode parent = (AbstractDataNode) this.getParent(); - Path myPath; - if (parent != null) { - myPath = parent.getPathFromRoot(); - return myPath.addLast(getField().getName(), parent.getField().getType().isArray()); + public synchronized Path getPathFromRoot() { + if (pathFromRoot == null) { + AbstractDataNode parent = (AbstractDataNode) this.getParent(); + Path myPath; + if (parent != null) { + myPath = parent.getPathFromRoot(); + pathFromRoot = myPath.addLast(getField().getName(), parent + .getField().getType().isArray()); + } + else { + pathFromRoot = Path.EMPTY_PATH; + } } - else { - return Path.EMPTY_PATH; - } + return pathFromRoot; } public Mapper getMapper() { @@ -462,7 +509,8 @@ public synchronized void addListener(DSHandleListener listener) { if (logger.isInfoEnabled()) { - logger.info("Adding handle listener \"" + listener + "\" to \"" + getIdentifyingString() + "\""); + logger.info("Adding handle listener \"" + listener + "\" to \"" + + getIdentifyingString() + "\""); } if (listeners == null) { listeners = new LinkedList(); @@ -480,7 +528,8 @@ DSHandleListener listener = (DSHandleListener) i.next(); i.remove(); if (logger.isInfoEnabled()) { - logger.info("Notifying listener \"" + listener + "\" about \"" + getIdentifyingString() + "\""); + logger.info("Notifying listener \"" + listener + + "\" about \"" + getIdentifyingString() + "\""); } listener.handleClosed(this); } @@ -494,6 +543,6 @@ String makeIdentifierURIString() { datasetIDCounter++; - return DATASET_URI_PREFIX + datasetIDPartialID + ":" + datasetIDCounter; + return DATASET_URI_PREFIX + datasetIDPartialID + ":" + datasetIDCounter; } } From noreply at svn.ci.uchicago.edu Mon Mar 9 15:05:10 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 15:05:10 -0500 (CDT) Subject: [Swift-commit] r2673 - trunk/src/org/griphyn/vdl/karajan/lib Message-ID: <20090309200510.27008228223@www.ci.uchicago.edu> Author: hategan Date: 2009-03-09 15:05:09 -0500 (Mon, 09 Mar 2009) New Revision: 2673 Modified: trunk/src/org/griphyn/vdl/karajan/lib/VDLFunction.java Log: made closeShallow static Modified: trunk/src/org/griphyn/vdl/karajan/lib/VDLFunction.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/VDLFunction.java 2009-03-09 20:04:18 UTC (rev 2672) +++ trunk/src/org/griphyn/vdl/karajan/lib/VDLFunction.java 2009-03-09 20:05:09 UTC (rev 2673) @@ -462,7 +462,7 @@ } } - protected void closeShallow(VariableStack stack, DSHandle handle) throws ExecutionException { + protected static void closeShallow(VariableStack stack, DSHandle handle) throws ExecutionException { synchronized(handle.getRoot()) { handle.closeShallow(); getFutureWrapperMap(stack).close(handle); From noreply at svn.ci.uchicago.edu Mon Mar 9 15:05:59 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 9 Mar 2009 15:05:59 -0500 (CDT) Subject: [Swift-commit] r2674 - trunk/src/org/griphyn/vdl/karajan/lib Message-ID: <20090309200559.4E9B82281D8@www.ci.uchicago.edu> Author: hategan Date: 2009-03-09 15:05:58 -0500 (Mon, 09 Mar 2009) New Revision: 2674 Modified: trunk/src/org/griphyn/vdl/karajan/lib/Range.java Log: made range return a lazy array Modified: trunk/src/org/griphyn/vdl/karajan/lib/Range.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/Range.java 2009-03-09 20:05:09 UTC (rev 2673) +++ trunk/src/org/griphyn/vdl/karajan/lib/Range.java 2009-03-09 20:05:58 UTC (rev 2674) @@ -3,13 +3,24 @@ */ package org.griphyn.vdl.karajan.lib; +import java.util.AbstractMap; +import java.util.AbstractSet; +import java.util.Collection; +import java.util.Collections; +import java.util.Iterator; +import java.util.Map; +import java.util.Set; + import org.globus.cog.karajan.arguments.Arg; import org.globus.cog.karajan.stack.VariableStack; import org.globus.cog.karajan.workflow.ExecutionException; import org.griphyn.vdl.mapping.AbstractDataNode; import org.griphyn.vdl.mapping.DSHandle; +import org.griphyn.vdl.mapping.HandleOpenException; +import org.griphyn.vdl.mapping.InvalidPathException; import org.griphyn.vdl.mapping.Path; import org.griphyn.vdl.mapping.RootArrayDataNode; +import org.griphyn.vdl.mapping.RootDataNode; import org.griphyn.vdl.type.Type; import org.griphyn.vdl.type.Types; @@ -22,27 +33,113 @@ setArguments(Range.class, new Arg[] { PA_FROM, PA_TO, OA_STEP }); } - public Object function(VariableStack stack) throws ExecutionException { + public Object function(final VariableStack stack) throws ExecutionException { // TODO: deal with expression - Type type = PA_FROM.getType(stack); - double start = PA_FROM.getDoubleValue(stack); - double stop = PA_TO.getDoubleValue(stack); - double incr = OA_STEP.getDoubleValue(stack); + final Type type = PA_FROM.getType(stack); + final double start = PA_FROM.getDoubleValue(stack); + final double stop = PA_TO.getDoubleValue(stack); + final double incr = OA_STEP.getDoubleValue(stack); // only deal with int and float try { - AbstractDataNode handle; + final AbstractDataNode handle; - handle = new RootArrayDataNode(type.arrayType()); - int index = 0; - for (double v = start; v <= stop; v += incr, index++) { - Path path = Path.EMPTY_PATH.addLast(String.valueOf(index), true); - DSHandle field = handle.getField(path); - field.setValue(new Double(v)); - closeShallow(stack, field); - } + handle = new RootArrayDataNode(type.arrayType()) { + final DSHandle h = this; + + public Collection getFields(Path path) + throws InvalidPathException, HandleOpenException { + if (path.size() > 1) { + throw new InvalidPathException(path, this); + } + else if (path.equals(Path.EMPTY_PATH)) { + return Collections.singletonList(this); + } + else { + int index = Integer.parseInt(path.getFirst()); + DSHandle value = new RootDataNode(type); + value.init(null); + value.setValue(new Double(start + incr * index)); + value.closeShallow(); + return Collections.singletonList(value); + } + } - closeShallow(stack, handle); + public Map getArrayValue() { + return new AbstractMap() { + public Set entrySet() { + return new AbstractSet() { + public Iterator iterator() { + return new Iterator() { + private double crt = start; + private int index = 0; + + public boolean hasNext() { + return crt <= stop; + } + + public Object next() { + try { + Map.Entry e = new Map.Entry() { + private DSHandle value; + private int key; + + { + value = new RootDataNode(type); + value.init(null); + value.setValue(new Double(crt)); + value.closeShallow(); + key = index; + } + + public Object getKey() { + return new Double(key); + } + + public Object getValue() { + return value; + } + + public Object setValue(Object value) { + throw new UnsupportedOperationException(); + } + }; + index++; + crt += incr; + if (crt > stop) { + VDLFunction.closeShallow(stack, h); + } + return e; + } + catch (Exception e) { + throw new RuntimeException(e); + } + } + + public void remove() { + throw new UnsupportedOperationException(); + } + }; + } + + public int size() { + return (int) ((stop - start) / incr); + } + }; + } + }; + } + + public boolean isClosed() { + //the need for this lie arises from: + //1. adding fields to a truly closed array throws an exception + //2. this is a lazy array not a future array. Consequently, we + // want the consumer(s) of this array to think that all values + // in the array are available + return true; + } + + }; return handle; } catch (Exception e) { From noreply at svn.ci.uchicago.edu Tue Mar 10 08:21:37 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 10 Mar 2009 08:21:37 -0500 (CDT) Subject: [Swift-commit] r2675 - trunk/tests/language-behaviour Message-ID: <20090310132137.B83352281F6@www.ci.uchicago.edu> Author: benc Date: 2009-03-10 08:21:33 -0500 (Tue, 10 Mar 2009) New Revision: 2675 Modified: trunk/tests/language-behaviour/066-many.swift Log: increase the range in many-iteration test to get up where stuff should behave lazily Modified: trunk/tests/language-behaviour/066-many.swift =================================================================== --- trunk/tests/language-behaviour/066-many.swift 2009-03-09 20:05:58 UTC (rev 2674) +++ trunk/tests/language-behaviour/066-many.swift 2009-03-10 13:21:33 UTC (rev 2675) @@ -5,6 +5,6 @@ } } -foreach i in [1:1000] { +foreach i in [1:3000] { p(); } From noreply at svn.ci.uchicago.edu Tue Mar 10 08:31:42 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 10 Mar 2009 08:31:42 -0500 (CDT) Subject: [Swift-commit] r2676 - in trunk: . tests/language-behaviour Message-ID: <20090310133142.8C0542281DA@www.ci.uchicago.edu> Author: benc Date: 2009-03-10 08:31:40 -0500 (Tue, 10 Mar 2009) New Revision: 2676 Added: trunk/tests/language-behaviour/161-star-dot.out.expected trunk/tests/language-behaviour/162-dot-on-array.out.expected trunk/tests/language-behaviour/162-dot-on-array.swift Removed: trunk/161-star-dot.out.expected trunk/162-dot-on-array.out.expected Log: tests for star-related syntax were messed up Deleted: trunk/161-star-dot.out.expected =================================================================== --- trunk/161-star-dot.out.expected 2009-03-10 13:21:33 UTC (rev 2675) +++ trunk/161-star-dot.out.expected 2009-03-10 13:31:40 UTC (rev 2676) @@ -1 +0,0 @@ -two-C Deleted: trunk/162-dot-on-array.out.expected =================================================================== --- trunk/162-dot-on-array.out.expected 2009-03-10 13:21:33 UTC (rev 2675) +++ trunk/162-dot-on-array.out.expected 2009-03-10 13:31:40 UTC (rev 2676) @@ -1 +0,0 @@ -two-C Copied: trunk/tests/language-behaviour/161-star-dot.out.expected (from rev 2675, trunk/161-star-dot.out.expected) =================================================================== --- trunk/tests/language-behaviour/161-star-dot.out.expected (rev 0) +++ trunk/tests/language-behaviour/161-star-dot.out.expected 2009-03-10 13:31:40 UTC (rev 2676) @@ -0,0 +1 @@ +two-C Copied: trunk/tests/language-behaviour/162-dot-on-array.out.expected (from rev 2675, trunk/162-dot-on-array.out.expected) =================================================================== --- trunk/tests/language-behaviour/162-dot-on-array.out.expected (rev 0) +++ trunk/tests/language-behaviour/162-dot-on-array.out.expected 2009-03-10 13:31:40 UTC (rev 2676) @@ -0,0 +1 @@ +two-C Added: trunk/tests/language-behaviour/162-dot-on-array.swift =================================================================== --- trunk/tests/language-behaviour/162-dot-on-array.swift (rev 0) +++ trunk/tests/language-behaviour/162-dot-on-array.swift 2009-03-10 13:31:40 UTC (rev 2676) @@ -0,0 +1,36 @@ +type messagefile; + +(messagefile t) greeting(string s) { + app { + echo s stdout=@filename(t); + } +} + +messagefile outfile <"162-dot-on-array.out">; + +type astruct { + string a; + string b; + string c; +}; + +astruct foo[]; + +foo[0].a = "zero-A"; +foo[0].b = "zero-B"; +foo[0].c = "zero-C"; + +foo[1].a = "one-A"; +foo[1].b = "one-B"; +foo[1].c = "one-C"; + +foo[2].a = "two-A"; +foo[2].b = "two-B"; +foo[2].c = "two-C"; + +string s[] = foo.c; + +string u = s[2]; + +outfile = greeting(u); + From noreply at svn.ci.uchicago.edu Tue Mar 10 14:23:13 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 10 Mar 2009 14:23:13 -0500 (CDT) Subject: [Swift-commit] r2677 - trunk/src/org/griphyn/vdl/karajan/lib Message-ID: <20090310192313.E47C72281D8@www.ci.uchicago.edu> Author: hategan Date: 2009-03-10 14:23:12 -0500 (Tue, 10 Mar 2009) New Revision: 2677 Modified: trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java Log: update according to changes in karajan Modified: trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java 2009-03-10 13:31:40 UTC (rev 2676) +++ trunk/src/org/griphyn/vdl/karajan/lib/ThrottledParallelFor.java 2009-03-10 19:23:12 UTC (rev 2677) @@ -59,7 +59,7 @@ } stack.setVar(VAR, var); setChildFailed(stack, false); - stack.setVar(CALLER, this); + stack.setCaller(this); initializeChannelBuffers(stack); initThreadCount(stack); citerate(stack, var, i); From noreply at svn.ci.uchicago.edu Wed Mar 11 04:03:32 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Wed, 11 Mar 2009 04:03:32 -0500 (CDT) Subject: [Swift-commit] r2678 - in SwiftApps: . pc3 Message-ID: <20090311090332.CC0BF2281D8@www.ci.uchicago.edu> Author: benc Date: 2009-03-11 04:03:31 -0500 (Wed, 11 Mar 2009) New Revision: 2678 Added: SwiftApps/pc3/ SwiftApps/pc3/README Log: make a working directory for pc3 Added: SwiftApps/pc3/README =================================================================== --- SwiftApps/pc3/README (rev 0) +++ SwiftApps/pc3/README 2009-03-11 09:03:31 UTC (rev 2678) @@ -0,0 +1 @@ +This is a work-space for the Swift entry to the third provenance challenge From noreply at svn.ci.uchicago.edu Wed Mar 11 10:15:42 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Wed, 11 Mar 2009 10:15:42 -0500 (CDT) Subject: [Swift-commit] r2679 - provenancedb Message-ID: <20090311151542.6E32822821B@www.ci.uchicago.edu> Author: benc Date: 2009-03-11 10:15:41 -0500 (Wed, 11 Mar 2009) New Revision: 2679 Modified: provenancedb/prov-to-opm.sh Log: use a real URI for swift-opm namespace Modified: provenancedb/prov-to-opm.sh =================================================================== --- provenancedb/prov-to-opm.sh 2009-03-11 09:03:31 UTC (rev 2678) +++ provenancedb/prov-to-opm.sh 2009-03-11 15:15:41 UTC (rev 2679) @@ -5,7 +5,7 @@ rm -f opm.xml # TODO make swift-opm-ns into a proper URI -echo "" > opm.xml +echo "" > opm.xml echo "" >> opm.xml From noreply at svn.ci.uchicago.edu Wed Mar 11 10:16:13 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Wed, 11 Mar 2009 10:16:13 -0500 (CDT) Subject: [Swift-commit] r2680 - provenancedb Message-ID: <20090311151613.E693222821B@www.ci.uchicago.edu> Author: benc Date: 2009-03-11 10:16:13 -0500 (Wed, 11 Mar 2009) New Revision: 2680 Modified: provenancedb/prov-to-opm.sh Log: Was reading the wrong fields from execute.global.event Modified: provenancedb/prov-to-opm.sh =================================================================== --- provenancedb/prov-to-opm.sh 2009-03-11 15:15:41 UTC (rev 2679) +++ provenancedb/prov-to-opm.sh 2009-03-11 15:16:13 UTC (rev 2680) @@ -11,11 +11,11 @@ echo "" >> opm.xml -while read time duration thread endstate app scratch; do +while read time duration globalthread localthread endstate app scratch; do -echo " " +echo " " echo " " -echo " " +echo " " # TODO no value here - this is some URI into an ontology, which is don't # really know how should be mapped from Swift echo " " From noreply at svn.ci.uchicago.edu Wed Mar 11 10:16:45 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Wed, 11 Mar 2009 10:16:45 -0500 (CDT) Subject: [Swift-commit] r2681 - provenancedb Message-ID: <20090311151645.062C022821B@www.ci.uchicago.edu> Author: benc Date: 2009-03-11 10:16:44 -0500 (Wed, 11 Mar 2009) New Revision: 2681 Modified: provenancedb/prov-to-opm.sh Log: separate IDs and URIs, so that IDs are now syntactically valid, and the global URI identifiers are separately represented Modified: provenancedb/prov-to-opm.sh =================================================================== --- provenancedb/prov-to-opm.sh 2009-03-11 15:16:13 UTC (rev 2680) +++ provenancedb/prov-to-opm.sh 2009-03-11 15:16:44 UTC (rev 2681) @@ -13,9 +13,12 @@ while read time duration globalthread localthread endstate app scratch; do -echo " " +threadid=$(echo $globalthread | sed 's/[:, at .]/-/g') + +echo " " echo " " echo " " +echo " $globalthread" # TODO no value here - this is some URI into an ontology, which is don't # really know how should be mapped from Swift echo " " @@ -44,8 +47,10 @@ cat tmp-dshandles.txt | sort | uniq > tmp-dshandles2.txt while read artifact ; do -echo " " +artifactid=$(echo $artifact | sed 's/[:, at .]/-/g') +echo " " echo " " +echo " $artifact" echo " " done < tmp-dshandles2.txt >> opm.xml @@ -59,18 +64,20 @@ # but for now do it in a single pass... while read thread direction dataset variable rest; do + datasetid=$(echo $dataset | sed 's/[:, at .]/-/g') + threadid=$(echo $thread | sed 's/[:, at .]/-/g') if [ "$direction" == "input" ] ; then echo " " - echo " " + echo " " echo " " - echo " " + echo " " echo " " echo " " else echo " " - echo " " + echo " " echo " " - echo " " + echo " " echo " " echo " " fi From noreply at svn.ci.uchicago.edu Fri Mar 13 07:52:39 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Fri, 13 Mar 2009 07:52:39 -0500 (CDT) Subject: [Swift-commit] r2682 - in trunk: resources src/org/griphyn/vdl/engine Message-ID: <20090313125239.7898922821B@www.ci.uchicago.edu> Author: benc Date: 2009-03-13 07:52:37 -0500 (Fri, 13 Mar 2009) New Revision: 2682 Modified: trunk/resources/Karajan.stg trunk/src/org/griphyn/vdl/engine/Karajan.java Log: get rid of explicit and/or templates and use general binary operator template for these Modified: trunk/resources/Karajan.stg =================================================================== --- trunk/resources/Karajan.stg 2009-03-11 15:16:44 UTC (rev 2681) +++ trunk/resources/Karajan.stg 2009-03-13 12:52:37 UTC (rev 2682) @@ -407,7 +407,9 @@ ">=":"vdlop:ge", ">":"vdlop:gt", "<":"vdlop:lt", - "==":"vdlop:eq" + "==":"vdlop:eq", + "&&":"vdlop:and", + "||":"vdlop:or" ] unaryNegation(exp, datatype) ::= << @@ -424,21 +426,6 @@ >> - -and(left,right,datatype) ::= << - - $left$ - $right$ - ->> - -or(left, right, datatype) ::= << - - $left$ - $right$ - ->> - not(exp, datatype) ::= << $exp$ Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/Karajan.java 2009-03-11 15:16:44 UTC (rev 2681) +++ trunk/src/org/griphyn/vdl/engine/Karajan.java 2009-03-13 12:52:37 UTC (rev 2682) @@ -844,10 +844,11 @@ QName expressionQName = new QName(namespaceURI, localName); if(expressionQName.equals(OR_EXPR)) { - StringTemplate st = template("or"); + StringTemplate st = template("binaryop"); BinaryOperator o = (BinaryOperator)expression; StringTemplate leftST = expressionToKarajan(o.getAbstractExpressionArray(0), scope); StringTemplate rightST = expressionToKarajan(o.getAbstractExpressionArray(1), scope); + st.setAttribute("op","||"); st.setAttribute("left", leftST); st.setAttribute("right", rightST); if (datatype(leftST).equals("boolean") && datatype(rightST).equals("boolean")) @@ -856,10 +857,11 @@ throw new CompilationException("Or operation can only be applied to parameters of type boolean."); return st; } else if (expressionQName.equals(AND_EXPR)) { - StringTemplate st = template("and"); + StringTemplate st = template("binaryop"); BinaryOperator o = (BinaryOperator)expression; StringTemplate leftST = expressionToKarajan(o.getAbstractExpressionArray(0), scope); StringTemplate rightST = expressionToKarajan(o.getAbstractExpressionArray(1), scope); + st.setAttribute("op","&&"); st.setAttribute("left", leftST); st.setAttribute("right", rightST); if (datatype(leftST).equals("boolean") && datatype(rightST).equals("boolean")) From noreply at svn.ci.uchicago.edu Sun Mar 15 11:56:02 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 15 Mar 2009 11:56:02 -0500 (CDT) Subject: [Swift-commit] r2683 - / trunk/bin trunk/libexec trunk/tests Message-ID: <20090315165602.9EB8E228235@www.ci.uchicago.edu> Author: benc Date: 2009-03-15 11:56:01 -0500 (Sun, 15 Mar 2009) New Revision: 2683 Added: trunk/bin/swift-log-to-cedps trunk/bin/swift-plot-log trunk/libexec/log-processing/ trunk/tests/log-processing/ Removed: log-processing/ Log: merge log-processing into main swift release Copied: trunk/bin/swift-log-to-cedps (from rev 2682, log-processing/bin/swift-log-to-cedps) =================================================================== --- trunk/bin/swift-log-to-cedps (rev 0) +++ trunk/bin/swift-log-to-cedps 2009-03-15 16:56:01 UTC (rev 2683) @@ -0,0 +1,52 @@ +#!/bin/bash + +ORIGDIR=$(pwd) +WORKINGDIR=$(mktemp -d /tmp/swift-plot-log-XXXXXXXXXXXXXXXX) + + +# $1 should be the pathname of the log file to plot + +LOG_CODE_HOME="`dirname $0`/../libexec/log-processing/" +export SWIFT_PLOT_HOME=$LOG_CODE_HOME + +export PATH=${LOG_CODE_HOME}:${PATH} + +FC="${1:0:1}" + +if [ "$FC" = "/" ]; then # absolute path + LOG_FILE_PATH=$1 +else + LOG_FILE_PATH="`pwd`/$1" +fi + +LOG_DIRECTORY=`dirname $LOG_FILE_PATH` + +LOG_FILE_BASE=`basename $LOG_FILE_PATH .log` + +echo Log file path is $LOG_FILE_PATH +echo Log is in directory $LOG_DIRECTORY +echo Log basename is $LOG_FILE_BASE + +cd $WORKINGDIR +echo Now in directory $(pwd) + +MAKEENV="-f ${LOG_CODE_HOME}/makefile -I ${LOG_CODE_HOME}" +MAKETARGETS="" + +INFODIR=${LOG_DIRECTORY}/${LOG_FILE_BASE}.d + +if [ -d $INFODIR ]; then + MAKEENV="$MAKEENV IDIR=${LOG_DIRECTORY}/${LOG_FILE_BASE}.d/" + MAKETARGETS="webpage.info" +fi + +make $MAKEENV SDL=${ORIGDIR}/report-${LOG_FILE_BASE} LOG=$LOG_FILE_PATH clean execute.cedps execute2.cedps karatasks.cedps + +cp -v *.cedps $ORIGDIR + +# webpage.kara webpage.weights karatasks.JOB_SUBMISSION.Queue.transitions karatasks.JOB_SUBMISSION.Queue.event karatasks.JOB_SUBMISSION.eip $MAKETARGETS webpage distribute + +# mv report-$LOG_FILE_BASE $LOG_DIRECTORY/ + +# rm -r $WORKINGDIR +echo not removing $WORKINGDIR Copied: trunk/bin/swift-plot-log (from rev 2682, log-processing/bin/swift-plot-log) =================================================================== --- trunk/bin/swift-plot-log (rev 0) +++ trunk/bin/swift-plot-log 2009-03-15 16:56:01 UTC (rev 2683) @@ -0,0 +1,52 @@ +#!/bin/bash + +ORIGDIR=$(pwd) +WORKINGDIR=$(mktemp -d /tmp/swift-plot-log-XXXXXXXXXXXXXXXX) + + +# $1 should be the pathname of the log file to plot + +LOG_CODE_HOME="`dirname $0`/../libexec/log-processing/" +export SWIFT_PLOT_HOME=$LOG_CODE_HOME + +export PATH=${LOG_CODE_HOME}:${PATH} + +FC="${1:0:1}" + +if [ "$FC" = "/" ]; then # absolute path + LOG_FILE_PATH=$1 +else + LOG_FILE_PATH="`pwd`/$1" +fi + +LOG_DIRECTORY=`dirname $LOG_FILE_PATH` + +LOG_FILE_BASE=`basename $LOG_FILE_PATH .log` + +echo Log file path is $LOG_FILE_PATH +echo Log is in directory $LOG_DIRECTORY +echo Log basename is $LOG_FILE_BASE + +cd $WORKINGDIR +echo Now in directory $(pwd) + +MAKEENV="-f ${LOG_CODE_HOME}/makefile -I ${LOG_CODE_HOME}" +MAKETARGETS="" + +INFODIR=${LOG_DIRECTORY}/${LOG_FILE_BASE}.d + +if [ -d $INFODIR ]; then + MAKEENV="$MAKEENV IDIR=${LOG_DIRECTORY}/${LOG_FILE_BASE}.d/" + MAKETARGETS="webpage.info" +fi + +if [ "X$2" = "X" ]; then +make $MAKEENV SDL=${ORIGDIR}/report-${LOG_FILE_BASE} LOG=$LOG_FILE_PATH clean webpage.kara webpage.weights karatasks.JOB_SUBMISSION.Queue.transitions karatasks.JOB_SUBMISSION.Queue.event karatasks.JOB_SUBMISSION.eip $MAKETARGETS webpage distribute +else +make $MAKEENV SDL=${ORIGDIR}/report-${LOG_FILE_BASE} LOG=$LOG_FILE_PATH clean $2 +cp $2 $ORIGDIR/ +fi + +# mv report-$LOG_FILE_BASE $LOG_DIRECTORY/ + +rm -r $WORKINGDIR Copied: trunk/libexec/log-processing (from rev 2682, log-processing/libexec/log-processing) Copied: trunk/tests/log-processing (from rev 2682, log-processing/tests/log-processing) From noreply at svn.ci.uchicago.edu Sun Mar 15 12:15:40 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 15 Mar 2009 12:15:40 -0500 (CDT) Subject: [Swift-commit] r2684 - trunk Message-ID: <20090315171540.CBC1D228236@www.ci.uchicago.edu> Author: benc Date: 2009-03-15 12:15:40 -0500 (Sun, 15 Mar 2009) New Revision: 2684 Modified: trunk/CHANGES.txt Log: CHANGES for log-processing merge Modified: trunk/CHANGES.txt =================================================================== --- trunk/CHANGES.txt 2009-03-15 16:56:01 UTC (rev 2683) +++ trunk/CHANGES.txt 2009-03-15 17:15:40 UTC (rev 2684) @@ -1,3 +1,8 @@ +(03/15/09) +*** The log-processing code, primarily exposed as the swift-plot-log command, + has been merged into the main Swift release, rather than being a separate + download. + (03/09/09) *** Added a throttling parallelFor and changed the swift foreach underlying implementation to it. The throttling parallelFor limits the number of From noreply at svn.ci.uchicago.edu Mon Mar 16 08:14:42 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 16 Mar 2009 08:14:42 -0500 (CDT) Subject: [Swift-commit] r2685 - trunk/bin Message-ID: <20090316131442.62D4722821B@www.ci.uchicago.edu> Author: benc Date: 2009-03-16 08:14:41 -0500 (Mon, 16 Mar 2009) New Revision: 2685 Modified: trunk/bin/swift-plot-log Log: Make swift-plot-log take multiple targets when used in make-like mode Modified: trunk/bin/swift-plot-log =================================================================== --- trunk/bin/swift-plot-log 2009-03-15 17:15:40 UTC (rev 2684) +++ trunk/bin/swift-plot-log 2009-03-16 13:14:41 UTC (rev 2685) @@ -43,8 +43,9 @@ if [ "X$2" = "X" ]; then make $MAKEENV SDL=${ORIGDIR}/report-${LOG_FILE_BASE} LOG=$LOG_FILE_PATH clean webpage.kara webpage.weights karatasks.JOB_SUBMISSION.Queue.transitions karatasks.JOB_SUBMISSION.Queue.event karatasks.JOB_SUBMISSION.eip $MAKETARGETS webpage distribute else -make $MAKEENV SDL=${ORIGDIR}/report-${LOG_FILE_BASE} LOG=$LOG_FILE_PATH clean $2 -cp $2 $ORIGDIR/ +shift +make $MAKEENV SDL=${ORIGDIR}/report-${LOG_FILE_BASE} LOG=$LOG_FILE_PATH clean $@ +cp $@ $ORIGDIR/ fi # mv report-$LOG_FILE_BASE $LOG_DIRECTORY/ From noreply at svn.ci.uchicago.edu Mon Mar 16 08:15:28 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 16 Mar 2009 08:15:28 -0500 (CDT) Subject: [Swift-commit] r2686 - trunk/libexec/log-processing Message-ID: <20090316131528.0370122821B@www.ci.uchicago.edu> Author: benc Date: 2009-03-16 08:15:27 -0500 (Mon, 16 Mar 2009) New Revision: 2686 Modified: trunk/libexec/log-processing/makefile Log: commandline interface to generate everylog-vs-versions.data - it can be generated using swift-plot-log, passing in the root directory to a log repository instead of an individual log file name Modified: trunk/libexec/log-processing/makefile =================================================================== --- trunk/libexec/log-processing/makefile 2009-03-16 13:14:41 UTC (rev 2685) +++ trunk/libexec/log-processing/makefile 2009-03-16 13:15:27 UTC (rev 2686) @@ -152,3 +152,6 @@ karatasks.JOB_SUBMISSION.annotated-execute2.transitions: karatasks.JOB_SUBMISSION.transitions execute2.event annotate-karatasks-with-execute2-id < karatasks.JOB_SUBMISSION.transitions > karatasks.JOB_SUBMISSION.annotated-execute2.transitions + +everylog-vs-versions.data: + create-everylog-vs-versions-data $(LOG) From noreply at svn.ci.uchicago.edu Mon Mar 16 08:16:17 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 16 Mar 2009 08:16:17 -0500 (CDT) Subject: [Swift-commit] r2687 - provenancedb Message-ID: <20090316131617.CFA3522821B@www.ci.uchicago.edu> Author: benc Date: 2009-03-16 08:16:17 -0500 (Mon, 16 Mar 2009) New Revision: 2687 Modified: provenancedb/import-all provenancedb/prepare-for-import provenancedb/prepare-provenance-chart Log: Make import scripts use the main swift-plot-log driver command, and stay away from directly messing with log-processing libexec components. Modified: provenancedb/import-all =================================================================== --- provenancedb/import-all 2009-03-16 13:15:27 UTC (rev 2686) +++ provenancedb/import-all 2009-03-16 13:16:17 UTC (rev 2687) @@ -8,7 +8,7 @@ source paths.source export PATH=$LOGDIR:$PROVDIR:$PATH -create-everylog-vs-versions-data $LOGREPO +swift-plot-log $LOGREPO everylog-vs-versions.data # TODO better tmp handling that always using the same name in a shared # directory Modified: provenancedb/prepare-for-import =================================================================== --- provenancedb/prepare-for-import 2009-03-16 13:15:27 UTC (rev 2686) +++ provenancedb/prepare-for-import 2009-03-16 13:16:17 UTC (rev 2687) @@ -4,9 +4,7 @@ # workflow has finished and will do everything necessary to import all # of the provenance information into the SQL provenance database. -pushd $LOGDIR -make LOG=$1 clean execute.global.event execute2.event workflow.event -popd +swift-plot-log $1 execute.global.event execute2.event workflow.event prepare-provenance-chart $1 Modified: provenancedb/prepare-provenance-chart =================================================================== --- provenancedb/prepare-provenance-chart 2009-03-16 13:15:27 UTC (rev 2686) +++ provenancedb/prepare-provenance-chart 2009-03-16 13:16:17 UTC (rev 2687) @@ -1,5 +1,4 @@ #!/bin/bash -pushd $LOGDIR # invent an ID. doesn't need to have any meaning. needs to be unique. # uuidgen for now. @@ -23,5 +22,4 @@ cat $1 | grep ' PROCEDURE ' | sed "s/^.*thread=$[^ ]*$ name=$[^ ]*$\$/${WFID}\1 \2/" > invocation-procedure-names.txt -popd From noreply at svn.ci.uchicago.edu Mon Mar 16 08:17:13 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 16 Mar 2009 08:17:13 -0500 (CDT) Subject: [Swift-commit] r2688 - provenancedb Message-ID: <20090316131713.3C04722821B@www.ci.uchicago.edu> Author: benc Date: 2009-03-16 08:17:12 -0500 (Mon, 16 Mar 2009) New Revision: 2688 Modified: provenancedb/drive-opm provenancedb/import-all provenancedb/paths.source.ci provenancedb/paths.source.soju Log: Remove expectation that log-processing lives in its own directory, as since Swift r2683 that has been in the main Swift codebase. Modified: provenancedb/drive-opm =================================================================== --- provenancedb/drive-opm 2009-03-16 13:16:17 UTC (rev 2687) +++ provenancedb/drive-opm 2009-03-16 13:17:12 UTC (rev 2688) @@ -4,7 +4,7 @@ filename=$1 -export PATH=$LOGPATH:$PATH +export PATH=$PATH echo log to OPM conversion for $filename Modified: provenancedb/import-all =================================================================== --- provenancedb/import-all 2009-03-16 13:16:17 UTC (rev 2687) +++ provenancedb/import-all 2009-03-16 13:17:12 UTC (rev 2688) @@ -6,7 +6,7 @@ # of individual runs will clean other files. source paths.source -export PATH=$LOGDIR:$PROVDIR:$PATH +export PATH=$PROVDIR:$PATH swift-plot-log $LOGREPO everylog-vs-versions.data Modified: provenancedb/paths.source.ci =================================================================== --- provenancedb/paths.source.ci 2009-03-16 13:16:17 UTC (rev 2687) +++ provenancedb/paths.source.ci 2009-03-16 13:17:12 UTC (rev 2688) @@ -1,10 +1,6 @@ # file to source that sets variables for the various paths that are # presently hardcoded -# this assumes that log-procesisng module is checked out in same place in -# relation to provenancedb as in the SVN. -export LOGDIR=`pwd`/../log-processing/ - # this is the path to log repo on benc's laptop export LOGREPO=~/swift-logs/ Modified: provenancedb/paths.source.soju =================================================================== --- provenancedb/paths.source.soju 2009-03-16 13:16:17 UTC (rev 2687) +++ provenancedb/paths.source.soju 2009-03-16 13:17:12 UTC (rev 2688) @@ -1,10 +1,6 @@ # file to source that sets variables for the various paths that are # presently hardcoded -# this assumes that log-procesisng module is checked out in same place in -# relation to provenancedb as in the SVN. -export LOGDIR=`pwd`/../log-processing/ - # this is the path to log repo on benc's laptop export LOGREPO=/Users/benc/work/everylog/ export IDIR=/Users/benc/work/everylog/ From noreply at svn.ci.uchicago.edu Mon Mar 16 08:18:00 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 16 Mar 2009 08:18:00 -0500 (CDT) Subject: [Swift-commit] r2689 - provenancedb Message-ID: <20090316131800.47ACD22821B@www.ci.uchicago.edu> Author: benc Date: 2009-03-16 08:17:59 -0500 (Mon, 16 Mar 2009) New Revision: 2689 Added: provenancedb/swift-prov-import-all-logs Removed: provenancedb/import-all Modified: provenancedb/provenance.xml Log: Rename user-exposed commands to swift-* Deleted: provenancedb/import-all =================================================================== --- provenancedb/import-all 2009-03-16 13:17:12 UTC (rev 2688) +++ provenancedb/import-all 2009-03-16 13:17:59 UTC (rev 2689) @@ -1,56 +0,0 @@ -#!/bin/bash - -PROVDIR=$(pwd)/$(dirname $0)/ - -# we need to keep this out of the log-proceesing dir because import -# of individual runs will clean other files. - -source paths.source -export PATH=$PROVDIR:$PATH - -swift-plot-log $LOGREPO everylog-vs-versions.data - -# TODO better tmp handling that always using the same name in a shared -# directory -cp everylog-vs-versions.data /tmp/ - -echo first commandline param is $1 -if [ "$1" == "rebuild" ]; then - echo CLEANING DATABASE - $SQLCMD < $PROVDIR/prov-init.sql -fi - -while read start version filename; do - if [ $version -ge 1538 ]; then - echo -n "Log: $filename ... " - -# TODO: this only works for psql, not for other DBs, so for now force it -# to be non-existing -#EXISTING=$(psql -p 5435 -d $PROVDB -U benc --tuples-only -c "select count(*) from known_workflows where workflow_log_filename='$filename';") -EXISTING=0 - -if [ "$EXISTING" -eq "0" ]; then -echo IMPORTING - -if grep --silent "DEBUG Loader Swift finished with no errors" $filename; then - - echo version $version in log file $filename - echo ============= will import ============= - prepare-for-import $filename - import-run-to-sql $filename -# import-run-to-xml $filename - $SQLCMD -c "INSERT INTO known_workflows (workflow_log_filename, version, importstatus) VALUES ('$filename','$version','imported');" - else - echo SKIP: Workflow does not report successful end. - $SQLCMD -c "INSERT INTO known_workflows (workflow_log_filename, version, importstatus) VALUES ('$filename','$version','skipped - workflow did not end sucessfully');" - fi -else -echo SKIP: Already known in workflow -fi - fi -done < /tmp/everylog-vs-versions.data - -# now pull the XML data into eXist, in as few runs as possible to avoid -# JVM startup. -# ls `pwd`/*.log.xml | time xargs ~/work/eXist/bin/client.sh -m /db/prov -ouri=xmldb:exist:// -p - Modified: provenancedb/provenance.xml =================================================================== --- provenancedb/provenance.xml 2009-03-16 13:17:12 UTC (rev 2688) +++ provenancedb/provenance.xml 2009-03-16 13:17:59 UTC (rev 2689) @@ -182,7 +182,7 @@ Now import your logs for the first time like this: -$ ./import-all rebuild +$ ./swift-prov-import-all-logs rebuild

Copied: provenancedb/swift-prov-import-all-logs (from rev 2688, provenancedb/import-all) =================================================================== --- provenancedb/swift-prov-import-all-logs (rev 0) +++ provenancedb/swift-prov-import-all-logs 2009-03-16 13:17:59 UTC (rev 2689) @@ -0,0 +1,56 @@ +#!/bin/bash + +PROVDIR=$(pwd)/$(dirname $0)/ + +# we need to keep this out of the log-proceesing dir because import +# of individual runs will clean other files. + +source paths.source +export PATH=$PROVDIR:$PATH + +swift-plot-log $LOGREPO everylog-vs-versions.data + +# TODO better tmp handling that always using the same name in a shared +# directory +cp everylog-vs-versions.data /tmp/ + +echo first commandline param is $1 +if [ "$1" == "rebuild" ]; then + echo CLEANING DATABASE + $SQLCMD < $PROVDIR/prov-init.sql +fi + +while read start version filename; do + if [ $version -ge 1538 ]; then + echo -n "Log: $filename ... " + +# TODO: this only works for psql, not for other DBs, so for now force it +# to be non-existing +#EXISTING=$(psql -p 5435 -d $PROVDB -U benc --tuples-only -c "select count(*) from known_workflows where workflow_log_filename='$filename';") +EXISTING=0 + +if [ "$EXISTING" -eq "0" ]; then +echo IMPORTING + +if grep --silent "DEBUG Loader Swift finished with no errors" $filename; then + + echo version $version in log file $filename + echo ============= will import ============= + prepare-for-import $filename + import-run-to-sql $filename +# import-run-to-xml $filename + $SQLCMD -c "INSERT INTO known_workflows (workflow_log_filename, version, importstatus) VALUES ('$filename','$version','imported');" + else + echo SKIP: Workflow does not report successful end. + $SQLCMD -c "INSERT INTO known_workflows (workflow_log_filename, version, importstatus) VALUES ('$filename','$version','skipped - workflow did not end sucessfully');" + fi +else +echo SKIP: Already known in workflow +fi + fi +done < /tmp/everylog-vs-versions.data + +# now pull the XML data into eXist, in as few runs as possible to avoid +# JVM startup. +# ls `pwd`/*.log.xml | time xargs ~/work/eXist/bin/client.sh -m /db/prov -ouri=xmldb:exist:// -p + From noreply at svn.ci.uchicago.edu Mon Mar 16 08:19:11 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 16 Mar 2009 08:19:11 -0500 (CDT) Subject: [Swift-commit] r2690 - in provenancedb: . etc Message-ID: <20090316131911.9F71822821B@www.ci.uchicago.edu> Author: benc Date: 2009-03-16 08:19:11 -0500 (Mon, 16 Mar 2009) New Revision: 2690 Added: provenancedb/etc/ provenancedb/etc/provenance.config.ci provenancedb/etc/provenance.config.soju Removed: provenancedb/paths.source.ci provenancedb/paths.source.soju Modified: provenancedb/provenance.xml provenancedb/swift-prov-import-all-logs Log: Change paths.source in base directory into a config file in etc/ Copied: provenancedb/etc/provenance.config.ci (from rev 2689, provenancedb/paths.source.ci) =================================================================== --- provenancedb/etc/provenance.config.ci (rev 0) +++ provenancedb/etc/provenance.config.ci 2009-03-16 13:19:11 UTC (rev 2690) @@ -0,0 +1,8 @@ +# file to source that sets variables for the various paths that are +# presently hardcoded + +# this is the path to log repo on benc's laptop +export LOGREPO=~/swift-logs/ + +# export SQLCMD="psql -p 5435 -d provdb " + Copied: provenancedb/etc/provenance.config.soju (from rev 2689, provenancedb/paths.source.soju) =================================================================== --- provenancedb/etc/provenance.config.soju (rev 0) +++ provenancedb/etc/provenance.config.soju 2009-03-16 13:19:11 UTC (rev 2690) @@ -0,0 +1,10 @@ +# file to source that sets variables for the various paths that are +# presently hardcoded + +# this is the path to log repo on benc's laptop +export LOGREPO=/Users/benc/work/everylog/ +export IDIR=/Users/benc/work/everylog/ + +export SQLCMD="sqlite3 provdb " +# export SQLCMD="psql -p 5435 -d provdb " + Deleted: provenancedb/paths.source.ci =================================================================== --- provenancedb/paths.source.ci 2009-03-16 13:17:59 UTC (rev 2689) +++ provenancedb/paths.source.ci 2009-03-16 13:19:11 UTC (rev 2690) @@ -1,8 +0,0 @@ -# file to source that sets variables for the various paths that are -# presently hardcoded - -# this is the path to log repo on benc's laptop -export LOGREPO=~/swift-logs/ - -# export SQLCMD="psql -p 5435 -d provdb " - Deleted: provenancedb/paths.source.soju =================================================================== --- provenancedb/paths.source.soju 2009-03-16 13:17:59 UTC (rev 2689) +++ provenancedb/paths.source.soju 2009-03-16 13:19:11 UTC (rev 2690) @@ -1,10 +0,0 @@ -# file to source that sets variables for the various paths that are -# presently hardcoded - -# this is the path to log repo on benc's laptop -export LOGREPO=/Users/benc/work/everylog/ -export IDIR=/Users/benc/work/everylog/ - -export SQLCMD="sqlite3 provdb " -# export SQLCMD="psql -p 5435 -d provdb " - Modified: provenancedb/provenance.xml =================================================================== --- provenancedb/provenance.xml 2009-03-16 13:17:59 UTC (rev 2689) +++ provenancedb/provenance.xml 2009-03-16 13:19:11 UTC (rev 2690) @@ -171,9 +171,9 @@ probably work under Linux. -Now create a paths.source file to define local +Now create a etc/provenance.config file to define local configuration. An example that I use on my laptop is present in -paths.source.soju. +provenance.config.soju. The SQLCMD indicates which command to run for SQL access. This is used by other scripts to access the database. The LOGREPO and IDIR variables should Modified: provenancedb/swift-prov-import-all-logs =================================================================== --- provenancedb/swift-prov-import-all-logs 2009-03-16 13:17:59 UTC (rev 2689) +++ provenancedb/swift-prov-import-all-logs 2009-03-16 13:19:11 UTC (rev 2690) @@ -5,7 +5,7 @@ # we need to keep this out of the log-proceesing dir because import # of individual runs will clean other files. -source paths.source +source $PROVDIR/etc/provenance.config export PATH=$PROVDIR:$PATH swift-plot-log $LOGREPO everylog-vs-versions.data From noreply at svn.ci.uchicago.edu Mon Mar 16 08:19:46 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 16 Mar 2009 08:19:46 -0500 (CDT) Subject: [Swift-commit] r2691 - provenancedb Message-ID: <20090316131946.464A322821B@www.ci.uchicago.edu> Author: benc Date: 2009-03-16 08:19:45 -0500 (Mon, 16 Mar 2009) New Revision: 2691 Modified: provenancedb/provenance.xml Log: Update documentation on basic use of provenance database Modified: provenancedb/provenance.xml =================================================================== --- provenancedb/provenance.xml 2009-03-16 13:19:11 UTC (rev 2690) +++ provenancedb/provenance.xml 2009-03-16 13:19:45 UTC (rev 2691) @@ -38,7 +38,7 @@

Getting started +

The old 'Getting started' section that you should ignore... This section details how to make basic use of some of the provenance database that I have running at CI. @@ -146,13 +146,14 @@

Check out the latest SVN code -Use the following two commands to check out the provenancedb -and log-processing modules: + +Use the following command to check out the provenancedb +module: svn co https://svn.ci.uchicago.edu/svn/vdl2/provenancedb -svn co https://svn.ci.uchicago.edu/svn/vdl2/log-processing +

@@ -186,9 +187,23 @@

Querying the newly generated database + +If you're using the SQLite database, you can get an interactive SQL +session to query your new provenance database like this: + +$ sqlite3 provdb +SQLite version 3.6.11 +Enter ".help" for instructions +Enter SQL statements terminated with a ";" +sqlite> + +

What this work does not address This work explicitly excludes a number of uses which traditionally From noreply at svn.ci.uchicago.edu Mon Mar 16 08:20:34 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 16 Mar 2009 08:20:34 -0500 (CDT) Subject: [Swift-commit] r2692 - provenancedb Message-ID: <20090316132034.1D6A82281AD@www.ci.uchicago.edu> Author: benc Date: 2009-03-16 08:20:33 -0500 (Mon, 16 Mar 2009) New Revision: 2692 Modified: provenancedb/provenance.xml provenancedb/swift-about-dataset provenancedb/swift-about-execution provenancedb/swift-about-filename Log: Make the existing swift-about-* commands which are hardcoded to postgres use configuration file to pick their database Modified: provenancedb/provenance.xml =================================================================== --- provenancedb/provenance.xml 2009-03-16 13:19:45 UTC (rev 2691) +++ provenancedb/provenance.xml 2009-03-16 13:20:33 UTC (rev 2692) @@ -51,69 +51,6 @@ -There are several swift-about- commands: - -swift-about-filename - returns the global dataset IDs for the specified -filename. Several runs may have output the same filename; the provenance -database cannot tell which run (if any) any file with that name that -exists now came from. - -Example: this looks for information about -001-echo.out which is the output of the first -test in the language-behaviour test suite: - -$ ./swift-about-filename 001-echo.out -Dataset IDs for files that have name file://localhost/001-echo.out - tag:benc at ci.uchicago.edu,2008:swift:dataset:20080114-1353-g1y3moc0:720000000001 - tag:benc at ci.uchicago.edu,2008:swift:dataset:20080107-1440-67vursv4:720000000001 - tag:benc at ci.uchicago.edu,2008:swift:dataset:20080107-2146-ja2r2z5f:720000000001 - tag:benc at ci.uchicago.edu,2008:swift:dataset:20080107-1608-itdd69l6:720000000001 - tag:benc at ci.uchicago.edu,2008:swift:dataset:20080303-1011-krz4g2y0:720000000001 - tag:benc at ci.uchicago.edu,2008:swift:dataset:20080303-1100-4in9a325:720000000001 - -Six different datasets in the provenance database have had that filename -(because six language behaviour test runs have been uploaded to the -database). - - -swift-about-dataset - returns information about a dataset, given -that dataset's uri. Returned information includes the IDs of a containing -dataset, datasets contained within this dataset, and IDs for executions -that used this dataset as input or output. - -Example: - -$ ./swift-about-dataset tag:benc at ci.uchicago.edu,2008:swift:dataset:20080114-1353-g1y3moc0:720000000001 -About dataset tag:benc at ci.uchicago.edu,2008:swift:dataset:20080114-1353-g1y3moc0:720000000001 -That dataset has these filename(s): - file://localhost/001-echo.out - -That dataset is part of these datasets: - -That dataset contains these datasets: - -That dataset was input to the following executions (as the specified named parameter): - -That dataset was output from the following executions (as the specified return parameter): - tag:benc at ci.uchicago.edu,2008:swiftlogs:execute:001-echo-20080114-1353-n7puv429:0 | t - -This shows that this dataset is not part of a more complicated dataset -structure, and was produced as an output parameter t from an execution. - -swift-about-execution - gives information about an execution, given -an execution ID - -$ ./swift-about-execution tag:benc at ci.uchicago.edu,2008:swiftlogs:execute:001-echo-20080114-1353-n7puv429:0 -About execution tag:benc at ci.uchicago.edu,2008:swiftlogs:execute:001-echo-20080114-1353-n7puv429:0 - id | starttime | duration | finalstate | app | scratch -----------------------------------------------------------------------------------------------------------------------------------+----------------+-------------------+----------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------- - tag:benc at ci.uchicago.edu,2008:swiftlogs:execute:001-echo-20080114-1353-n7puv429:0 | 1200318839.393 | 0.743000030517578 | 0 | END_SUCCESS | echo -(1 row) - -This shows some basic information about the execution - the start time, -the duration, the name of the application, the final status. - -

Direct database access on terminable @@ -189,6 +126,10 @@

Querying the newly generated database +You can use swift-about-* commands, described in +the commands section. + + If you're using the SQLite database, you can get an interactive SQL session to query your new provenance database like this: @@ -204,6 +145,73 @@

swift-about-* commands +There are several swift-about- commands: + +swift-about-filename - returns the global dataset IDs for the specified +filename. Several runs may have output the same filename; the provenance +database cannot tell which run (if any) any file with that name that +exists now came from. + +Example: this looks for information about +001-echo.out which is the output of the first +test in the language-behaviour test suite: + +$ ./swift-about-filename 001-echo.out +Dataset IDs for files that have name file://localhost/001-echo.out + tag:benc at ci.uchicago.edu,2008:swift:dataset:20080114-1353-g1y3moc0:720000000001 + tag:benc at ci.uchicago.edu,2008:swift:dataset:20080107-1440-67vursv4:720000000001 + tag:benc at ci.uchicago.edu,2008:swift:dataset:20080107-2146-ja2r2z5f:720000000001 + tag:benc at ci.uchicago.edu,2008:swift:dataset:20080107-1608-itdd69l6:720000000001 + tag:benc at ci.uchicago.edu,2008:swift:dataset:20080303-1011-krz4g2y0:720000000001 + tag:benc at ci.uchicago.edu,2008:swift:dataset:20080303-1100-4in9a325:720000000001 + +Six different datasets in the provenance database have had that filename +(because six language behaviour test runs have been uploaded to the +database). + + +swift-about-dataset - returns information about a dataset, given +that dataset's uri. Returned information includes the IDs of a containing +dataset, datasets contained within this dataset, and IDs for executions +that used this dataset as input or output. + +Example: + +$ ./swift-about-dataset tag:benc at ci.uchicago.edu,2008:swift:dataset:20080114-1353-g1y3moc0:720000000001 +About dataset tag:benc at ci.uchicago.edu,2008:swift:dataset:20080114-1353-g1y3moc0:720000000001 +That dataset has these filename(s): + file://localhost/001-echo.out + +That dataset is part of these datasets: + +That dataset contains these datasets: + +That dataset was input to the following executions (as the specified named parameter): + +That dataset was output from the following executions (as the specified return parameter): + tag:benc at ci.uchicago.edu,2008:swiftlogs:execute:001-echo-20080114-1353-n7puv429:0 | t + +This shows that this dataset is not part of a more complicated dataset +structure, and was produced as an output parameter t from an execution. + +swift-about-execution - gives information about an execution, given +an execution ID + +$ ./swift-about-execution tag:benc at ci.uchicago.edu,2008:swiftlogs:execute:001-echo-20080114-1353-n7puv429:0 +About execution tag:benc at ci.uchicago.edu,2008:swiftlogs:execute:001-echo-20080114-1353-n7puv429:0 + id | starttime | duration | finalstate | app | scratch +----------------------------------------------------------------------------------------------------------------------------------+----------------+-------------------+----------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------- + tag:benc at ci.uchicago.edu,2008:swiftlogs:execute:001-echo-20080114-1353-n7puv429:0 | 1200318839.393 | 0.743000030517578 | 0 | END_SUCCESS | echo +(1 row) + +This shows some basic information about the execution - the start time, +the duration, the name of the application, the final status. + + +

What this work does not address This work explicitly excludes a number of uses which traditionally Modified: provenancedb/swift-about-dataset =================================================================== --- provenancedb/swift-about-dataset 2009-03-16 13:19:45 UTC (rev 2691) +++ provenancedb/swift-about-dataset 2009-03-16 13:20:33 UTC (rev 2692) @@ -1,5 +1,8 @@ #!/bin/bash +PROVDIR=$(pwd)/$(dirname $0)/ +source $PROVDIR/etc/provenance.config + ID=$1 #tag:benc at ci.uchicago.edu,2008:swift:dataset:20080114-1353-g1y3moc0:720000000001 @@ -7,18 +10,21 @@ echo About dataset $ID echo "That dataset has these filename(s):" -psql -p 5435 -d provdb -U benc --tuples-only -c "select filename from dataset_filenames where dataset_id='$ID';" +echo "select filename from dataset_filenames where dataset_id='$ID';" | $SQLCMD +echo echo "That dataset is part of these datasets:" -psql -p 5435 -d provdb -U benc --tuples-only -c "select outer_dataset_id from dataset_containment where inner_dataset_id='$ID';" +echo "select outer_dataset_id from dataset_containment where inner_dataset_id='$ID';" | $SQLCMD +echo echo "That dataset contains these datasets:" -psql -p 5435 -d provdb -U benc --tuples-only -c "select inner_dataset_id from dataset_containment where outer_dataset_id='$ID';" +echo "select inner_dataset_id from dataset_containment where outer_dataset_id='$ID';" | $SQLCMD echo "That dataset was input to the following executions (as the specified named parameter):" +echo "select execute_id, param_name from dataset_usage where dataset_id='$ID' and direction='I';" | $SQLCMD +echo -psql -p 5435 -d provdb -U benc --tuples-only -c "select execute_id, param_name from dataset_usage where dataset_id='$ID' and direction='I';" - echo "That dataset was output from the following executions (as the specified return parameter):" -psql -p 5435 -d provdb -U benc --tuples-only -c "select execute_id, param_name from dataset_usage where dataset_id='$ID' and direction='O';" +echo "select execute_id, param_name from dataset_usage where dataset_id='$ID' and direction='O';" | $SQLCMD +echo Modified: provenancedb/swift-about-execution =================================================================== --- provenancedb/swift-about-execution 2009-03-16 13:19:45 UTC (rev 2691) +++ provenancedb/swift-about-execution 2009-03-16 13:20:33 UTC (rev 2692) @@ -1,17 +1,22 @@ #!/bin/bash +PROVDIR=$(pwd)/$(dirname $0)/ +source $PROVDIR/etc/provenance.config + EXECUTEID=$1 echo About execution $EXECUTEID -psql -p 5435 -d provdb -U benc -c "select * from executes where id='$EXECUTEID';" +echo "select * from executes where id='$EXECUTEID';" | $SQLCMD +echo - echo Name of SwiftScript procedure which invoked this: -psql -p 5435 -d provdb -U benc --tuples-only -c "select procedure_name from invocation_procedure_names where execute_id='$EXECUTEID';" +echo "select procedure_name from invocation_procedure_names where execute_id='$EXECUTEID';" | $SQLCMD +echo echo Input datasets: -psql -p 5435 -d provdb -U benc -c "select dataset_id,param_name from dataset_usage where execute_id='$EXECUTEID' and direction='I';" +echo "select dataset_id,param_name from dataset_usage where execute_id='$EXECUTEID' and direction='I';" | $SQLCMD +echo echo Output datasets: -psql -p 5435 -d provdb -U benc -c "select dataset_id,param_name from dataset_usage where execute_id='$EXECUTEID' and direction='O';" +echo "select dataset_id,param_name from dataset_usage where execute_id='$EXECUTEID' and direction='O';" | $SQLCMD Modified: provenancedb/swift-about-filename =================================================================== --- provenancedb/swift-about-filename 2009-03-16 13:19:45 UTC (rev 2691) +++ provenancedb/swift-about-filename 2009-03-16 13:20:33 UTC (rev 2692) @@ -1,10 +1,17 @@ #!/bin/bash +PROVDIR=$(pwd)/$(dirname $0)/ +source $PROVDIR/etc/provenance.config + FILENAME="file://localhost/$1" echo Dataset IDs for files that have name $FILENAME -psql -p 5435 -d provdb -U benc --tuples-only -c "select dataset_id from dataset_filenames where filename='$FILENAME';" +# when used with postgres, --tuples-only is useful here; but it doesn't +# generalise +echo "select dataset_id from dataset_filenames where filename='$FILENAME';" | $SQLCMD + + From noreply at svn.ci.uchicago.edu Mon Mar 16 13:17:21 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 16 Mar 2009 13:17:21 -0500 (CDT) Subject: [Swift-commit] r2693 - provenancedb Message-ID: <20090316181721.594EA2281F0@www.ci.uchicago.edu> Author: benc Date: 2009-03-16 13:17:20 -0500 (Mon, 16 Mar 2009) New Revision: 2693 Modified: provenancedb/swift-prov-import-all-logs Log: change $SQLCMD invocation syntax to something that works with sqlite3 as well as postgres Modified: provenancedb/swift-prov-import-all-logs =================================================================== --- provenancedb/swift-prov-import-all-logs 2009-03-16 13:20:33 UTC (rev 2692) +++ provenancedb/swift-prov-import-all-logs 2009-03-16 18:17:20 UTC (rev 2693) @@ -39,10 +39,10 @@ prepare-for-import $filename import-run-to-sql $filename # import-run-to-xml $filename - $SQLCMD -c "INSERT INTO known_workflows (workflow_log_filename, version, importstatus) VALUES ('$filename','$version','imported');" + echo "INSERT INTO known_workflows (workflow_log_filename, version, importstatus) VALUES ('$filename','$version','imported');" | $SQLCMD else echo SKIP: Workflow does not report successful end. - $SQLCMD -c "INSERT INTO known_workflows (workflow_log_filename, version, importstatus) VALUES ('$filename','$version','skipped - workflow did not end sucessfully');" + echo "INSERT INTO known_workflows (workflow_log_filename, version, importstatus) VALUES ('$filename','$version','skipped - workflow did not end sucessfully');" | $SQLCMD fi else echo SKIP: Already known in workflow From noreply at svn.ci.uchicago.edu Mon Mar 16 13:18:19 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 16 Mar 2009 13:18:19 -0500 (CDT) Subject: [Swift-commit] r2694 - provenancedb Message-ID: <20090316181819.5F4762281F0@www.ci.uchicago.edu> Author: benc Date: 2009-03-16 13:18:18 -0500 (Mon, 16 Mar 2009) New Revision: 2694 Modified: provenancedb/prov-to-sql.sh Log: Intermediate files are now found in pwd, but some code was not looking there. Modified: provenancedb/prov-to-sql.sh =================================================================== --- provenancedb/prov-to-sql.sh 2009-03-16 18:17:20 UTC (rev 2693) +++ provenancedb/prov-to-sql.sh 2009-03-16 18:18:18 UTC (rev 2694) @@ -6,7 +6,7 @@ while read time duration thread endstate app scratch; do echo "INSERT INTO executes (id, starttime, duration, finalstate, app, scratch) VALUES ('$thread', $time, $duration, '$endstate', '$app', '$scratch');" >> tmp-import.sql -done < $LOGDIR/execute.global.event +done < execute.global.event while read thread direction dataset variable rest; do if [ "$direction" == "input" ] ; then @@ -15,24 +15,24 @@ dir=O fi echo "INSERT INTO dataset_usage (execute_id, direction, dataset_id, param_name) VALUES ('$thread', '$dir', '$dataset', '$variable');" >> tmp-import.sql -done < $LOGDIR/tie-data-invocs.txt +done < tie-data-invocs.txt while read thread appname; do echo "INSERT INTO invocation_procedure_names (execute_id, procedure_name) VALUES ('$thread', '$appname');" >> tmp-import.sql -done < $LOGDIR/invocation-procedure-names.txt +done < invocation-procedure-names.txt while read outer inner; do echo "INSERT INTO dataset_containment (outer_dataset_id, inner_dataset_id) VALUES ('$outer', '$inner');" >> tmp-import.sql -done < $LOGDIR/tie-containers.txt +done < tie-containers.txt while read dataset filename; do echo "INSERT INTO dataset_filenames (dataset_id, filename) VALUES ('$dataset', '$filename');" >> tmp-import.sql -done < $LOGDIR/dataset-filenames.txt +done < dataset-filenames.txt while read start duration wfid; do echo "INSERT INTO workflow_events (workflow_id,starttime, duration) VALUES ('$wfid', '$start', '$duration');" >> tmp-import.sql -done < $LOGDIR/workflow.event +done < workflow.event echo Sending SQL to DB $SQLCMD < tmp-import.sql From noreply at svn.ci.uchicago.edu Mon Mar 16 13:19:29 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 16 Mar 2009 13:19:29 -0500 (CDT) Subject: [Swift-commit] r2695 - provenancedb Message-ID: <20090316181929.42EA02281F0@www.ci.uchicago.edu> Author: benc Date: 2009-03-16 13:19:28 -0500 (Mon, 16 Mar 2009) New Revision: 2695 Modified: provenancedb/swift-about-dataset provenancedb/swift-about-execution provenancedb/swift-about-filename provenancedb/swift-prov-import-all-logs Log: better detection of PROVDIR Modified: provenancedb/swift-about-dataset =================================================================== --- provenancedb/swift-about-dataset 2009-03-16 18:18:18 UTC (rev 2694) +++ provenancedb/swift-about-dataset 2009-03-16 18:19:28 UTC (rev 2695) @@ -1,6 +1,9 @@ #!/bin/bash -PROVDIR=$(pwd)/$(dirname $0)/ +PROVDIR=$(dirname $0) +pushd $PROVDIR +PROVDIR=$(pwd) +popd source $PROVDIR/etc/provenance.config ID=$1 Modified: provenancedb/swift-about-execution =================================================================== --- provenancedb/swift-about-execution 2009-03-16 18:18:18 UTC (rev 2694) +++ provenancedb/swift-about-execution 2009-03-16 18:19:28 UTC (rev 2695) @@ -1,6 +1,10 @@ #!/bin/bash -PROVDIR=$(pwd)/$(dirname $0)/ +PROVDIR=$(dirname $0) +pushd $PROVDIR +PROVDIR=$(pwd) +popd + source $PROVDIR/etc/provenance.config EXECUTEID=$1 Modified: provenancedb/swift-about-filename =================================================================== --- provenancedb/swift-about-filename 2009-03-16 18:18:18 UTC (rev 2694) +++ provenancedb/swift-about-filename 2009-03-16 18:19:28 UTC (rev 2695) @@ -1,6 +1,10 @@ #!/bin/bash -PROVDIR=$(pwd)/$(dirname $0)/ +PROVDIR=$(dirname $0) +pushd $PROVDIR +PROVDIR=$(pwd) +popd + source $PROVDIR/etc/provenance.config FILENAME="file://localhost/$1" Modified: provenancedb/swift-prov-import-all-logs =================================================================== --- provenancedb/swift-prov-import-all-logs 2009-03-16 18:18:18 UTC (rev 2694) +++ provenancedb/swift-prov-import-all-logs 2009-03-16 18:19:28 UTC (rev 2695) @@ -1,6 +1,9 @@ #!/bin/bash -PROVDIR=$(pwd)/$(dirname $0)/ +PROVDIR=$(dirname $0) +pushd $PROVDIR +PROVDIR=$(pwd) +popd # we need to keep this out of the log-proceesing dir because import # of individual runs will clean other files. From noreply at svn.ci.uchicago.edu Mon Mar 16 13:20:20 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 16 Mar 2009 13:20:20 -0500 (CDT) Subject: [Swift-commit] r2696 - provenancedb Message-ID: <20090316182020.2711D2281F0@www.ci.uchicago.edu> Author: benc Date: 2009-03-16 13:20:19 -0500 (Mon, 16 Mar 2009) New Revision: 2696 Modified: provenancedb/swift-prov-import-all-logs Log: fail harder when swift-plot-log does not run Modified: provenancedb/swift-prov-import-all-logs =================================================================== --- provenancedb/swift-prov-import-all-logs 2009-03-16 18:19:28 UTC (rev 2695) +++ provenancedb/swift-prov-import-all-logs 2009-03-16 18:20:19 UTC (rev 2696) @@ -13,6 +13,11 @@ swift-plot-log $LOGREPO everylog-vs-versions.data +if [ "$?" != "0" ]; then + echo swift-plot-log failed when building everylog-vs-versions.data + exit 1 +fi + # TODO better tmp handling that always using the same name in a shared # directory cp everylog-vs-versions.data /tmp/ @@ -40,7 +45,16 @@ echo version $version in log file $filename echo ============= will import ============= prepare-for-import $filename + if [ "$?" != "0" ]; then + echo prepare-for-import failed + exit 2 + fi import-run-to-sql $filename + if [ "$?" != "0" ]; then + echo import-run-to-sql failed + exit 3 + fi + # import-run-to-xml $filename echo "INSERT INTO known_workflows (workflow_log_filename, version, importstatus) VALUES ('$filename','$version','imported');" | $SQLCMD else From noreply at svn.ci.uchicago.edu Mon Mar 16 13:30:11 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 16 Mar 2009 13:30:11 -0500 (CDT) Subject: [Swift-commit] r2698 - provenancedb/tests/provenancedb Message-ID: <20090316183011.542A12281AD@www.ci.uchicago.edu> Author: benc Date: 2009-03-16 13:30:10 -0500 (Mon, 16 Mar 2009) New Revision: 2698 Added: provenancedb/tests/provenancedb/counts.out.expected provenancedb/tests/provenancedb/counts.sql provenancedb/tests/provenancedb/run Log: Some tests for the provenance code, based around sqlite3 and a set of supplied log files. Added: provenancedb/tests/provenancedb/counts.out.expected =================================================================== --- provenancedb/tests/provenancedb/counts.out.expected (rev 0) +++ provenancedb/tests/provenancedb/counts.out.expected 2009-03-16 18:30:10 UTC (rev 2698) @@ -0,0 +1,8 @@ +505 +629 +419 +226 +0 +259 +145 +144 Added: provenancedb/tests/provenancedb/counts.sql =================================================================== --- provenancedb/tests/provenancedb/counts.sql (rev 0) +++ provenancedb/tests/provenancedb/counts.sql 2009-03-16 18:30:10 UTC (rev 2698) @@ -0,0 +1,9 @@ +select count(*) from dataset_containment; +select count(*) from dataset_filenames; +select count(*) from dataset_usage; +select count(*) from executes; +select count(*) from executes_in_workflows; +select count(*) from invocation_procedure_names; +select count(*) from known_workflows; +select count(*) from workflow_events; + Added: provenancedb/tests/provenancedb/run =================================================================== --- provenancedb/tests/provenancedb/run (rev 0) +++ provenancedb/tests/provenancedb/run 2009-03-16 18:30:10 UTC (rev 2698) @@ -0,0 +1,44 @@ +#!/bin/bash + +# assumes that provenance directory is on path (to get the commands) +# and that an appropriate configuration for sqlite3 exists. + +# the latter could (and should) be automated to get logs from this +# test directory + +echo starting provenance tests + +# delete the db + +rm -f provdb + +if [ -f "provdb" ]; then + echo FAIL - provdb should have been deleted but still exists + exit 1 +fi + +# import + +swift-prov-import-all-logs rebuild + +if [ "$?" != "0" ]; then + echo FAIL - import of all logs returned non-zero + exit 2 +fi + +if ! [ -f "provdb" ]; then + echo FAIL - provdb should now exist but does not + exit 3 +fi + +echo checking row counts: + +cat counts.sql | sqlite3 provdb > counts.out +diff -q counts.out counts.out.expected +if [ "$?" -ne "0" ]; then + echo FAIL - row counts differ + exit 4 +fi + +echo ending provenance tests without failures +exit 0 Property changes on: provenancedb/tests/provenancedb/run ___________________________________________________________________ Name: svn:executable + * From noreply at svn.ci.uchicago.edu Mon Mar 16 13:31:11 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 16 Mar 2009 13:31:11 -0500 (CDT) Subject: [Swift-commit] r2699 - provenancedb Message-ID: <20090316183111.75A8B2281AD@www.ci.uchicago.edu> Author: benc Date: 2009-03-16 13:31:11 -0500 (Mon, 16 Mar 2009) New Revision: 2699 Modified: provenancedb/provenance.xml Log: More work on making provenance code easily switchable between sqlite and postgres. Modified: provenancedb/provenance.xml =================================================================== --- provenancedb/provenance.xml 2009-03-16 18:30:10 UTC (rev 2698) +++ provenancedb/provenance.xml 2009-03-16 18:31:11 UTC (rev 2699) @@ -97,6 +97,11 @@

Configuring your SQL database +Follow the instructions in one of the following sections, to configure your +database either for sqlite3 or for postgres. + +

Configuring your sqlite3 SQL database + This section describes configuring the SQL scripts to use sqlite, which is appropriate for a single-user installation. @@ -109,6 +114,63 @@ probably work under Linux. +In the next section, you will create a provenance.config +file. In that, you should configure the use of sqlite3 by specifying: + +export SQLCMD="sqlite3 provdb " + +(note the trailing space before the closing quote) + +

+ +

Configuring your own postgres 8.3 SQL database + +This section describes configuring a postgres 8.3 database, which is +appropriate for a large installation (where large means lots of log +files or multiple users) + + +First install and start postgres as appropriate for your platform +(using apt-get or port for example). + + +As user postgres, create a database: + +$ /opt/local/lib/postgresql83/bin/createdb provtest1 + + + +Check that you can connect and see the empty database: + +$ psql83 -d provtest1 -U postgres +Welcome to psql83 8.3.6, the PostgreSQL interactive terminal. + +Type: \copyright for distribution terms + \h for help with SQL commands + \? for help with psql commands + \g or terminate with semicolon to execute query + \q to quit + +provtest1=# \dt +No relations found. +provtest1=# \q + + + +In the next section, when configuring provenance.config, +specify the use of postgres like this: + +export SQLCMD="psql83 -d provtest1 -U postgres " + +Note the trailing space before the final quote. Also, note that if you +fiddled the above test command line to make it work, you will have to make +similar fiddles in the SQLCMD configuration line. + +

+ +

Import your logs + Now create a etc/provenance.config file to define local configuration. An example that I use on my laptop is present in provenance.config.soju. @@ -123,7 +185,9 @@ $ ./swift-prov-import-all-logs rebuild +

Querying the newly generated database You can use swift-about-* commands, described in From noreply at svn.ci.uchicago.edu Mon Mar 16 13:32:11 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 16 Mar 2009 13:32:11 -0500 (CDT) Subject: [Swift-commit] r2700 - provenancedb Message-ID: <20090316183211.15DE02281AD@www.ci.uchicago.edu> Author: benc Date: 2009-03-16 13:32:10 -0500 (Mon, 16 Mar 2009) New Revision: 2700 Modified: provenancedb/prov-init.sql Log: work on the documentation of the SQL schema Modified: provenancedb/prov-init.sql =================================================================== --- provenancedb/prov-init.sql 2009-03-16 18:31:11 UTC (rev 2699) +++ provenancedb/prov-init.sql 2009-03-16 18:32:10 UTC (rev 2700) @@ -1,5 +1,6 @@ --- sqlite3 monkey < this file +-- this is the schema definition used for the main relational provenance +-- implementation (in both sqlite3 and postgres) DROP TABLE executes; DROP TABLE dataset_usage; @@ -10,11 +11,18 @@ DROP TABLE known_workflows; DROP TABLE workflow_events; + +-- executes_in_workflow is unused at the moment, but is intended to associate +-- each execute with its containing workflow CREATE TABLE executes_in_workflows (workflow_id char(128), execute_id char(128) ); + +-- this gives information about each execute. +-- each execute is identified by a unique URI. other information from +-- swift logs is also stored here CREATE TABLE executes (id char(128) PRIMARY KEY, starttime numeric, @@ -24,7 +32,13 @@ scratch char(128) ); --- no primary key here. should probably index both on execute_id and on + +-- dataset_usage records usage relationships between executes and datasets; +-- in SwiftScript terms, the input and output parameters for each +-- application procedure invocation; in OPM terms, the artificts which are +-- input to and output from each process that is a Swift execution + +-- TODO: no primary key here. should probably index both on execute_id and on -- dataset_id for common queries? maybe add arbitrary ID for sake of it? CREATE TABLE dataset_usage @@ -36,39 +50,66 @@ -- this dataset was bound to. ); --- probably desirable that this is part of executes table + +-- invocation_procedure_name maps each execute ID to the name of its +-- SwiftScript procedure + +-- TODO probably desirable that this is part of executes table -- but for now this is the easiest to pull data from logs. + +-- TODO primary key should be execute_id CREATE TABLE invocation_procedure_names (execute_id char(128), procedure_name char(128) ); --- outer_dataset_id contains inner_dataset_id +-- dataset_containment stores the containment hierarchy between +-- container datasets (arrays and structs) and their contents. + +-- outer_dataset_id contains inner_dataset_id + +-- TODO this should perhaps be replaced with a more OPM-like model of +-- constructors and accessors, rather than, or in addition to, +-- a containment hierarchy. The relationship (such as array index or +-- structure member name) should also be stored in this table. CREATE TABLE dataset_containment ( outer_dataset_id char(128), inner_dataset_id char(128) ); --- dataset_filesnames (dataset_id, filename) + +-- dataset_filenames stores the filename mapped to each dataset. As some +-- datasets do not have filenames, it should not be expected that +-- every dataset will have a row in this table + +-- TODO dataset_id should be primary key CREATE TABLE dataset_filenames ( dataset_id char(128), filename char(128) ); + +-- known_workflows stores some information about each workflow log that has +-- been seen by the importer: the log filename, swift version and import +-- status. CREATE TABLE known_workflows ( workflow_log_filename char(128), version char(128), importstatus char(128) ); + +-- workflow_events stores the start time and duration for each workflow +-- that has been successfully imported. CREATE TABLE workflow_events ( workflow_id char(128), starttime numeric, duration numeric ); --- this does not work for sqlite; you'll get a syntax error but --- ignore it + +-- this GRANT does not work for sqlite; you'll get a syntax error but +-- ignore it, as it is not needed in sqlite grant all on dataset_containment, dataset_filenames, dataset_usage, executes_in_workflows, invocation_procedure_names, known_workflows, workflow_events to public; From noreply at svn.ci.uchicago.edu Mon Mar 16 13:33:13 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 16 Mar 2009 13:33:13 -0500 (CDT) Subject: [Swift-commit] r2701 - provenancedb Message-ID: <20090316183313.628C02281F0@www.ci.uchicago.edu> Author: benc Date: 2009-03-16 13:33:11 -0500 (Mon, 16 Mar 2009) New Revision: 2701 Modified: provenancedb/provenance.xml Log: move old getting-started section to the end - I want to reuse it later but it is not a good place for people to start with at the moment Modified: provenancedb/provenance.xml =================================================================== --- provenancedb/provenance.xml 2009-03-16 18:32:10 UTC (rev 2700) +++ provenancedb/provenance.xml 2009-03-16 18:33:11 UTC (rev 2701) @@ -37,46 +37,6 @@

- -

The old 'Getting started' section that you should ignore... - -This section details how to make basic use of some of the provenance database -that I have running at CI. - -

basic commands - -ssh to terminable. Set up path to the provenance commands like this: - -$ export PATH=~benc/swift-svn/provenancedb:$PATH - - - -

- -

Direct database access on terminable - - -Data is imported into two databases - an SQL database, and an XML database. -Approximately the same data is imported into both - you can experiment with -querying using SQL from the SQL language and XPath and XQuery from the -XML database. - -At the moment, the database is generated from my log repository once -per day. Logs that you put in the log repository a day ago should appear in -the databases; logs that you added more recently may not. - - -To query SQL, ssh into terminable, and type: - -$ psql -p 5435 -d provdb -U benc - - - -Details of the tables can be found in the 'plain sql' section later. - -

Running your own provenance database This section details running your own SQL-based provenance database on servers of your own control. @@ -2569,5 +2529,45 @@

The old 'Getting started' section that you should ignore... + +This section details how to make basic use of some of the provenance database +that I have running at CI. + +

basic commands + +ssh to terminable. Set up path to the provenance commands like this: + +$ export PATH=~benc/swift-svn/provenancedb:$PATH + + + +

+ +

Direct database access on terminable + + +Data is imported into two databases - an SQL database, and an XML database. +Approximately the same data is imported into both - you can experiment with +querying using SQL from the SQL language and XPath and XQuery from the +XML database. + +At the moment, the database is generated from my log repository once +per day. Logs that you put in the log repository a day ago should appear in +the databases; logs that you added more recently may not. + + +To query SQL, ssh into terminable, and type: + +$ psql -p 5435 -d provdb -U benc + + + +Details of the tables can be found in the 'plain sql' section later. + +

+ + From noreply at svn.ci.uchicago.edu Tue Mar 17 10:16:14 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 17 Mar 2009 10:16:14 -0500 (CDT) Subject: [Swift-commit] r2702 - trunk/src/org/griphyn/vdl/karajan/lib Message-ID: <20090317151614.BA87222821B@www.ci.uchicago.edu> Author: benc Date: 2009-03-17 10:16:13 -0500 (Tue, 17 Mar 2009) New Revision: 2702 Modified: trunk/src/org/griphyn/vdl/karajan/lib/CreateArray.java Log: remove a TODO that has been done Modified: trunk/src/org/griphyn/vdl/karajan/lib/CreateArray.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/CreateArray.java 2009-03-16 18:33:11 UTC (rev 2701) +++ trunk/src/org/griphyn/vdl/karajan/lib/CreateArray.java 2009-03-17 15:16:13 UTC (rev 2702) @@ -22,10 +22,6 @@ Object value = PA_VALUE.getValue(stack); try { -//TODO figure out the type here based on the type(s) of the array contents -// and throw appropriate exception in the case of inconsistency - - if (!(value instanceof List)) { throw new RuntimeException( "An array variable can only be initialized with a list of values"); From noreply at svn.ci.uchicago.edu Tue Mar 17 10:17:16 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 17 Mar 2009 10:17:16 -0500 (CDT) Subject: [Swift-commit] r2703 - in trunk: resources src/org/griphyn/vdl/engine Message-ID: <20090317151716.0800022820E@www.ci.uchicago.edu> Author: benc Date: 2009-03-17 10:17:15 -0500 (Tue, 17 Mar 2009) New Revision: 2703 Modified: trunk/resources/Karajan.stg trunk/src/org/griphyn/vdl/engine/Karajan.java Log: Separate the extractarrayelement template into the two templates that it really is. Modified: trunk/resources/Karajan.stg =================================================================== --- trunk/resources/Karajan.stg 2009-03-17 15:16:13 UTC (rev 2702) +++ trunk/resources/Karajan.stg 2009-03-17 15:17:15 UTC (rev 2703) @@ -436,25 +436,18 @@ $var$ >> -// TODO this would more cleanly be two templates, one for structure -// access and one for array access. - -extractarrayelement(parent, arraychild, memberchild, datatype) ::= << - -$if(arraychild)$ +extractarrayelement(parent, arraychild, datatype) ::= << $parent$ $arraychild$ -$endif$ +>> -$if(memberchild)$ +extractstructelement(parent, memberchild, datatype) ::= << $parent$ $memberchild$ -$endif$ - >> slicearray(parent, memberchild, datatype) ::= << Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/Karajan.java 2009-03-17 15:16:13 UTC (rev 2702) +++ trunk/src/org/griphyn/vdl/engine/Karajan.java 2009-03-17 15:17:15 UTC (rev 2703) @@ -1047,7 +1047,7 @@ newst.setAttribute("datatype", actualType); return newst; } else { - StringTemplate newst = template("extractarrayelement"); + StringTemplate newst = template("extractstructelement"); newst.setAttribute("parent", parentST); newst.setAttribute("memberchild", sm.getMemberName()); newst.setAttribute("datatype", actualType); From noreply at svn.ci.uchicago.edu Tue Mar 17 10:18:43 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 17 Mar 2009 10:18:43 -0500 (CDT) Subject: [Swift-commit] r2704 - provenancedb Message-ID: <20090317151843.17F7F22820E@www.ci.uchicago.edu> Author: benc Date: 2009-03-17 10:18:41 -0500 (Tue, 17 Mar 2009) New Revision: 2704 Modified: provenancedb/provenance.xml Log: some opm notes Modified: provenancedb/provenance.xml =================================================================== --- provenancedb/provenance.xml 2009-03-17 15:17:15 UTC (rev 2703) +++ provenancedb/provenance.xml 2009-03-17 15:18:41 UTC (rev 2704) @@ -1882,10 +1882,70 @@

OPM links Open Provenance Model at ipaw.info

+ +

Swift specific OPM considerations + + +non-strictness: Swift sometimes lazily constructs collections (leading to +the notion in Swift of an array being closed, which means that we know no +more contents will be created, somewhat like knowing we've reached the end +of a list). It may be that an array is never closed during a run, but that +we still have sufficient provenance information to answer useful queries +(for example, if we specify a list [1:100000] and only refer to the 5th +element in that array, we probably never generate most of the DSHandles... +so an explicit representation of that array in terms of datasets cannot be +expressed - though a higher level representation of it in terms of its +constructor parameters can be made) (?) + + + +aliasing: (this is related to some similar ambiguity in other parts of +Swift, to do with dataset roots - not provenance related). It is possible to +construct arrays by explicitly listing their members: + +int i = 8; +int j = 100; +int a[] = [i,j]; +int k = a[1]; +// here, k = 8 + +The dataset contained in i is an artifact (a literal, so +some input artifact that has no creating process). The array +a is an artifact created by the explicit array construction +syntax [memberlist] (which is an OPM process). If we +then model the array accessor syntax a[1] as an OPM +process, what artifact does it return? The same one or a different one? +In OPM, we want it to return a different artifact; but in Swift we want this +to be the same dataset... (perhaps explaining this with int +type variables is not the best way - using file-mapped data might be better) +TODO: what are the reasons we want files to have a single dataset +representation in Swift? dependency ordering - definitely. cache management? +Does this lead to a stronger notion of aliasing in Swift? + + + +Provenance of array indices: It seems fairly natural to represent arrays as OPM +collections, with array element extraction being a process. However, in OPM, +the index of an array is indicated with a role (with suggestions that it might +be a simple number or an XPath expression). In Swift arrays, the index is +a number, but it has its own provenance, so by recording only an integer there, +we lose provenance information about where that integer came from - that +integer is a Swift dataset in its own right, which has its own provenance. +It would be nice to be able to represent that (even if its not standardised +in OPM). I think that needs re-ification of roles so that they can be +described; or it needs treatment of [] as being like any other binary +operator (which is what happens inside swift) - where the LHS and RHS are +artifacts, and the role is not used for identifying the member (which would +also be an argument for making array element extraction be treated more +like a plain binary operator inside the Swift compiler and runtime) + +

stuff TODO transcribe info from the pile of papers I wrote. esp query analysis @@ -2569,5 +2629,36 @@

Representation of dataset containment and procedure execution in r2681 and how it could change. + + +Representation of processes that transform one dataset into another dataset +at present only occurs for app procedures, in logging of +vdl:execute invocations, in lines like this: + +2009-03-12 12:20:29,772+0100 INFO vdl:parameterlog PARAM thread=0-10-1 direction=input variable=s provenanceid=tag:benc at ci.uchicago.edu,2008:swift:dataset:20090312-1220-md2mfc24:720000000033 + +and dataset containment is represented at closing of the containing DSHandle by this: + +2009-03-12 12:20:30,205+0100 INFO AbstractDataNode CONTAINMENT parent=tag:benc at ci.uchicago.edu,2008:swift:dataset:20090312-1220-md2mfc24:720000000020 child=tag:benc at ci.uchicago.edu,2008:swift:dataset:20090312-1220-md2mfc24:720000000086 +2009-03-12 12:20:30,205+0100 INFO AbstractDataNode ROOTPATH dataset=tag:benc at ci.uchicago.edu,2008:swift:dataset:20090312-1220-md2mfc24:720000000086 path=[2] + + + + +This representation does not represent the relationship between datasets when +they are related by @functions or operators. Nor does it represent causal +relationships between collections and their members - instead it represents +containment. + + + +Adding representation of operators (including array construction) and of + at function invocations would give substantially more information about +the provenance of many more datasets. + + +

The Site Catalog - sites.xml Modified: trunk/libexec/_swiftwrap =================================================================== --- trunk/libexec/_swiftwrap 2009-03-27 20:24:17 UTC (rev 2747) +++ trunk/libexec/_swiftwrap 2009-03-27 21:14:45 UTC (rev 2748) @@ -194,6 +194,10 @@ export PATH=$PATHPREFIX:$PATH fi +if [ "$SWIFT_EXTRA_INFO" != "" ]; then + log "EXTRAINFO=$($SWIFT_EXTRA_INFO)" +fi + if [ "X${EXEC:0:1}" != "X/" ] ; then export ORIGEXEC=$EXEC export EXEC=$(which $EXEC) From noreply at svn.ci.uchicago.edu Sat Mar 28 10:55:54 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sat, 28 Mar 2009 10:55:54 -0500 (CDT) Subject: [Swift-commit] r2749 - in usertools/cio: . falkon swift Message-ID: <20090328155555.14A3622817C@www.ci.uchicago.edu> Author: aespinosa Date: 2009-03-28 10:55:52 -0500 (Sat, 28 Mar 2009) New Revision: 2749 Added: usertools/cio/falkon/ usertools/cio/falkon/falkon-start-bgp-ram-iptorus.sh-mosa usertools/cio/falkon/run.worker-c-bgp-iptorus.sh-mosa Modified: usertools/cio/README usertools/cio/swift/modswift.sh Log: Imported BGP Falkon CIO startup scripts Modified: usertools/cio/README =================================================================== --- usertools/cio/README 2009-03-27 21:14:45 UTC (rev 2748) +++ usertools/cio/README 2009-03-28 15:55:52 UTC (rev 2749) @@ -6,3 +6,5 @@ tools/ +falkon/* : Patched Falkon startup scripts to initiate CIO services + Added: usertools/cio/falkon/falkon-start-bgp-ram-iptorus.sh-mosa =================================================================== --- usertools/cio/falkon/falkon-start-bgp-ram-iptorus.sh-mosa (rev 0) +++ usertools/cio/falkon/falkon-start-bgp-ram-iptorus.sh-mosa 2009-03-28 15:55:52 UTC (rev 2749) @@ -0,0 +1,152 @@ +#!/bin/bash + + + if [ -z "$3" ]; then + echo "usage: $0 " + echo "usage: $0 prod 1024 60" + echo "-- or --" + echo "usage: $0 " + echo "usage: $0 prod 1024 60 4 1 1" + exit 1 + fi + + +if [ -z "${FALKON_HOME}" ]; then + echo "ERROR: environment variable FALKON_HOME not defined" + exit 1 +fi + + +QUEUE_NAME=$1 +#PROFILE_NAME="iskra-mpich" +PROFILE_NAME="zeptocn" +#PROFILE_NAME="zeptocn-exp" +NUM_NODES=$2 +let NUM_ION=NUM_NODES/64 +MAX_TIME_MIN=$3 +SERVICE_IP="192.168.1.254" +SERVICE_PORT1=55000 +SERVICE_PORT2=55001 +WORKERS_PER_NODE=4 +DATA_NUM=1 +STR=1 +if [ ! -z $4 ];then + WORKERS_PER_NODE=$4 +fi +if [ ! -z $5 ];then + DATA_NUM=$5 +fi +if [ ! -z $6 ];then + STR=$6 +fi + + +cp ${FALKON_HOME}/bin/zoid-user-script.sh ${HOME}/zoid-user-script.sh +chmod +x ${HOME}/zoid-user-script.sh + +FALKON_JOB_ID=`falkon-id-get.sh N/A` +EXIT_CODE=$? + + + if [ "${EXIT_CODE}" -ne "0" ]; then + echo "Error in geting a unique falkon ID.. 'falkon-get-id.sh N/A'" + cqdel ${ZOID_JOB_ID} + exit ${EXIT_CODE} + fi + + + +ZOID_JOB_ID=`cqsub -q ${QUEUE_NAME} -k ${PROFILE_NAME} -C ${HOME} -t ${MAX_TIME_MIN} -n ${NUM_NODES} -e LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib /bgsys/linux/1.2.020080512/bin/bash /fuse/gpfs/home/espinosa/experiments/falkon/run.worker-c-bgp-iptorus.sh-mosa ${SERVICE_IP} ${SERVICE_PORT1} ${SERVICE_PORT2} ${WORKERS_PER_NODE} ${USER} ${FALKON_JOB_ID} ${FALKON_ROOT} ${NUM_NODES} ${DATA_NUM} ${STR}` + +EXIT_CODE=$? + + if [ "${EXIT_CODE}" -ne "0" ]; then + echo "Error in submitting job to Cobalt.. 'cqsub -q ${QUEUE_NAME} -k ${PROFILE_NAME} -C ${HOME} -t ${MAX_TIME_MIN} -n ${NUM_NODES} -e LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib /bgsys/linux/1.2.020080512/bin/bash /fuse/${FALKON_WORKER_HOME}/run.worker-c-bgp.sh-mosa ${SERVICE_IP} ${SERVICE_PORT1} ${SERVICE_PORT2} ${WORKERS_PER_NODE} ${USER} ${FALKON_JOB_ID} ${FALKON_ROOT}' " + exit ${EXIT_CODE} + fi + +falkon-id-update.sh ${FALKON_JOB_ID} ${ZOID_JOB_ID} + +EXIT_CODE=$? + + if [ "${EXIT_CODE}" -ne "0" ]; then + echo "Error in updating cobalt job info for falkon.. 'falkon-update-id.sh ${ZOID_JOB_ID}'" + cqdel ${ZOID_JOB_ID} + exit ${EXIT_CODE} + fi + + +FALKON_JOBID_HOME=${FALKON_ROOT}/users/${USER}/${FALKON_JOB_ID} + +echo "Submitted job ${ZOID_JOB_ID} to Cobalt, creating the job specific Falkon tree for logs and configuration in ${FALKON_JOBID_HOME}..." + +mkdir -p ${FALKON_JOBID_HOME} + +cp ${FALKON_HOME}/falkon.env.bgp* ${FALKON_JOBID_HOME}/ +cp -r ${FALKON_HOME}/config ${FALKON_JOBID_HOME}/ +cp ${FALKON_JOBID_HOME}/config/Client-service-URIs.config2 ${FALKON_JOBID_HOME}/config/Client-service-URIs.config +mkdir -p ${FALKON_JOBID_HOME}/logs/client ${FALKON_JOBID_HOME}/logs/service ${FALKON_JOBID_HOME}/logs/provisioner ${FALKON_JOBID_HOME}/logs/worker + + +DATE=`date +%s` +echo "$DATE: pre-creating log dirs for Falkon service..." +RACK_START=0 +RACK_END=48 +SEQUENCE_DIR=`seq -w ${RACK_START} ${RACK_END}` +PSET_START=1 +PSET_END=16 +for a in ${SEQUENCE_DIR} +do + for ((b=${PSET_START}; b <= ${PSET_END} ; b++)) # Double parentheses, and "LIMIT" with no "$". + do + DIR_NAME="ion-R${a}-${b}" + mkdir -p ${FALKON_JOBID_HOME}/logs/service/$DIR_NAME + done +done + +for ((b=${PSET_START}; b <= ${PSET_END} ; b++)) # Double parentheses, and "LIMIT" with no "$". +do + DIR_NAME="ion-${b}" + mkdir -p ${FALKON_JOBID_HOME}/logs/service/$DIR_NAME +done + + +DATE=`date +%s` +echo "$DATE: done creating log dirs for Falkon service!" + + + +FALKON_HOME_RAM=/tmp/${USER}/falkon + +ln -s ${FALKON_HOME}/apps ${FALKON_JOBID_HOME}/apps +ln -s ${FALKON_HOME_RAM}/container ${FALKON_JOBID_HOME}/container +ln -s ${FALKON_HOME}/service ${FALKON_JOBID_HOME}/service +ln -s ${FALKON_HOME}/worker ${FALKON_JOBID_HOME}/worker +ln -s ${FALKON_HOME}/AstroPortal ${FALKON_JOBID_HOME}/AstroPortal +ln -s ${FALKON_HOME}/client ${FALKON_JOBID_HOME}/client +ln -s ${FALKON_HOME}/monitor ${FALKON_JOBID_HOME}/monitor +ln -s ${FALKON_HOME}/bin ${FALKON_JOBID_HOME}/bin +ln -s ${FALKON_HOME}/config ${FALKON_JOBID_HOME}/config +ln -s ${FALKON_HOME}/ploticus ${FALKON_JOBID_HOME}/ploticus +ln -s ${FALKON_HOME}/webserver ${FALKON_JOBID_HOME}/webserver +ln -s ${FALKON_HOME}/workloads ${FALKON_JOBID_HOME}/workloads +ln -s ${FALKON_HOME}/id ${FALKON_JOBID_HOME}/id +ln -s ${FALKON_HOME}/apache-ant-1.7.0 ${FALKON_JOBID_HOME}/apache-ant-1.7.0 +ln -s ${FALKON_HOME}/ibm-java2-ppc64-50 ${FALKON_JOBID_HOME}/ibm-java2-ppc64-50 +ln -s ${FALKON_HOME_RAM}/ibm-java2-ppc-50 ${FALKON_JOBID_HOME}/ibm-java2-ppc-50 +ln -s ${FALKON_HOME}/falkon.tgz ${FALKON_JOBID_HOME}/falkon.tgz + + +if [ ! -d "${FALKON_JOBID_HOME}" ]; then + echo "ERROR: invalid path ${FALKON_JOBID_HOME}... exiting" + cqdel ${ZOID_JOB_ID} + exit 1 +fi + +echo "Succesfully submitted the job to Cobalt, and setup job specific Falkon tree!" +echo "To monitor the job status, type 'cqstat | grep ${USER}'; once it is in running state, you can use the Falkon specific command ...." +echo "To submit your Falkon-based workload, type: ....; you can do this any time, the falkon workload will wait for the resources to come online, and will only be submitted when everything is ready; the script is run in the background, so the workload will run even if the ssh session gets disconnected." +echo "" +echo "Remember, your job id is ${ZOID_JOB_ID}, and if you need to look through the logs manually for anything, remember that you can find them at ${HOME}/${ZOID_JOB_ID}.output, ${HOME}/${ZOID_JOB_ID}.error, and ${FALKON_JOBID_HOME}/logs/..." + + Property changes on: usertools/cio/falkon/falkon-start-bgp-ram-iptorus.sh-mosa ___________________________________________________________________ Name: svn:executable + * Added: usertools/cio/falkon/run.worker-c-bgp-iptorus.sh-mosa =================================================================== --- usertools/cio/falkon/run.worker-c-bgp-iptorus.sh-mosa (rev 0) +++ usertools/cio/falkon/run.worker-c-bgp-iptorus.sh-mosa 2009-03-28 15:55:52 UTC (rev 2749) @@ -0,0 +1,152 @@ +#!/fuse/bin/bash + +#ERROR_LOG=${FALKON_ROOT_PATH}/users/error.log +#ERROR_LOG=/home/falkon/users/error-workers.log +ERROR_LOG=/dev/null +#ERROR_LOG=/logfile + +if [ -z "$8" ]; then + echo "usage: $0 " >> ${ERROR_LOG} + echo "$0 192.168.1.254 55000 55001 4 iraicu 0001 [FALKON_ROOT] 1024 1 1" + echo "$0 192.168.1.254 55000 55001 4 iraicu 0001 [FALKON_R00T] 1024 1 1" >> ${ERROR_LOG} + exit 2 +fi + +# Initiate the IP stack of the Torus network +# -aespinosa + +/home/espinosa/bin/cn-ipfwd & + +while [ ! -f /tmp/ifconfig.cmd ]; do + sleep 1 +done + +. /tmp/ifconfig.cmd + +#source /home/iraicu/.bashrc + +IO_RANK=`/home/espinosa/bin/iorank` +BG_RANK=`/home/espinosa/bin/rank` +PSET=`/home/espinosa/bin/pset` + +# size of the allocated partition. also known as the total number of CNs +PART_SIZE=$8 +# number of data CNs in partition +DATA_NUM=$9 +STR=${10} + +ROLE=`/home/espinosa/bin/isdata $PART_SIZE $DATA_NUM $STR` +ISDATA=$ROLE +tmp=`/home/espinosa/bin/getdata $PART_SIZE $DATA_NUM $STR` +DATACN=`/fuse/gpfs/home/espinosa/bin/ipcal 10.128.0.0 $tmp` + +if [ "$ISDATA" != 'app' ] +then + if [ "$ROLE" = 'head' ] + then +# if [ $STR -eq 1 ] +# then +# echo "[ $BG_RANK@$IO_RANK-$PSET ] Starting Chirp server..." >&2 +# /home/espinosa/experiments/util/startchirp.sh >&2 +# else + echo "[ $BG_RANK@$IO_RANK-$PSET ] Starting Mosa Benefactor managed by $DATACN" >&2 + echo "[ $BG_RANK@$IO_RANK-$PSET ] Starting Mosa Manager across $STR CNs" >&2 + /home/espinosa/experiments/util/startchirp-wmosa.sh $DATACN $STR >&2 +# fi + else + echo "[ $BG_RANK@$IO_RANK-$PSET ] Starting Mosa Benefactor managed by $DATACN" >&2 + /home/espinosa/experiments/util/benefactor.sh $DATACN >&2 + fi +else + echo "[ $BG_RANK@$IO_RANK-$PSET ] Mounting Mosa data CN filesystem from $DATACN" >&2 + /bin/mkdir /dataifs + export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/lib:/fuse/lib:/usr/lib + sleep 30 + /home/espinosa/experiments/util/mosamount.sh $DATACN >&2 + + echo "worker started..." >> ${ERROR_LOG} + + FALKON_SERVICE_IP=$1 + TCPCORE_PORT1=$2 + TCPCORE_PORT2=$3 + NUM_WORKERS=$4 + USER_ID=$5 + FALKON_JOB_ID=$6 + FALKON_ROOT_PATH=$7 + + + ln -s /fuse/gpfs1 /gpfs1 + #FALKON_HOME=/gpfs1/falkon/falkon + #HOME=/gpfs1/falkon + + + + #echo "Worker for job id ${ZOID_JOB_ID} starting..." >> ${ERROR_LOG} + + FALKON_HOME=${FALKON_ROOT_PATH}/users/${USER_ID}/${FALKON_JOB_ID} + + echo "FALKON_HOME = ${FALKON_HOME}" >> ${ERROR_LOG} + + #FALKON_HOME=/home/falkon/falkon + + HOME=${FALKON_ROOT_PATH} + echo "HOME = ${HOME}" >> ${ERROR_LOG} + #HOME=${FALKON_HOME} + + if [ -z "${FALKON_HOME}" ]; then + echo "ERROR: environment variable FALKON_HOME not defined" + echo "ERROR: environment variable FALKON_HOME not defined" >> ${ERROR_LOG} + exit 1 + fi + + if [ ! -d "${FALKON_HOME}" ]; then + echo "ERROR: invalid FALKON_HOME set: $FALKON_HOME" + echo "ERROR: invalid FALKON_HOME set: $FALKON_HOME" >> ${ERROR_LOG} + exit 1 + fi + + cd ${FALKON_HOME} + source falkon.env.bgp-cn + + + + HOME_PATH=`pwd` + export MACH_ID=`uname -i` + RAND_ID2=`/home/espinosa/bin/rank` + + a0=${RAND_ID2%%=*}; rest=${RAND_ID2#*=} + RAND_ID=${rest%%=*}; + + #PLATFORM=`uname -m` + + + + EXP_START=`date +%Y.%m.%d_%H.%M.%S` + + + for ((b=1; b < ${NUM_WORKERS} ; b++)) + do + #LOG_FILE=${FALKON_LOGS}/worker/${RAND_ID}.${b}.${EXP_START}.txt + LOG_FILE=/dev/null + + #${FALKON_WORKER_HOME}/src-c/BGexec_${PLATFORM} ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf >> "${LOG_FILE}" 2>&1 & + #echo "starting .... ${FALKON_WORKER_HOME}/src-c/BGexec ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf" + echo "starting worker ${RAND_ID} CPU ${b}..." >> ${ERROR_LOG} + ${FALKON_WORKER_HOME}/src-c/BGexec ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf >> ${LOG_FILE} 2>&1 & + #${HOME_PATH}/src-c/BGexec_${PLATFORM} ${FALKON_SERVICE_IP} 55000 55001 -debug + /bin/sleep 1 + done + + #LOG_FILE=${FALKON_LOGS}/worker/${RAND_ID}.${NUM_WORKERS}.${EXP_START}.txt + LOG_FILE=/dev/null + echo "starting worker ${RAND_ID} CPU ${NUM_WORKERS}..." >> ${ERROR_LOG} + + #${FALKON_WORKER_HOME}/src-c/BGexec_${PLATFORM} ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf >> "${LOG_FILE}" 2>&1 + #echo "starting .... ${FALKON_WORKER_HOME}/src-c/BGexec ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf" + + ${FALKON_WORKER_HOME}/src-c/BGexec ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf >> ${LOG_FILE} 2>&1 + + + echo "finished!" >> ${ERROR_LOG} +fi Property changes on: usertools/cio/falkon/run.worker-c-bgp-iptorus.sh-mosa ___________________________________________________________________ Name: svn:executable + * Modified: usertools/cio/swift/modswift.sh =================================================================== --- usertools/cio/swift/modswift.sh 2009-03-27 21:14:45 UTC (rev 2748) +++ usertools/cio/swift/modswift.sh 2009-03-28 15:55:52 UTC (rev 2749) @@ -2,3 +2,15 @@ # this script should copy all the mods below this dir into the right place in th swift source tree. +COGDIR=src/cogkit +SWIFTDIR=$COGDIR/modules/swift + +if [ ! -f $SWIFTDIR/libexec/wrapper.sh.vanilla ]; then + mv $SWIFTDIR/libexec/wrapper.sh $SWIFTDIR/libexec/wrapper.sh.vanilla +fi +cp -v libexec/wrapper.sh $SWIFTDIR/libexec/wrapper.sh + +if [ ! -f $SWIFTDIR/libexec/vdl-int.k.vanilla ]; then + mv $SWIFTDIR/libexec/vdl-int.k $SWIFTDIR/libexec/vdl-int.k.vanilla +fi +cp -v libexec/vdl-int.k $SWIFTDIR/libexec/vdl-int.k From noreply at svn.ci.uchicago.edu Sat Mar 28 10:56:09 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sat, 28 Mar 2009 10:56:09 -0500 (CDT) Subject: [Swift-commit] r2750 - usertools/cio/falkon Message-ID: <20090328155609.524DE22817C@www.ci.uchicago.edu> Author: aespinosa Date: 2009-03-28 10:56:07 -0500 (Sat, 28 Mar 2009) New Revision: 2750 Added: usertools/cio/falkon/cio-logic.sh usertools/cio/falkon/falkon-start-bgp-cio.sh Removed: usertools/cio/falkon/falkon-start-bgp-ram-iptorus.sh-mosa usertools/cio/falkon/run.worker-c-bgp-iptorus.sh-mosa Log: Initial cleanup of CIO logic startup scripts on BGP Copied: usertools/cio/falkon/cio-logic.sh (from rev 2749, usertools/cio/falkon/run.worker-c-bgp-iptorus.sh-mosa) =================================================================== --- usertools/cio/falkon/cio-logic.sh (rev 0) +++ usertools/cio/falkon/cio-logic.sh 2009-03-28 15:56:07 UTC (rev 2750) @@ -0,0 +1,137 @@ +#!/fuse/bin/bash + +CIOROOT=/home/espinosa/cio +ERROR_LOG=/dev/null + +if [ $# -lt 7 ]; then + cat << EOF +Usage: $0 \ + +Example: $0 192.168.1.254 55000 55001 4 iraicu 0001 [FALKON_ROOT] \ + 1024 1 1" +EOF + exit 2 +fi + +# Initiate the IP stack of the Torus network +# -aespinosa + +/home/iskra/ZeptoOS/packages/cnip/prebuilt/cn-ipfwd.2409 & + +while [ ! -f /tmp/ifconfig.cmd ]; do + sleep 1 +done + +. /tmp/ifconfig.cmd + +RANK=`/home/espinosa/bin/rank` + +# size of the allocated partition. also known as the total number of CNs +PART_SIZE=$8 +# number of data CNs in partition +DATA_NUM=$9 +STR=${10} + +ROLE=`/home/espinosa/bin/isdata $PART_SIZE $DATA_NUM $STR` +ISDATA=$ROLE +tmp=`/home/espinosa/bin/getdata $PART_SIZE $DATA_NUM $STR` +DATACN=`/fuse/gpfs/home/espinosa/bin/ipcal 10.128.0.0 $tmp` + +if [ $RANK -eq 0 ]; then + # Run Central Hash Service + $CIOROOT/hashserver.rb +elif [ "$ISDATA" != 'app' ]; then + if [ "$ROLE" = 'head' ]; then + echo "[ $BG_RANK@$IO_RANK-$PSET ] Starting Mosa Benefactor managed by $DATACN" >&2 + echo "[ $BG_RANK@$IO_RANK-$PSET ] Starting Mosa Manager across $STR CNs" >&2 + /home/espinosa/experiments/util/startchirp-wmosa.sh $DATACN $STR >&2 + else + echo "[ $BG_RANK@$IO_RANK-$PSET ] Starting Mosa Benefactor managed by $DATACN" >&2 + /home/espinosa/experiments/util/benefactor.sh $DATACN >&2 + fi +else + echo "[ $BG_RANK@$IO_RANK-$PSET ] Mounting Mosa data CN filesystem from $DATACN" >&2 + /bin/mkdir /dataifs + export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/lib:/fuse/lib:/usr/lib + sleep 30 + /home/espinosa/experiments/util/mosamount.sh $DATACN >&2 + + echo "worker started..." >> ${ERROR_LOG} + + # Regular Falon worker startup script + + FALKON_SERVICE_IP=$1 + TCPCORE_PORT1=$2 + TCPCORE_PORT2=$3 + NUM_WORKERS=$4 + USER_ID=$5 + FALKON_JOB_ID=$6 + FALKON_ROOT_PATH=$7 + + + FALKON_HOME=${FALKON_ROOT_PATH}/users/${USER_ID}/${FALKON_JOB_ID} + + echo "FALKON_HOME = ${FALKON_HOME}" >> ${ERROR_LOG} + + #FALKON_HOME=/home/falkon/falkon + + HOME=${FALKON_ROOT_PATH} + echo "HOME = ${HOME}" >> ${ERROR_LOG} + #HOME=${FALKON_HOME} + + if [ -z "${FALKON_HOME}" ]; then + echo "ERROR: environment variable FALKON_HOME not defined" + echo "ERROR: environment variable FALKON_HOME not defined" >> ${ERROR_LOG} + exit 1 + fi + + if [ ! -d "${FALKON_HOME}" ]; then + echo "ERROR: invalid FALKON_HOME set: $FALKON_HOME" + echo "ERROR: invalid FALKON_HOME set: $FALKON_HOME" >> ${ERROR_LOG} + exit 1 + fi + + cd ${FALKON_HOME} + source falkon.env.bgp-cn + + + + HOME_PATH=`pwd` + export MACH_ID=`uname -i` + RAND_ID2=`/home/espinosa/bin/rank` + + a0=${RAND_ID2%%=*}; rest=${RAND_ID2#*=} + RAND_ID=${rest%%=*}; + + #PLATFORM=`uname -m` + + + + EXP_START=`date +%Y.%m.%d_%H.%M.%S` + + + for ((b=1; b < ${NUM_WORKERS} ; b++)) + do + #LOG_FILE=${FALKON_LOGS}/worker/${RAND_ID}.${b}.${EXP_START}.txt + LOG_FILE=/dev/null + + #${FALKON_WORKER_HOME}/src-c/BGexec_${PLATFORM} ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf >> "${LOG_FILE}" 2>&1 & + #echo "starting .... ${FALKON_WORKER_HOME}/src-c/BGexec ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf" + echo "starting worker ${RAND_ID} CPU ${b}..." >> ${ERROR_LOG} + ${FALKON_WORKER_HOME}/src-c/BGexec ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf >> ${LOG_FILE} 2>&1 & + #${HOME_PATH}/src-c/BGexec_${PLATFORM} ${FALKON_SERVICE_IP} 55000 55001 -debug + /bin/sleep 1 + done + + #LOG_FILE=${FALKON_LOGS}/worker/${RAND_ID}.${NUM_WORKERS}.${EXP_START}.txt + LOG_FILE=/dev/null + echo "starting worker ${RAND_ID} CPU ${NUM_WORKERS}..." >> ${ERROR_LOG} + + #${FALKON_WORKER_HOME}/src-c/BGexec_${PLATFORM} ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf >> "${LOG_FILE}" 2>&1 + #echo "starting .... ${FALKON_WORKER_HOME}/src-c/BGexec ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf" + + ${FALKON_WORKER_HOME}/src-c/BGexec ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf >> ${LOG_FILE} 2>&1 + + + echo "finished!" >> ${ERROR_LOG} +fi Copied: usertools/cio/falkon/falkon-start-bgp-cio.sh (from rev 2749, usertools/cio/falkon/falkon-start-bgp-ram-iptorus.sh-mosa) =================================================================== --- usertools/cio/falkon/falkon-start-bgp-cio.sh (rev 0) +++ usertools/cio/falkon/falkon-start-bgp-cio.sh 2009-03-28 15:56:07 UTC (rev 2750) @@ -0,0 +1,153 @@ +#!/bin/bash + +if [ $# -lt 3 ]; then + cat << EOF +Usage: $0 +Example: $0 prod 1024 60 +--or-- +Usage: $0 \ + +EOF + exit 1 +fi + +if [ -z $FALKON_HOME ]; then + echo "ERROR: environment variable FALKON_HOME not defined" + exit 1 +fi + +PROFILE_NAME="zeptocn" +QUEUE_NAME=$1 +NUM_NODES=$2 +let NUM_ION=NUM_NODES/64 +MAX_TIME_MIN=$3 +SERVICE_IP="192.168.1.254" +SERVICE_PORT1=55000 +SERVICE_PORT2=55001 + +WORKERS_PER_NODE=4 +DATA_NUM=1 +STR=1 +if [ -n $4 ];then + WORKERS_PER_NODE=$4 +fi +if [ -n $5 ];then + DATA_NUM=$5 +fi +if [ -n $6 ];then + STR=$6 +fi + + +cp ${FALKON_HOME}/bin/zoid-user-script.sh ${HOME}/zoid-user-script.sh +chmod +x ${HOME}/zoid-user-script.sh + +FALKON_JOB_ID=`falkon-id-get.sh N/A` +EXIT_CODE=$? + +if [ "${EXIT_CODE}" -ne "0" ]; then + echo "Error in geting a unique falkon ID.. 'falkon-get-id.sh N/A'" + cqdel ${ZOID_JOB_ID} + exit ${EXIT_CODE} +fi + + + +WORKER_SCRIPT=cio-logic.sh +ZOID_JOB_ID=`cqsub -q ${QUEUE_NAME} -k ${PROFILE_NAME} -C ${HOME} -t \ + ${MAX_TIME_MIN} -n ${NUM_NODES} \ + -e LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib \ + /bgsys/linux/1.2.020080512/bin/bash $WORKER_SCRIPT \ + ${SERVICE_IP} ${SERVICE_PORT1} ${SERVICE_PORT2} ${WORKERS_PER_NODE} \ + ${USER} ${FALKON_JOB_ID} ${FALKON_ROOT} ${NUM_NODES} ${DATA_NUM} ${STR}` + +EXIT_CODE=$? + +if [ "${EXIT_CODE}" -ne "0" ]; then + echo "Error in submitting job to Cobalt.. 'cqsub -q ${QUEUE_NAME} -k ${PROFILE_NAME} -C ${HOME} -t ${MAX_TIME_MIN} -n ${NUM_NODES} -e LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib /bgsys/linux/1.2.020080512/bin/bash /fuse/${FALKON_WORKER_HOME}/run.worker-c-bgp.sh-mosa ${SERVICE_IP} ${SERVICE_PORT1} ${SERVICE_PORT2} ${WORKERS_PER_NODE} ${USER} ${FALKON_JOB_ID} ${FALKON_ROOT}' " + exit ${EXIT_CODE} +fi + +falkon-id-update.sh ${FALKON_JOB_ID} ${ZOID_JOB_ID} + +EXIT_CODE=$? + +if [ "${EXIT_CODE}" -ne "0" ]; then + echo "Error in updating cobalt job info for falkon.. 'falkon-update-id.sh ${ZOID_JOB_ID}'" + cqdel ${ZOID_JOB_ID} + exit ${EXIT_CODE} +fi + + +FALKON_JOBID_HOME=${FALKON_ROOT}/users/${USER}/${FALKON_JOB_ID} + +echo "Submitted job ${ZOID_JOB_ID} to Cobalt, creating the job specific Falkon tree for logs and configuration in ${FALKON_JOBID_HOME}..." + +mkdir -p ${FALKON_JOBID_HOME} + +cp ${FALKON_HOME}/falkon.env.bgp* ${FALKON_JOBID_HOME}/ +cp -r ${FALKON_HOME}/config ${FALKON_JOBID_HOME}/ +cp ${FALKON_JOBID_HOME}/config/Client-service-URIs.config2 ${FALKON_JOBID_HOME}/config/Client-service-URIs.config +mkdir -p ${FALKON_JOBID_HOME}/logs/client ${FALKON_JOBID_HOME}/logs/service ${FALKON_JOBID_HOME}/logs/provisioner ${FALKON_JOBID_HOME}/logs/worker + + +DATE=`date +%s` +echo "$DATE: pre-creating log dirs for Falkon service..." +RACK_START=0 +RACK_END=48 +SEQUENCE_DIR=`seq -w ${RACK_START} ${RACK_END}` +PSET_START=1 +PSET_END=16 +for a in ${SEQUENCE_DIR} +do + for ((b=${PSET_START}; b <= ${PSET_END} ; b++)) # Double parentheses, and "LIMIT" with no "$". + do + DIR_NAME="ion-R${a}-${b}" + mkdir -p ${FALKON_JOBID_HOME}/logs/service/$DIR_NAME + done +done + +for ((b=${PSET_START}; b <= ${PSET_END} ; b++)) # Double parentheses, and "LIMIT" with no "$". +do + DIR_NAME="ion-${b}" + mkdir -p ${FALKON_JOBID_HOME}/logs/service/$DIR_NAME +done + + +DATE=`date +%s` +echo "$DATE: done creating log dirs for Falkon service!" + +FALKON_HOME_RAM=/tmp/${USER}/falkon + +ln -s ${FALKON_HOME}/apps ${FALKON_JOBID_HOME}/apps +ln -s ${FALKON_HOME_RAM}/container ${FALKON_JOBID_HOME}/container +ln -s ${FALKON_HOME}/service ${FALKON_JOBID_HOME}/service +ln -s ${FALKON_HOME}/worker ${FALKON_JOBID_HOME}/worker +ln -s ${FALKON_HOME}/AstroPortal ${FALKON_JOBID_HOME}/AstroPortal +ln -s ${FALKON_HOME}/client ${FALKON_JOBID_HOME}/client +ln -s ${FALKON_HOME}/monitor ${FALKON_JOBID_HOME}/monitor +ln -s ${FALKON_HOME}/bin ${FALKON_JOBID_HOME}/bin +ln -s ${FALKON_HOME}/config ${FALKON_JOBID_HOME}/config +ln -s ${FALKON_HOME}/ploticus ${FALKON_JOBID_HOME}/ploticus +ln -s ${FALKON_HOME}/webserver ${FALKON_JOBID_HOME}/webserver +ln -s ${FALKON_HOME}/workloads ${FALKON_JOBID_HOME}/workloads +ln -s ${FALKON_HOME}/id ${FALKON_JOBID_HOME}/id +ln -s ${FALKON_HOME}/apache-ant-1.7.0 ${FALKON_JOBID_HOME}/apache-ant-1.7.0 +ln -s ${FALKON_HOME}/ibm-java2-ppc64-50 ${FALKON_JOBID_HOME}/ibm-java2-ppc64-50 +ln -s ${FALKON_HOME_RAM}/ibm-java2-ppc-50 ${FALKON_JOBID_HOME}/ibm-java2-ppc-50 +ln -s ${FALKON_HOME}/falkon.tgz ${FALKON_JOBID_HOME}/falkon.tgz + + +if [ ! -d "${FALKON_JOBID_HOME}" ]; then + echo "ERROR: invalid path ${FALKON_JOBID_HOME}... exiting" + cqdel ${ZOID_JOB_ID} + exit 1 +fi + +echo "Succesfully submitted the job to Cobalt, and setup job specific Falkon tree!" +echo "To monitor the job status, type 'cqstat | grep ${USER}'; once it is in running state, you can use the Falkon specific command ...." +echo "To submit your Falkon-based workload, type: ....; you can do this any time, the falkon workload will wait for the resources to come online, and will only be submitted when everything is ready; the script is run in the background, so the workload will run even if the ssh session gets disconnected." +echo "" +echo "Remember, your job id is ${ZOID_JOB_ID}, and if you need to look through the logs manually for anything, remember that you can find them at ${HOME}/${ZOID_JOB_ID}.output, ${HOME}/${ZOID_JOB_ID}.error, and ${FALKON_JOBID_HOME}/logs/..." + + Deleted: usertools/cio/falkon/falkon-start-bgp-ram-iptorus.sh-mosa =================================================================== --- usertools/cio/falkon/falkon-start-bgp-ram-iptorus.sh-mosa 2009-03-28 15:55:52 UTC (rev 2749) +++ usertools/cio/falkon/falkon-start-bgp-ram-iptorus.sh-mosa 2009-03-28 15:56:07 UTC (rev 2750) @@ -1,152 +0,0 @@ -#!/bin/bash - - - if [ -z "$3" ]; then - echo "usage: $0 " - echo "usage: $0 prod 1024 60" - echo "-- or --" - echo "usage: $0 " - echo "usage: $0 prod 1024 60 4 1 1" - exit 1 - fi - - -if [ -z "${FALKON_HOME}" ]; then - echo "ERROR: environment variable FALKON_HOME not defined" - exit 1 -fi - - -QUEUE_NAME=$1 -#PROFILE_NAME="iskra-mpich" -PROFILE_NAME="zeptocn" -#PROFILE_NAME="zeptocn-exp" -NUM_NODES=$2 -let NUM_ION=NUM_NODES/64 -MAX_TIME_MIN=$3 -SERVICE_IP="192.168.1.254" -SERVICE_PORT1=55000 -SERVICE_PORT2=55001 -WORKERS_PER_NODE=4 -DATA_NUM=1 -STR=1 -if [ ! -z $4 ];then - WORKERS_PER_NODE=$4 -fi -if [ ! -z $5 ];then - DATA_NUM=$5 -fi -if [ ! -z $6 ];then - STR=$6 -fi - - -cp ${FALKON_HOME}/bin/zoid-user-script.sh ${HOME}/zoid-user-script.sh -chmod +x ${HOME}/zoid-user-script.sh - -FALKON_JOB_ID=`falkon-id-get.sh N/A` -EXIT_CODE=$? - - - if [ "${EXIT_CODE}" -ne "0" ]; then - echo "Error in geting a unique falkon ID.. 'falkon-get-id.sh N/A'" - cqdel ${ZOID_JOB_ID} - exit ${EXIT_CODE} - fi - - - -ZOID_JOB_ID=`cqsub -q ${QUEUE_NAME} -k ${PROFILE_NAME} -C ${HOME} -t ${MAX_TIME_MIN} -n ${NUM_NODES} -e LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib /bgsys/linux/1.2.020080512/bin/bash /fuse/gpfs/home/espinosa/experiments/falkon/run.worker-c-bgp-iptorus.sh-mosa ${SERVICE_IP} ${SERVICE_PORT1} ${SERVICE_PORT2} ${WORKERS_PER_NODE} ${USER} ${FALKON_JOB_ID} ${FALKON_ROOT} ${NUM_NODES} ${DATA_NUM} ${STR}` - -EXIT_CODE=$? - - if [ "${EXIT_CODE}" -ne "0" ]; then - echo "Error in submitting job to Cobalt.. 'cqsub -q ${QUEUE_NAME} -k ${PROFILE_NAME} -C ${HOME} -t ${MAX_TIME_MIN} -n ${NUM_NODES} -e LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib /bgsys/linux/1.2.020080512/bin/bash /fuse/${FALKON_WORKER_HOME}/run.worker-c-bgp.sh-mosa ${SERVICE_IP} ${SERVICE_PORT1} ${SERVICE_PORT2} ${WORKERS_PER_NODE} ${USER} ${FALKON_JOB_ID} ${FALKON_ROOT}' " - exit ${EXIT_CODE} - fi - -falkon-id-update.sh ${FALKON_JOB_ID} ${ZOID_JOB_ID} - -EXIT_CODE=$? - - if [ "${EXIT_CODE}" -ne "0" ]; then - echo "Error in updating cobalt job info for falkon.. 'falkon-update-id.sh ${ZOID_JOB_ID}'" - cqdel ${ZOID_JOB_ID} - exit ${EXIT_CODE} - fi - - -FALKON_JOBID_HOME=${FALKON_ROOT}/users/${USER}/${FALKON_JOB_ID} - -echo "Submitted job ${ZOID_JOB_ID} to Cobalt, creating the job specific Falkon tree for logs and configuration in ${FALKON_JOBID_HOME}..." - -mkdir -p ${FALKON_JOBID_HOME} - -cp ${FALKON_HOME}/falkon.env.bgp* ${FALKON_JOBID_HOME}/ -cp -r ${FALKON_HOME}/config ${FALKON_JOBID_HOME}/ -cp ${FALKON_JOBID_HOME}/config/Client-service-URIs.config2 ${FALKON_JOBID_HOME}/config/Client-service-URIs.config -mkdir -p ${FALKON_JOBID_HOME}/logs/client ${FALKON_JOBID_HOME}/logs/service ${FALKON_JOBID_HOME}/logs/provisioner ${FALKON_JOBID_HOME}/logs/worker - - -DATE=`date +%s` -echo "$DATE: pre-creating log dirs for Falkon service..." -RACK_START=0 -RACK_END=48 -SEQUENCE_DIR=`seq -w ${RACK_START} ${RACK_END}` -PSET_START=1 -PSET_END=16 -for a in ${SEQUENCE_DIR} -do - for ((b=${PSET_START}; b <= ${PSET_END} ; b++)) # Double parentheses, and "LIMIT" with no "$". - do - DIR_NAME="ion-R${a}-${b}" - mkdir -p ${FALKON_JOBID_HOME}/logs/service/$DIR_NAME - done -done - -for ((b=${PSET_START}; b <= ${PSET_END} ; b++)) # Double parentheses, and "LIMIT" with no "$". -do - DIR_NAME="ion-${b}" - mkdir -p ${FALKON_JOBID_HOME}/logs/service/$DIR_NAME -done - - -DATE=`date +%s` -echo "$DATE: done creating log dirs for Falkon service!" - - - -FALKON_HOME_RAM=/tmp/${USER}/falkon - -ln -s ${FALKON_HOME}/apps ${FALKON_JOBID_HOME}/apps -ln -s ${FALKON_HOME_RAM}/container ${FALKON_JOBID_HOME}/container -ln -s ${FALKON_HOME}/service ${FALKON_JOBID_HOME}/service -ln -s ${FALKON_HOME}/worker ${FALKON_JOBID_HOME}/worker -ln -s ${FALKON_HOME}/AstroPortal ${FALKON_JOBID_HOME}/AstroPortal -ln -s ${FALKON_HOME}/client ${FALKON_JOBID_HOME}/client -ln -s ${FALKON_HOME}/monitor ${FALKON_JOBID_HOME}/monitor -ln -s ${FALKON_HOME}/bin ${FALKON_JOBID_HOME}/bin -ln -s ${FALKON_HOME}/config ${FALKON_JOBID_HOME}/config -ln -s ${FALKON_HOME}/ploticus ${FALKON_JOBID_HOME}/ploticus -ln -s ${FALKON_HOME}/webserver ${FALKON_JOBID_HOME}/webserver -ln -s ${FALKON_HOME}/workloads ${FALKON_JOBID_HOME}/workloads -ln -s ${FALKON_HOME}/id ${FALKON_JOBID_HOME}/id -ln -s ${FALKON_HOME}/apache-ant-1.7.0 ${FALKON_JOBID_HOME}/apache-ant-1.7.0 -ln -s ${FALKON_HOME}/ibm-java2-ppc64-50 ${FALKON_JOBID_HOME}/ibm-java2-ppc64-50 -ln -s ${FALKON_HOME_RAM}/ibm-java2-ppc-50 ${FALKON_JOBID_HOME}/ibm-java2-ppc-50 -ln -s ${FALKON_HOME}/falkon.tgz ${FALKON_JOBID_HOME}/falkon.tgz - - -if [ ! -d "${FALKON_JOBID_HOME}" ]; then - echo "ERROR: invalid path ${FALKON_JOBID_HOME}... exiting" - cqdel ${ZOID_JOB_ID} - exit 1 -fi - -echo "Succesfully submitted the job to Cobalt, and setup job specific Falkon tree!" -echo "To monitor the job status, type 'cqstat | grep ${USER}'; once it is in running state, you can use the Falkon specific command ...." -echo "To submit your Falkon-based workload, type: ....; you can do this any time, the falkon workload will wait for the resources to come online, and will only be submitted when everything is ready; the script is run in the background, so the workload will run even if the ssh session gets disconnected." -echo "" -echo "Remember, your job id is ${ZOID_JOB_ID}, and if you need to look through the logs manually for anything, remember that you can find them at ${HOME}/${ZOID_JOB_ID}.output, ${HOME}/${ZOID_JOB_ID}.error, and ${FALKON_JOBID_HOME}/logs/..." - - Deleted: usertools/cio/falkon/run.worker-c-bgp-iptorus.sh-mosa =================================================================== --- usertools/cio/falkon/run.worker-c-bgp-iptorus.sh-mosa 2009-03-28 15:55:52 UTC (rev 2749) +++ usertools/cio/falkon/run.worker-c-bgp-iptorus.sh-mosa 2009-03-28 15:56:07 UTC (rev 2750) @@ -1,152 +0,0 @@ -#!/fuse/bin/bash - -#ERROR_LOG=${FALKON_ROOT_PATH}/users/error.log -#ERROR_LOG=/home/falkon/users/error-workers.log -ERROR_LOG=/dev/null -#ERROR_LOG=/logfile - -if [ -z "$8" ]; then - echo "usage: $0 " >> ${ERROR_LOG} - echo "$0 192.168.1.254 55000 55001 4 iraicu 0001 [FALKON_ROOT] 1024 1 1" - echo "$0 192.168.1.254 55000 55001 4 iraicu 0001 [FALKON_R00T] 1024 1 1" >> ${ERROR_LOG} - exit 2 -fi - -# Initiate the IP stack of the Torus network -# -aespinosa - -/home/espinosa/bin/cn-ipfwd & - -while [ ! -f /tmp/ifconfig.cmd ]; do - sleep 1 -done - -. /tmp/ifconfig.cmd - -#source /home/iraicu/.bashrc - -IO_RANK=`/home/espinosa/bin/iorank` -BG_RANK=`/home/espinosa/bin/rank` -PSET=`/home/espinosa/bin/pset` - -# size of the allocated partition. also known as the total number of CNs -PART_SIZE=$8 -# number of data CNs in partition -DATA_NUM=$9 -STR=${10} - -ROLE=`/home/espinosa/bin/isdata $PART_SIZE $DATA_NUM $STR` -ISDATA=$ROLE -tmp=`/home/espinosa/bin/getdata $PART_SIZE $DATA_NUM $STR` -DATACN=`/fuse/gpfs/home/espinosa/bin/ipcal 10.128.0.0 $tmp` - -if [ "$ISDATA" != 'app' ] -then - if [ "$ROLE" = 'head' ] - then -# if [ $STR -eq 1 ] -# then -# echo "[ $BG_RANK@$IO_RANK-$PSET ] Starting Chirp server..." >&2 -# /home/espinosa/experiments/util/startchirp.sh >&2 -# else - echo "[ $BG_RANK@$IO_RANK-$PSET ] Starting Mosa Benefactor managed by $DATACN" >&2 - echo "[ $BG_RANK@$IO_RANK-$PSET ] Starting Mosa Manager across $STR CNs" >&2 - /home/espinosa/experiments/util/startchirp-wmosa.sh $DATACN $STR >&2 -# fi - else - echo "[ $BG_RANK@$IO_RANK-$PSET ] Starting Mosa Benefactor managed by $DATACN" >&2 - /home/espinosa/experiments/util/benefactor.sh $DATACN >&2 - fi -else - echo "[ $BG_RANK@$IO_RANK-$PSET ] Mounting Mosa data CN filesystem from $DATACN" >&2 - /bin/mkdir /dataifs - export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/lib:/fuse/lib:/usr/lib - sleep 30 - /home/espinosa/experiments/util/mosamount.sh $DATACN >&2 - - echo "worker started..." >> ${ERROR_LOG} - - FALKON_SERVICE_IP=$1 - TCPCORE_PORT1=$2 - TCPCORE_PORT2=$3 - NUM_WORKERS=$4 - USER_ID=$5 - FALKON_JOB_ID=$6 - FALKON_ROOT_PATH=$7 - - - ln -s /fuse/gpfs1 /gpfs1 - #FALKON_HOME=/gpfs1/falkon/falkon - #HOME=/gpfs1/falkon - - - - #echo "Worker for job id ${ZOID_JOB_ID} starting..." >> ${ERROR_LOG} - - FALKON_HOME=${FALKON_ROOT_PATH}/users/${USER_ID}/${FALKON_JOB_ID} - - echo "FALKON_HOME = ${FALKON_HOME}" >> ${ERROR_LOG} - - #FALKON_HOME=/home/falkon/falkon - - HOME=${FALKON_ROOT_PATH} - echo "HOME = ${HOME}" >> ${ERROR_LOG} - #HOME=${FALKON_HOME} - - if [ -z "${FALKON_HOME}" ]; then - echo "ERROR: environment variable FALKON_HOME not defined" - echo "ERROR: environment variable FALKON_HOME not defined" >> ${ERROR_LOG} - exit 1 - fi - - if [ ! -d "${FALKON_HOME}" ]; then - echo "ERROR: invalid FALKON_HOME set: $FALKON_HOME" - echo "ERROR: invalid FALKON_HOME set: $FALKON_HOME" >> ${ERROR_LOG} - exit 1 - fi - - cd ${FALKON_HOME} - source falkon.env.bgp-cn - - - - HOME_PATH=`pwd` - export MACH_ID=`uname -i` - RAND_ID2=`/home/espinosa/bin/rank` - - a0=${RAND_ID2%%=*}; rest=${RAND_ID2#*=} - RAND_ID=${rest%%=*}; - - #PLATFORM=`uname -m` - - - - EXP_START=`date +%Y.%m.%d_%H.%M.%S` - - - for ((b=1; b < ${NUM_WORKERS} ; b++)) - do - #LOG_FILE=${FALKON_LOGS}/worker/${RAND_ID}.${b}.${EXP_START}.txt - LOG_FILE=/dev/null - - #${FALKON_WORKER_HOME}/src-c/BGexec_${PLATFORM} ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf >> "${LOG_FILE}" 2>&1 & - #echo "starting .... ${FALKON_WORKER_HOME}/src-c/BGexec ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf" - echo "starting worker ${RAND_ID} CPU ${b}..." >> ${ERROR_LOG} - ${FALKON_WORKER_HOME}/src-c/BGexec ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf >> ${LOG_FILE} 2>&1 & - #${HOME_PATH}/src-c/BGexec_${PLATFORM} ${FALKON_SERVICE_IP} 55000 55001 -debug - /bin/sleep 1 - done - - #LOG_FILE=${FALKON_LOGS}/worker/${RAND_ID}.${NUM_WORKERS}.${EXP_START}.txt - LOG_FILE=/dev/null - echo "starting worker ${RAND_ID} CPU ${NUM_WORKERS}..." >> ${ERROR_LOG} - - #${FALKON_WORKER_HOME}/src-c/BGexec_${PLATFORM} ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf >> "${LOG_FILE}" 2>&1 - #echo "starting .... ${FALKON_WORKER_HOME}/src-c/BGexec ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf" - - ${FALKON_WORKER_HOME}/src-c/BGexec ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf >> ${LOG_FILE} 2>&1 - - - echo "finished!" >> ${ERROR_LOG} -fi From noreply at svn.ci.uchicago.edu Sat Mar 28 10:56:19 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sat, 28 Mar 2009 10:56:19 -0500 (CDT) Subject: [Swift-commit] r2751 - usertools/cio/falkon Message-ID: <20090328155619.883CA22817C@www.ci.uchicago.edu> Author: aespinosa Date: 2009-03-28 10:56:19 -0500 (Sat, 28 Mar 2009) New Revision: 2751 Added: usertools/cio/falkon/ciologic-bgp.sh Removed: usertools/cio/falkon/cio-logic.sh Log: Rename to separate BGP specific script Deleted: usertools/cio/falkon/cio-logic.sh =================================================================== --- usertools/cio/falkon/cio-logic.sh 2009-03-28 15:56:07 UTC (rev 2750) +++ usertools/cio/falkon/cio-logic.sh 2009-03-28 15:56:19 UTC (rev 2751) @@ -1,137 +0,0 @@ -#!/fuse/bin/bash - -CIOROOT=/home/espinosa/cio -ERROR_LOG=/dev/null - -if [ $# -lt 7 ]; then - cat << EOF -Usage: $0 \ - -Example: $0 192.168.1.254 55000 55001 4 iraicu 0001 [FALKON_ROOT] \ - 1024 1 1" -EOF - exit 2 -fi - -# Initiate the IP stack of the Torus network -# -aespinosa - -/home/iskra/ZeptoOS/packages/cnip/prebuilt/cn-ipfwd.2409 & - -while [ ! -f /tmp/ifconfig.cmd ]; do - sleep 1 -done - -. /tmp/ifconfig.cmd - -RANK=`/home/espinosa/bin/rank` - -# size of the allocated partition. also known as the total number of CNs -PART_SIZE=$8 -# number of data CNs in partition -DATA_NUM=$9 -STR=${10} - -ROLE=`/home/espinosa/bin/isdata $PART_SIZE $DATA_NUM $STR` -ISDATA=$ROLE -tmp=`/home/espinosa/bin/getdata $PART_SIZE $DATA_NUM $STR` -DATACN=`/fuse/gpfs/home/espinosa/bin/ipcal 10.128.0.0 $tmp` - -if [ $RANK -eq 0 ]; then - # Run Central Hash Service - $CIOROOT/hashserver.rb -elif [ "$ISDATA" != 'app' ]; then - if [ "$ROLE" = 'head' ]; then - echo "[ $BG_RANK@$IO_RANK-$PSET ] Starting Mosa Benefactor managed by $DATACN" >&2 - echo "[ $BG_RANK@$IO_RANK-$PSET ] Starting Mosa Manager across $STR CNs" >&2 - /home/espinosa/experiments/util/startchirp-wmosa.sh $DATACN $STR >&2 - else - echo "[ $BG_RANK@$IO_RANK-$PSET ] Starting Mosa Benefactor managed by $DATACN" >&2 - /home/espinosa/experiments/util/benefactor.sh $DATACN >&2 - fi -else - echo "[ $BG_RANK@$IO_RANK-$PSET ] Mounting Mosa data CN filesystem from $DATACN" >&2 - /bin/mkdir /dataifs - export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/lib:/fuse/lib:/usr/lib - sleep 30 - /home/espinosa/experiments/util/mosamount.sh $DATACN >&2 - - echo "worker started..." >> ${ERROR_LOG} - - # Regular Falon worker startup script - - FALKON_SERVICE_IP=$1 - TCPCORE_PORT1=$2 - TCPCORE_PORT2=$3 - NUM_WORKERS=$4 - USER_ID=$5 - FALKON_JOB_ID=$6 - FALKON_ROOT_PATH=$7 - - - FALKON_HOME=${FALKON_ROOT_PATH}/users/${USER_ID}/${FALKON_JOB_ID} - - echo "FALKON_HOME = ${FALKON_HOME}" >> ${ERROR_LOG} - - #FALKON_HOME=/home/falkon/falkon - - HOME=${FALKON_ROOT_PATH} - echo "HOME = ${HOME}" >> ${ERROR_LOG} - #HOME=${FALKON_HOME} - - if [ -z "${FALKON_HOME}" ]; then - echo "ERROR: environment variable FALKON_HOME not defined" - echo "ERROR: environment variable FALKON_HOME not defined" >> ${ERROR_LOG} - exit 1 - fi - - if [ ! -d "${FALKON_HOME}" ]; then - echo "ERROR: invalid FALKON_HOME set: $FALKON_HOME" - echo "ERROR: invalid FALKON_HOME set: $FALKON_HOME" >> ${ERROR_LOG} - exit 1 - fi - - cd ${FALKON_HOME} - source falkon.env.bgp-cn - - - - HOME_PATH=`pwd` - export MACH_ID=`uname -i` - RAND_ID2=`/home/espinosa/bin/rank` - - a0=${RAND_ID2%%=*}; rest=${RAND_ID2#*=} - RAND_ID=${rest%%=*}; - - #PLATFORM=`uname -m` - - - - EXP_START=`date +%Y.%m.%d_%H.%M.%S` - - - for ((b=1; b < ${NUM_WORKERS} ; b++)) - do - #LOG_FILE=${FALKON_LOGS}/worker/${RAND_ID}.${b}.${EXP_START}.txt - LOG_FILE=/dev/null - - #${FALKON_WORKER_HOME}/src-c/BGexec_${PLATFORM} ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf >> "${LOG_FILE}" 2>&1 & - #echo "starting .... ${FALKON_WORKER_HOME}/src-c/BGexec ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf" - echo "starting worker ${RAND_ID} CPU ${b}..." >> ${ERROR_LOG} - ${FALKON_WORKER_HOME}/src-c/BGexec ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf >> ${LOG_FILE} 2>&1 & - #${HOME_PATH}/src-c/BGexec_${PLATFORM} ${FALKON_SERVICE_IP} 55000 55001 -debug - /bin/sleep 1 - done - - #LOG_FILE=${FALKON_LOGS}/worker/${RAND_ID}.${NUM_WORKERS}.${EXP_START}.txt - LOG_FILE=/dev/null - echo "starting worker ${RAND_ID} CPU ${NUM_WORKERS}..." >> ${ERROR_LOG} - - #${FALKON_WORKER_HOME}/src-c/BGexec_${PLATFORM} ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf >> "${LOG_FILE}" 2>&1 - #echo "starting .... ${FALKON_WORKER_HOME}/src-c/BGexec ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf" - - ${FALKON_WORKER_HOME}/src-c/BGexec ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf >> ${LOG_FILE} 2>&1 - - - echo "finished!" >> ${ERROR_LOG} -fi Copied: usertools/cio/falkon/ciologic-bgp.sh (from rev 2750, usertools/cio/falkon/cio-logic.sh) =================================================================== --- usertools/cio/falkon/ciologic-bgp.sh (rev 0) +++ usertools/cio/falkon/ciologic-bgp.sh 2009-03-28 15:56:19 UTC (rev 2751) @@ -0,0 +1,137 @@ +#!/fuse/bin/bash + +CIOROOT=/home/espinosa/cio +ERROR_LOG=/dev/null + +if [ $# -lt 7 ]; then + cat << EOF +Usage: $0 \ + +Example: $0 192.168.1.254 55000 55001 4 iraicu 0001 [FALKON_ROOT] \ + 1024 1 1" +EOF + exit 2 +fi + +# Initiate the IP stack of the Torus network +# -aespinosa + +/home/iskra/ZeptoOS/packages/cnip/prebuilt/cn-ipfwd.2409 & + +while [ ! -f /tmp/ifconfig.cmd ]; do + sleep 1 +done + +. /tmp/ifconfig.cmd + +RANK=`/home/espinosa/bin/rank` + +# size of the allocated partition. also known as the total number of CNs +PART_SIZE=$8 +# number of data CNs in partition +DATA_NUM=$9 +STR=${10} + +ROLE=`/home/espinosa/bin/isdata $PART_SIZE $DATA_NUM $STR` +ISDATA=$ROLE +tmp=`/home/espinosa/bin/getdata $PART_SIZE $DATA_NUM $STR` +DATACN=`/fuse/gpfs/home/espinosa/bin/ipcal 10.128.0.0 $tmp` + +if [ $RANK -eq 0 ]; then + # Run Central Hash Service + $CIOROOT/hashserver.rb +elif [ "$ISDATA" != 'app' ]; then + if [ "$ROLE" = 'head' ]; then + echo "[ $BG_RANK@$IO_RANK-$PSET ] Starting Mosa Benefactor managed by $DATACN" >&2 + echo "[ $BG_RANK@$IO_RANK-$PSET ] Starting Mosa Manager across $STR CNs" >&2 + /home/espinosa/experiments/util/startchirp-wmosa.sh $DATACN $STR >&2 + else + echo "[ $BG_RANK@$IO_RANK-$PSET ] Starting Mosa Benefactor managed by $DATACN" >&2 + /home/espinosa/experiments/util/benefactor.sh $DATACN >&2 + fi +else + echo "[ $BG_RANK@$IO_RANK-$PSET ] Mounting Mosa data CN filesystem from $DATACN" >&2 + /bin/mkdir /dataifs + export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/lib:/fuse/lib:/usr/lib + sleep 30 + /home/espinosa/experiments/util/mosamount.sh $DATACN >&2 + + echo "worker started..." >> ${ERROR_LOG} + + # Regular Falon worker startup script + + FALKON_SERVICE_IP=$1 + TCPCORE_PORT1=$2 + TCPCORE_PORT2=$3 + NUM_WORKERS=$4 + USER_ID=$5 + FALKON_JOB_ID=$6 + FALKON_ROOT_PATH=$7 + + + FALKON_HOME=${FALKON_ROOT_PATH}/users/${USER_ID}/${FALKON_JOB_ID} + + echo "FALKON_HOME = ${FALKON_HOME}" >> ${ERROR_LOG} + + #FALKON_HOME=/home/falkon/falkon + + HOME=${FALKON_ROOT_PATH} + echo "HOME = ${HOME}" >> ${ERROR_LOG} + #HOME=${FALKON_HOME} + + if [ -z "${FALKON_HOME}" ]; then + echo "ERROR: environment variable FALKON_HOME not defined" + echo "ERROR: environment variable FALKON_HOME not defined" >> ${ERROR_LOG} + exit 1 + fi + + if [ ! -d "${FALKON_HOME}" ]; then + echo "ERROR: invalid FALKON_HOME set: $FALKON_HOME" + echo "ERROR: invalid FALKON_HOME set: $FALKON_HOME" >> ${ERROR_LOG} + exit 1 + fi + + cd ${FALKON_HOME} + source falkon.env.bgp-cn + + + + HOME_PATH=`pwd` + export MACH_ID=`uname -i` + RAND_ID2=`/home/espinosa/bin/rank` + + a0=${RAND_ID2%%=*}; rest=${RAND_ID2#*=} + RAND_ID=${rest%%=*}; + + #PLATFORM=`uname -m` + + + + EXP_START=`date +%Y.%m.%d_%H.%M.%S` + + + for ((b=1; b < ${NUM_WORKERS} ; b++)) + do + #LOG_FILE=${FALKON_LOGS}/worker/${RAND_ID}.${b}.${EXP_START}.txt + LOG_FILE=/dev/null + + #${FALKON_WORKER_HOME}/src-c/BGexec_${PLATFORM} ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf >> "${LOG_FILE}" 2>&1 & + #echo "starting .... ${FALKON_WORKER_HOME}/src-c/BGexec ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf" + echo "starting worker ${RAND_ID} CPU ${b}..." >> ${ERROR_LOG} + ${FALKON_WORKER_HOME}/src-c/BGexec ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf >> ${LOG_FILE} 2>&1 & + #${HOME_PATH}/src-c/BGexec_${PLATFORM} ${FALKON_SERVICE_IP} 55000 55001 -debug + /bin/sleep 1 + done + + #LOG_FILE=${FALKON_LOGS}/worker/${RAND_ID}.${NUM_WORKERS}.${EXP_START}.txt + LOG_FILE=/dev/null + echo "starting worker ${RAND_ID} CPU ${NUM_WORKERS}..." >> ${ERROR_LOG} + + #${FALKON_WORKER_HOME}/src-c/BGexec_${PLATFORM} ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf >> "${LOG_FILE}" 2>&1 + #echo "starting .... ${FALKON_WORKER_HOME}/src-c/BGexec ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf" + + ${FALKON_WORKER_HOME}/src-c/BGexec ${FALKON_SERVICE_IP} ${TCPCORE_PORT1} ${TCPCORE_PORT2} -perf >> ${LOG_FILE} 2>&1 + + + echo "finished!" >> ${ERROR_LOG} +fi From noreply at svn.ci.uchicago.edu Sat Mar 28 10:56:36 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sat, 28 Mar 2009 10:56:36 -0500 (CDT) Subject: [Swift-commit] r2752 - in usertools/cio: . falkon tools tools/src Message-ID: <20090328155636.9FE8D22817C@www.ci.uchicago.edu> Author: aespinosa Date: 2009-03-28 10:56:36 -0500 (Sat, 28 Mar 2009) New Revision: 2752 Added: usertools/cio/tools/ usertools/cio/tools/ifshead.sh usertools/cio/tools/ifsmount.sh usertools/cio/tools/ifsstripe.sh usertools/cio/tools/src/ usertools/cio/tools/src/getdata.c usertools/cio/tools/src/ipcal.c usertools/cio/tools/src/isdata.c usertools/cio/tools/src/rank.c Modified: usertools/cio/falkon/ciologic-bgp.sh Log: Imported support tools in the CIO logic Modified: usertools/cio/falkon/ciologic-bgp.sh =================================================================== --- usertools/cio/falkon/ciologic-bgp.sh 2009-03-28 15:56:19 UTC (rev 2751) +++ usertools/cio/falkon/ciologic-bgp.sh 2009-03-28 15:56:36 UTC (rev 2752) @@ -39,22 +39,22 @@ if [ $RANK -eq 0 ]; then # Run Central Hash Service - $CIOROOT/hashserver.rb + $CIOROOT/tools/hashserver.rb elif [ "$ISDATA" != 'app' ]; then if [ "$ROLE" = 'head' ]; then - echo "[ $BG_RANK@$IO_RANK-$PSET ] Starting Mosa Benefactor managed by $DATACN" >&2 - echo "[ $BG_RANK@$IO_RANK-$PSET ] Starting Mosa Manager across $STR CNs" >&2 - /home/espinosa/experiments/util/startchirp-wmosa.sh $DATACN $STR >&2 + # Run head IFS service + $CIOROOT/tools/ifshead.sh $DATACN $STR >&2 else - echo "[ $BG_RANK@$IO_RANK-$PSET ] Starting Mosa Benefactor managed by $DATACN" >&2 - /home/espinosa/experiments/util/benefactor.sh $DATACN >&2 + # Contribute stripes to the IFS service + $CIOROOT/tools/ifsslave.sh $DATACN >&2 fi else echo "[ $BG_RANK@$IO_RANK-$PSET ] Mounting Mosa data CN filesystem from $DATACN" >&2 /bin/mkdir /dataifs export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/lib:/fuse/lib:/usr/lib sleep 30 - /home/espinosa/experiments/util/mosamount.sh $DATACN >&2 + # Contact / mount assigned IFS head + $CIOROOT/tools/ifsmount.sh $DATACN >&2 echo "worker started..." >> ${ERROR_LOG} Added: usertools/cio/tools/ifshead.sh =================================================================== --- usertools/cio/tools/ifshead.sh (rev 0) +++ usertools/cio/tools/ifshead.sh 2009-03-28 15:56:36 UTC (rev 2752) @@ -0,0 +1,63 @@ +#!/bin/sh + +# Script : ifshead.sh +# Description : Starts the IFS directory service + + +DATACN=$1 +STR=$2 + +export LD_LIBRARY_PATH=/lib:/fuse/lib:/usr/lib + +if [ -d /dev/shm/indiv ]; +then + rm -rf /dev/shm/indiv +fi +mkdir /dev/shm/indiv +cat > /tmp/benefactor_config.cfg << EOF +# the hostname or the IP address of the manager +manager_name = $DATACN + +# manager port number +manager_port = 7005 + +# The path to the local directory where the benefactor will store the chunks +benefactor_path = /dev/shm/indiv + +# Aggregation type, this could be or +# is a typical setting for general workloads +# is under development +aggregation_type = DISK + +# The donated disk space size in MB +disk_space_size = 512 + +# The donated memory space size in MB, this is effective if aggregation type= MEMORY +memory_space_size = 512 + +# The manager update period in seconds +update_period = 5 + +# (Optional) The local address the benefactor should use - specially in multihomed machines +#benefactor_address = +EOF + +cat > /tmp/manager_config.cfg << EOF +# the recommended stripe width +# this is the number of benefactors the client will strip the data among in the write operation +stripe_width = $STR + +# the size fo the chunk - not fully implemented yet +# chunk_size = 1048576 + +# the maximum possible number of benefactors in the system +max_num_ben = 4096 +EOF +RANK=`/home/espinosa/bin/rank` +cd /tmp +#/home/espinosa/bin/manager 7005 > /home/espinosa/log/$RANK.mgr 2> /home/espinosa/log/$RANK-err.mgr & +/home/espinosa/bin/manager 7005 2>&1 > /dev/null & + +sleep 30 +#/home/espinosa/bin/benefactor -d >&2 2> /home/espinosa/log/$RANK.ben +/home/espinosa/bin/benefactor 2>&1 > /dev/null Property changes on: usertools/cio/tools/ifshead.sh ___________________________________________________________________ Name: svn:executable + * Added: usertools/cio/tools/ifsmount.sh =================================================================== --- usertools/cio/tools/ifsmount.sh (rev 0) +++ usertools/cio/tools/ifsmount.sh 2009-03-28 15:56:36 UTC (rev 2752) @@ -0,0 +1,87 @@ +#!/bin/sh + + +# Script : ifsmount.sh +# Description : Mounts the IFS associated with this Worker CN + +DATACN=$1 +RANK=`/home/espinosa/bin/rank` + +export LD_LIBRARY_PATH=/lib:/fuse/lib:/usr/lib +cd /tmp + +cat > /tmp/flfs_config.cfg << EOF +# the hostname or the IP address of the manager +manager_name = $DATACN + + +# manager port number +manager_port = 7005 + + +# execution mode - not fully implemented yet - DEBUG is the only option for now +execution_mode = DEBUG + + +# Naming scheme - to name chunk by sequence number or by hash +# the chunks will be named by sequence number, this is a typical setting for general workloads +# this option is for content addressablity feature +chunk_naming = SEQNUM + + +# Commit scheme - specifies whether to overwrite, non-overwrite or version the previous copy. +# possible values include , , and +# : writing a new file with the same name as an existing file will fail. +# : if a new file is stored in the system with the same file name as an existing file name, +# the new file will overwrite the old file +# : if a new file is stored in the system with the same file name as an existing file name, +# the new file will be store as a new version of the file +commit_scheme = NOOVERWRITE + + +# Number of chunks to reserve in the repository +num_reserve_chunks = 1024 + +# Write Interface type - to select the write interface type, the following are the write interfaces +# : Sliding window write interface, this is the typical setting. +# : Incremental write interface +# : The Complete local write interface +# and use the local disk in the write operation. these two are not extensively tested. +write_interface_type = SLIDINGWINDOWWRITE + + +#the memory space allocated for the buffers in the write operations, in MB, +# effects Sliding window interface only +memory_size = 256 + + +# if is the selected write interface spacifies +# the size of the temporary local files in number of chunks +inc_write_file_size = 64 + + +# if of is the selected write interface spacifies +# the path to the directory where the temporary files will be saved +local_write_directory = /tmp/FLIncW + +# Read Interface type - to select the read interface type, currently is only implemented +# : Fixed Buffer per Request read interface +read_interface_type = FBR + + +# if is the selected read interface specifies +# the number of chunks allocated for every request +fbr_request_buffer_size = 4 + +# Number of threads per write agent ( there is an agent per benefactor ) +num_threads_per_agent = 1 +# Cache update period in seconds +cache_update_period = 5 +dbg_file_name = /dev/null + + +EOF +#/home/espinosa/bin/mosastore -o direct_io -o sync_read /dataifs +#/home/espinosa/bin/mosastore -o sync_read /dataifs -d >&2 2> /home/espinosa/log/$RANK.mos & +/home/espinosa/bin/mosastore -o sync_read /dataifs & + Property changes on: usertools/cio/tools/ifsmount.sh ___________________________________________________________________ Name: svn:executable + * Added: usertools/cio/tools/ifsstripe.sh =================================================================== --- usertools/cio/tools/ifsstripe.sh (rev 0) +++ usertools/cio/tools/ifsstripe.sh 2009-03-28 15:56:36 UTC (rev 2752) @@ -0,0 +1,48 @@ +#!/fuse/bin/bash + +# Script : ifsslave.sh +# Description : Allocates space in the compute node to contribute +# space to the head IFS. + +cd /tmp + +DATACN=$1 + +export LD_LIBRARY_PATH=/lib:/fuse/lib:/usr/lib + +if [ -d /dev/shm/indiv ]; then + rm -rf /dev/shm/indiv +fi +mkdir /dev/shm/indiv +cat > /tmp/benefactor_config.cfg << EOF +# the hostname or the IP address of the manager +manager_name = $DATACN + +# manager port number +manager_port = 7005 + +# The path to the local directory where the benefactor will store the chunks +benefactor_path = /dev/shm/indiv + +# Aggregation type, this could be or +# is a typical setting for general workloads +# is under development +aggregation_type = DISK + +# The donated disk space size in MB +disk_space_size = 512 + +# The donated memory space size in MB, this is effective if aggregation type= MEMORY +memory_space_size = 512 + +# The manager update period in seconds +update_period = 5 + +# (Optional) The local address the benefactor should use - specially in multihomed machines +#benefactor_address = +EOF +cd /tmp +sleep 30 +#RANK=`/home/espinosa/bin/rank` +#/home/espinosa/bin/benefactor -d >&2 2> /home/espinosa/log/$RANK.ben +/home/espinosa/bin/benefactor 2>&1 > /dev/null Property changes on: usertools/cio/tools/ifsstripe.sh ___________________________________________________________________ Name: svn:executable + * Added: usertools/cio/tools/src/getdata.c =================================================================== --- usertools/cio/tools/src/getdata.c (rev 0) +++ usertools/cio/tools/src/getdata.c 2009-03-28 15:56:36 UTC (rev 2752) @@ -0,0 +1,72 @@ +/* + getdata.c + Fetch Rank of assigned IFS service with this compute node + TODO: Make this a more general script + + gcc -mpowerpc -Wall -I/bgsys/drivers/ppcfloor/arch/include/ -o isdata isdata.c + + aespinosa at cs.uchicago.edu +*/ + +#include +#include +#include +#include +#include +#include +#include + +#include + +static _BGP_Personality_t pers; + +int main(int argc, char* argv[]) +{ + int fd; + int rc; + + fd = open("/proc/personality", O_RDONLY); + if( fd < 0 ) + { + fprintf(stderr,"Unable to open /proc/personality"); + exit(1); + } + rc = read(fd, (char*)&pers, sizeof(_BGP_Personality_t)); + if( rc!=sizeof(_BGP_Personality_t) ) + { + fprintf(stderr,"Failed to read /proc/personality"); + exit(1); + } + close(fd); +/* + * usage: + * isdata [PART_SIZE] [DNUM] [STR] + * [ PART_SIZE ] - total number of nodes/ size of job partition + * [ DNUM ] - number of data CNs in the partition + * [ STR ] - number of data CNs to stripe together + * + */ + + int psize, dnum, str; + sscanf( argv[1], "%d", &psize ); + sscanf( argv[2], "%d", &dnum ); + sscanf( argv[3], "%d", &str ); + int *target_rank = (int*) malloc( dnum * sizeof(int) ); + + int i; + for ( i = 0; i < dnum; i++ ) + { + target_rank[i] = psize * ( 2*i + 1 ) / ( 2*dnum ); + } + + for ( i = 0; i < dnum; i++ ) + { + int rank = pers.Network_Config.Rank; + if ( psize * i / dnum <= rank && rank < psize * (i+1) / dnum ) + { + printf("%d", target_rank[i]); + break; + } + } + return 0; +} Added: usertools/cio/tools/src/ipcal.c =================================================================== --- usertools/cio/tools/src/ipcal.c (rev 0) +++ usertools/cio/tools/src/ipcal.c 2009-03-28 15:56:36 UTC (rev 2752) @@ -0,0 +1,18 @@ +#include +#include + +int main(int argc, char* argv[]) +{ + unsigned long network; + unsigned long rank; + char dot; + unsigned long a, b, c, d; + network = 0; + sscanf(argv[1], "%d.%d.%d.%d", &a, &b, &c, &d); + network = network | (a << 24) | (b << 16) | (c << 8) | d; + sscanf( argv[2], "%d", &rank ); + unsigned long ip = network | rank; + printf("%ld.%ld.%ld.%ld\n", (ip & 0xff000000)>>24, (ip & 0x00ff0000)>>16, (ip & 0x0000ff00)>>8, ip & 0x000000ff); + return 0; +} + Added: usertools/cio/tools/src/isdata.c =================================================================== --- usertools/cio/tools/src/isdata.c (rev 0) +++ usertools/cio/tools/src/isdata.c 2009-03-28 15:56:36 UTC (rev 2752) @@ -0,0 +1,80 @@ +/* + isdata.c + Utility to get the role of the CN in the CIO ecosystem + + gcc -mpowerpc -Wall -I/bgsys/drivers/ppcfloor/arch/include/ -o isdata isdata.c + + aespinosa at cs.uchicago.edu +*/ + +#include +#include +#include +#include +#include +#include +#include + +#include + +static _BGP_Personality_t pers; + +int main(int argc, char* argv[]) +{ + int fd; + int rc; + + fd = open("/proc/personality", O_RDONLY); + if( fd < 0 ) + { + fprintf(stderr,"Unable to open /proc/personality"); + exit(1); + } + rc = read(fd, (char*)&pers, sizeof(_BGP_Personality_t)); + if( rc!=sizeof(_BGP_Personality_t) ) + { + fprintf(stderr,"Failed to read /proc/personality"); + exit(1); + } + close(fd); +/* + * usage: + * isdata [PART_SIZE] [DNUM] [STR] + * [ PART_SIZE ] - total number of nodes/ size of job partition + * [ DNUM ] - number of data CNs in the partition + * [ STR ] - number of data CNs to stripe together + * + */ + + int psize, dnum, str; + sscanf( argv[1], "%d", &psize ); + sscanf( argv[2], "%d", &dnum ); + sscanf( argv[3], "%d", &str ); + int *target_rank = (int*) malloc( dnum * sizeof(int) ); + + int i; + int flag = 0; + + for ( i = 0; i < dnum; i++ ) + { + target_rank[i] = psize * ( 2*i + 1 ) / ( 2*dnum ); + if ( pers.Network_Config.Rank == target_rank[i] ) + { + printf("head"); + flag = 1; + break; + } + else if ( target_rank[i] < pers.Network_Config.Rank + && pers.Network_Config.Rank < target_rank[i] + str ) + { + printf("slave"); + flag = 1; + break; + } + } + if( !flag ) + { + printf("app"); + } + return 0; +} Added: usertools/cio/tools/src/rank.c =================================================================== --- usertools/cio/tools/src/rank.c (rev 0) +++ usertools/cio/tools/src/rank.c 2009-03-28 15:56:36 UTC (rev 2752) @@ -0,0 +1,43 @@ +/* + BGP personality sample code + + + gcc -mpowerpc -Wall -I/bgsys/drivers/ppcfloor/arch/include/ -o pers pers.c + + kazutomo at mcs.anl.gov +*/ + +#include +#include +#include +#include +#include +#include +#include + +#include + +static _BGP_Personality_t pers; + +int main() +{ + int fd; + int rc; + + fd = open("/proc/personality", O_RDONLY); + if( fd < 0 ) { + fprintf(stderr,"Unable to open /proc/personality"); + exit(1); + } + rc = read(fd, (char*)&pers, sizeof(_BGP_Personality_t)); + if( rc!=sizeof(_BGP_Personality_t) ) { + fprintf(stderr,"Failed to read /proc/personality"); + exit(1); + } + close(fd); + + printf("%d\n",pers.Network_Config.Rank); + + return 0; +} + From noreply at svn.ci.uchicago.edu Sat Mar 28 10:59:11 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sat, 28 Mar 2009 10:59:11 -0500 (CDT) Subject: [Swift-commit] r2753 - SwiftApps/LigandAtlas Message-ID: <20090328155911.4803422817C@www.ci.uchicago.edu> Author: aespinosa Date: 2009-03-28 10:59:10 -0500 (Sat, 28 Mar 2009) New Revision: 2753 Added: SwiftApps/LigandAtlas/dock6.swift Log: Old dock workflow from Zhao Added: SwiftApps/LigandAtlas/dock6.swift =================================================================== --- SwiftApps/LigandAtlas/dock6.swift (rev 0) +++ SwiftApps/LigandAtlas/dock6.swift 2009-03-28 15:59:10 UTC (rev 2753) @@ -0,0 +1,94 @@ +type Mol {} +type Bin {} +type Common {} +type DOCKOut {} +type Molout{} +type DockRunSummary {} +type Bcastout {} + +/*app (DOCKOut ofile, Molout moloutfile) rundock_zhao (Mol molfile, Bin binary, Common flex_def, Common +flex_tbl, Common grid_bmp, Common grid_in, Common grid_nrg, Common grid_out, +Common rec_box, Common receptor, Common sample_grid, Common selected_spheres, +Common template_in, Common vdw, Common awkscript)*/ + +app (DOCKOut ofile) rundock_zhao (Mol molfile, Bin binary, Common flex_def, Common +flex_tbl, Common grid_bmp, Common grid_in, Common grid_nrg, Common grid_out, +Common rec_box, Common receptor, Common sample_grid, Common selected_spheres, +Common template_in, Common vdw, Common awkscript) +{ + rundock_zhao @filename(molfile) @filename(binary) @filename(flex_def) + @filename(flex_tbl) @filename(grid_bmp) @filename(grid_in) + @filename(grid_nrg) @filename(grid_out) @filename(rec_box) + @filename(receptor) @filename(sample_grid) @filename(selected_spheres) + @filename(template_in) @filename(vdw) @filename(awkscript) stdout=@filename(ofile); +} + +app (DockRunSummary finalfile) sumdockresults(DOCKOut r[] ) +{ + summary @filenames(r) stdout=@filename(finalfile); +} + + +Mol texts[] ; +//Mol texts[] ; + + +(DOCKOut result[])doall(Mol texts[]) +{ + Bin binary <"common/dock6.O3.cn">; + Common flex_def <"common/flex.defn">; + Common flex_tbl <"common/flex_drive.tbl">; + Common grid_bmp <"common/grid.bmp">; + Common grid_in <"common/grid.in">; + Common grid_nrg <"common/grid.nrg">; + Common grid_out <"common/grid.out">; + Common rec_box <"common/rec_box.pdb">; + Common receptor <"common/receptor_charged.mol2">; + Common sample_grid <"common/sample_grid.in">; + Common selected_spheres <"common/selected_spheres.sph">; + Common template_in <"common/template.in">; + Common vdw <"common/vdw_AMBER_parm99.defn">; + Common awkscript <"common/awkscript">; + + foreach p,i in texts { + /*DOCKOut r ;*/ + + /*Molout mof ;*/ + + result[i] = rundock_zhao(p, binary, flex_def, flex_tbl, grid_bmp, + grid_in, grid_nrg, grid_out, rec_box, receptor, sample_grid, + selected_spheres, template_in, vdw, awkscript); + } +} + +// Main + +DockRunSummary summary <"summary.txt">; +DOCKOut result[]; +//Bcastout bout <"bcast.sig">; +Bin binary <"common/dock6.O3.cn">; + Common flex_def <"common/flex.defn">; + Common flex_tbl <"common/flex_drive.tbl">; + Common grid_bmp <"common/grid.bmp">; + Common grid_in <"common/grid.in">; + Common grid_nrg <"common/grid.nrg">; + Common grid_out <"common/grid.out">; + Common rec_box <"common/rec_box.pdb">; + Common receptor <"common/receptor_charged.mol2">; + Common sample_grid <"common/sample_grid.in">; + Common selected_spheres <"common/selected_spheres.sph">; + Common template_in <"common/template.in">; + Common vdw <"common/vdw_AMBER_parm99.defn">; + Common awkscript <"common/awkscript">; + Bcastout bout; + +result=doall(texts); +summary = sumdockresults(result); From noreply at svn.ci.uchicago.edu Sat Mar 28 10:59:19 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sat, 28 Mar 2009 10:59:19 -0500 (CDT) Subject: [Swift-commit] r2754 - SwiftApps/LigandAtlas Message-ID: <20090328155919.62F2322817C@www.ci.uchicago.edu> Author: aespinosa Date: 2009-03-28 10:59:19 -0500 (Sat, 28 Mar 2009) New Revision: 2754 Modified: SwiftApps/LigandAtlas/dock6.swift Log: Collective operations introduced in the workflow Modified: SwiftApps/LigandAtlas/dock6.swift =================================================================== --- SwiftApps/LigandAtlas/dock6.swift 2009-03-28 15:59:10 UTC (rev 2753) +++ SwiftApps/LigandAtlas/dock6.swift 2009-03-28 15:59:19 UTC (rev 2754) @@ -4,8 +4,23 @@ type DOCKOut {} type Molout{} type DockRunSummary {} -type Bcastout {} +type file {} +(external bout)bcast (Bin binary, Common flex_def, Common flex_tbl, Common grid_bmp, Common +grid_in, Common grid_nrg, Common grid_out, Common rec_box, Common receptor, +Common sample_grid, Common selected_spheres, Common template_in, Common vdw, +Common awkscript) +{ + app { + bcast @filename(binary) @filename(flex_def) + @filename(flex_tbl) @filename(grid_bmp) @filename(grid_in) + @filename(grid_nrg) @filename(grid_out) @filename(rec_box) + @filename(receptor) @filename(sample_grid) + @filename(selected_spheres) + @filename(template_in) @filename(vdw) @filename(awkscript); + } +} + /*app (DOCKOut ofile, Molout moloutfile) rundock_zhao (Mol molfile, Bin binary, Common flex_def, Common flex_tbl, Common grid_bmp, Common grid_in, Common grid_nrg, Common grid_out, Common rec_box, Common receptor, Common sample_grid, Common selected_spheres, @@ -14,7 +29,7 @@ app (DOCKOut ofile) rundock_zhao (Mol molfile, Bin binary, Common flex_def, Common flex_tbl, Common grid_bmp, Common grid_in, Common grid_nrg, Common grid_out, Common rec_box, Common receptor, Common sample_grid, Common selected_spheres, -Common template_in, Common vdw, Common awkscript) +Common template_in, Common vdw, Common awkscript, external bout) { rundock_zhao @filename(molfile) @filename(binary) @filename(flex_def) @filename(flex_tbl) @filename(grid_bmp) @filename(grid_in) @@ -50,6 +65,10 @@ Common vdw <"common/vdw_AMBER_parm99.defn">; Common awkscript <"common/awkscript">; + external bout; + + bout=bcast(binary, flex_def, flex_tbl, grid_bmp, grid_in, grid_nrg, grid_out, rec_box, receptor, sample_grid, selected_spheres, template_in, vdw, awkscript); + foreach p,i in texts { /*DOCKOut r ;*/ - result[i] = rundock_zhao(p, binary, flex_def, flex_tbl, grid_bmp, - grid_in, grid_nrg, grid_out, rec_box, receptor, sample_grid, - selected_spheres, template_in, vdw, awkscript); + result[i] = rundock_zhao(p, binary, flex_def, flex_tbl, grid_bmp, grid_in, + grid_nrg, grid_out, rec_box, receptor, sample_grid, selected_spheres, template_in, vdw, awkscript, bout); } + + } // Main DockRunSummary summary <"summary.txt">; DOCKOut result[]; -//Bcastout bout <"bcast.sig">; -Bin binary <"common/dock6.O3.cn">; - Common flex_def <"common/flex.defn">; - Common flex_tbl <"common/flex_drive.tbl">; - Common grid_bmp <"common/grid.bmp">; - Common grid_in <"common/grid.in">; - Common grid_nrg <"common/grid.nrg">; - Common grid_out <"common/grid.out">; - Common rec_box <"common/rec_box.pdb">; - Common receptor <"common/receptor_charged.mol2">; - Common sample_grid <"common/sample_grid.in">; - Common selected_spheres <"common/selected_spheres.sph">; - Common template_in <"common/template.in">; - Common vdw <"common/vdw_AMBER_parm99.defn">; - Common awkscript <"common/awkscript">; - Bcastout bout; - result=doall(texts); summary = sumdockresults(result); From noreply at svn.ci.uchicago.edu Sun Mar 29 14:23:35 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 29 Mar 2009 14:23:35 -0500 (CDT) Subject: [Swift-commit] r2755 - trunk/resources Message-ID: <20090329192335.DE997228217@www.ci.uchicago.edu> Author: hategan Date: 2009-03-29 14:23:34 -0500 (Sun, 29 Mar 2009) New Revision: 2755 Modified: trunk/resources/swiftscript.stg trunk/resources/swiftscript.xsd Log: source location for procedures Modified: trunk/resources/swiftscript.stg =================================================================== --- trunk/resources/swiftscript.stg 2009-03-28 15:59:19 UTC (rev 2754) +++ trunk/resources/swiftscript.stg 2009-03-29 19:23:34 UTC (rev 2755) @@ -105,7 +105,7 @@ >> function(name,outputs,inputs,statements,config,sourcelocation) ::= << - + $outputs;separator="\n"$ $inputs;separator="\n"$ $statements;separator="\n"$ Modified: trunk/resources/swiftscript.xsd =================================================================== --- trunk/resources/swiftscript.xsd 2009-03-28 15:59:19 UTC (rev 2754) +++ trunk/resources/swiftscript.xsd 2009-03-29 19:23:34 UTC (rev 2755) @@ -111,6 +111,8 @@ the name of the procedure + + From noreply at svn.ci.uchicago.edu Sun Mar 29 14:24:53 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 29 Mar 2009 14:24:53 -0500 (CDT) Subject: [Swift-commit] r2756 - trunk/resources Message-ID: <20090329192453.735C4228217@www.ci.uchicago.edu> Author: hategan Date: 2009-03-29 14:24:51 -0500 (Sun, 29 Mar 2009) New Revision: 2756 Modified: trunk/resources/Karajan.stg Log: log line numbers and start/end for foreach and procs Modified: trunk/resources/Karajan.stg =================================================================== --- trunk/resources/Karajan.stg 2009-03-29 19:23:34 UTC (rev 2755) +++ trunk/resources/Karajan.stg 2009-03-29 19:24:51 UTC (rev 2756) @@ -53,13 +53,13 @@ >> -procedure(name,outputs,inputs,arguments,optargs,binding,declarations,statements,config) ::= << +procedure(name,outputs,inputs,arguments,optargs,binding,declarations,statements,config,line) ::= << $optargs:default_arg();separator="\n"$ $outputs:typecheck();separator="\n"$ $inputs:typecheck();separator="\n"$ $if(binding)$ - $vdl_execute(outputs=outputs,inputs=inputs,application=binding.application, name=name)$ + $vdl_execute(outputs=outputs,inputs=inputs,application=binding.application, name=name,line=line)$ $else$ $compound(outputs=outputs,inputs=inputs,declarations=declarations,statements=statements,name=name)$ $endif$ @@ -97,11 +97,8 @@ >> -vdl_execute(outputs,inputs,application,name) ::= << - - info - PROCEDURE thread={#thread} name=$name$ - +vdl_execute(outputs,inputs,application,name,line) ::= << + $inputs:vdl_log_input();separator="\n"$ $outputs:vdl_log_output();separator="\n"$ @@ -111,6 +108,7 @@ $vdl_arguments(arguments=application.arguments, stdin=application.stdin,stdout=application.stdout,stderr=application.stderr)$ $outputs:vdl_closedataset();separator="\n"$ + >> vdl_log_input() ::= << @@ -199,7 +197,7 @@ >> -foreach(var,in,indexVar,declarations,statements) ::= << +foreach(var,in,indexVar,declarations,statements,line) ::= << $in$ @@ -210,10 +208,12 @@ $endif$ + $declarations;separator="\n"$ $if(statements)$ $parallel(statements=statements)$ $endif$ + >> From noreply at svn.ci.uchicago.edu Sun Mar 29 14:26:13 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 29 Mar 2009 14:26:13 -0500 (CDT) Subject: [Swift-commit] r2757 - trunk/src/org/griphyn/vdl/engine Message-ID: <20090329192613.33A0A228217@www.ci.uchicago.edu> Author: hategan Date: 2009-03-29 14:26:12 -0500 (Sun, 29 Mar 2009) New Revision: 2757 Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java Log: pass line numbers to the vdlx Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/Karajan.java 2009-03-29 19:24:51 UTC (rev 2756) +++ trunk/src/org/griphyn/vdl/engine/Karajan.java 2009-03-29 19:26:12 UTC (rev 2757) @@ -199,6 +199,7 @@ VariableScope innerScope = new VariableScope(this, outerScope, VariableScope.ENCLOSURE_NONE); StringTemplate procST = template("procedure"); containingScope.bodyTemplate.setAttribute("procedures", procST); + procST.setAttribute("line", proc.getSrc().substring(proc.getSrc().indexOf(' ') + 1)); procST.setAttribute("name", proc.getName()); for (int i = 0; i < proc.sizeOfOutputArray(); i++) { FormalParameter param = proc.getOutputArray(i); @@ -603,6 +604,7 @@ StringTemplate foreachST = template("foreach"); foreachST.setAttribute("var", foreach.getVar()); + foreachST.setAttribute("line", foreach.getSrc().substring(foreach.getSrc().indexOf(' ') + 1)); XmlObject in = foreach.getIn().getAbstractExpression(); StringTemplate inST = expressionToKarajan(in, scope); From noreply at svn.ci.uchicago.edu Sun Mar 29 14:30:44 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 29 Mar 2009 14:30:44 -0500 (CDT) Subject: [Swift-commit] r2759 - trunk/src/org/griphyn/vdl/karajan Message-ID: <20090329193044.217F522817C@www.ci.uchicago.edu> Author: hategan Date: 2009-03-29 14:30:43 -0500 (Sun, 29 Mar 2009) New Revision: 2759 Modified: trunk/src/org/griphyn/vdl/karajan/InHook.java Log: disable this for now Modified: trunk/src/org/griphyn/vdl/karajan/InHook.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/InHook.java 2009-03-29 19:30:00 UTC (rev 2758) +++ trunk/src/org/griphyn/vdl/karajan/InHook.java 2009-03-29 19:30:43 UTC (rev 2759) @@ -14,9 +14,9 @@ public static final Logger logger = Logger.getLogger(InHook.class); public synchronized static void install(Monitor m) { - if (!(System.in instanceof InHook)) { + /*if (!(System.in instanceof InHook)) { System.setIn(new InHook(System.in, m)); - } + }*/ } private BufferedInputStream is; From noreply at svn.ci.uchicago.edu Sun Mar 29 14:31:30 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 29 Mar 2009 14:31:30 -0500 (CDT) Subject: [Swift-commit] r2760 - trunk/src/org/griphyn/vdl/karajan Message-ID: <20090329193130.249A422817C@www.ci.uchicago.edu> Author: hategan Date: 2009-03-29 14:31:29 -0500 (Sun, 29 Mar 2009) New Revision: 2760 Modified: trunk/src/org/griphyn/vdl/karajan/VDL2ExecutionContext.java Log: log stack to help getting progress ticker information Modified: trunk/src/org/griphyn/vdl/karajan/VDL2ExecutionContext.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/VDL2ExecutionContext.java 2009-03-29 19:30:43 UTC (rev 2759) +++ trunk/src/org/griphyn/vdl/karajan/VDL2ExecutionContext.java 2009-03-29 19:31:29 UTC (rev 2760) @@ -61,4 +61,11 @@ public void setRunID(String runID) { this.runID = runID; } + + public void start(VariableStack stack) { + if (logger.isInfoEnabled()) { + logger.info(stack); + } + super.start(stack); + } } From noreply at svn.ci.uchicago.edu Sun Mar 29 14:32:13 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 29 Mar 2009 14:32:13 -0500 (CDT) Subject: [Swift-commit] r2761 - trunk/src/org/griphyn/vdl/karajan/lib Message-ID: <20090329193213.E898122817C@www.ci.uchicago.edu> Author: hategan Date: 2009-03-29 14:32:13 -0500 (Sun, 29 Mar 2009) New Revision: 2761 Modified: trunk/src/org/griphyn/vdl/karajan/lib/Log.java Log: added message argument to Log - simpler xml Modified: trunk/src/org/griphyn/vdl/karajan/lib/Log.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/Log.java 2009-03-29 19:31:29 UTC (rev 2760) +++ trunk/src/org/griphyn/vdl/karajan/lib/Log.java 2009-03-29 19:32:13 UTC (rev 2761) @@ -19,9 +19,10 @@ public class Log extends AbstractSequentialWithArguments { public static final Arg LEVEL = new Arg.Positional("level"); + public static final Arg MESSAGE = new Arg.Optional("message", null); static { - setArguments(Log.class, new Arg[] { LEVEL, Arg.VARGS }); + setArguments(Log.class, new Arg[] { LEVEL, MESSAGE, Arg.VARGS }); } private static Map loggers = new HashMap(); @@ -71,12 +72,18 @@ Level lvl = getLevel(TypeUtil.toString(LEVEL.getValue(stack))); Logger logger = getLogger(cls); if (logger.isEnabledFor(lvl)) { - Object[] msg = Arg.VARGS.asArray(stack); - StringBuffer sb = new StringBuffer(); - for (int i = 0; i < msg.length; i++) { - sb.append(TypeUtil.toString(msg[i])); - } - logger.log(lvl, sb.toString()); + Object smsg = MESSAGE.getValue(stack); + if (smsg != null) { + logger.log(lvl, smsg); + } + else { + Object[] msg = Arg.VARGS.asArray(stack); + StringBuffer sb = new StringBuffer(); + for (int i = 0; i < msg.length; i++) { + sb.append(TypeUtil.toString(msg[i])); + } + logger.log(lvl, sb.toString()); + } } super.post(stack); } From noreply at svn.ci.uchicago.edu Sun Mar 29 14:32:56 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 29 Mar 2009 14:32:56 -0500 (CDT) Subject: [Swift-commit] r2762 - trunk/src/org/griphyn/vdl/karajan/lib Message-ID: <20090329193256.3337722817C@www.ci.uchicago.edu> Author: hategan Date: 2009-03-29 14:32:55 -0500 (Sun, 29 Mar 2009) New Revision: 2762 Modified: trunk/src/org/griphyn/vdl/karajan/lib/RuntimeStats.java Log: cleaned up runtime stats Modified: trunk/src/org/griphyn/vdl/karajan/lib/RuntimeStats.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/RuntimeStats.java 2009-03-29 19:32:13 UTC (rev 2761) +++ trunk/src/org/griphyn/vdl/karajan/lib/RuntimeStats.java 2009-03-29 19:32:55 UTC (rev 2762) @@ -8,6 +8,7 @@ import java.util.Map; import org.globus.cog.karajan.arguments.Arg; +import org.globus.cog.karajan.stack.VariableNotFoundException; import org.globus.cog.karajan.stack.VariableStack; import org.globus.cog.karajan.util.TypeUtil; import org.globus.cog.karajan.workflow.ExecutionException; @@ -18,6 +19,9 @@ proof of concept. */ public class RuntimeStats extends FunctionsCollection { + + public static final String TICKER = "#swift-runtime-progress-ticker"; + public static final String PROGRESS = "#swift-runtime-progress"; public static final Arg PA_STATE = new Arg.Positional("state"); public static final int MIN_PERIOD_MS=1000; @@ -43,12 +47,28 @@ setArguments("vdl_setprogress", new Arg[] { PA_STATE } ); setArguments("vdl_initprogressstate", new Arg[] { PA_STATE }); } + + public static void setTicker(VariableStack stack, ProgressTicker ticker) { + stack.setGlobal(TICKER, ticker); + } + + public static ProgressTicker getTicker(VariableStack stack) { + return (ProgressTicker) stack.getGlobal(TICKER); + } + + public static void setProgress(VariableStack stack, RuntimeProgress p) { + stack.parentFrame().setVar(PROGRESS, p); + } + + public static RuntimeProgress getProgress(VariableStack stack) throws VariableNotFoundException { + return (RuntimeProgress) stack.getDeepVar(PROGRESS); + } public Object vdl_startprogressticker(VariableStack stack) throws ExecutionException { ProgressTicker t = new ProgressTicker(); t.setDaemon(true); t.start(); - stack.parentFrame().setVar("#swift-runtime-progress-ticker",t); + setTicker(stack, t); return null; } @@ -59,27 +79,25 @@ } static public void setProgress(VariableStack stack, String newState) throws ExecutionException { - RuntimeProgress rp = (RuntimeProgress)stack.getVar("#swift-runtime-progress"); - rp.status = newState; - ProgressTicker p = (ProgressTicker)stack.getVar("#swift-runtime-progress-ticker"); - p.dumpState(); + getProgress(stack).status = newState; + getTicker(stack).dumpState(); } public Object vdl_initprogressstate(VariableStack stack) throws ExecutionException { RuntimeProgress rp = new RuntimeProgress(); - ProgressTicker p = (ProgressTicker)stack.getVar("#swift-runtime-progress-ticker"); + ProgressTicker p = getTicker(stack); synchronized (p.states) { p.states.add(rp); } - stack.parentFrame().setVar("#swift-runtime-progress",rp); + setProgress(stack, rp); p.dumpState(); return null; } public synchronized Object vdl_stopprogressticker(VariableStack stack) throws ExecutionException { - ProgressTicker p = (ProgressTicker)stack.getVar("#swift-runtime-progress-ticker"); + ProgressTicker p = getTicker(stack); p.finalDumpState(); - p.stop(); + p.shutdown(); return null; } @@ -90,6 +108,7 @@ long lastDumpTime = 0; private boolean disabled; + private boolean shutdown; public ProgressTicker() { super("Progress ticker"); @@ -108,17 +127,20 @@ if (disabled) { return; } - while(true) { + while(!shutdown) { dumpState(); try { Thread.sleep(MAX_PERIOD_MS); } catch(InterruptedException e) { - System.err.println("Runtime ticker interrupted. Looping immediately."); } } } + + void shutdown() { + shutdown = true; + } void dumpState() { if (disabled) { @@ -136,26 +158,30 @@ } printStates("Final status: "); } + + public Map getSummary() { + Map summary = new HashMap(); + synchronized(states) { + Iterator stateIterator = states.iterator(); + // summarize details of known states into summary, with + // one entry per state type, storing the number of + // jobs in that state. + while(stateIterator.hasNext()) { + String key = ((RuntimeProgress)stateIterator.next()).status; + Integer count = (Integer) summary.get(key); + if(count == null) { + summary.put(key,new Integer(1)); + } else { + summary.put(key,new Integer(count.intValue()+1)); + } + } + } + return summary; + } + void printStates(String header) { - Map summary = new HashMap(); - synchronized(states) { - Iterator stateIterator = states.iterator(); - - // summarize details of known states into summary, with - // one entry per state type, storing the number of - // jobs in that state. - while(stateIterator.hasNext()) { - String key = ((RuntimeProgress)stateIterator.next()).status; - Integer count = (Integer) summary.get(key); - if(count == null) { - summary.put(key,new Integer(1)); - } else { - summary.put(key,new Integer(count.intValue()+1)); - } - } - } - + Map summary = getSummary(); // output the results of summarization, in a relatively // pretty form - first the preferred order listed elements, // and then anything remaining @@ -182,5 +208,4 @@ class RuntimeProgress { String status = "uninitialized"; } - } From noreply at svn.ci.uchicago.edu Sun Mar 29 14:38:39 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 29 Mar 2009 14:38:39 -0500 (CDT) Subject: [Swift-commit] r2763 - trunk/src/org/griphyn/vdl/karajan Message-ID: <20090329193839.5422222817C@www.ci.uchicago.edu> Author: hategan Date: 2009-03-29 14:38:38 -0500 (Sun, 29 Mar 2009) New Revision: 2763 Modified: trunk/src/org/griphyn/vdl/karajan/Loader.java Log: added tui command line arg and the corresponding setup/cleanup actions Modified: trunk/src/org/griphyn/vdl/karajan/Loader.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/Loader.java 2009-03-29 19:32:55 UTC (rev 2762) +++ trunk/src/org/griphyn/vdl/karajan/Loader.java 2009-03-29 19:38:38 UTC (rev 2763) @@ -27,16 +27,20 @@ import org.apache.log4j.FileAppender; import org.apache.log4j.Level; import org.apache.log4j.Logger; +import org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler; import org.globus.cog.karajan.stack.LinkedStack; import org.globus.cog.karajan.stack.VariableStack; import org.globus.cog.karajan.util.Monitor; import org.globus.cog.karajan.workflow.ElementTree; import org.globus.cog.karajan.workflow.PrintStreamChannel; import org.globus.cog.karajan.workflow.nodes.FlowElement; +import org.globus.cog.karajan.workflow.nodes.grid.AbstractGridNode; import org.globus.cog.util.ArgumentParser; import org.globus.cog.util.ArgumentParserException; import org.griphyn.vdl.engine.Karajan; import org.griphyn.vdl.karajan.functions.ConfigProperty; +import org.griphyn.vdl.karajan.lib.Execute; +import org.griphyn.vdl.karajan.monitor.MonitorAppender; import org.griphyn.vdl.toolkit.VDLt2VDLx; import org.griphyn.vdl.toolkit.VDLt2VDLx.IncorrectInvocationException; import org.griphyn.vdl.toolkit.VDLt2VDLx.ParsingException; @@ -45,425 +49,478 @@ import org.griphyn.vdl.util.VDL2ConfigProperties.PropInfo; public class Loader extends org.globus.cog.karajan.Loader { - private static final Logger logger = Logger.getLogger(Loader.class); + private static final Logger logger = Logger.getLogger(Loader.class); - public static final String ARG_HELP = "help"; - public static final String ARG_MONITOR = "monitor"; - public static final String ARG_RESUME = "resume"; - public static final String ARG_INSTANCE_CONFIG = "config"; - public static final String ARG_TYPECHECK = "typecheck"; - public static final String ARG_DRYRUN = "dryrun"; - public static final String ARG_VERBOSE = "verbose"; - public static final String ARG_DEBUG = "debug"; - public static final String ARG_LOGFILE = "logfile"; - public static final String ARG_RUNID = "runid"; + public static final String ARG_HELP = "help"; + public static final String ARG_MONITOR = "monitor"; + public static final String ARG_RESUME = "resume"; + public static final String ARG_INSTANCE_CONFIG = "config"; + public static final String ARG_TYPECHECK = "typecheck"; + public static final String ARG_DRYRUN = "dryrun"; + public static final String ARG_VERBOSE = "verbose"; + public static final String ARG_DEBUG = "debug"; + public static final String ARG_LOGFILE = "logfile"; + public static final String ARG_RUNID = "runid"; + public static final String ARG_TUI = "tui"; - public static final String CONST_VDL_OPERATION = "vdl:operation"; - public static final String VDL_OPERATION_RUN = "run"; - public static final String VDL_OPERATION_TYPECHECK = "typecheck"; - public static final String VDL_OPERATION_DRYRUN = "dryrun"; + public static final String CONST_VDL_OPERATION = "vdl:operation"; + public static final String VDL_OPERATION_RUN = "run"; + public static final String VDL_OPERATION_TYPECHECK = "typecheck"; + public static final String VDL_OPERATION_DRYRUN = "dryrun"; - public static String buildVersion; + public static String buildVersion; - public static void main(String[] argv) { - logger.debug("Loader started"); - ArgumentParser ap = buildArgumentParser(); - String project = null; - try { - ap.parse(argv); + public static void main(String[] argv) { + logger.debug("Loader started"); + ArgumentParser ap = buildArgumentParser(); + String project = null; + try { + ap.parse(argv); - if (ap.isPresent(ARG_HELP)) { - ap.usage(); - System.exit(0); - } - if (ap.isPresent(ARG_MONITOR)) { - new Monitor().start(); - } - if (!ap.hasValue(ArgumentParser.DEFAULT)) { - error("No SwiftScript program specified"); - } - project = ap.getStringValue(ArgumentParser.DEFAULT); - } - catch (ArgumentParserException e) { - System.err.println("Error parsing arguments: " + e.getMessage() + "\n"); - shortUsage(); - System.exit(1); - } - - if (!new File(project).exists()) { - System.err.println("SwiftScript program does not exist: " + project + "\n"); - shortUsage(); - System.exit(1); - } + if (ap.isPresent(ARG_HELP)) { + ap.usage(); + System.exit(0); + } + if (ap.isPresent(ARG_MONITOR)) { + new Monitor().start(); + } + if (!ap.hasValue(ArgumentParser.DEFAULT)) { + error("No SwiftScript program specified"); + } + project = ap.getStringValue(ArgumentParser.DEFAULT); + } + catch (ArgumentParserException e) { + System.err.println("Error parsing arguments: " + e.getMessage() + + "\n"); + shortUsage(); + System.exit(1); + } - boolean runerror = false; - String projectName = projectName(project); + if (!new File(project).exists()) { + System.err.println("SwiftScript program does not exist: " + project + + "\n"); + shortUsage(); + System.exit(1); + } - String runID; - if(ap.isPresent(ARG_RUNID)) { - runID = ap.getStringValue(ARG_RUNID); - } else { - runID = getUUID(); - } + boolean runerror = false; + String projectName = projectName(project); - try { - setupLogging(ap, projectName, runID); + String runID; + if (ap.isPresent(ARG_RUNID)) { + runID = ap.getStringValue(ARG_RUNID); + } + else { + runID = getUUID(); + } - if(!(new File(project).exists())) { - logger.error("Input file "+project+" does not exist."); - System.exit(4); - } + try { + setupLogging(ap, projectName, runID); - /** This deprecation should stay until at least after Swift 0.4 is - released; sometime after that, this test and the subsequent - support for .dtm extensions should be removed. */ - if (project.endsWith(".dtm")) { - logger.warn("Source file extension .dtm is deprecated and will not be supported in future versions. Use .swift extension instead."); - } + if (!(new File(project).exists())) { + logger.error("Input file " + project + " does not exist."); + System.exit(4); + } - if (project.endsWith(".dtm") || project.endsWith(".swift")) { - try { - project = compile(project); - } catch(ParsingException pe) { - // the compiler should have already logged useful - // error messages, so this log line is just for - // debugging - logger.debug("Exception when compiling "+project,pe); - System.exit(3); - } - } - ElementTree tree = null; - if (project != null) { - tree = load(project); - } - else { - System.err.println("No source file specified"); - shortUsage(); - System.exit(1); - } + /** + * This deprecation should stay until at least after Swift 0.4 is + * released; sometime after that, this test and the subsequent + * support for .dtm extensions should be removed. + */ + if (project.endsWith(".dtm")) { + logger + .warn("Source file extension .dtm is deprecated and will not be supported in future versions. Use .swift extension instead."); + } - tree.setName(projectName + "-" + runID); - tree.getRoot().setProperty(FlowElement.FILENAME, project); + if (project.endsWith(".dtm") || project.endsWith(".swift")) { + try { + project = compile(project); + } + catch (ParsingException pe) { + // the compiler should have already logged useful + // error messages, so this log line is just for + // debugging + logger.debug("Exception when compiling " + project, pe); + System.exit(3); + } + } + ElementTree tree = null; + if (project != null) { + tree = load(project); + } + else { + System.err.println("No source file specified"); + shortUsage(); + System.exit(1); + } - VDL2ExecutionContext ec = new VDL2ExecutionContext(tree, projectName); - ec.setRunID(runID); - // no order - ec.setStdout(new PrintStreamChannel(System.out, true)); + tree.setName(projectName + "-" + runID); + tree.getRoot().setProperty(FlowElement.FILENAME, project); - + VDL2ExecutionContext ec = new VDL2ExecutionContext(tree, + projectName); + ec.setRunID(runID); + // no order + ec.setStdout(new PrintStreamChannel(System.out, true)); - VariableStack stack = new LinkedStack(ec); - VDL2Config config = loadConfig(ap, stack); - addCommandLineProperties(config, ap); + VariableStack stack = new LinkedStack(ec); + VDL2Config config = loadConfig(ap, stack); + addCommandLineProperties(config, ap); - if (ap.isPresent(ARG_DRYRUN)) { - stack.setGlobal(CONST_VDL_OPERATION, VDL_OPERATION_DRYRUN); - } - else if (ap.isPresent(ARG_TYPECHECK)) { - stack.setGlobal(CONST_VDL_OPERATION, VDL_OPERATION_TYPECHECK); - } - else { - stack.setGlobal(CONST_VDL_OPERATION, VDL_OPERATION_RUN); - } - // TODO should not be referring to vds.home any more - stack.setGlobal("vds.home", System.getProperty("vds.home")); + if (ap.isPresent(ARG_DRYRUN)) { + stack.setGlobal(CONST_VDL_OPERATION, VDL_OPERATION_DRYRUN); + } + else if (ap.isPresent(ARG_TYPECHECK)) { + stack.setGlobal(CONST_VDL_OPERATION, VDL_OPERATION_TYPECHECK); + } + else { + stack.setGlobal(CONST_VDL_OPERATION, VDL_OPERATION_RUN); + } + // TODO should not be referring to vds.home any more + stack.setGlobal("vds.home", System.getProperty("vds.home")); - List arguments = ap.getArguments(); - if (ap.hasValue(ARG_RESUME)) { - arguments.add("-rlog:resume=" + ap.getStringValue(ARG_RESUME)); - } - ec.setArguments(arguments); + List arguments = ap.getArguments(); + if (ap.hasValue(ARG_RESUME)) { + arguments.add("-rlog:resume=" + ap.getStringValue(ARG_RESUME)); + } + ec.setArguments(arguments); - ec.start(stack); - ec.waitFor(); - if (ec.isFailed()) { - runerror = true; - } - } - catch (Exception e) { - logger.debug("Detailed exception:", e); - error("Could not start execution.\n\t" + e.getMessage()); - } + ec.start(stack); + ec.waitFor(); + if (ec.isFailed()) { + runerror = true; + } + } + catch (Exception e) { + logger.debug("Detailed exception:", e); + error("Could not start execution.\n\t" + e.getMessage()); + } - if(runerror) { - logger.debug("Swift finished with errors"); - } else { - logger.debug("Swift finished with no errors"); - } - System.exit(runerror ? 2 : 0); - } + if (runerror) { + logger.debug("Swift finished with errors"); + } + else { + logger.debug("Swift finished with no errors"); + } + if (ap.isPresent(ARG_TUI)) { + ma.close(); + } + System.exit(runerror ? 2 : 0); + } - private static void shortUsage() { - System.err.print("For usage information: swift -help\n\n"); - } + private static void shortUsage() { + System.err.print("For usage information: swift -help\n\n"); + } - public static String compile(String project) - throws FileNotFoundException, ParsingException, - IncorrectInvocationException, CompilationException, - IOException { - File swiftscript = new File(project); - String projectBase = project.substring(0, project.lastIndexOf('.')); - File xml = new File(projectBase + ".xml"); - File kml = new File(projectBase + ".kml"); + public static String compile(String project) throws FileNotFoundException, + ParsingException, IncorrectInvocationException, + CompilationException, IOException { + File swiftscript = new File(project); + String projectBase = project.substring(0, project.lastIndexOf('.')); + File xml = new File(projectBase + ".xml"); + File kml = new File(projectBase + ".kml"); - loadBuildVersion(); + loadBuildVersion(); - boolean recompile = false; + boolean recompile = false; - if (swiftscript.lastModified() > kml.lastModified()) { - logger.info(project + ": source file is new. Recompiling."); - recompile = true; - } + if (swiftscript.lastModified() > kml.lastModified()) { + logger.info(project + ": source file is new. Recompiling."); + recompile = true; + } - if (kml.exists()) { - // read first line of kml - Reader fr = new FileReader(kml); - BufferedReader br = new BufferedReader(fr); - String firstLine = br.readLine(); - if(firstLine == null) { - logger.debug("KML file is empty. Recompiling."); - recompile = true; - } else { - String prefix = ""); - String kmlversion = cut.substring(0,endOffset); - logger.debug("kmlversion is >"+kmlversion+"<"); - logger.debug("build version is >"+buildVersion+"<"); - if(!(kmlversion.equals(buildVersion))) { - logger.info(project + ": source file was compiled with a different version of Swift. Recompiling."); - recompile=true; - } - } - } - } + if (kml.exists()) { + // read first line of kml + Reader fr = new FileReader(kml); + BufferedReader br = new BufferedReader(fr); + String firstLine = br.readLine(); + if (firstLine == null) { + logger.debug("KML file is empty. Recompiling."); + recompile = true; + } + else { + String prefix = ""); + String kmlversion = cut.substring(0, endOffset); + logger.debug("kmlversion is >" + kmlversion + "<"); + logger.debug("build version is >" + buildVersion + "<"); + if (!(kmlversion.equals(buildVersion))) { + logger.info(project + + ": source file was" + + " compiled with a different version of Swift. Recompiling."); + recompile = true; + } + } + } + } - if (recompile) { - VDLt2VDLx.compile(new FileInputStream(swiftscript), new PrintStream(new FileOutputStream(xml))); + if (recompile) { + VDLt2VDLx.compile(new FileInputStream(swiftscript), + new PrintStream(new FileOutputStream(xml))); - try { - FileOutputStream f = new FileOutputStream(kml); - Karajan.compile(xml.getAbsolutePath(), new PrintStream(f)); - f.close(); - } - catch (Error e) { - kml.delete(); - throw e; - } - catch(CompilationException e) { - // if we leave a kml file around, then a subsequent - // re-run will skip recompiling and cause a different - // error message for the user - kml.delete(); - throw e; - } - catch(Exception e) { - // if we leave a kml file around, then a subsequent - // re-run will skip recompiling and cause a different - // error message for the user - kml.delete(); - throw new CompilationException("Failed to convert .xml to .kml for "+project,e); - } - } - else { - logger.debug("Recompilation suppressed."); - } - return kml.getAbsolutePath(); - } + try { + FileOutputStream f = new FileOutputStream(kml); + Karajan.compile(xml.getAbsolutePath(), new PrintStream(f)); + f.close(); + } + catch (Error e) { + kml.delete(); + throw e; + } + catch (CompilationException e) { + // if we leave a kml file around, then a subsequent + // re-run will skip recompiling and cause a different + // error message for the user + kml.delete(); + throw e; + } + catch (Exception e) { + // if we leave a kml file around, then a subsequent + // re-run will skip recompiling and cause a different + // error message for the user + kml.delete(); + throw new CompilationException( + "Failed to convert .xml to .kml for " + project, e); + } + } + else { + logger.debug("Recompilation suppressed."); + } + return kml.getAbsolutePath(); + } - private static void loadBuildVersion() throws IOException { - File f = new File(System.getProperty("swift.home")+"/libexec/buildid.txt"); - BufferedReader br = new BufferedReader(new FileReader(f)); - buildVersion = br.readLine(); - } + private static void loadBuildVersion() { + try { + File f = new File(System.getProperty("swift.home") + + "/libexec/buildid.txt"); + BufferedReader br = new BufferedReader(new FileReader(f)); + buildVersion = br.readLine(); + } + catch (IOException e) { + buildVersion = null; + } + } - private static VDL2Config loadConfig(ArgumentParser ap, VariableStack stack) throws IOException { - VDL2Config conf; - if (ap.hasValue(ARG_INSTANCE_CONFIG)) { - String configFile = ap.getStringValue(ARG_INSTANCE_CONFIG); - stack.setGlobal(ConfigProperty.INSTANCE_CONFIG_FILE, configFile); - conf = VDL2Config.getConfig(configFile); - } - else { - conf = (VDL2Config) VDL2Config.getConfig().clone(); - } - stack.setGlobal(ConfigProperty.INSTANCE_CONFIG, conf); - return conf; - } + private static VDL2Config loadConfig(ArgumentParser ap, VariableStack stack) + throws IOException { + VDL2Config conf; + if (ap.hasValue(ARG_INSTANCE_CONFIG)) { + String configFile = ap.getStringValue(ARG_INSTANCE_CONFIG); + stack.setGlobal(ConfigProperty.INSTANCE_CONFIG_FILE, configFile); + conf = VDL2Config.getConfig(configFile); + } + else { + conf = (VDL2Config) VDL2Config.getConfig().clone(); + } + stack.setGlobal(ConfigProperty.INSTANCE_CONFIG, conf); + return conf; + } - private static void addCommandLineProperties(VDL2Config config, ArgumentParser ap) { - Map desc = VDL2ConfigProperties.getPropertyDescriptions(); - Iterator i = desc.entrySet().iterator(); - while (i.hasNext()) { - Map.Entry e = (Map.Entry) i.next(); - String name = (String) e.getKey(); - if (ap.isPresent(name)) { - config.setProperty(name, ap.getStringValue(name)); - } - } - } + private static void addCommandLineProperties(VDL2Config config, + ArgumentParser ap) { + Map desc = VDL2ConfigProperties.getPropertyDescriptions(); + Iterator i = desc.entrySet().iterator(); + while (i.hasNext()) { + Map.Entry e = (Map.Entry) i.next(); + String name = (String) e.getKey(); + if (ap.isPresent(name)) { + config.setProperty(name, ap.getStringValue(name)); + } + } + } - private static ArgumentParser buildArgumentParser() { - ArgumentParser ap = new ArgumentParser(); - ap.setArguments(true); - ap.setExecutableName("swift"); - ap.addOption(ArgumentParser.DEFAULT, "A file (.dtm or .kml) to execute", "file", - ArgumentParser.OPTIONAL); + private static ArgumentParser buildArgumentParser() { + ArgumentParser ap = new ArgumentParser(); + ap.setArguments(true); + ap.setExecutableName("swift"); + ap + .addOption(ArgumentParser.DEFAULT, + "A file (.dtm or .kml) to execute", "file", + ArgumentParser.OPTIONAL); - ap.addFlag(ARG_HELP, "Display usage information"); - ap.addAlias(ARG_HELP, "h"); + ap.addFlag(ARG_HELP, "Display usage information"); + ap.addAlias(ARG_HELP, "h"); - ap.addFlag(ARG_TYPECHECK, "Does a typecheck instead of executing the SwiftScript program"); + ap.addFlag(ARG_TYPECHECK, + "Does a typecheck instead of executing the SwiftScript program"); - ap.addFlag(ARG_DRYRUN, - "Runs the SwiftScript program without submitting any jobs (can be used to get a graph)"); + ap + .addFlag( + ARG_DRYRUN, + "Runs the SwiftScript program without submitting any jobs (can be used to get a graph)"); - ap.addFlag(ARG_MONITOR, "Shows a graphical resource monitor"); + ap.addFlag(ARG_MONITOR, "Shows a graphical resource monitor"); - ap.addOption(ARG_RESUME, "Resumes the execution using a log file", "file", - ArgumentParser.OPTIONAL); - ap.addOption( - ARG_INSTANCE_CONFIG, - "Indicates the Swift configuration file to be used for this run." - + " Properties in this configuration file will override the default properties. " - + "If individual command line arguments are used for properties, they will override " - + "the contents of this file.", "file", ArgumentParser.OPTIONAL); - ap.addFlag(ARG_VERBOSE, - "Increases the level of output that Swift produces on the console to include more detail " - + "about the execution"); - ap.addAlias(ARG_VERBOSE, "v"); - ap.addFlag(ARG_DEBUG, - "Increases the level of output that Swift produces on the console to include lots of " - + "detail about the execution"); - ap.addAlias(ARG_DEBUG, "d"); - ap.addOption( - ARG_LOGFILE, - "Specifies a file where log messages should go to. By default Swift " - + "uses the name of the SwiftScript program being run and additional information to make the name unique.", - "file", ArgumentParser.OPTIONAL); - ap.addOption( - ARG_RUNID, - "Specifies the run identifier. This must be unique for every invocation of a workflow and is used in several places to keep files from different executions cleanly separated. By default, a datestamp and random number are used to generate a run identifier.", - "string", ArgumentParser.OPTIONAL); + ap.addOption(ARG_RESUME, "Resumes the execution using a log file", + "file", ArgumentParser.OPTIONAL); + ap + .addOption( + ARG_INSTANCE_CONFIG, + "Indicates the Swift configuration file to be used for this run." + + " Properties in this configuration file will override the default properties. " + + "If individual command line arguments are used for properties, they will override " + + "the contents of this file.", "file", + ArgumentParser.OPTIONAL); + ap + .addFlag( + ARG_VERBOSE, + "Increases the level of output that Swift produces on the console to include more detail " + + "about the execution"); + ap.addAlias(ARG_VERBOSE, "v"); + ap + .addFlag( + ARG_DEBUG, + "Increases the level of output that Swift produces on the console to include lots of " + + "detail about the execution"); + ap.addAlias(ARG_DEBUG, "d"); + ap + .addOption( + ARG_LOGFILE, + "Specifies a file where log messages should go to. By default Swift " + + "uses the name of the SwiftScript program being run and additional information to make the name unique.", + "file", ArgumentParser.OPTIONAL); + ap + .addOption( + ARG_RUNID, + "Specifies the run identifier. This must be unique for every invocation of a workflow and is used in several places to keep files from different executions cleanly separated. By default, a datestamp and random number are used to generate a run identifier.", + "string", ArgumentParser.OPTIONAL); + ap.addFlag(ARG_TUI); - Map desc = VDL2ConfigProperties.getPropertyDescriptions(); - Iterator i = desc.entrySet().iterator(); - while (i.hasNext()) { - Map.Entry e = (Map.Entry) i.next(); - PropInfo pi = (PropInfo) e.getValue(); - ap.addOption((String) e.getKey(), pi.desc, pi.validValues, ArgumentParser.OPTIONAL); - } - return ap; - } + Map desc = VDL2ConfigProperties.getPropertyDescriptions(); + Iterator i = desc.entrySet().iterator(); + while (i.hasNext()) { + Map.Entry e = (Map.Entry) i.next(); + PropInfo pi = (PropInfo) e.getValue(); + ap.addOption((String) e.getKey(), pi.desc, pi.validValues, + ArgumentParser.OPTIONAL); + } + return ap; + } - protected static void setupLogging(ArgumentParser ap, String projectName, String runID) - throws IOException { - String logfile; - if (ap.isPresent(ARG_LOGFILE)) { - logfile = ap.getStringValue(ARG_LOGFILE); - } - else { - logfile = projectName + "-" + runID + ".log"; - } + private static MonitorAppender ma; - File f = new File(logfile); + protected static void setupLogging(ArgumentParser ap, String projectName, + String runID) throws IOException { + String logfile; + if (ap.isPresent(ARG_LOGFILE)) { + logfile = ap.getStringValue(ARG_LOGFILE); + } + else { + logfile = projectName + "-" + runID + ".log"; + } - FileAppender fa = (FileAppender) getAppender(FileAppender.class); - if (fa == null) { - logger.warn("Failed to configure log file name"); - } - else { - fa.setFile(f.getAbsolutePath()); - fa.activateOptions(); - } - Level level = Level.WARN; - if (ap.isPresent(ARG_VERBOSE)) { - level = Level.INFO; - } - if (ap.isPresent(ARG_DEBUG)) { - level = Level.DEBUG; - } - ConsoleAppender ca = (ConsoleAppender) getAppender(ConsoleAppender.class); - if (ca == null) { - logger.warn("Failed to configure console log level"); - } - else { - ca.setThreshold(level); - ca.activateOptions(); - } - } + File f = new File(logfile); - protected static Appender getAppender(Class cls) { - Logger root = Logger.getRootLogger(); - Enumeration e = root.getAllAppenders(); - while (e.hasMoreElements()) { - Appender a = (Appender) e.nextElement(); - if (cls.isAssignableFrom(a.getClass())) { - return a; - } - } - return null; - } + FileAppender fa = (FileAppender) getAppender(FileAppender.class); + if (fa == null) { + logger.warn("Failed to configure log file name"); + } + else { + fa.setFile(f.getAbsolutePath()); + fa.activateOptions(); + } + Level level = Level.WARN; + if (ap.isPresent(ARG_VERBOSE)) { + level = Level.INFO; + } + if (ap.isPresent(ARG_DEBUG)) { + level = Level.DEBUG; + } + ConsoleAppender ca = (ConsoleAppender) getAppender(ConsoleAppender.class); + if (ca == null) { + logger.warn("Failed to configure console log level"); + } + else { + ca.setThreshold(level); + ca.activateOptions(); + } + if (ap.isPresent(ARG_TUI)) { + ma = new MonitorAppender(projectName); + Logger.getRootLogger().addAppender(ma); + Logger.getLogger(AbstractGridNode.class).setLevel(Level.DEBUG); + Logger.getLogger(Execute.class).setLevel(Level.DEBUG); + Logger.getLogger(VDL2ExecutionContext.class).setLevel(Level.INFO); + Logger.getLogger(WeightedHostScoreScheduler.class).setLevel( + Level.INFO); + } + } - protected static String projectName(String project) { - project = project.substring(project.lastIndexOf(File.separatorChar) + 1); - return project.substring(0, project.lastIndexOf('.')); - } + protected static Appender getAppender(Class cls) { + Logger root = Logger.getRootLogger(); + Enumeration e = root.getAllAppenders(); + while (e.hasMoreElements()) { + Appender a = (Appender) e.nextElement(); + if (cls.isAssignableFrom(a.getClass())) { + return a; + } + } + return null; + } - private static long lastTime = 0; - - private static DateFormat UID_DATE_FORMAT = new SimpleDateFormat("yyyyMMdd-HHmm"); + protected static String projectName(String project) { + project = project + .substring(project.lastIndexOf(File.separatorChar) + 1); + return project.substring(0, project.lastIndexOf('.')); + } - public static synchronized String getUUID() { - long l; - // we want seconds and milliseconds from this one - // that's 16 bits - while (true) { - l = System.currentTimeMillis() % (60*1000); - if (l != lastTime) { - lastTime = l; - break; - } - else { - try { - Thread.sleep(1000); - } - catch (InterruptedException e) { - } - } - } - // and for the msbs, some random stuff - int rnd; - try { - SecureRandom prng = SecureRandom.getInstance("SHA1PRNG"); - rnd = prng.nextInt(); - } - catch (NoSuchAlgorithmException e) { - rnd = (int) (Math.random() * 0xffffff); - } - //and 24 bits - rnd &= 0x00ffffff; - l += ((long) rnd) << 16; - return UID_DATE_FORMAT.format(new Date()) + '-' + alphanum(l); - } + private static long lastTime = 0; - public static final String codes = "0123456789abcdefghijklmnopqrstuvxyz"; + private static DateFormat UID_DATE_FORMAT = new SimpleDateFormat( + "yyyyMMdd-HHmm"); - protected static String alphanum(long val) { - StringBuffer sb = new StringBuffer(); - int base = codes.length(); - for (int i = 0; i < 8; i++) { - int c = (int) (val % base); - sb.append(codes.charAt(c)); - val = val / base; - } - return sb.toString(); - } + public static synchronized String getUUID() { + long l; + // we want seconds and milliseconds from this one + // that's 16 bits + while (true) { + l = System.currentTimeMillis() % (60 * 1000); + if (l != lastTime) { + lastTime = l; + break; + } + else { + try { + Thread.sleep(1000); + } + catch (InterruptedException e) { + } + } + } + // and for the msbs, some random stuff + int rnd; + try { + SecureRandom prng = SecureRandom.getInstance("SHA1PRNG"); + rnd = prng.nextInt(); + } + catch (NoSuchAlgorithmException e) { + rnd = (int) (Math.random() * 0xffffff); + } + // and 24 bits + rnd &= 0x00ffffff; + l += ((long) rnd) << 16; + return UID_DATE_FORMAT.format(new Date()) + '-' + alphanum(l); + } + public static final String codes = "0123456789abcdefghijklmnopqrstuvxyz"; + + protected static String alphanum(long val) { + StringBuffer sb = new StringBuffer(); + int base = codes.length(); + for (int i = 0; i < 8; i++) { + int c = (int) (val % base); + sb.append(codes.charAt(c)); + val = val / base; + } + return sb.toString(); + } + } From noreply at svn.ci.uchicago.edu Sun Mar 29 14:39:42 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 29 Mar 2009 14:39:42 -0500 (CDT) Subject: [Swift-commit] r2764 - in usertools/cio: falkon tools Message-ID: <20090329193942.CE44D228232@www.ci.uchicago.edu> Author: aespinosa Date: 2009-03-29 14:39:42 -0500 (Sun, 29 Mar 2009) New Revision: 2764 Added: usertools/cio/tools/hashserver.rb Modified: usertools/cio/falkon/ciologic-bgp.sh usertools/cio/falkon/falkon-start-bgp-cio.sh Log: Added ruby hash server. Cleanup and fix of falkon startup scripts Modified: usertools/cio/falkon/ciologic-bgp.sh =================================================================== --- usertools/cio/falkon/ciologic-bgp.sh 2009-03-29 19:38:38 UTC (rev 2763) +++ usertools/cio/falkon/ciologic-bgp.sh 2009-03-29 19:39:42 UTC (rev 2764) @@ -49,16 +49,15 @@ $CIOROOT/tools/ifsslave.sh $DATACN >&2 fi else - echo "[ $BG_RANK@$IO_RANK-$PSET ] Mounting Mosa data CN filesystem from $DATACN" >&2 /bin/mkdir /dataifs export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/lib:/fuse/lib:/usr/lib sleep 30 - # Contact / mount assigned IFS head + # Contact and mount assigned IFS head $CIOROOT/tools/ifsmount.sh $DATACN >&2 echo "worker started..." >> ${ERROR_LOG} - # Regular Falon worker startup script + # Regular Falkon worker startup script FALKON_SERVICE_IP=$1 TCPCORE_PORT1=$2 Modified: usertools/cio/falkon/falkon-start-bgp-cio.sh =================================================================== --- usertools/cio/falkon/falkon-start-bgp-cio.sh 2009-03-29 19:38:38 UTC (rev 2763) +++ usertools/cio/falkon/falkon-start-bgp-cio.sh 2009-03-29 19:39:42 UTC (rev 2764) @@ -1,5 +1,7 @@ #!/bin/bash +CIOROOT=/home/espinosa/cio + if [ $# -lt 3 ]; then cat << EOF Usage: $0 @@ -53,7 +55,7 @@ -WORKER_SCRIPT=cio-logic.sh +WORKER_SCRIPT=$CIOROOT/falkon/ciologic-bgp.sh ZOID_JOB_ID=`cqsub -q ${QUEUE_NAME} -k ${PROFILE_NAME} -C ${HOME} -t \ ${MAX_TIME_MIN} -n ${NUM_NODES} \ -e LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib \ @@ -64,7 +66,12 @@ EXIT_CODE=$? if [ "${EXIT_CODE}" -ne "0" ]; then - echo "Error in submitting job to Cobalt.. 'cqsub -q ${QUEUE_NAME} -k ${PROFILE_NAME} -C ${HOME} -t ${MAX_TIME_MIN} -n ${NUM_NODES} -e LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib /bgsys/linux/1.2.020080512/bin/bash /fuse/${FALKON_WORKER_HOME}/run.worker-c-bgp.sh-mosa ${SERVICE_IP} ${SERVICE_PORT1} ${SERVICE_PORT2} ${WORKERS_PER_NODE} ${USER} ${FALKON_JOB_ID} ${FALKON_ROOT}' " + echo "Error in submitting job to Cobalt.. 'cqsub -q ${QUEUE_NAME} \ + -k ${PROFILE_NAME} -C ${HOME} -t ${MAX_TIME_MIN} -n ${NUM_NODES} \ + -e LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib \ + /bgsys/linux/1.2.020080512/bin/bash $WORKER_SCRIPT \ + ${SERVICE_IP} ${SERVICE_PORT1} ${SERVICE_PORT2} \ + ${WORKERS_PER_NODE} ${USER} ${FALKON_JOB_ID} ${FALKON_ROOT}' " exit ${EXIT_CODE} fi Added: usertools/cio/tools/hashserver.rb =================================================================== --- usertools/cio/tools/hashserver.rb (rev 0) +++ usertools/cio/tools/hashserver.rb 2009-03-29 19:39:42 UTC (rev 2764) @@ -0,0 +1,26 @@ +#!/home/espinosa/local/bin/ruby + +require 'drb' + +class HashServer + attr_reader :file + + def initialize + @file = {} + end + + def put(fname, rank) + @file[fname] = rank + end + + def get(fname) + @file[fname] + end + +end + +server = HashServer.new + +DRb.start_service('druby://*:9000', server) +DRb.thread.join + Property changes on: usertools/cio/tools/hashserver.rb ___________________________________________________________________ Name: svn:executable + * From noreply at svn.ci.uchicago.edu Sun Mar 29 14:41:39 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 29 Mar 2009 14:41:39 -0500 (CDT) Subject: [Swift-commit] r2765 - trunk/libexec Message-ID: <20090329194139.9768C228110@www.ci.uchicago.edu> Author: hategan Date: 2009-03-29 14:41:38 -0500 (Sun, 29 Mar 2009) New Revision: 2765 Modified: trunk/libexec/vdl-int.k Log: tweaked provenance graph a bit Modified: trunk/libexec/vdl-int.k =================================================================== --- trunk/libexec/vdl-int.k 2009-03-29 19:39:42 UTC (rev 2764) +++ trunk/libexec/vdl-int.k 2009-03-29 19:41:38 UTC (rev 2765) @@ -276,15 +276,15 @@ element(graphStuff, [tr, stagein, stageout, err, optional(args)] if( vdl:configProperty("pgraph") != "false" then( - errprops := if(err ",color=lightsalmon" "") + errprops := if(err ",color=lightsalmon" ",color=lightsteelblue1") tp := vdl:threadPrefix() to(graph, - concat(str:quote(tp), " [label=", str:quote("{tr}{errprops}"), "]") + concat(str:quote(tp), " [label=", str:quote(tr), "{errprops}]") ) for(si, stagein si := basename(si) to(graph - concat(str:quote(si), " [shape=parallelogram{errprops}]") + concat(str:quote(si), " [shape=parallelogram]") concat(str:quote(si), " -> ", str:quote(tp)) ) ) @@ -295,7 +295,7 @@ label := vdl:niceName(var, path = path) to(graph concat(str:quote(file), " [shape=parallelogram,label=", - str:quote("{label}{errprops}"), "]") + str:quote(label), "]") concat(str:quote(tp), " -> ", str:quote(file)) ) ) From noreply at svn.ci.uchicago.edu Sun Mar 29 14:42:10 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 29 Mar 2009 14:42:10 -0500 (CDT) Subject: [Swift-commit] r2766 - in trunk: . lib Message-ID: <20090329194210.80242228110@www.ci.uchicago.edu> Author: hategan Date: 2009-03-29 14:42:10 -0500 (Sun, 29 Mar 2009) New Revision: 2766 Added: trunk/lib/jline-0.9.94.jar Modified: trunk/project.properties Log: added jline library Added: trunk/lib/jline-0.9.94.jar =================================================================== (Binary files differ) Property changes on: trunk/lib/jline-0.9.94.jar ___________________________________________________________________ Name: svn:mime-type + application/octet-stream Modified: trunk/project.properties =================================================================== --- trunk/project.properties 2009-03-29 19:41:38 UTC (rev 2765) +++ trunk/project.properties 2009-03-29 19:42:10 UTC (rev 2766) @@ -8,4 +8,4 @@ # stable branch should be a change of version number) version = svn project = Swift workflow system -lib.deps = gvds.*, antlr*.*, jsr173_1.0_api.*, resolver.*, stringtemplate.*, vdldefinitions.*, xbean.*, xbean_xpath.*, jug*.*, jaxrpc*.* +lib.deps = gvds.*, antlr*.*, jsr173_1.0_api.*, resolver.*, stringtemplate.*, vdldefinitions.*, xbean.*, xbean_xpath.*, jug*.*, jaxrpc*.*, jline*.jar From noreply at svn.ci.uchicago.edu Sun Mar 29 15:35:43 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 29 Mar 2009 15:35:43 -0500 (CDT) Subject: [Swift-commit] r2767 - trunk/src/org/griphyn/vdl/karajan/monitor/monitors/ansi/tui Message-ID: <20090329203543.BE40722817C@www.ci.uchicago.edu> Author: hategan Date: 2009-03-29 15:35:43 -0500 (Sun, 29 Mar 2009) New Revision: 2767 Modified: trunk/src/org/griphyn/vdl/karajan/monitor/monitors/ansi/tui/ANSIContext.java Log: OS X terminal has different f1-f4 and some weird sync issues when replies to VTxxx queries happen at the same time as keyboard hits (or so I think) Modified: trunk/src/org/griphyn/vdl/karajan/monitor/monitors/ansi/tui/ANSIContext.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/monitor/monitors/ansi/tui/ANSIContext.java 2009-03-29 19:42:10 UTC (rev 2766) +++ trunk/src/org/griphyn/vdl/karajan/monitor/monitors/ansi/tui/ANSIContext.java 2009-03-29 20:35:43 UTC (rev 2767) @@ -15,7 +15,11 @@ import jline.Terminal; import jline.UnixTerminal; +import org.apache.log4j.Logger; + public class ANSIContext { + public static final Logger logger = Logger.getLogger(ANSIContext.class); + private OutputStreamWriter os; private InputStream is; private Screen screen; @@ -131,24 +135,32 @@ return terminal.isANSISupported(); } - public int[] querySize() throws IOException { - os.write(ANSI.AESC + "18t"); - os.flush(); - try { - expect(ANSI.AESC, 250); - List nums = readNums(); - if (nums.size() == 3) { - nums.remove(0); - } - int[] sz = new int[2]; - sz[0] = ((Integer) nums.get(1)).intValue(); - sz[1] = ((Integer) nums.get(0)).intValue(); - return sz; - } - catch (UnsupportedOperationException e) { - return null; - } - } + public int[] querySize() throws IOException { + os.write(ANSI.AESC + "18t"); + os.flush(); + try { + expect(ANSI.AESC, 250); + List nums = readNums(); + if (nums == null || nums.size() < 2) { + if (buf != null) { + return new int[] { buf.getWidth(), buf.getHeight() }; + } + else { + return new int[] { 80, 24 }; + } + } + if (nums.size() == 3) { + nums.remove(0); + } + int[] sz = new int[2]; + sz[0] = ((Integer) nums.get(1)).intValue(); + sz[1] = ((Integer) nums.get(0)).intValue(); + return sz; + } + catch (UnsupportedOperationException e) { + return null; + } + } protected void expect(char c) throws IOException { if (is.read() != c) { @@ -200,7 +212,12 @@ do { c = is.read(); if (!Character.isDigit((char) c)) { - nums.add(new Integer(sb.toString())); + try { + nums.add(new Integer(sb.toString())); + } + catch (NumberFormatException e) { + return null; + } sb = new StringBuffer(); if (c != ';') { return nums; @@ -282,6 +299,11 @@ key = new Key(0, Key.KEYPAD + c0); } } + else if (c == 'O') { + // OS X F1 - F4 + int c0 = read(); + key = new Key(0, Key.F1 + (c0 - 'P')); + } else { key = new Key(Key.MOD_ALT, c); } @@ -324,6 +346,7 @@ } catch (Exception e) { if (!done) { + logger.warn("Rendering exception", e); moveTo(1, 1); bgColor(ANSI.RED); fgColor(ANSI.WHITE); @@ -483,10 +506,10 @@ public void echo(boolean b) throws IOException { if (b) { - //terminal.enableEcho(); + // terminal.enableEcho(); } else { - //terminal.disableEcho(); + // terminal.disableEcho(); } } From noreply at svn.ci.uchicago.edu Sun Mar 29 15:39:04 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 29 Mar 2009 15:39:04 -0500 (CDT) Subject: [Swift-commit] r2768 - trunk/src/org/griphyn/vdl/karajan/monitor/monitors/ansi/tui Message-ID: <20090329203904.9F95522817C@www.ci.uchicago.edu> Author: hategan Date: 2009-03-29 15:39:03 -0500 (Sun, 29 Mar 2009) New Revision: 2768 Modified: trunk/src/org/griphyn/vdl/karajan/monitor/monitors/ansi/tui/ANSIContext.java Log: one more Modified: trunk/src/org/griphyn/vdl/karajan/monitor/monitors/ansi/tui/ANSIContext.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/monitor/monitors/ansi/tui/ANSIContext.java 2009-03-29 20:35:43 UTC (rev 2767) +++ trunk/src/org/griphyn/vdl/karajan/monitor/monitors/ansi/tui/ANSIContext.java 2009-03-29 20:39:03 UTC (rev 2768) @@ -245,7 +245,7 @@ if (count == 0) { count = 10; int[] size = querySize(); - if (size[0] != buf.getWidth() || size[1] != buf.getHeight()) { + if (size != null && size[0] != buf.getWidth() || size[1] != buf.getHeight()) { buf.resize(size[0], size[1]); screen.setSize(size[0], size[1]); screen.invalidate(); From noreply at svn.ci.uchicago.edu Sun Mar 29 15:39:41 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 29 Mar 2009 15:39:41 -0500 (CDT) Subject: [Swift-commit] r2769 - trunk Message-ID: <20090329203941.D113B2281A1@www.ci.uchicago.edu> Author: hategan Date: 2009-03-29 15:39:41 -0500 (Sun, 29 Mar 2009) New Revision: 2769 Modified: trunk/CHANGES.txt Log: changes Modified: trunk/CHANGES.txt =================================================================== --- trunk/CHANGES.txt 2009-03-29 20:39:03 UTC (rev 2768) +++ trunk/CHANGES.txt 2009-03-29 20:39:41 UTC (rev 2769) @@ -1,3 +1,7 @@ +(03/29/09) +*** Added console text interface to provide live information about swift + runs. + (03/27/09) *** Environment variable SWIFT_EXTRA_INFO, when set in an environment profile, is executed with the result being recorded in wrapper logs. This allows From noreply at svn.ci.uchicago.edu Sun Mar 29 15:48:35 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 29 Mar 2009 15:48:35 -0500 (CDT) Subject: [Swift-commit] r2770 - trunk/src/org/griphyn/vdl/karajan/monitor/monitors/ansi/tui Message-ID: <20090329204835.91AC622817C@www.ci.uchicago.edu> Author: hategan Date: 2009-03-29 15:48:34 -0500 (Sun, 29 Mar 2009) New Revision: 2770 Modified: trunk/src/org/griphyn/vdl/karajan/monitor/monitors/ansi/tui/ANSIContext.java Log: && has higher priority than || Modified: trunk/src/org/griphyn/vdl/karajan/monitor/monitors/ansi/tui/ANSIContext.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/monitor/monitors/ansi/tui/ANSIContext.java 2009-03-29 20:39:41 UTC (rev 2769) +++ trunk/src/org/griphyn/vdl/karajan/monitor/monitors/ansi/tui/ANSIContext.java 2009-03-29 20:48:34 UTC (rev 2770) @@ -245,7 +245,7 @@ if (count == 0) { count = 10; int[] size = querySize(); - if (size != null && size[0] != buf.getWidth() || size[1] != buf.getHeight()) { + if (size != null && (size[0] != buf.getWidth() || size[1] != buf.getHeight())) { buf.resize(size[0], size[1]); screen.setSize(size[0], size[1]); screen.invalidate(); From noreply at svn.ci.uchicago.edu Sun Mar 29 23:52:05 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Sun, 29 Mar 2009 23:52:05 -0500 (CDT) Subject: [Swift-commit] r2771 - in usertools/cio: falkon tools tools/zoid Message-ID: <20090330045206.12742228110@www.ci.uchicago.edu> Author: aespinosa Date: 2009-03-29 23:52:04 -0500 (Sun, 29 Mar 2009) New Revision: 2771 Added: usertools/cio/falkon/cobaltrun.job usertools/cio/falkon/falkon-start-bgp-logging.sh usertools/cio/tools/zoid/ usertools/cio/tools/zoid/logging-script.sh Log: Added tools for logging GPFS traffic on ZeptoOS ION. also included a sample script to run Added: usertools/cio/falkon/cobaltrun.job =================================================================== --- usertools/cio/falkon/cobaltrun.job (rev 0) +++ usertools/cio/falkon/cobaltrun.job 2009-03-30 04:52:04 UTC (rev 2771) @@ -0,0 +1 @@ +./falkon-start-bgp-logging.sh default 64 60 4 Added: usertools/cio/falkon/falkon-start-bgp-logging.sh =================================================================== --- usertools/cio/falkon/falkon-start-bgp-logging.sh (rev 0) +++ usertools/cio/falkon/falkon-start-bgp-logging.sh 2009-03-30 04:52:04 UTC (rev 2771) @@ -0,0 +1,148 @@ +#!/bin/bash + +# Patched falkon-start-bgp-ram.sh which logs GPFS IO activity in ZeptoOS +CIOROOT=/home/espinosa/cio + + + if [ -z "$3" ]; then + echo "usage: $0 " + echo "usage: $0 prod 1024 60" + echo "-- or --" + echo "usage: $0 " + echo "usage: $0 prod 1024 60 4" + exit 1 + fi + + +if [ -z "${FALKON_HOME}" ]; then + echo "ERROR: environment variable FALKON_HOME not defined" + exit 1 +fi + + +QUEUE_NAME=$1 +PROFILE_NAME="zeptocn-log" +NUM_NODES=$2 +let NUM_ION=NUM_NODES/64 +MAX_TIME_MIN=$3 +SERVICE_IP="192.168.1.254" +SERVICE_PORT1=55000 +SERVICE_PORT2=55001 +WORKERS_PER_NODE=4 +if [ ! -z $4 ];then + WORKERS_PER_NODE=$4 +fi + + + +cp $CIOROOT/tools/zoid/logging-script.sh ${HOME}/zoid-user-script.sh +chmod +x ${HOME}/zoid-user-script.sh + +FALKON_JOB_ID=`falkon-id-get.sh N/A` +EXIT_CODE=$? + + + if [ "${EXIT_CODE}" -ne "0" ]; then + echo "Error in geting a unique falkon ID.. 'falkon-get-id.sh N/A'" + cqdel ${ZOID_JOB_ID} + exit ${EXIT_CODE} + fi + + + +ZOID_JOB_ID=`cqsub -q ${QUEUE_NAME} -k ${PROFILE_NAME} -C ${HOME} -t ${MAX_TIME_MIN} -n ${NUM_NODES} -e LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib /bgsys/linux/1.2.020080512/bin/bash /fuse/${FALKON_WORKER_HOME}/run.worker-c-bgp.sh ${SERVICE_IP} ${SERVICE_PORT1} ${SERVICE_PORT2} ${WORKERS_PER_NODE} ${USER} ${FALKON_JOB_ID} ${FALKON_ROOT}` + +echo $ZOID_JOB_ID $FALKON_JOB_ID $QUEUE_NAME $NUM_NODES $MAX_TIME_MIN $WORKERS_PER_NODE >>$HOME/.falkonjobs + +EXIT_CODE=$? + + if [ "${EXIT_CODE}" -ne "0" ]; then + echo "Error in submitting job to Cobalt.. 'cqsub -q ${QUEUE_NAME} -k ${PROFILE_NAME} -C ${HOME} -t ${MAX_TIME_MIN} -n ${NUM_NODES} -e LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib /bgsys/linux/1.2.020080512/bin/bash /fuse/${FALKON_WORKER_HOME}/run.worker-c-bgp.sh ${SERVICE_IP} ${SERVICE_PORT1} ${SERVICE_PORT2} ${WORKERS_PER_NODE} ${USER} ${FALKON_JOB_ID} ${FALKON_ROOT}' " + exit ${EXIT_CODE} + fi + +falkon-id-update.sh ${FALKON_JOB_ID} ${ZOID_JOB_ID} + +EXIT_CODE=$? + + if [ "${EXIT_CODE}" -ne "0" ]; then + echo "Error in updating cobalt job info for falkon.. 'falkon-update-id.sh ${ZOID_JOB_ID}'" + cqdel ${ZOID_JOB_ID} + exit ${EXIT_CODE} + fi + + +FALKON_JOBID_HOME=${FALKON_ROOT}/users/${USER}/${FALKON_JOB_ID} + +echo "Submitted job ${ZOID_JOB_ID} to Cobalt, creating the job specific Falkon tree for logs and configuration in ${FALKON_JOBID_HOME}..." + +mkdir -p ${FALKON_JOBID_HOME} + +cp ${FALKON_HOME}/falkon.env.bgp* ${FALKON_JOBID_HOME}/ +cp -r ${FALKON_HOME}/config ${FALKON_JOBID_HOME}/ +cp ${FALKON_JOBID_HOME}/config/Client-service-URIs.config2 ${FALKON_JOBID_HOME}/config/Client-service-URIs.config +mkdir -p ${FALKON_JOBID_HOME}/logs/client ${FALKON_JOBID_HOME}/logs/service ${FALKON_JOBID_HOME}/logs/provisioner ${FALKON_JOBID_HOME}/logs/worker + + +DATE=`date +%s` +echo "$DATE: pre-creating log dirs for Falkon service..." +RACK_START=0 +RACK_END=48 +SEQUENCE_DIR=`seq -w ${RACK_START} ${RACK_END}` +PSET_START=1 +PSET_END=16 +for a in ${SEQUENCE_DIR} +do + for ((b=${PSET_START}; b <= ${PSET_END} ; b++)) # Double parentheses, and "LIMIT" with no "$". + do + DIR_NAME="ion-R${a}-${b}" + mkdir -p ${FALKON_JOBID_HOME}/logs/service/$DIR_NAME + done +done + +for ((b=${PSET_START}; b <= ${PSET_END} ; b++)) # Double parentheses, and "LIMIT" with no "$". +do + DIR_NAME="ion-${b}" + mkdir -p ${FALKON_JOBID_HOME}/logs/service/$DIR_NAME +done + + +DATE=`date +%s` +echo "$DATE: done creating log dirs for Falkon service!" + + + +FALKON_HOME_RAM=/tmp/${USER}/falkon + +ln -s ${FALKON_HOME}/apps ${FALKON_JOBID_HOME}/apps +ln -s ${FALKON_HOME_RAM}/container ${FALKON_JOBID_HOME}/container +ln -s ${FALKON_HOME}/service ${FALKON_JOBID_HOME}/service +ln -s ${FALKON_HOME}/worker ${FALKON_JOBID_HOME}/worker +ln -s ${FALKON_HOME}/AstroPortal ${FALKON_JOBID_HOME}/AstroPortal +ln -s ${FALKON_HOME}/client ${FALKON_JOBID_HOME}/client +ln -s ${FALKON_HOME}/monitor ${FALKON_JOBID_HOME}/monitor +ln -s ${FALKON_HOME}/bin ${FALKON_JOBID_HOME}/bin +ln -s ${FALKON_HOME}/config ${FALKON_JOBID_HOME}/config +ln -s ${FALKON_HOME}/ploticus ${FALKON_JOBID_HOME}/ploticus +ln -s ${FALKON_HOME}/webserver ${FALKON_JOBID_HOME}/webserver +ln -s ${FALKON_HOME}/workloads ${FALKON_JOBID_HOME}/workloads +ln -s ${FALKON_HOME}/id ${FALKON_JOBID_HOME}/id +ln -s ${FALKON_HOME}/apache-ant-1.7.0 ${FALKON_JOBID_HOME}/apache-ant-1.7.0 +ln -s ${FALKON_HOME}/ibm-java2-ppc64-50 ${FALKON_JOBID_HOME}/ibm-java2-ppc64-50 +ln -s ${FALKON_HOME_RAM}/ibm-java2-ppc-50 ${FALKON_JOBID_HOME}/ibm-java2-ppc-50 +ln -s ${FALKON_HOME}/falkon.tgz ${FALKON_JOBID_HOME}/falkon.tgz + + +if [ ! -d "${FALKON_JOBID_HOME}" ]; then + echo "ERROR: invalid path ${FALKON_JOBID_HOME}... exiting" + cqdel ${ZOID_JOB_ID} + exit 1 +fi + +echo "Succesfully submitted the job to Cobalt, and setup job specific Falkon tree!" +echo "To monitor the job status, type 'cqstat | grep ${USER}'; once it is in running state, you can use the Falkon specific command ...." +echo "To submit your Falkon-based workload, type: ....; you can do this any time, the falkon workload will wait for the resources to come online, and will only be submitted when everything is ready; the script is run in the background, so the workload will run even if the ssh session gets disconnected." +echo "" +echo "Remember, your job id is ${ZOID_JOB_ID}, and if you need to look through the logs manually for anything, remember that you can find them at ${HOME}/${ZOID_JOB_ID}.output, ${HOME}/${ZOID_JOB_ID}.error, and ${FALKON_JOBID_HOME}/logs/..." + + Property changes on: usertools/cio/falkon/falkon-start-bgp-logging.sh ___________________________________________________________________ Name: svn:executable + * Added: usertools/cio/tools/zoid/logging-script.sh =================================================================== --- usertools/cio/tools/zoid/logging-script.sh (rev 0) +++ usertools/cio/tools/zoid/logging-script.sh 2009-03-30 04:52:04 UTC (rev 2771) @@ -0,0 +1,101 @@ +#!/bin/bash + +# Added GPFS activity logging of IO calls -ame +COBALT_JOBID=`echo $ZOID_JOB_ENV | sed 's/^.*COBALT_JOBID=$[^:]*$/\1/'` + +TARGET_DIR=$HOME/performance/$COBALT_JOBID + +mkdir -p $TARGET_DIR || exit 1 + +echo -n $TARGET_DIR/gpfs.`hostname` >/tmp/gpfs_log_name.new +touch $TARGET_DIR/gpfs.`hostname` || exit 1 +mv /tmp/gpfs_log_name.new /tmp/gpfs_log_name +echo -n $TARGET_DIR/zoid.`hostname` >/tmp/zoid_log_name.new +mv /tmp/zoid_log_name.new /tmp/zoid_log_name +# end addition + +#startup + if [ "${1}" -eq "1" ]; then + +#ERROR_LOG=/home/falkon/users/error.log +ERROR_LOG=/dev/null + + echo "starting zoid-user-script.sh..." >> ${ERROR_LOG} + #echo "reviewing the command and arguements: $0 $1 $2 $3 $4 $5 $6 $7..." >> ${ERROR_LOG} + #tENV=`env` + #echo "reviewing the envirnoment: $tENV..." >> ${ERROR_LOG} + +original=$ZOID_JOB_ARGS +a0=${original%%:*}; rest=${original#*:} +a1=${rest%%:*}; rest=${rest#*:} +a2=${rest%%:*}; rest=${rest#*:} +a3=${rest%%:*}; rest=${rest#*:} +a4=${rest%%:*}; rest=${rest#*:} +a5=${rest%%:*}; rest=${rest#*:} +a6=${rest%%:*}; rest=${rest#*:} +a7=${rest%%:*}; + + + echo "reviewing the command and arguements: $a0 $a1 $a2 $a3 $a4 $a5 $a6 $a7..." >> ${ERROR_LOG} + + +#FALKON_JOBID_HOME=/home/falkon/users/${USER}/${ZOID_JOB_ID} +FALKON_JOBID_HOME=${a7}/users/${a5}/${a6} +FALKON_HOME=${a7}/users/${a5}/${a6} + echo "FALKON_JOBID_HOME: $FALKON_JOBID_HOME" >> ${ERROR_LOG} + + +if [ ! -d "${FALKON_JOBID_HOME}" ]; then + echo "ERROR: invalid path ${FALKON_JOBID_HOME}... exiting" >> ${ERROR_LOG} + exit 1 +fi + + + + MACH_ID=`uname -n` + #this should be done via some configuration parameter + + cd ${FALKON_JOBID_HOME} + #source falkon.env.bgp-io + + echo "ION at ${MACH_ID} ==> ${FALKON_HOME}..." >> ${ERROR_LOG} + + LOG_FILE_DEBUG="${FALKON_HOME}/logs/provisioner/STATUS_ION_${MACH_ID}" + DATE=`date` + echo "${DATE} : falkon-ion-start : starting the I/O node Falkon startup in the background..." >> ${LOG_FILE_DEBUG} + + +if [ ! -e "${FALKON_HOME}/bin/falkon-ion-start-blocking-zeptocn.sh" ]; then + echo "ERROR: invalid script ${FALKON_HOME}/bin/falkon-ion-start-blocking-zeptocn.sh" >> ${ERROR_LOG} + exit 1 +fi + + #used to run from GPFS + #${FALKON_JOBID_HOME}/bin/falkon-ion-start-blocking-zeptocn.sh ${FALKON_JOBID_HOME} & + #used to run from RAM + ${FALKON_JOBID_HOME}/bin/falkon-ion-start-blocking-zeptocn-ram.sh ${FALKON_JOBID_HOME} & + + + EXIT_CODE=$? + DATE=`date` + + echo "${DATE} : falkon-ion-start : completed the I/O node Falkon startup in the background..." >> ${LOG_FILE_DEBUG} + + echo "ION at ${MACH_ID} finished OK!" >> ${ERROR_LOG} + + +exit ${EXIT_CODE} + + + fi + + + +#cleanup + if [ "${1}" -eq "0" ]; then + + killall -9 java + exit 0 + + fi + Property changes on: usertools/cio/tools/zoid/logging-script.sh ___________________________________________________________________ Name: svn:executable + * From noreply at svn.ci.uchicago.edu Mon Mar 30 00:24:57 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 30 Mar 2009 00:24:57 -0500 (CDT) Subject: [Swift-commit] r2773 - in trunk: . src/org/griphyn/vdl/karajan Message-ID: <20090330052457.DBF6F2281AD@www.ci.uchicago.edu> Author: benc Date: 2009-03-30 00:24:57 -0500 (Mon, 30 Mar 2009) New Revision: 2773 Modified: trunk/CHANGES.txt trunk/src/org/griphyn/vdl/karajan/Loader.java Log: Removed support for .dtm file extension which was deprecated in Swift 0.4 Modified: trunk/CHANGES.txt =================================================================== --- trunk/CHANGES.txt 2009-03-30 05:23:56 UTC (rev 2772) +++ trunk/CHANGES.txt 2009-03-30 05:24:57 UTC (rev 2773) @@ -1,3 +1,6 @@ +(03/30/09) +*** Removed support for .dtm file extension which was deprecated in Swift 0.4 + (03/29/09) *** Added console text interface to provide live information about swift runs. Modified: trunk/src/org/griphyn/vdl/karajan/Loader.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/Loader.java 2009-03-30 05:23:56 UTC (rev 2772) +++ trunk/src/org/griphyn/vdl/karajan/Loader.java 2009-03-30 05:24:57 UTC (rev 2773) @@ -122,17 +122,7 @@ System.exit(4); } - /** - * This deprecation should stay until at least after Swift 0.4 is - * released; sometime after that, this test and the subsequent - * support for .dtm extensions should be removed. - */ - if (project.endsWith(".dtm")) { - logger - .warn("Source file extension .dtm is deprecated and will not be supported in future versions. Use .swift extension instead."); - } - - if (project.endsWith(".dtm") || project.endsWith(".swift")) { + if (project.endsWith(".swift")) { try { project = compile(project); } @@ -344,7 +334,7 @@ ap.setExecutableName("swift"); ap .addOption(ArgumentParser.DEFAULT, - "A file (.dtm or .kml) to execute", "file", + "A file (.swift or .kml) to execute", "file", ArgumentParser.OPTIONAL); ap.addFlag(ARG_HELP, "Display usage information"); From noreply at svn.ci.uchicago.edu Mon Mar 30 00:25:50 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 30 Mar 2009 00:25:50 -0500 (CDT) Subject: [Swift-commit] r2774 - trunk/src/org/griphyn/vdl/karajan/lib Message-ID: <20090330052550.C33BA2281AD@www.ci.uchicago.edu> Author: benc Date: 2009-03-30 00:25:50 -0500 (Mon, 30 Mar 2009) New Revision: 2774 Modified: trunk/src/org/griphyn/vdl/karajan/lib/RuntimeStats.java Log: the joy of whitespace Modified: trunk/src/org/griphyn/vdl/karajan/lib/RuntimeStats.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/RuntimeStats.java 2009-03-30 05:24:57 UTC (rev 2773) +++ trunk/src/org/griphyn/vdl/karajan/lib/RuntimeStats.java 2009-03-30 05:25:50 UTC (rev 2774) @@ -19,10 +19,10 @@ proof of concept. */ public class RuntimeStats extends FunctionsCollection { - - public static final String TICKER = "#swift-runtime-progress-ticker"; - public static final String PROGRESS = "#swift-runtime-progress"; + public static final String TICKER = "#swift-runtime-progress-ticker"; + public static final String PROGRESS = "#swift-runtime-progress"; + public static final Arg PA_STATE = new Arg.Positional("state"); public static final int MIN_PERIOD_MS=1000; public static final int MAX_PERIOD_MS=30000; @@ -53,17 +53,17 @@ } public static ProgressTicker getTicker(VariableStack stack) { - return (ProgressTicker) stack.getGlobal(TICKER); - } + return (ProgressTicker) stack.getGlobal(TICKER); + } public static void setProgress(VariableStack stack, RuntimeProgress p) { - stack.parentFrame().setVar(PROGRESS, p); - } - - public static RuntimeProgress getProgress(VariableStack stack) throws VariableNotFoundException { - return (RuntimeProgress) stack.getDeepVar(PROGRESS); - } + stack.parentFrame().setVar(PROGRESS, p); + } + public static RuntimeProgress getProgress(VariableStack stack) throws VariableNotFoundException { + return (RuntimeProgress) stack.getDeepVar(PROGRESS); + } + public Object vdl_startprogressticker(VariableStack stack) throws ExecutionException { ProgressTicker t = new ProgressTicker(); t.setDaemon(true); @@ -160,24 +160,24 @@ } public Map getSummary() { - Map summary = new HashMap(); - synchronized(states) { - Iterator stateIterator = states.iterator(); + Map summary = new HashMap(); + synchronized(states) { + Iterator stateIterator = states.iterator(); - // summarize details of known states into summary, with - // one entry per state type, storing the number of - // jobs in that state. - while(stateIterator.hasNext()) { - String key = ((RuntimeProgress)stateIterator.next()).status; - Integer count = (Integer) summary.get(key); - if(count == null) { - summary.put(key,new Integer(1)); - } else { - summary.put(key,new Integer(count.intValue()+1)); - } - } - } - return summary; + // summarize details of known states into summary, with + // one entry per state type, storing the number of + // jobs in that state. + while(stateIterator.hasNext()) { + String key = ((RuntimeProgress)stateIterator.next()).status; + Integer count = (Integer) summary.get(key); + if(count == null) { + summary.put(key,new Integer(1)); + } else { + summary.put(key,new Integer(count.intValue()+1)); + } + } + } + return summary; } void printStates(String header) { From noreply at svn.ci.uchicago.edu Mon Mar 30 00:26:23 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 30 Mar 2009 00:26:23 -0500 (CDT) Subject: [Swift-commit] r2775 - trunk/docs Message-ID: <20090330052623.C5F4E2281AD@www.ci.uchicago.edu> Author: benc Date: 2009-03-30 00:26:23 -0500 (Mon, 30 Mar 2009) New Revision: 2775 Modified: trunk/docs/userguide.xml Log: document -tui commandline Modified: trunk/docs/userguide.xml =================================================================== --- trunk/docs/userguide.xml 2009-03-30 05:25:50 UTC (rev 2774) +++ trunk/docs/userguide.xml 2009-03-30 05:26:23 UTC (rev 2775) @@ -1439,6 +1439,12 @@ + -tui + + Displays an interactive text mode monitor during a run. + + + In addition, the following Swift properties can be set on the From noreply at svn.ci.uchicago.edu Mon Mar 30 00:26:56 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 30 Mar 2009 00:26:56 -0500 (CDT) Subject: [Swift-commit] r2776 - trunk/docs Message-ID: <20090330052656.969F02281AD@www.ci.uchicago.edu> Author: benc Date: 2009-03-30 00:26:56 -0500 (Mon, 30 Mar 2009) New Revision: 2776 Modified: trunk/docs/userguide.xml Log: split a very long doc line Modified: trunk/docs/userguide.xml =================================================================== --- trunk/docs/userguide.xml 2009-03-30 05:26:23 UTC (rev 2775) +++ trunk/docs/userguide.xml 2009-03-30 05:26:56 UTC (rev 2776) @@ -1423,9 +1423,13 @@ -runid identifier -Specifies the run identifier. This must be unique for every invocation and is used in several places to keep files from different executions cleanly separated. By default, a datestamp and random number are used to generate a run identifier. When using this parameter, care should be taken to ensure -that the run ID remains unique with respect to all other run IDs that might -be used, irrespective of (at least) expected execution sites, program or user. + Specifies the run identifier. This must be unique for every invocation + and is used in several places to keep files from different executions + cleanly separated. By default, a datestamp and random number are used + to generate a run identifier. When using this parameter, care should be + taken to ensure that the run ID remains unique with respect to all + other run IDs that might be used, irrespective of (at least) expected + execution sites, program or user. From noreply at svn.ci.uchicago.edu Mon Mar 30 00:27:31 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 30 Mar 2009 00:27:31 -0500 (CDT) Subject: [Swift-commit] r2777 - trunk/docs Message-ID: <20090330052731.1D61D228223@www.ci.uchicago.edu> Author: benc Date: 2009-03-30 00:27:28 -0500 (Mon, 30 Mar 2009) New Revision: 2777 Modified: trunk/docs/userguide.xml Log: update path for CI users logs Modified: trunk/docs/userguide.xml =================================================================== --- trunk/docs/userguide.xml 2009-03-30 05:26:56 UTC (rev 2776) +++ trunk/docs/userguide.xml 2009-03-30 05:27:28 UTC (rev 2777) @@ -3546,7 +3546,7 @@ -rsync --ignore-existing *.log *.d login.ci.uchicago.edu:/home/benc/swift-logs/ --verbose +rsync --ignore-existing *.log *.d login.ci.uchicago.edu:/disks/ci-gpfs/swift/swift-logs/ --verbose

The Site Catalog - sites.xml From noreply at svn.ci.uchicago.edu Mon Mar 30 11:47:52 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 30 Mar 2009 11:47:52 -0500 (CDT) Subject: [Swift-commit] r2782 - in SwiftApps/LigandAtlas: . dock Message-ID: <20090330164753.022072281AD@www.ci.uchicago.edu> Author: zzhang Date: 2009-03-30 11:47:51 -0500 (Mon, 30 Mar 2009) New Revision: 2782 Added: SwiftApps/LigandAtlas/dock/ SwiftApps/LigandAtlas/dock/dock6.swift SwiftApps/LigandAtlas/dock/dock7.swift Log: swift script on BGP Added: SwiftApps/LigandAtlas/dock/dock6.swift =================================================================== --- SwiftApps/LigandAtlas/dock/dock6.swift (rev 0) +++ SwiftApps/LigandAtlas/dock/dock6.swift 2009-03-30 16:47:51 UTC (rev 2782) @@ -0,0 +1,97 @@ +type Mol {} +type Bin {} +type Common {} +type DOCKOut {} +type Molout{} +type DockRunSummary {} +type file {} + +(external bout)bcast (Bin binary, Common flex_def, Common flex_tbl, Common grid_bmp, Common +grid_in, Common grid_nrg, Common grid_out, Common rec_box, Common receptor, +Common sample_grid, Common selected_spheres, Common template_in, Common vdw, +Common awkscript) +{ + app { + bcast @filename(binary) @filename(flex_def) + @filename(flex_tbl) @filename(grid_bmp) @filename(grid_in) + @filename(grid_nrg) @filename(grid_out) @filename(rec_box) + @filename(receptor) @filename(sample_grid) + @filename(selected_spheres) + @filename(template_in) @filename(vdw) @filename(awkscript); + } +} + +/*app (DOCKOut ofile, Molout moloutfile) rundock_zhao (Mol molfile, Bin binary, Common flex_def, Common +flex_tbl, Common grid_bmp, Common grid_in, Common grid_nrg, Common grid_out, +Common rec_box, Common receptor, Common sample_grid, Common selected_spheres, +Common template_in, Common vdw, Common awkscript)*/ + +app (DOCKOut ofile) rundock_zhao (Mol molfile, Bin binary, Common flex_def, Common +flex_tbl, Common grid_bmp, Common grid_in, Common grid_nrg, Common grid_out, +Common rec_box, Common receptor, Common sample_grid, Common selected_spheres, +Common template_in, Common vdw, Common awkscript, external bout) +{ + rundock_zhao @filename(molfile) @filename(binary) @filename(flex_def) + @filename(flex_tbl) @filename(grid_bmp) @filename(grid_in) + @filename(grid_nrg) @filename(grid_out) @filename(rec_box) + @filename(receptor) @filename(sample_grid) @filename(selected_spheres) + @filename(template_in) @filename(vdw) @filename(awkscript) stdout=@filename(ofile); +} + +app (DockRunSummary finalfile) sumdockresults(DOCKOut r[] ) +{ + summary @filenames(r) stdout=@filename(finalfile); +} + + +Mol texts[] ; +//Mol texts[] ; + + +(DOCKOut result[])doall(Mol texts[]) +{ + Bin binary <"common/dock6.O3.cn">; + Common flex_def <"common/flex.defn">; + Common flex_tbl <"common/flex_drive.tbl">; + Common grid_bmp <"common/grid.bmp">; + Common grid_in <"common/grid.in">; + Common grid_nrg <"common/grid.nrg">; + Common grid_out <"common/grid.out">; + Common rec_box <"common/rec_box.pdb">; + Common receptor <"common/receptor_charged.mol2">; + Common sample_grid <"common/sample_grid.in">; + Common selected_spheres <"common/selected_spheres.sph">; + Common template_in <"common/template.in">; + Common vdw <"common/vdw_AMBER_parm99.defn">; + Common awkscript <"common/awkscript">; + + external bout; + + bout=bcast(binary, flex_def, flex_tbl, grid_bmp, grid_in, grid_nrg, grid_out, rec_box, receptor, sample_grid, selected_spheres, template_in, vdw, awkscript); + + foreach p,i in texts { + /*DOCKOut r ;*/ + + /*Molout mof ;*/ + + result[i] = rundock_zhao(p, binary, flex_def, flex_tbl, grid_bmp, grid_in, + grid_nrg, grid_out, rec_box, receptor, sample_grid, selected_spheres, template_in, vdw, awkscript, bout); + } + + +} + +// Main + +DockRunSummary summary <"summary.txt">; +DOCKOut result[]; +result=doall(texts); +summary = sumdockresults(result); Added: SwiftApps/LigandAtlas/dock/dock7.swift =================================================================== --- SwiftApps/LigandAtlas/dock/dock7.swift (rev 0) +++ SwiftApps/LigandAtlas/dock/dock7.swift 2009-03-30 16:47:51 UTC (rev 2782) @@ -0,0 +1,91 @@ +type Mol {} +type Bin {} +type Common {} +type DOCKOut {} +type Molout{} +type DockRunSummary {} +type file {} + +(external bout)bcast (Bin binary, Common flex_def, Common flex_tbl, Common grid_bmp, Common +grid_in, Common grid_nrg, Common grid_out, Common rec_box, Common receptor, +Common sample_grid, Common selected_spheres, Common template_in, Common vdw, +Common awkscript) +{ + app { + bcast @filename(binary) @filename(flex_def) + @filename(flex_tbl) @filename(grid_bmp) @filename(grid_in) + @filename(grid_nrg) @filename(grid_out) @filename(rec_box) + @filename(receptor) @filename(sample_grid) + @filename(selected_spheres) + @filename(template_in) @filename(vdw) @filename(awkscript); + } +} + +app (DOCKOut ofile, Molout moloutfile) rundock_zhao (Mol molfile, Bin binary, +Common flex_def, Common flex_tbl, Common grid_bmp, Common grid_in, Common +grid_nrg, Common grid_out, Common rec_box, Common receptor, Common sample_grid, Common selected_spheres, +Common template_in, Common vdw, Common awkscript, external bout) +{ + rundock_zhao @filename(molfile) @filename(binary) @filename(flex_def) + @filename(flex_tbl) @filename(grid_bmp) @filename(grid_in) + @filename(grid_nrg) @filename(grid_out) @filename(rec_box) + @filename(receptor) @filename(sample_grid) @filename(selected_spheres) + @filename(template_in) @filename(vdw) @filename(awkscript) stdout=@filename(ofile); +} + +app (DockRunSummary finalfile) sumdockresults(DOCKOut r[] ) +{ + summary @filenames(r) stdout=@filename(finalfile); +} + + +Mol texts[] ; +//Mol texts[] ; + + +(DOCKOut result[])doall(Mol texts[]) +{ + Bin binary <"common/dock6.O3.cn">; + Common flex_def <"common/flex.defn">; + Common flex_tbl <"common/flex_drive.tbl">; + Common grid_bmp <"common/grid.bmp">; + Common grid_in <"common/grid.in">; + Common grid_nrg <"common/grid.nrg">; + Common grid_out <"common/grid.out">; + Common rec_box <"common/rec_box.pdb">; + Common receptor <"common/receptor_charged.mol2">; + Common sample_grid <"common/sample_grid.in">; + Common selected_spheres <"common/selected_spheres.sph">; + Common template_in <"common/template.in">; + Common vdw <"common/vdw_AMBER_parm99.defn">; + Common awkscript <"common/awkscript">; + + external bout; + + bout=bcast(binary, flex_def, flex_tbl, grid_bmp, grid_in, grid_nrg, grid_out, rec_box, receptor, sample_grid, selected_spheres, template_in, vdw, awkscript); + + foreach p,i in texts { + /*DOCKOut r ;*/ + + Molout mof ; + + (result[i], mof) = rundock_zhao(p, binary, flex_def, flex_tbl, grid_bmp, grid_in, grid_nrg, grid_out, rec_box, receptor, sample_grid, selected_spheres, template_in, vdw, awkscript, bout); + } + + +} + +// Main + +DockRunSummary summary <"summary.txt">; +DOCKOut result[]; +result=doall(texts); +summary = sumdockresults(result); From noreply at svn.ci.uchicago.edu Mon Mar 30 11:48:40 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 30 Mar 2009 11:48:40 -0500 (CDT) Subject: [Swift-commit] r2783 - in SwiftApps/LigandAtlas: . swift Message-ID: <20090330164840.6B8C62281AD@www.ci.uchicago.edu> Author: zzhang Date: 2009-03-30 11:48:40 -0500 (Mon, 30 Mar 2009) New Revision: 2783 Added: SwiftApps/LigandAtlas/swift/ SwiftApps/LigandAtlas/swift/run_swift.sh SwiftApps/LigandAtlas/swift/vdl-int.k SwiftApps/LigandAtlas/swift/wrapper.sh Log: changes Zhao has done to swift to make MTIO working Added: SwiftApps/LigandAtlas/swift/run_swift.sh =================================================================== --- SwiftApps/LigandAtlas/swift/run_swift.sh (rev 0) +++ SwiftApps/LigandAtlas/swift/run_swift.sh 2009-03-30 16:48:40 UTC (rev 2783) @@ -0,0 +1,34 @@ +#!/bin/bash + + if [ -z "$4" ]; then + echo "usage: $0 " + exit 1 + fi + +CUR_DIR=`pwd` + + +SERVICE_LIST_FILE=/home/falkon/users/${USER}/${1}/config/Client-service-URIs.config + + +NUM_NODES=$2 +SWIFT_SCRIPT=$3 +WORKING=$4 +let NUM_ION=NUM_NODES/64 + +echo "waiting for at least ${NUM_NODES} nodes to register before submitting workload..." +falkon-wait-for-allocation.sh ${SERVICE_LIST_FILE} ${NUM_ION} + +echo "found at least ${NUM_NODES} registered, submitting workload..." + +rm -rf sites.xml + +echo " +" > sites.xml + +cat $SERVICE_LIST_FILE | awk -v var=$WORKING 'NR>1 {printf "\n\t\n\t\n\t%s\n\t8\n\t1000\n\n\n", NR-2, $(1), var}' >> sites.xml + +echo "" >> sites.xml + +time swift -sites.file ./sites.xml -tc.file ./tc.data $3 Property changes on: SwiftApps/LigandAtlas/swift/run_swift.sh ___________________________________________________________________ Name: svn:executable + * Added: SwiftApps/LigandAtlas/swift/vdl-int.k =================================================================== --- SwiftApps/LigandAtlas/swift/vdl-int.k (rev 0) +++ SwiftApps/LigandAtlas/swift/vdl-int.k 2009-03-30 16:48:40 UTC (rev 2783) @@ -0,0 +1,514 @@ +import("sys.k") +import("task.k") +import("vdl-lib.xml") +/* + * Things that are not exposed to the translated file + */ + +global(LOG:DEBUG, "debug") +global(LOG:INFO, "info") +global(LOG:WARN, "warn") +global(LOG:ERROR, "error") +global(LOG:FATAL, "fatal") + +namespace("vdl" + export( + element(rmdir, [dir, host] + parallelFor(entry, file:list(dir, host=host) + epath := concat(dir, "/", entry) + if( + file:isDirectory(epath, host=host) rmdir(epath, host) + file:remove(epath, host=host) + ) + ) + dir:remove(dir, host=host) + ) + + element(createdirs, [path, dir, host] + dc := dircat(dir, path) + log(LOG:INFO, "START path={path} dir={dir} - Creating directory structure") + + dir:make(dc, host=host) + ) + + element(isDone, [stageout] + sys:and( + for(pv, stageout + [path, var] := each(pv) + vdl:isLogged(var, path) + ) + sys:not(isEmpty(stageout)) + ) + ) + + element(mark, [restarts, err, optional(mapping)] + if( + err for(pv, restarts + [path, var] := each(pv) + vdl:setFutureFault(var, path=path, mapping=mapping) + ) + ) + ) + + element(flatten, [...] + if ( + isEmpty(...) "" + + concat( + for(i, butLast(...), if(isList(i) flatten(i) i), "^") last(...) + ) + ) + ) + + element(checkJobStatus, [rhost, wfdir, jobid, tr, jobdir] + log(LOG:DEBUG, "START jobid={jobid}") + try( + sequential( + /* + * This is a bit of optimization, but I'm not completely + * sure of its correctness. The goal is to both detect + * the presence of the success file and remove it, all + * in one operation. It relies on file:remove() throwing + * an exception if the file is not there. + */ + file:remove("{wfdir}/status/{jobdir}/{jobid}-success", host=rhost) + log(LOG:INFO, "SUCCESS jobid={jobid} - Success file found") + ) + if( + file:exists("{wfdir}/status/{jobdir}/{jobid}-error", host=rhost) then( + log(LOG:INFO, "FAILURE jobid={jobid} - Failure file found") + task:transfer(srchost=rhost, srcdir="{wfdir}/status/{jobdir}", srcfile="{jobid}-error") + error := parallel( + file:remove("{wfdir}/status/{jobdir}/{jobid}-error", host=rhost) + sequential( + str:strip(file:read("{jobid}-error")) + file:remove("{jobid}-error") + ) + ) + throw(error) + ) + else ( + log(LOG:INFO, "NO_STATUS_FILE jobid={jobid} - Both status files are missing") + throw("No status file was found. Check the shared filesystem on {rhost}") + ) + ) + ) + ) + + element(initSharedDir, [rhost] + once(list(rhost, "shared") + vdl:setprogress("Initializing site shared directory") + + log(LOG:INFO, "START host={rhost} - Initializing shared directory") + + wfdir := "{VDL:SCRIPTNAME}-{VDL:RUNID}" + sharedDir := dircat(wfdir, "shared") + dir:make(sharedDir, host=rhost) + transfer(srcdir="{vds.home}/libexec/", srcfile="wrapper.sh", destdir=sharedDir, desthost=rhost) + transfer(srcdir="{vds.home}/libexec/", srcfile="seq.sh", destdir=sharedDir, desthost=rhost) + dir:make(dircat(wfdir, "kickstart"), host=rhost) + dir:make(dircat(wfdir, "status"), host=rhost) + dir:make(dircat(wfdir, "info"), host=rhost) + /*l := list(for(i, sys:range(97, 122), chr(i)), for(i, sys:range(48, 57), chr(i))) + for(one, l, + for(two, l, + dir:make(dircat("{wfdir}/shared/" , str:concat(one , "/" , two)), host=rhost) + ) + ) + for(one, l, + for(two, l, + dir:make(dircat("{wfdir}/info/" , str:concat(one , "/" , two)), host=rhost) + ) + )*/ + wfdir, sharedDir + //we send the cleanup data to vdl:main() + to(cleanup, list(wfdir, rhost)) + log(LOG:INFO, "END host={rhost} - Done initializing shared directory") + ) + ) + + element(inFileDirs, [stageins] + for(file, stageins + reldirname(file) + ) + ) + + element(outFileDirs, [stageouts] + for(pv, stageouts + [path, var] := each(pv) + + file := vdl:filename(vdl:getfield(var, path = path)) + + dirname(file) + ) + ) + + element(inFiles, [stageins] + pathnames(stageins) + ) + + element(outFiles, [stageouts] + for(pv, stageouts + [path, var] := each(pv) + + file := vdl:filename(vdl:getfield(var, path = path)) + + file + ) + ) + + element(fileDirs, [stageins, stageouts] + list( + unique( + inFileDirs(stageins) + outFileDirs(stageouts) + ) + ) + ) + + element(createDirSet, [jobid, destdir, host, dirs] + /* + * Ideally this would be done by creating a tree of the directories + * to be created and (eventually) exploiting the concurrency in that. + */ + log(LOG:INFO, "START jobid={jobid} host={host} - Initializing directory structure") + for(u, dirs + cacheOn(list(u, destdir, host) + createdirs(u, destdir, host) + ) + ) + log(LOG:INFO, "END jobid={jobid} - Done initializing directory structure") + ) + + element(cleanup, [dir, host] + log(LOG:INFO, "START dir={dir} host={host}") + if(vdl:configProperty("sitedir.keep") == "false" + task:execute("/bin/rm", arguments="-rf {dir}", host=host, batch=true) + ) + log(LOG:INFO, "END dir={dir} host={host}") + ) + + element(cleanups, [cleanup] + log(LOG:INFO, "START cleanups={cleanup}") + parallelFor(i, cleanup + [dir, host] := each(i) + try( + cleanup(dir, host) + catch(".*", + log(LOG:DEBUG, "EXCEPTION - Exception caught while cleaning up", exception) + to(warnings, exception("Cleanup on {host} failed", exception)) + ) + ) + ) + ) + + element(cleanupFiles, [files, host] + uParallelFor(r, files + log(LOG:INFO, "Purging ", r, " on ", host) + file:remove(r, host=host) + vdl:cacheFileRemoved(r, host) + ) + ) + + element(doStagein, [jobid, files, dir, host] + log(LOG:INFO, "START jobid={jobid} - Staging in files") + uParallelFor(file, files + provider := provider(file) + srchost := hostname(file) + srcdir := dirname(file) + destdir := dircat(dir, reldirname(file)) + filename := basename(file) + size := file:size("{srcdir}/{filename}", host=srchost, provider=provider) + + vdl:cacheAddAndLockFile(filename, destdir, host, size + cleanupFiles(cacheFilesToRemove, host) + log(LOG:DEBUG, "FILE_STAGE_IN_START file={file} srchost={srchost} srcdir={srcdir} srcname={filename} ", + "desthost={host} destdir={destdir} provider={provider}") + restartOnError(".*", 2 + task:transfer(srcprovider=provider, srchost=srchost, srcfile=filename, + srcdir=srcdir, desthost=host, destdir=destdir) + ) + log(LOG:DEBUG, "FILE_STAGE_IN_END file={file} srchost={srchost} srcdir={srcdir} srcname={filename} ", + "desthost={host} destdir={destdir} provider={provider}") + ) + ) + log(LOG:INFO, "END jobid={jobid} - Staging in finished") + ) + + element(doStageout, [jobid, stageouts, dir, host] + log(LOG:INFO, "START jobid={jobid} - Staging out files") + done := list( + uparallelFor(pv, stageouts + [path, var] := each(pv) + file := vdl:absfilename(vdl:getfield(var, path = path)) + provider := vdl:provider(file) + dhost := vdl:hostname(file) + rdir := dircat(dir, reldirname(file)) + bname := basename(file) + ldir := dirname(file) + fullLocal := dircat(ldir, bname) + fullRemote := dircat(rdir, bname) + + log(LOG:DEBUG, "FILE_STAGE_OUT_START srcname={bname} srcdir={rdir} srchost={host} ", + "destdir={ldir} desthost={dhost} provider={provider}") + //make sure we do have the directory on the client side + /*dir:make(ldir) + restartOnError(".*", 2 + task:transfer(srchost=host, srcfile=bname, + srcdir=rdir, destdir=ldir, desthost=dhost, destprovider=provider) + )*/ + log(LOG:DEBUG, "FILE_STAGE_OUT_END srcname={bname} srcdir={rdir} srchost={host} ", + "destdir={ldir} desthost={dhost} provider={provider}") + + list(bname, rdir, host, file:size(fullLocal)) + ) + ) + uParallelFor(f, done + [bname, rdir, host, size] := each(f) + vdl:cacheAddFile(bname, rdir, host, size + cleanupFiles(cacheFilesToRemove, host) + ) + ) + log(LOG:INFO, "END jobid={jobid} - Staging out finished") + ) + + element(doRestartlog, [restartouts] + uParallelFor(f, restartouts, + [path, var] := each(f) + vdl:logvar(var, path) + ) + ) + + element(graphStuff, [tr, stagein, stageout, err, optional(args)] + if( + vdl:configProperty("pgraph") != "false" then( + errprops := if(err ",color=lightsalmon" "") + tp := vdl:threadPrefix() + to(graph, + concat(str:quote(tp), " [label=", str:quote("{tr}{errprops}"), "]") + ) + for(si, stagein + si := basename(si) + to(graph + concat(str:quote(si), " [shape=parallelogram{errprops}]") + concat(str:quote(si), " -> ", str:quote(tp)) + ) + ) + for(pv, stageout + [path, var] := each(pv) + file := vdl:fileName(vdl:getfield(var, path=path)) + file := basename(file) + label := vdl:niceName(var, path = path) + to(graph + concat(str:quote(file), " [shape=parallelogram,label=", + str:quote("{label}{errprops}"), "]") + concat(str:quote(tp), " -> ", str:quote(file)) + ) + ) + ) + ) + ) + + element(fileSizes, [files] + math:sum( + for(f, files, file:size(file)) + ) + ) + + element(transferStandardFiles, [rhost, tmpdir, jobid, stdout, stderr] + concat( + for(f, list(list("stderr.txt", stderr), list("stdout.txt", stdout)) + [name, file] := each(f) + destfile := "{jobid}-{file}" + nl() + "{name}: " + try( + sequential( + task:transfer(srchost=rhost, srcdir=tmpdir, srcfile=file, + destfile=destfile) + + file:read(destfile) + ) + nl() + ) + maybe(file:remove(destfile)) + ) + ) + ) + + element(transferKickstartRec, [rhost, wfdir, jobid, jobdir] + if(sys:not(file:exists("{VDL:SCRIPTNAME}-{VDL:RUNID}.d")) + task:dir:make("{VDL:SCRIPTNAME}-{VDL:RUNID}.d") + ) + recfile := "{jobid}-kickstart.xml" + srcdir := dircat(concat(wfdir, "/kickstart/"), jobdir) + try( + task:transfer(srchost=rhost, srcdir=srcdir, srcfile=recfile, destdir="{VDL:SCRIPTNAME}-{VDL:RUNID}.d/") + ( + maybe(file:remove(recfile)) + log(LOG:WARN, "Failed to transfer kickstart records from {srcdir}/{rhost}", exception) + ) + ) + recfile + ) + + element(transferWrapperLog, [rhost, wfdir, jobid, jobdir] + if(sys:not(file:exists("{VDL:SCRIPTNAME}-{VDL:RUNID}.d")) + task:dir:make("{VDL:SCRIPTNAME}-{VDL:RUNID}.d") + ) + recfile := "{jobid}-info" + srcdir := dircat(concat(wfdir, "/info/"), jobdir) + try( + task:transfer(srchost=rhost, srcdir=srcdir, srcfile=recfile, destdir="{VDL:SCRIPTNAME}-{VDL:RUNID}.d/") + ( + maybe(file:remove(recfile)) + log(LOG:WARN, "Failed to transfer wrapper log from {srcdir}/{rhost}") + log(LOG:DEBUG, "Exception for wrapper log failure from {srcdir}/{rhost}: ", exception) + ) + ) + recfile + ) + + element(execute2, [tr, optional(arguments, stdin, stdout, stderr), stagein, stageout, restartout, + replicationGroup, replicationChannel] + stagein := list(unique(each(stagein))) + stageout := list(unique(each(stageout))) + allocateHost(rhost, constraints=vdl:jobConstraints(tr) + + [wfdir, sharedDir] := try( + initSharedDir(rhost) + throw(exception("Could not initialize shared directory on {rhost}", exception)) + ) + + uid := uid() + jobdir := concat(substring(uid, from=0,to=1), "/", substring(uid, from=1, to=2)) + jobid := concat(tr, "-", uid) + + log(LOG:DEBUG, "THREAD_ASSOCIATION jobid={jobid} thread={#thread} host={rhost} replicationGroup={replicationGroup}") + vdl:setprogress("Stage in") + tmpdir := dircat(concat(wfdir, "/jobs/", jobdir), jobid) + + stdout := try(stdout, "stdout.txt") + stderr := try(stderr, "stderr.txt") + + kickstart := vdl:kickstart(rhost) + + try( + sequential( + fileDirs := fileDirs(stagein, stageout) + + createDirSet(jobid, sharedDir, rhost, fileDirs) + doStagein(jobid, stagein, sharedDir, rhost) + + log(LOG:DEBUG, "JOB_START jobid={jobid} tr={tr}", maybe(" arguments=", arguments), " tmpdir={tmpdir} host={rhost}") + + vdl:setprogress("Submitting") + vdl:execute("/bin/bash", + list("shared/wrapper.sh", jobid, + "-jobdir", jobdir, + "-e", vdl:executable(tr, rhost), + "-out", stdout, + "-err", stderr, + "-i", maybe(stdin), + "-d", flatten(each(fileDirs)), + "-if", flatten(infiles(stagein)), + "-of", flatten(outfiles(stageout)), + "-k", kickstart, + "-a", maybe(each(arguments))) + directory=wfdir + redirect=false + host=rhost + vdl:tcprofile(tr, rhost) //this gets various app params from the tc, such as environment, walltime, etc + replicationGroup=replicationGroup + replicationChannel=replicationChannel + jobid=jobid + ) + + + log(LOG:DEBUG, "STAGING_OUT jobid={jobid}") + + + /* need to stage the files to upper scratch area in case they are not transfered to another site + before all the files get cleaned out */ + + + vdl:setprogress("Stage out") + doStageout(jobid, stageout, sharedDir, rhost) + doRestartlog(restartout) + if( + kickstart != "" & vdl:configProperty("kickstart.always.transfer") == "true" + discard(transferKickstartRec(rhost, wfdir, jobid, jobdir)) + ) + if( + vdl:configProperty("wrapperlog.always.transfer") == "true" + discard(transferWrapperLog(rhost, wfdir, jobid, jobdir)) + ) + vdl:cacheUnlockFiles(stagein, sharedDir, rhost, cleanupFiles(cacheFilesToRemove, rhost)) + log(LOG:DEBUG, "JOB_END jobid={jobid}") + ) + catch("^Abort$" + log(LOG:DEBUG, "JOB_CANCELED jobid={jobid}") + vdl:cacheUnlockFiles(stagein, sharedDir, rhost, force=false + cleanupFiles(cacheFilesToRemove, rhost) + ) + throw(exception) + ) + catch("^(?!Abort$).*" + vdl:setprogress("Failed but can retry") + log(LOG:DEBUG, "APPLICATION_EXCEPTION jobid={jobid} - Application exception: ", exception) + vdl:cacheUnlockFiles(stagein, sharedDir, rhost, force=false + cleanupFiles(cacheFilesToRemove, rhost) + ) + + outs := transferStandardFiles(rhost, tmpdir, jobid, stdout, stderr) + + discard(transferWrapperLog(rhost, wfdir, jobid, jobdir)) + kickstartRec := if( + kickstart == "" "" + else( + try( + ( + recfile := transferKickstartRec(rhost, wfdir, jobid, jobdir) + "KickstartRecord: {recfile}" + ) + "" + ) + ) + ) + + throw( + exception( + concat( + "Exception in {tr}:", nl(), + maybe("Arguments: {arguments}", nl()), + "Host: {rhost}", nl(), + "Directory: {tmpdir}", + "{outs}", nl(), + "----", nl(), + kickstartRec + ) + exception + ) + ) + ) + ) + ) + ) + + element(generateProvenanceGraph, [gdata] + pgraph := vdl:configProperty("pgraph") + gname := if(pgraph == "true" "{VDL:SCRIPTNAME}-{VDL:RUNID}.dot" pgraph) + file:write(gname + "digraph SwiftProvenance {{", nl() + " graph [", vdl:configProperty("pgraph.graph.options"), "];", nl() + " node [", vdl:configProperty("pgraph.node.options"), "];", nl() + + for(i, gdata + " ", i, nl() + ) + "}", nl() + ) + log(LOG:INFO, "Provenance graph saved in ", gname) + ) + ) +) Added: SwiftApps/LigandAtlas/swift/wrapper.sh =================================================================== --- SwiftApps/LigandAtlas/swift/wrapper.sh (rev 0) +++ SwiftApps/LigandAtlas/swift/wrapper.sh 2009-03-30 16:48:40 UTC (rev 2783) @@ -0,0 +1,288 @@ +# this script must be invoked inside of bash, not plain sh +PATH=/fuse/bin:/fuse/usr/bin:$PATH +infosection() { + echo >& "$INFO" + echo "_____________________________________________________________________________" >& "$INFO" + echo >& "$INFO" + echo " $1" >& "$INFO" + echo "_____________________________________________________________________________" >& "$INFO" + echo >& "$INFO" +} + +info() { + infosection "uname -a" + uname -a 2>&1 >& "$INFO" + infosection "id" + id 2>&1 >& "$INFO" + infosection "env" + env 2>&1 >& "$INFO" + infosection "df" + df 2>&1 >& "$INFO" + infosection "/proc/cpuinfo" + cat /proc/cpuinfo 2>&1 >& "$INFO" + infosection "/proc/meminfo" + cat /proc/meminfo 2>&1 >& "$INFO" + infosection "command line" + echo $COMMANDLINE 2>&1 >& "$INFO" +} + +logstate() { + echo "Progress " `date +"%Y-%m-%d %H:%M:%S"` " $@" >& "$INFO" +} + +log() { + echo "$@" >& "$INFO" +} + +fail() { + EC=$1 + shift + log $@ + info + exit $EC +} + +checkError() { + if [ "$?" != "0" ]; then + fail $@ + fi +} + +checkEmpty() { + if [ "$1" == "" ]; then + shift + fail 254 $@ + fi +} + +getarg() { + NAME=$1 + shift + VALUE="" + SHIFTCOUNT=0 + if [ "$1" == "$NAME" ]; then + shift + let "SHIFTCOUNT=$SHIFTCOUNT+1" + while [ "${1:0:1}" != "-" ] && [ "$#" != "0" ]; do + VALUE="$VALUE $1" + shift + let "SHIFTCOUNT=$SHIFTCOUNT+1" + done + else + fail 254 "Missing $NAME argument" + fi + VALUE="${VALUE:1}" +} + +openinfo() { + exec 3<> $1 + INFO=3 +} + +closeinfo() { + exec 3>&- +} + +cioinput() { + INPUT=$1 + FILEPATH=`dirname $INPUT` + FILENAME=`basename $INPUT` + TYPE=${INPUT%%/*} + echo "INPUT_TYPE: $TYPE" >> /dev/shm/cio + if [ "$TYPE" == "common" ] && [ -e /dev/shm/share/$FILENAME ]; then + echo "cioinput(): link for common input $INPUT" >> /dev/shm/cio + ln -s "/dev/shm/share/$FILENAME" "$DIR/$L" + elif [ "$TYPE" == "_concurrent" ]; then + echo "cioinput(): toruscp for intermediate data $INPUT" >> /dev/shm/cio + mkdir -p $FILEPATH + echo "cioinput(): $DIR/$FILEPATH" >> /dev/shm/cio + /home/zzhang/cio/bin/toruscp.sh $FILENAME $DIR/$FILEPATH + else + echo "cioinput(): copy from GPFS $INPUT" >> /dev/shm/cio + cp "$PWD/shared/$L" "$DIR/$L" + fi +} +ciooutput() { + OUTPUT=$1 + FILEPATH=`dirname $OUTPUT` + FILENAME=`basename $OUTPUT` + TYPE=${OUTPUT%%/*} + echo "OUTPUT_TYPE: $TYPE" >> /dev/shm/cio + + if [ "$TYPE" == "_concurrent" ]; then + echo "ciooutput(): write intermediate data $OUTPUT" >> /dev/shm/cio + /home/zzhang/cio/bin/chtput.rb $FILENAME $RANK + cp $OUTPUT /dev/shm/share/ + else + echo "ciooutput(): write regular data $OUTPUT" >> /dev/shm/cio + #dd if="$OUTPUT" of="$WFDIR/shared/$OUTPUT" bs=128k + echo "$OUTPUT /chirp/multi/${CHIRP_ADD}@stripe/" >> /dev/shm/chirp_add + cp "$OUTPUT" /chirp/multi/${CHIRP_ADD}@stripe/ + fi +} +echo $@ >> /dev/shm/log +RANK=`echo $CONTROL_INIT | awk -F, '{print $4}'` +echo "RANK: $RANK" >> /dev/shm/log +COMMANDLINE=$@ +WFDIR=$PWD +ID=$1 +checkEmpty "$ID" "Missing job ID" + +shift + +getarg "-jobdir" "$@" +JOBDIR=$VALUE +shift $SHIFTCOUNT + +checkEmpty "$JOBDIR" "Missing job directory prefix" +mkdir -p /dev/shm/swift-info/$JOBDIR + +closeinfo +openinfo "/dev/shm/swift-info/$JOBDIR/${ID}-info" +#openinfo "/dev/null" + +logstate "LOG_START" + +getarg "-e" "$@" +EXEC=$VALUE +shift $SHIFTCOUNT + +getarg "-out" "$@" +STDOUT=$VALUE +shift $SHIFTCOUNT + +getarg "-err" "$@" +STDERR=$VALUE +shift $SHIFTCOUNT + +getarg "-i" "$@" +STDIN=$VALUE +shift $SHIFTCOUNT + +getarg "-d" "$@" +DIRS=$VALUE +shift $SHIFTCOUNT + +getarg "-if" "$@" +INF=$VALUE +shift $SHIFTCOUNT + +getarg "-of" "$@" +OUTF=$VALUE +shift $SHIFTCOUNT + +getarg "-k" "$@" +KICKSTART=$VALUE +shift $SHIFTCOUNT + +if [ "$1" == "-a" ]; then + shift +else + fail 254 "Missing arguments (-a option)" +fi + +if [ "X$SWIFT_JOBDIR_PATH" != "X" ]; then + DIR=${SWIFT_JOBDIR_PATH}/$JOBDIR/$ID + COPYNOTLINK=1 +else + DIR=/dev/shm/swift-work/$JOBDIR/$ID + COPYNOTLINK=0 +fi + +PATH=$PATH:/bin:/usr/bin + +if [ "$PATHPREFIX" != "" ]; then +export PATH=$PATHPREFIX:$PATH +fi + +IFS="^" + +logstate "CREATE_JOBDIR" +mkdir -p $DIR + +logstate "CREATE_INPUTDIR" + +for D in $DIRS ; do + mkdir -p "$DIR/$D" + checkError 254 "Failed to create input directory $D" +done + +#cd $DIR +logstate "LINK_INPUTS" +for L in $INF ; do + if [ $COPYNOTLINK = 1 ]; then + cp "$PWD/shared/$L" "$DIR/$L" + checkError 254 "Failed to copy input file $L" + else + cioinput $L + #cp "$PWD/shared/$L" "$DIR/$L" + checkError 254 "Failed to link input file $L" + fi +done + +logstate "EXECUTE" + +cd $DIR + +if [ "$KICKSTART" == "" ]; then + if [ "$STDIN" == "" ]; then + "$EXEC" "$@" 1>"$STDOUT" 2>"$STDERR" + else + "$EXEC" "$@" 1>"$STDOUT" 2>"$STDERR" <"$STDIN" + fi + checkError $? "Exit code $?" +else + if [ ! -f "$KICKSTART" ]; then + fail 254 "The Kickstart executable ($KICKSTART) was not found" + elif [ ! -x "$KICKSTART" ]; then + fail 254 "The Kickstart executable ($KICKSTART) does not have the executable bit set" + else + mkdir -p $WFDIR/kickstart/$JOBDIR + if [ "$STDIN" == "" ]; then + "$KICKSTART" -H -o "$STDOUT" -e "$STDERR" "$TMPEXEC" "$@" 1>kickstart.xml 2>"$STDERR" + else + "$KICKSTART" -H -o "$STDOUT" -i "$STDIN" -e "$STDERR" "$TMPEXEC" "$@" 1>kickstart.xml 2>"$STDERR" + fi + export APPEXIT=$? + mv -f kickstart.xml "$WFDIR/kickstart/$JOBDIR/$ID-kickstart.xml" 2>&1 >& "$INFO" + checkError 254 "Failed to copy Kickstart record to shared directory" + if [ "$APPEXIT" != "0" ]; then + fail $APPEXIT "Exit code $APPEXIT" + fi + fi +fi + +logstate "EXECUTE_DONE" + +MISSING= +for O in $OUTF ; do + if [ ! -f "$DIR/$O" ]; then + if [ "$MISSING" == "" ]; then + MISSING=$O + else + MISSING="$MISSING, $O" + fi + fi +done +if [ "$MISSING" != "" ]; then + fail 254 "The following output files were not created by the application: $MISSING" +fi + +logstate "COPYING_OUTPUTS" +for O in $OUTF ; do + #cp "$DIR/$O" "$WFDIR/shared/$O" 2>&1 >& "$INFO" + #cp "$DIR/$O" "$WFDIR/shared/$O" + #dd if="$DIR/$O" of="$WFDIR/shared/$JOBDIR/$O" bs=128k + #dd if="$DIR/$O" of="$WFDIR/shared/$O" bs=128k + ciooutput $O + checkError 254 "Failed to copy output file $O to shared directory" +done + +logstate "RM_JOBDIR" + +closeinfo +#rm -f "$WFDIR/info/$JOBDIR/${ID}-info" +#echo "$WFDIR/info/$JOBDIR/${ID}-info" >> /dev/shm/log +#mkdir -p "$WFDIR/info/$JOBDIR/" +#dd if=/dev/shm/swift-info/$JOBDIR/${ID}-info of="$WFDIR/info/$JOBDIR/${ID}-info" bs=128k +#dd if=/dev/shm/swift-info/$JOBDIR/${ID}-info of="/fuse/tmp/${ID}-info" bs=128k From noreply at svn.ci.uchicago.edu Mon Mar 30 11:49:57 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 30 Mar 2009 11:49:57 -0500 (CDT) Subject: [Swift-commit] r2784 - in SwiftApps/LigandAtlas: . MTIO MTIO/bin MTIO/lib Message-ID: <20090330164957.935502281AD@www.ci.uchicago.edu> Author: zzhang Date: 2009-03-30 11:49:57 -0500 (Mon, 30 Mar 2009) New Revision: 2784 Added: SwiftApps/LigandAtlas/MTIO/ SwiftApps/LigandAtlas/MTIO/bin/ SwiftApps/LigandAtlas/MTIO/bin/.__acl SwiftApps/LigandAtlas/MTIO/bin/bashrc SwiftApps/LigandAtlas/MTIO/bin/bcast.sh SwiftApps/LigandAtlas/MTIO/bin/broadcast.sh SwiftApps/LigandAtlas/MTIO/bin/chtclient.rb SwiftApps/LigandAtlas/MTIO/bin/chtget.rb SwiftApps/LigandAtlas/MTIO/bin/chtput.rb SwiftApps/LigandAtlas/MTIO/bin/cn-ipfwd.2409 SwiftApps/LigandAtlas/MTIO/bin/cnip-work.sh SwiftApps/LigandAtlas/MTIO/bin/cnip.sh SwiftApps/LigandAtlas/MTIO/bin/collector.sh SwiftApps/LigandAtlas/MTIO/bin/hashserver.rb SwiftApps/LigandAtlas/MTIO/bin/ipcal SwiftApps/LigandAtlas/MTIO/bin/torus-falkon-start-bgp-ram.sh SwiftApps/LigandAtlas/MTIO/bin/toruscp.sh SwiftApps/LigandAtlas/MTIO/bin/wrapper.sh SwiftApps/LigandAtlas/MTIO/bin/zoid-user-script.sh SwiftApps/LigandAtlas/MTIO/lib/ SwiftApps/LigandAtlas/MTIO/lib/libcrypt.so.1 SwiftApps/LigandAtlas/MTIO/lib/libnsl.so.1 SwiftApps/LigandAtlas/MTIO/lib/libresolv.so.2 Log: The collection of APIs that MTIO is built on. Added: SwiftApps/LigandAtlas/MTIO/bin/.__acl =================================================================== --- SwiftApps/LigandAtlas/MTIO/bin/.__acl (rev 0) +++ SwiftApps/LigandAtlas/MTIO/bin/.__acl 2009-03-30 16:49:57 UTC (rev 2784) @@ -0,0 +1,2 @@ +address:192.168.1.* rwlda +address:10.* rwlda Added: SwiftApps/LigandAtlas/MTIO/bin/bashrc =================================================================== --- SwiftApps/LigandAtlas/MTIO/bin/bashrc (rev 0) +++ SwiftApps/LigandAtlas/MTIO/bin/bashrc 2009-03-30 16:49:57 UTC (rev 2784) @@ -0,0 +1,3 @@ +export PATH=/home/zzhang/chirp/bin:$PATH +export PATH=/home/zzhang/ruby-1.8.7-p72/bin/bin:$PATH +PATH=/fuse/bin:/fuse/usr/bin:$PATH \ No newline at end of file Added: SwiftApps/LigandAtlas/MTIO/bin/bcast.sh =================================================================== --- SwiftApps/LigandAtlas/MTIO/bin/bcast.sh (rev 0) +++ SwiftApps/LigandAtlas/MTIO/bin/bcast.sh 2009-03-30 16:49:57 UTC (rev 2784) @@ -0,0 +1,16 @@ +#!/fuse/bin/bash + +export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/zzhang/cio/lib +PATH=/fuse/bin:/fuse/usr/bin:$PATH + +IP=`/sbin/ifconfig | grep inet | tail -n 1 | cut -d ':' -f 2 |awk '{print $1}'` +#mkdir -p /dev/shm/share/common +#cp -r common/* /dev/shm/share/ +#chmod -R 755 /dev/shm/share/common +#exit 1 +# tree network path +DESTHOSTS=`seq 0 63 | sed "s/^/10.128.0./" | xargs` +echo ${DESTHOSTS/$IP/" "} +/home/zzhang/chirp/bin/chirp_distribute -a address -D 127.0.0.1 / ${DESTHOSTS/$IP/" "} + +exit 0 \ No newline at end of file Property changes on: SwiftApps/LigandAtlas/MTIO/bin/bcast.sh ___________________________________________________________________ Name: svn:executable + * Added: SwiftApps/LigandAtlas/MTIO/bin/broadcast.sh =================================================================== --- SwiftApps/LigandAtlas/MTIO/bin/broadcast.sh (rev 0) +++ SwiftApps/LigandAtlas/MTIO/bin/broadcast.sh 2009-03-30 16:49:57 UTC (rev 2784) @@ -0,0 +1,16 @@ +#!/fuse/bin/bash + +export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/zzhang/cio/lib +PATH=/fuse/bin:/fuse/usr/bin:$PATH + +IP=`/sbin/ifconfig | grep inet | tail -n 1 | cut -d ':' -f 2 |awk '{print $1}'` +#mkdir -p /dev/shm/share/common +cp -r common/* /dev/shm/share/ +#chmod -R 755 /dev/shm/share/common +#exit 1 +# tree network path +DESTHOSTS=`seq 0 63 | sed "s/^/10.128.0./" | xargs` +#DESTHOSTS1=`seq 0 255 | sed "s/^/10.128.1./" | xargs` +echo ${DESTHOSTS/$IP/" "} +echo start +/home/zzhang/chirp/bin/chirp_distribute -a address -D 127.0.0.1 / ${DESTHOSTS/$IP/" "} Property changes on: SwiftApps/LigandAtlas/MTIO/bin/broadcast.sh ___________________________________________________________________ Name: svn:executable + * Added: SwiftApps/LigandAtlas/MTIO/bin/chtclient.rb =================================================================== --- SwiftApps/LigandAtlas/MTIO/bin/chtclient.rb (rev 0) +++ SwiftApps/LigandAtlas/MTIO/bin/chtclient.rb 2009-03-30 16:49:57 UTC (rev 2784) @@ -0,0 +1,75 @@ +#!/home/espinosa/local/bin/ruby + +require 'drb' + +DRb.start_service +cht = DRbObject.new(nil, 'druby://192.168.1.64:9000') + +cht.put("/x/0", 0) +cht.put("/x/1", 1) +cht.put("/x/2", 2) +cht.put("/x/3", 3) +cht.put("/x/4", 4) +cht.put("/x/5", 5) +cht.put("/x/6", 6) +cht.put("/x/7", 7) +cht.put("/x/8", 8) +cht.put("/x/9", 9) +cht.put("/x/10", 10) +cht.put("/x/11", 11) +cht.put("/x/12", 12) +cht.put("/x/13", 13) +cht.put("/x/14", 14) +cht.put("/x/15", 15) +cht.put("/x/16", 16) +cht.put("/x/17", 17) +cht.put("/x/18", 18) +cht.put("/x/19", 19) +cht.put("/x/20", 20) +cht.put("/x/21", 21) +cht.put("/x/22", 22) +cht.put("/x/23", 23) +cht.put("/x/24", 24) +cht.put("/x/25", 25) +cht.put("/x/26", 26) +cht.put("/x/27", 27) +cht.put("/x/28", 28) +cht.put("/x/29", 29) +cht.put("/x/20", 20) +cht.put("/x/31", 31) +cht.put("/x/32", 32) +cht.put("/x/33", 33) +cht.put("/x/34", 34) +cht.put("/x/35", 35) +cht.put("/x/36", 36) +cht.put("/x/37", 37) +cht.put("/x/38", 38) +cht.put("/x/39", 39) +cht.put("/x/40", 40) +cht.put("/x/41", 41) +cht.put("/x/42", 42) +cht.put("/x/43", 43) +cht.put("/x/44", 44) +cht.put("/x/45", 45) +cht.put("/x/46", 46) +cht.put("/x/47", 47) +cht.put("/x/48", 48) +cht.put("/x/49", 49) +cht.put("/x/50", 50) +cht.put("/x/51", 51) +cht.put("/x/52", 52) +cht.put("/x/53", 53) +cht.put("/x/54", 54) +cht.put("/x/55", 55) +cht.put("/x/56", 56) +cht.put("/x/57", 57) +cht.put("/x/58", 58) +cht.put("/x/59", 59) +cht.put("/x/60", 60) +cht.put("/x/61", 61) +cht.put("/x/62", 62) +cht.put("/x/63", 63) +#ARGV.each do|a| +# puts "Argument: #{a}" +# puts cht.get"#{a}" +#end Property changes on: SwiftApps/LigandAtlas/MTIO/bin/chtclient.rb ___________________________________________________________________ Name: svn:executable + * Added: SwiftApps/LigandAtlas/MTIO/bin/chtget.rb =================================================================== --- SwiftApps/LigandAtlas/MTIO/bin/chtget.rb (rev 0) +++ SwiftApps/LigandAtlas/MTIO/bin/chtget.rb 2009-03-30 16:49:57 UTC (rev 2784) @@ -0,0 +1,16 @@ +#!/home/espinosa/local/bin/ruby + +require 'drb' + +DRb.start_service +cht = DRbObject.new(nil, 'druby://10.128.0.0:9000') + +#cht.put("/x/y", 0) +#cht.put("/x/z", 0) +#cht.put("/x/a", 3) +#cht.put("/x/b", 2) +#cht.put("/x/c", 2) +#cht.put("/x/d", 2) + + puts cht.get(ARGV[0]) + Property changes on: SwiftApps/LigandAtlas/MTIO/bin/chtget.rb ___________________________________________________________________ Name: svn:executable + * Added: SwiftApps/LigandAtlas/MTIO/bin/chtput.rb =================================================================== --- SwiftApps/LigandAtlas/MTIO/bin/chtput.rb (rev 0) +++ SwiftApps/LigandAtlas/MTIO/bin/chtput.rb 2009-03-30 16:49:57 UTC (rev 2784) @@ -0,0 +1,16 @@ +#!/home/espinosa/local/bin/ruby + +require 'drb' + +DRb.start_service +cht = DRbObject.new(nil, 'druby://10.128.0.0:9000') + +#cht.put("/x/y", 0) +#cht.put("/x/z", 0) +#cht.put("/x/a", 3) +#cht.put("/x/b", 2) +#cht.put("/x/c", 2) +puts ARGV[0] +puts ARGV[1] +cht.put(ARGV[0], ARGV[1]) + Property changes on: SwiftApps/LigandAtlas/MTIO/bin/chtput.rb ___________________________________________________________________ Name: svn:executable + * Added: SwiftApps/LigandAtlas/MTIO/bin/cn-ipfwd.2409 =================================================================== (Binary files differ) Property changes on: SwiftApps/LigandAtlas/MTIO/bin/cn-ipfwd.2409 ___________________________________________________________________ Name: svn:executable + * Name: svn:mime-type + application/octet-stream Added: SwiftApps/LigandAtlas/MTIO/bin/cnip-work.sh =================================================================== --- SwiftApps/LigandAtlas/MTIO/bin/cnip-work.sh (rev 0) +++ SwiftApps/LigandAtlas/MTIO/bin/cnip-work.sh 2009-03-30 16:49:57 UTC (rev 2784) @@ -0,0 +1,69 @@ +#!/fuse/bin/bash +LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib +PATH=/fuse/bin:/fuse/usr/bin:$PATH + +#Torus START +/home/iskra/ZeptoOS/packages/cnip/prebuilt/cn-ipfwd.2409 & + +while [ ! -f /tmp/ifconfig.cmd ]; do + sleep 1 +done + +. /tmp/ifconfig.cmd + +#IP Calculation +RANK=`echo $CONTROL_INIT | awk -F, '{print $4}'` +FIRST=`expr $RANK % 256` +SECOND=`expr $RANK / 256` + +BASE=`expr $FIRST / 64` +MODULO=`expr $FIRST % 64` +CHIRP_BASE=`expr $BASE \* 64` +CHIRP_1=`expr $CHIRP_BASE + 1` +CHIRP_2=`expr $CHIRP_BASE + 2` +CHIRP_3=`expr $CHIRP_BASE + 3` +CHIRP_4=`expr $CHIRP_BASE + 4` +export CHIRP_ADD=10.128.$SECOND.$CHIRP_1 +#Chirp START +if [ -d /dev/shm/share ]; + then + rm -rf /dev/shm/share +fi +mkdir /dev/shm/share +echo "address:192.168.1.* rwlda" >>/dev/shm/share/.__acl +echo "address:10.* rwlda" >> /dev/shm/share/.__acl +echo "address:127.0.0.1 rwlda" >> /dev/shm/share/.__acl + +/home/zzhang/chirp/bin/chirp_server -r /dev/shm/share & + +#MAIN +if [ "$RANK" = "0" ] +then + /home/zzhang/cio/bin/hashserver.rb & +elif [ "$MODULO" = "1" ] +then + cd /dev/shm/share + mkdir stripe + cd stripe + cp ../.__acl . + mkdir root + cp .__acl root/ + echo 10.128.$SECOND.$CHIRP_1 >> hosts + echo 10.128.$SECOND.$CHIRP_2 >> hosts + echo 10.128.$SECOND.$CHIRP_3 >> hosts + echo 10.128.$SECOND.$CHIRP_4 >> hosts + echo bigvolfiles > key + + mkdir /chirp + /home/zzhang/chirp/bin/chirp_fuse -a address /chirp + /home/zzhang/cio/bin/collector.sh & +elif [ "$MODULO" = "2" ] || [ "$MODULO" = "3" ] || [ "$MODULO" = "4" ] +then + sleep 3600 +else + mkdir /chirp + /home/zzhang/chirp/bin/chirp_fuse -a address /chirp + /home/zzhang/falkon/worker/run.worker-c-bgp.sh $1 $2 $3 $4 $5 $6 $7 +fi + +sleep 3600 Property changes on: SwiftApps/LigandAtlas/MTIO/bin/cnip-work.sh ___________________________________________________________________ Name: svn:executable + * Added: SwiftApps/LigandAtlas/MTIO/bin/cnip.sh =================================================================== --- SwiftApps/LigandAtlas/MTIO/bin/cnip.sh (rev 0) +++ SwiftApps/LigandAtlas/MTIO/bin/cnip.sh 2009-03-30 16:49:57 UTC (rev 2784) @@ -0,0 +1,34 @@ +#!/fuse/bin/bash +LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib +PATH=/fuse/bin:/fuse/usr/bin:$PATH + +/home/iskra/ZeptoOS/packages/cnip/prebuilt/cn-ipfwd.2409 & + +while [ ! -f /tmp/ifconfig.cmd ]; do + sleep 1 +done + +. /tmp/ifconfig.cmd + +RANK=`echo $CONTROL_INIT | awk -F, '{print $4}'` + +if [ -d /dev/shm/share ]; + then + rm -rf /dev/shm/share +fi +mkdir /dev/shm/share +echo "address:192.168.1.* rwlda" >>/dev/shm/share/.__acl +echo "address:10.* rwlda" >> /dev/shm/share/.__acl +echo "address:127.0.0.1 rwlda" >> /dev/shm/share/.__acl + +/home/zzhang/chirp/bin/chirp_server -r /dev/shm/share & + +if [ "$RANK" = "0" ]; +then + /home/zzhang/cio/bin/hashserver.rb & +else + /home/zzhang/falkon/worker/run.worker-c-bgp.sh $1 $2 $3 $4 $5 $6 $7 +fi + +echo $RANK > /dev/shm/rank +sleep 3600 Property changes on: SwiftApps/LigandAtlas/MTIO/bin/cnip.sh ___________________________________________________________________ Name: svn:executable + * Added: SwiftApps/LigandAtlas/MTIO/bin/collector.sh =================================================================== --- SwiftApps/LigandAtlas/MTIO/bin/collector.sh (rev 0) +++ SwiftApps/LigandAtlas/MTIO/bin/collector.sh 2009-03-30 16:49:57 UTC (rev 2784) @@ -0,0 +1,60 @@ +#!/fuse/bin/bash +LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib +PATH=/fuse/bin:/fuse/usr/bin:$PATH + +i=0 +RANK=`echo $CONTROL_INIT | awk -F, '{print $4}'` +IOPATH=`cat /fuse/tmp/iohost` +CHIRP_ADDR=`echo $CHIRP_ADD` +LOG=/dev/shm/collector +echo "CHIRP: $CHIRP_ADD" >> $LOG 2>&1 +echo "START RANK: $RANK" >> $LOG 2>&1 +function send() +{ + ext=$1 + cd /chirp/multi/${CHIRP_ADDR}@stripe/ + time tar cf /tmp/${RANK}_${i}_${ext}.tar `ls /dev/shm/share/stripe/root/` >> $LOG 2>&1 + DATE=`date` + echo "${DATE}: ${i}th tar done with exit code $?" >> $LOG 2>&1 + time dd if=/tmp/${RANK}_${i}_${ext}.tar of=${IOPATH}/${RANK}_${i}_${ext}.tar bs=128k >> $LOG 2>&1 + DATE=`date` + echo "${DATE}: ${i}th copy done with exit code $?" >> $LOG 2>&1 + rm /tmp/${MACH_ID}_${i}_${ext}.tar + i=$[$i+1] +} + +function loopminute() +{ + while true; do + sleep 60 + + num=`ls -l /dev/shm/share/stripe/root/ | wc -l` >> $LOG 2>&1 + DATE=`date` + echo "${DATE}: there are $num result files in output" >> $LOG 2>&1 + if [ "${num}" -gt "1" ]; then + send minute + fi + DATE=`date` + echo "${DATE}: loopminute over" >> $LOG 2>&1 + done +} + +function loopsecond() +{ + while true; do + sleep 1 + filesize=`ls -l /dev/shm/share/stripe/root/ | awk 'NR>1 {a[1]=a[1]+$5} END{print a[1]}'` + if [ "${filesize}" -gt "268000000" ]; then + DATE=`date` + echo "${DATE}: $filesize reaches limit" >> $LOG 2>&1 + mkdir ${FLAG} + send seconds + rmdir ${FLAG} + DATE=`date` + echo "${DATE}: $filesize reaches limit over" >> $LOG 2>&1 + fi + done +} + +#loopsecond & +loopminute \ No newline at end of file Property changes on: SwiftApps/LigandAtlas/MTIO/bin/collector.sh ___________________________________________________________________ Name: svn:executable + * Added: SwiftApps/LigandAtlas/MTIO/bin/hashserver.rb =================================================================== --- SwiftApps/LigandAtlas/MTIO/bin/hashserver.rb (rev 0) +++ SwiftApps/LigandAtlas/MTIO/bin/hashserver.rb 2009-03-30 16:49:57 UTC (rev 2784) @@ -0,0 +1,26 @@ +#!/home/espinosa/local/bin/ruby + +require 'drb' + +class HashServer + attr_reader :file + + def initialize + @file = {} + end + + def put(fname, rank) + @file[fname] = rank + end + + def get(fname) + @file[fname] + end + +end + +server = HashServer.new + +DRb.start_service('druby://*:9000', server) +DRb.thread.join + Property changes on: SwiftApps/LigandAtlas/MTIO/bin/hashserver.rb ___________________________________________________________________ Name: svn:executable + * Added: SwiftApps/LigandAtlas/MTIO/bin/ipcal =================================================================== (Binary files differ) Property changes on: SwiftApps/LigandAtlas/MTIO/bin/ipcal ___________________________________________________________________ Name: svn:executable + * Name: svn:mime-type + application/octet-stream Added: SwiftApps/LigandAtlas/MTIO/bin/torus-falkon-start-bgp-ram.sh =================================================================== --- SwiftApps/LigandAtlas/MTIO/bin/torus-falkon-start-bgp-ram.sh (rev 0) +++ SwiftApps/LigandAtlas/MTIO/bin/torus-falkon-start-bgp-ram.sh 2009-03-30 16:49:57 UTC (rev 2784) @@ -0,0 +1,174 @@ +#!/bin/bash + + + if [ -z "$3" ]; then + echo "usage: $0 " + echo "usage: $0 prod 1024 60" + echo "-- or --" + echo "usage: $0 " + echo "usage: $0 prod 1024 60 4" + exit 1 + fi + + +if [ -z "${FALKON_HOME}" ]; then + echo "ERROR: environment variable FALKON_HOME not defined" + exit 1 +fi + + +QUEUE_NAME=$1 +PROFILE_NAME="zeptocn" +NUM_NODES=$2 +let NUM_ION=NUM_NODES/64 +MAX_TIME_MIN=$3 +SERVICE_IP="192.168.1.254" +SERVICE_PORT1=55000 +SERVICE_PORT2=55001 +WORKERS_PER_NODE=4 +if [ ! -z $4 ];then + WORKERS_PER_NODE=$4 +fi + + +cp /home/zzhang/cio/bin/zoid-user-script.sh ${HOME}/zoid-user-script.sh +chmod +x ${HOME}/zoid-user-script.sh + +FALKON_JOB_ID=`falkon-id-get.sh N/A` +EXIT_CODE=$? + + + if [ "${EXIT_CODE}" -ne "0" ]; then + echo "Error in geting a unique falkon ID.. 'falkon-get-id.sh N/A'" + cqdel ${ZOID_JOB_ID} + exit ${EXIT_CODE} + fi + + + +#ZOID_JOB_ID=`cqsub -q ${QUEUE_NAME} -k ${PROFILE_NAME} -C ${HOME} -t ${MAX_TIME_MIN} -n ${NUM_NODES} -e LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib /bgsys/linux/1.2.020080512/bin/bash /home/zzhang/falkon/worker/run.worker-c-bgp.sh ${SERVICE_IP} ${SERVICE_PORT1} ${SERVICE_PORT2} ${WORKERS_PER_NODE} ${USER} ${FALKON_JOB_ID} ${FALKON_ROOT}` + +ZOID_JOB_ID=`cqsub -q ${QUEUE_NAME} -k ${PROFILE_NAME} -C ${HOME} -t ${MAX_TIME_MIN} -n ${NUM_NODES} -e LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib /bgsys/linux/1.2.020080512/bin/bash /home/zzhang/cio/bin/cnip-work.sh ${SERVICE_IP} ${SERVICE_PORT1} ${SERVICE_PORT2} ${WORKERS_PER_NODE} ${USER} ${FALKON_JOB_ID} ${FALKON_ROOT}` + +#ZOID_JOB_ID=`cqsub -q ${QUEUE_NAME} -k ${PROFILE_NAME} -C ${HOME} -t ${MAX_TIME_MIN} -n ${NUM_NODES} -e LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib /bin/sleep 1800` + +EXIT_CODE=$? + + if [ "${EXIT_CODE}" -ne "0" ]; then + echo "Error in submitting job to Cobalt.. 'cqsub -q ${QUEUE_NAME} -k ${PROFILE_NAME} -C ${HOME} -t ${MAX_TIME_MIN} -n ${NUM_NODES} -e LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib /bgsys/linux/1.2.020080512/bin/bash /fuse/${FALKON_WORKER_HOME}/run.worker-c-bgp.sh ${SERVICE_IP} ${SERVICE_PORT1} ${SERVICE_PORT2} ${WORKERS_PER_NODE} ${USER} ${FALKON_JOB_ID} ${FALKON_ROOT}' " + exit ${EXIT_CODE} + fi + +falkon-id-update.sh ${FALKON_JOB_ID} ${ZOID_JOB_ID} + +EXIT_CODE=$? + + if [ "${EXIT_CODE}" -ne "0" ]; then + echo "Error in updating cobalt job info for falkon.. 'falkon-update-id.sh ${ZOID_JOB_ID}'" + cqdel ${ZOID_JOB_ID} + exit ${EXIT_CODE} + fi + + +FALKON_JOBID_HOME=${FALKON_ROOT}/users/${USER}/${FALKON_JOB_ID} + +echo "Submitted job ${ZOID_JOB_ID} to Cobalt, creating the job specific Falkon tree for logs and configuration in ${FALKON_JOBID_HOME}..." + +mkdir -p ${FALKON_JOBID_HOME} + +cp ${FALKON_HOME}/falkon.env.bgp* ${FALKON_JOBID_HOME}/ +cp -r ${FALKON_HOME}/config ${FALKON_JOBID_HOME}/ +cp ${FALKON_JOBID_HOME}/config/Client-service-URIs.config2 ${FALKON_JOBID_HOME}/config/Client-service-URIs.config +mkdir -p ${FALKON_JOBID_HOME}/logs/client ${FALKON_JOBID_HOME}/logs/service ${FALKON_JOBID_HOME}/logs/provisioner ${FALKON_JOBID_HOME}/logs/worker + + +DATE=`date +%s` +echo "$DATE: pre-creating log dirs for Falkon service..." +RACK_START=0 +RACK_END=48 +SEQUENCE_DIR=`seq -w ${RACK_START} ${RACK_END}` +PSET_START=1 +PSET_END=16 +for a in ${SEQUENCE_DIR} +do + for ((b=${PSET_START}; b <= ${PSET_END} ; b++)) # Double parentheses, and "LIMIT" with no "$". + do + DIR_NAME="ion-R${a}-${b}" + mkdir -p ${FALKON_JOBID_HOME}/logs/service/$DIR_NAME + done +done + +for ((b=${PSET_START}; b <= ${PSET_END} ; b++)) # Double parentheses, and "LIMIT" with no "$". +do + DIR_NAME="ion-${b}" + mkdir -p ${FALKON_JOBID_HOME}/logs/service/$DIR_NAME +done + + +DATE=`date +%s` +echo "$DATE: done creating log dirs for Falkon service!" + + + +DATE=`date +%s` +echo "$DATE: pre-creating data dirs for Falkon service..." +RACK_START=0 +RACK_END=48 +SEQUENCE_DIR=`seq -w ${RACK_START} ${RACK_END}` +PSET_START=1 +PSET_END=16 +for a in ${SEQUENCE_DIR} +do + for ((b=${PSET_START}; b <= ${PSET_END} ; b++)) # Double parentheses, and "LIMIT" with no "$". + do + DIR_NAME="ion-R${a}-${b}" + mkdir -p ${FALKON_JOBID_HOME}/data/$DIR_NAME + done +done + +for ((b=${PSET_START}; b <= ${PSET_END} ; b++)) # Double parentheses, and "LIMIT" with no "$". +do + DIR_NAME="ion-${b}" + mkdir -p ${FALKON_JOBID_HOME}/data/$DIR_NAME +done + + +DATE=`date +%s` +echo "$DATE: done creating data dirs for Falkon service!" + + + +FALKON_HOME_RAM=/tmp/${USER}/falkon + +ln -s ${FALKON_HOME}/apps ${FALKON_JOBID_HOME}/apps +ln -s ${FALKON_HOME_RAM}/container ${FALKON_JOBID_HOME}/container +ln -s ${FALKON_HOME}/service ${FALKON_JOBID_HOME}/service +ln -s ${FALKON_HOME}/worker ${FALKON_JOBID_HOME}/worker +ln -s ${FALKON_HOME}/AstroPortal ${FALKON_JOBID_HOME}/AstroPortal +ln -s ${FALKON_HOME}/client ${FALKON_JOBID_HOME}/client +ln -s ${FALKON_HOME}/monitor ${FALKON_JOBID_HOME}/monitor +ln -s ${FALKON_HOME}/bin ${FALKON_JOBID_HOME}/bin +ln -s ${FALKON_HOME}/config ${FALKON_JOBID_HOME}/config +ln -s ${FALKON_HOME}/ploticus ${FALKON_JOBID_HOME}/ploticus +ln -s ${FALKON_HOME}/webserver ${FALKON_JOBID_HOME}/webserver +ln -s ${FALKON_HOME}/workloads ${FALKON_JOBID_HOME}/workloads +ln -s ${FALKON_HOME}/id ${FALKON_JOBID_HOME}/id +ln -s ${FALKON_HOME}/apache-ant-1.7.0 ${FALKON_JOBID_HOME}/apache-ant-1.7.0 +ln -s ${FALKON_HOME}/ibm-java2-ppc64-50 ${FALKON_JOBID_HOME}/ibm-java2-ppc64-50 +ln -s ${FALKON_HOME_RAM}/ibm-java2-ppc-50 ${FALKON_JOBID_HOME}/ibm-java2-ppc-50 +ln -s ${FALKON_HOME}/falkon.tgz ${FALKON_JOBID_HOME}/falkon.tgz + + +if [ ! -d "${FALKON_JOBID_HOME}" ]; then + echo "ERROR: invalid path ${FALKON_JOBID_HOME}... exiting" + cqdel ${ZOID_JOB_ID} + exit 1 +fi + +echo "Succesfully submitted the job to Cobalt, and setup job specific Falkon tree!" +echo "To monitor the job status, type 'cqstat | grep ${USER}'; once it is in running state, you can use the Falkon specific command ...." +echo "To submit your Falkon-based workload, type: ....; you can do this any time, the falkon workload will wait for the resources to come online, and will only be submitted when everything is ready; the script is run in the background, so the workload will run even if the ssh session gets disconnected." +echo "" +echo "Remember, your job id is ${ZOID_JOB_ID}, and if you need to look through the logs manually for anything, remember that you can find them at ${HOME}/${ZOID_JOB_ID}.output, ${HOME}/${ZOID_JOB_ID}.error, and ${FALKON_JOBID_HOME}/logs/..." + + Property changes on: SwiftApps/LigandAtlas/MTIO/bin/torus-falkon-start-bgp-ram.sh ___________________________________________________________________ Name: svn:executable + * Added: SwiftApps/LigandAtlas/MTIO/bin/toruscp.sh =================================================================== --- SwiftApps/LigandAtlas/MTIO/bin/toruscp.sh (rev 0) +++ SwiftApps/LigandAtlas/MTIO/bin/toruscp.sh 2009-03-30 16:49:57 UTC (rev 2784) @@ -0,0 +1,18 @@ +#!/fuse/bin/bash + +if [ -z "$2" ]; then + echo "usage: $0 " + echo "usage: $0 hello /dev/shm" + exit 1 +fi + +FILENAME=$1 +FILEPATH=$2 +echo "Full name : $FILENAME" +FILE=`basename $FILENAME` +echo "FILE name: $FILENAME" +RANK=`/home/zzhang/cio/bin/chtget.rb $FILENAME` +echo "RANK: $RANK" +IP=`/home/zzhang/cio/bin/ipcal 10.128.0.0 $RANK` +echo "IP: $IP" +/home/zzhang/chirp/bin/chirp_get -a address $IP $FILE $FILEPATH/$FILE Property changes on: SwiftApps/LigandAtlas/MTIO/bin/toruscp.sh ___________________________________________________________________ Name: svn:executable + * Added: SwiftApps/LigandAtlas/MTIO/bin/wrapper.sh =================================================================== --- SwiftApps/LigandAtlas/MTIO/bin/wrapper.sh (rev 0) +++ SwiftApps/LigandAtlas/MTIO/bin/wrapper.sh 2009-03-30 16:49:57 UTC (rev 2784) @@ -0,0 +1,288 @@ +# this script must be invoked inside of bash, not plain sh +PATH=/fuse/bin:/fuse/usr/bin:$PATH +infosection() { + echo >& "$INFO" + echo "_____________________________________________________________________________" >& "$INFO" + echo >& "$INFO" + echo " $1" >& "$INFO" + echo "_____________________________________________________________________________" >& "$INFO" + echo >& "$INFO" +} + +info() { + infosection "uname -a" + uname -a 2>&1 >& "$INFO" + infosection "id" + id 2>&1 >& "$INFO" + infosection "env" + env 2>&1 >& "$INFO" + infosection "df" + df 2>&1 >& "$INFO" + infosection "/proc/cpuinfo" + cat /proc/cpuinfo 2>&1 >& "$INFO" + infosection "/proc/meminfo" + cat /proc/meminfo 2>&1 >& "$INFO" + infosection "command line" + echo $COMMANDLINE 2>&1 >& "$INFO" +} + +logstate() { + echo "Progress " `date +"%Y-%m-%d %H:%M:%S"` " $@" >& "$INFO" +} + +log() { + echo "$@" >& "$INFO" +} + +fail() { + EC=$1 + shift + log $@ + info + exit $EC +} + +checkError() { + if [ "$?" != "0" ]; then + fail $@ + fi +} + +checkEmpty() { + if [ "$1" == "" ]; then + shift + fail 254 $@ + fi +} + +getarg() { + NAME=$1 + shift + VALUE="" + SHIFTCOUNT=0 + if [ "$1" == "$NAME" ]; then + shift + let "SHIFTCOUNT=$SHIFTCOUNT+1" + while [ "${1:0:1}" != "-" ] && [ "$#" != "0" ]; do + VALUE="$VALUE $1" + shift + let "SHIFTCOUNT=$SHIFTCOUNT+1" + done + else + fail 254 "Missing $NAME argument" + fi + VALUE="${VALUE:1}" +} + +openinfo() { + exec 3<> $1 + INFO=3 +} + +closeinfo() { + exec 3>&- +} + +cioinput() { + INPUT=$1 + FILEPATH=`dirname $INPUT` + FILENAME=`basename $INPUT` + TYPE=${INPUT%%/*} + echo "INPUT_TYPE: $TYPE" >> /dev/shm/cio + if [ "$TYPE" == "common" ] && [ -e /dev/shm/share/$FILENAME ]; then + echo "cioinput(): link for common input $INPUT" >> /dev/shm/cio + ln -s "/dev/shm/share/$FILENAME" "$DIR/$L" + elif [ "$TYPE" == "_concurrent" ]; then + echo "cioinput(): toruscp for intermediate data $INPUT" >> /dev/shm/cio + mkdir -p $FILEPATH + echo "cioinput(): $DIR/$FILEPATH" >> /dev/shm/cio + /home/zzhang/cio/bin/toruscp.sh $FILENAME $DIR/$FILEPATH + else + echo "cioinput(): copy from GPFS $INPUT" >> /dev/shm/cio + cp "$PWD/shared/$L" "$DIR/$L" + fi +} +ciooutput() { + OUTPUT=$1 + FILEPATH=`dirname $OUTPUT` + FILENAME=`basename $OUTPUT` + TYPE=${OUTPUT%%/*} + echo "OUTPUT_TYPE: $TYPE" >> /dev/shm/cio + + if [ "$TYPE" == "_concurrent" ]; then + echo "ciooutput(): write intermediate data $OUTPUT" >> /dev/shm/cio + /home/zzhang/cio/bin/chtput.rb $FILENAME $RANK + cp $OUTPUT /dev/shm/share/ + else + echo "ciooutput(): write regular data $OUTPUT" >> /dev/shm/cio + #dd if="$OUTPUT" of="$WFDIR/shared/$OUTPUT" bs=128k + echo "$OUTPUT /chirp/multi/${CHIRP_ADD}@stripe/" >> /dev/shm/chirp_add + cp "$OUTPUT" /chirp/multi/${CHIRP_ADD}@stripe/ + fi +} +echo $@ >> /dev/shm/log +RANK=`echo $CONTROL_INIT | awk -F, '{print $4}'` +echo "RANK: $RANK" >> /dev/shm/log +COMMANDLINE=$@ +WFDIR=$PWD +ID=$1 +checkEmpty "$ID" "Missing job ID" + +shift + +getarg "-jobdir" "$@" +JOBDIR=$VALUE +shift $SHIFTCOUNT + +checkEmpty "$JOBDIR" "Missing job directory prefix" +mkdir -p /dev/shm/swift-info/$JOBDIR + +closeinfo +openinfo "/dev/shm/swift-info/$JOBDIR/${ID}-info" +#openinfo "/dev/null" + +logstate "LOG_START" + +getarg "-e" "$@" +EXEC=$VALUE +shift $SHIFTCOUNT + +getarg "-out" "$@" +STDOUT=$VALUE +shift $SHIFTCOUNT + +getarg "-err" "$@" +STDERR=$VALUE +shift $SHIFTCOUNT + +getarg "-i" "$@" +STDIN=$VALUE +shift $SHIFTCOUNT + +getarg "-d" "$@" +DIRS=$VALUE +shift $SHIFTCOUNT + +getarg "-if" "$@" +INF=$VALUE +shift $SHIFTCOUNT + +getarg "-of" "$@" +OUTF=$VALUE +shift $SHIFTCOUNT + +getarg "-k" "$@" +KICKSTART=$VALUE +shift $SHIFTCOUNT + +if [ "$1" == "-a" ]; then + shift +else + fail 254 "Missing arguments (-a option)" +fi + +if [ "X$SWIFT_JOBDIR_PATH" != "X" ]; then + DIR=${SWIFT_JOBDIR_PATH}/$JOBDIR/$ID + COPYNOTLINK=1 +else + DIR=/dev/shm/swift-work/$JOBDIR/$ID + COPYNOTLINK=0 +fi + +PATH=$PATH:/bin:/usr/bin + +if [ "$PATHPREFIX" != "" ]; then +export PATH=$PATHPREFIX:$PATH +fi + +IFS="^" + +logstate "CREATE_JOBDIR" +mkdir -p $DIR + +logstate "CREATE_INPUTDIR" + +for D in $DIRS ; do + mkdir -p "$DIR/$D" + checkError 254 "Failed to create input directory $D" +done + +#cd $DIR +logstate "LINK_INPUTS" +for L in $INF ; do + if [ $COPYNOTLINK = 1 ]; then + cp "$PWD/shared/$L" "$DIR/$L" + checkError 254 "Failed to copy input file $L" + else + cioinput $L + #cp "$PWD/shared/$L" "$DIR/$L" + checkError 254 "Failed to link input file $L" + fi +done + +logstate "EXECUTE" + +cd $DIR + +if [ "$KICKSTART" == "" ]; then + if [ "$STDIN" == "" ]; then + "$EXEC" "$@" 1>"$STDOUT" 2>"$STDERR" + else + "$EXEC" "$@" 1>"$STDOUT" 2>"$STDERR" <"$STDIN" + fi + checkError $? "Exit code $?" +else + if [ ! -f "$KICKSTART" ]; then + fail 254 "The Kickstart executable ($KICKSTART) was not found" + elif [ ! -x "$KICKSTART" ]; then + fail 254 "The Kickstart executable ($KICKSTART) does not have the executable bit set" + else + mkdir -p $WFDIR/kickstart/$JOBDIR + if [ "$STDIN" == "" ]; then + "$KICKSTART" -H -o "$STDOUT" -e "$STDERR" "$TMPEXEC" "$@" 1>kickstart.xml 2>"$STDERR" + else + "$KICKSTART" -H -o "$STDOUT" -i "$STDIN" -e "$STDERR" "$TMPEXEC" "$@" 1>kickstart.xml 2>"$STDERR" + fi + export APPEXIT=$? + mv -f kickstart.xml "$WFDIR/kickstart/$JOBDIR/$ID-kickstart.xml" 2>&1 >& "$INFO" + checkError 254 "Failed to copy Kickstart record to shared directory" + if [ "$APPEXIT" != "0" ]; then + fail $APPEXIT "Exit code $APPEXIT" + fi + fi +fi + +logstate "EXECUTE_DONE" + +MISSING= +for O in $OUTF ; do + if [ ! -f "$DIR/$O" ]; then + if [ "$MISSING" == "" ]; then + MISSING=$O + else + MISSING="$MISSING, $O" + fi + fi +done +if [ "$MISSING" != "" ]; then + fail 254 "The following output files were not created by the application: $MISSING" +fi + +logstate "COPYING_OUTPUTS" +for O in $OUTF ; do + #cp "$DIR/$O" "$WFDIR/shared/$O" 2>&1 >& "$INFO" + #cp "$DIR/$O" "$WFDIR/shared/$O" + #dd if="$DIR/$O" of="$WFDIR/shared/$JOBDIR/$O" bs=128k + #dd if="$DIR/$O" of="$WFDIR/shared/$O" bs=128k + ciooutput $O + checkError 254 "Failed to copy output file $O to shared directory" +done + +logstate "RM_JOBDIR" + +closeinfo +#rm -f "$WFDIR/info/$JOBDIR/${ID}-info" +#echo "$WFDIR/info/$JOBDIR/${ID}-info" >> /dev/shm/log +#mkdir -p "$WFDIR/info/$JOBDIR/" +#dd if=/dev/shm/swift-info/$JOBDIR/${ID}-info of="$WFDIR/info/$JOBDIR/${ID}-info" bs=128k +#dd if=/dev/shm/swift-info/$JOBDIR/${ID}-info of="/fuse/tmp/${ID}-info" bs=128k Added: SwiftApps/LigandAtlas/MTIO/bin/zoid-user-script.sh =================================================================== --- SwiftApps/LigandAtlas/MTIO/bin/zoid-user-script.sh (rev 0) +++ SwiftApps/LigandAtlas/MTIO/bin/zoid-user-script.sh 2009-03-30 16:49:57 UTC (rev 2784) @@ -0,0 +1,90 @@ +#!/bin/bash + +#IP=`hostname -i` +#echo ${IP} >> /home/falkon/falkon/logs/provisioner/zoid-log.txt + +#startup + if [ "${1}" -eq "1" ]; then + +#ERROR_LOG=/home/falkon/users/error.log +ERROR_LOG=/dev/null + + echo "starting zoid-user-script.sh..." >> ${ERROR_LOG} + #echo "reviewing the command and arguements: $0 $1 $2 $3 $4 $5 $6 $7..." >> ${ERROR_LOG} + #tENV=`env` + #echo "reviewing the envirnoment: $tENV..." >> ${ERROR_LOG} + +original=$ZOID_JOB_ARGS +a0=${original%%:*}; rest=${original#*:} +a1=${rest%%:*}; rest=${rest#*:} +a2=${rest%%:*}; rest=${rest#*:} +a3=${rest%%:*}; rest=${rest#*:} +a4=${rest%%:*}; rest=${rest#*:} +a5=${rest%%:*}; rest=${rest#*:} +a6=${rest%%:*}; rest=${rest#*:} +a7=${rest%%:*}; + + + echo "reviewing the command and arguements: $a0 $a1 $a2 $a3 $a4 $a5 $a6 $a7..." >> ${ERROR_LOG} + + +#FALKON_JOBID_HOME=/home/falkon/users/${USER}/${ZOID_JOB_ID} +FALKON_JOBID_HOME=${a7}/users/${a5}/${a6} +FALKON_HOME=${a7}/users/${a5}/${a6} + echo "FALKON_JOBID_HOME: $FALKON_JOBID_HOME" >> ${ERROR_LOG} +MACH_ID=`uname -n` + #this should be done via some configuration parameter + +echo ${FALKON_JOBID_HOME}/data/${MACH_ID} >> /tmp/iohost +if [ ! -d "${FALKON_JOBID_HOME}" ]; then + echo "ERROR: invalid path ${FALKON_JOBID_HOME}... exiting" >> ${ERROR_LOG} + exit 1 +fi + + + + + cd ${FALKON_JOBID_HOME} + #source falkon.env.bgp-io + + echo "ION at ${MACH_ID} ==> ${FALKON_HOME}..." >> ${ERROR_LOG} + + LOG_FILE_DEBUG="${FALKON_HOME}/logs/provisioner/STATUS_ION_${MACH_ID}" + DATE=`date` + echo "${DATE} : falkon-ion-start : starting the I/O node Falkon startup in the background..." >> ${LOG_FILE_DEBUG} + + +if [ ! -e "${FALKON_HOME}/bin/falkon-ion-start-blocking-zeptocn.sh" ]; then + echo "ERROR: invalid script ${FALKON_HOME}/bin/falkon-ion-start-blocking-zeptocn.sh" >> ${ERROR_LOG} + exit 1 +fi + + #used to run from GPFS + #${FALKON_JOBID_HOME}/bin/falkon-ion-start-blocking-zeptocn.sh ${FALKON_JOBID_HOME} & + #used to run from RAM + ${FALKON_JOBID_HOME}/bin/falkon-ion-start-blocking-zeptocn-ram.sh ${FALKON_JOBID_HOME} & + + + EXIT_CODE=$? + DATE=`date` + + echo "${DATE} : falkon-ion-start : completed the I/O node Falkon startup in the background..." >> ${LOG_FILE_DEBUG} + + echo "ION at ${MACH_ID} finished OK!" >> ${ERROR_LOG} + + +exit ${EXIT_CODE} + + + fi + + + +#cleanup + if [ "${1}" -eq "0" ]; then + + killall -9 java + exit 0 + + fi + Property changes on: SwiftApps/LigandAtlas/MTIO/bin/zoid-user-script.sh ___________________________________________________________________ Name: svn:executable + * Added: SwiftApps/LigandAtlas/MTIO/lib/libcrypt.so.1 =================================================================== (Binary files differ) Property changes on: SwiftApps/LigandAtlas/MTIO/lib/libcrypt.so.1 ___________________________________________________________________ Name: svn:executable + * Name: svn:mime-type + application/octet-stream Added: SwiftApps/LigandAtlas/MTIO/lib/libnsl.so.1 =================================================================== (Binary files differ) Property changes on: SwiftApps/LigandAtlas/MTIO/lib/libnsl.so.1 ___________________________________________________________________ Name: svn:executable + * Name: svn:mime-type + application/octet-stream Added: SwiftApps/LigandAtlas/MTIO/lib/libresolv.so.2 =================================================================== (Binary files differ) Property changes on: SwiftApps/LigandAtlas/MTIO/lib/libresolv.so.2 ___________________________________________________________________ Name: svn:executable + * Name: svn:mime-type + application/octet-stream From noreply at svn.ci.uchicago.edu Mon Mar 30 19:21:38 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 30 Mar 2009 19:21:38 -0500 (CDT) Subject: [Swift-commit] r2785 - trunk Message-ID: <20090331002138.B74FA228223@www.ci.uchicago.edu> Author: leggett Date: 2009-03-30 19:21:37 -0500 (Mon, 30 Mar 2009) New Revision: 2785 Added: trunk/leggett Log: - testing commits Added: trunk/leggett =================================================================== From noreply at svn.ci.uchicago.edu Mon Mar 30 19:22:13 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 30 Mar 2009 19:22:13 -0500 (CDT) Subject: [Swift-commit] r2786 - trunk Message-ID: <20090331002213.898A1228223@www.ci.uchicago.edu> Author: leggett Date: 2009-03-30 19:22:13 -0500 (Mon, 30 Mar 2009) New Revision: 2786 Removed: trunk/leggett Log: - removing my test Deleted: trunk/leggett =================================================================== From noreply at svn.ci.uchicago.edu Mon Mar 30 19:31:33 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 30 Mar 2009 19:31:33 -0500 (CDT) Subject: [Swift-commit] r2787 - trunk/src/org/griphyn/vdl/karajan/monitor/monitors/ansi/tui Message-ID: <20090331003134.11E47228223@www.ci.uchicago.edu> Author: hategan Date: 2009-03-30 19:31:33 -0500 (Mon, 30 Mar 2009) New Revision: 2787 Modified: trunk/src/org/griphyn/vdl/karajan/monitor/monitors/ansi/tui/ANSIContext.java Log: double ESC for escape; ESC+n for Fn emulation Modified: trunk/src/org/griphyn/vdl/karajan/monitor/monitors/ansi/tui/ANSIContext.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/monitor/monitors/ansi/tui/ANSIContext.java 2009-03-31 00:22:13 UTC (rev 2786) +++ trunk/src/org/griphyn/vdl/karajan/monitor/monitors/ansi/tui/ANSIContext.java 2009-03-31 00:31:33 UTC (rev 2787) @@ -245,7 +245,8 @@ if (count == 0) { count = 10; int[] size = querySize(); - if (size != null && (size[0] != buf.getWidth() || size[1] != buf.getHeight())) { + if (size != null + && (size[0] != buf.getWidth() || size[1] != buf.getHeight())) { buf.resize(size[0], size[1]); screen.setSize(size[0], size[1]); screen.invalidate(); @@ -266,48 +267,46 @@ Key key; int c = read(); if (c == 27) { - try { - Thread.sleep(25); + c = read(); + if (c == 27) { + key = new Key(0, 27); } - catch (InterruptedException e) { - return; + else if (c >= '1' && c <= '5') { + key = new Key(0, Key.F1 + c - '1'); } - if (is.available() == 0) { - key = new Key(0, c); + else if (c >= '6' && c <= '9') { + key = new Key(0, Key.F6 + c - '6'); } - else { - c = read(); - if (c == '[') { - int c0 = read(); - if (c0 <= 54) { - int c1 = read(); - if (c1 == '~') { - key = new Key(0, Key.KEYPAD2 + c0); + else if (c == '[') { + int c0 = read(); + if (c0 <= 54) { + int c1 = read(); + if (c1 == '~') { + key = new Key(0, Key.KEYPAD2 + c0); + } + else { + int c2 = read(); + if (c2 == '~') { + key = new Key(c0, c1, 0); } else { - int c2 = read(); - if (c2 == '~') { - key = new Key(c0, c1, 0); - } - else { - key = new Key(c0, c1, is.read()); - read(); - } + key = new Key(c0, c1, is.read()); + read(); } } - else { - key = new Key(0, Key.KEYPAD + c0); - } } - else if (c == 'O') { - // OS X F1 - F4 - int c0 = read(); - key = new Key(0, Key.F1 + (c0 - 'P')); - } else { - key = new Key(Key.MOD_ALT, c); + key = new Key(0, Key.KEYPAD + c0); } } + else if (c == 'O') { + // OS X F1 - F4 + int c0 = read(); + key = new Key(0, Key.F1 + (c0 - 'P')); + } + else { + key = new Key(Key.MOD_ALT, c); + } } else if (c < 32 && c != 0x0a && c != 0x0d) { key = new Key(Key.MOD_CTRL, c + 96); @@ -335,13 +334,7 @@ e.printStackTrace(); } } - if (!done) { - moveTo(1, 1); - bgColor(ANSI.RED); - fgColor(ANSI.WHITE); - text(key.toString()); - sync(); - } + //screen.status(key.toString()); } } catch (Exception e) { @@ -377,7 +370,7 @@ } } int c = is.read(); - System.err.print("\n" + c + " - " + (char) c); + //logger.warn("CONSOLE IN: " + c + " - " + (char) c); return c; } @@ -506,10 +499,10 @@ public void echo(boolean b) throws IOException { if (b) { - // terminal.enableEcho(); + terminal.enableEcho(); } else { - // terminal.disableEcho(); + terminal.disableEcho(); } } @@ -518,16 +511,26 @@ } public synchronized void lock() { - /* - * while (lock > 0) { try { wait(); } catch (InterruptedException e) { - * e.printStackTrace(); } } lock++; - */ + if (!doubleBuffered) { + while (lock > 0) { + try { + wait(); + } + catch (InterruptedException e) { + e.printStackTrace(); + } + } + lock++; + } } public synchronized void unlock() { - /* - * lock--; if (lock == 0) { notifyAll(); } - */ + if (!doubleBuffered) { + lock--; + if (lock == 0) { + notifyAll(); + } + } } public void exit() { From noreply at svn.ci.uchicago.edu Mon Mar 30 21:41:23 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Mon, 30 Mar 2009 21:41:23 -0500 (CDT) Subject: [Swift-commit] r2788 - in usertools/cio: . bin Message-ID: <20090331024123.1BE5A22817C@www.ci.uchicago.edu> Author: zzhang Date: 2009-03-30 21:41:22 -0500 (Mon, 30 Mar 2009) New Revision: 2788 Added: usertools/cio/bin/ usertools/cio/bin/.__acl usertools/cio/bin/bashrc usertools/cio/bin/bcast.sh usertools/cio/bin/broadcast.sh usertools/cio/bin/chtclient.rb usertools/cio/bin/chtget.rb usertools/cio/bin/chtput.rb usertools/cio/bin/cn-ipfwd.2409 usertools/cio/bin/cnip-work.sh usertools/cio/bin/cnip.sh usertools/cio/bin/collector.sh usertools/cio/bin/hashserver.rb usertools/cio/bin/ipcal usertools/cio/bin/torus-falkon-start-bgp-ram.sh usertools/cio/bin/toruscp.sh usertools/cio/bin/wrapper.sh usertools/cio/bin/zoid-user-script.sh Log: Basic APIs for MTIO Added: usertools/cio/bin/.__acl =================================================================== --- usertools/cio/bin/.__acl (rev 0) +++ usertools/cio/bin/.__acl 2009-03-31 02:41:22 UTC (rev 2788) @@ -0,0 +1,2 @@ +address:192.168.1.* rwlda +address:10.* rwlda Added: usertools/cio/bin/bashrc =================================================================== --- usertools/cio/bin/bashrc (rev 0) +++ usertools/cio/bin/bashrc 2009-03-31 02:41:22 UTC (rev 2788) @@ -0,0 +1,3 @@ +export PATH=/home/zzhang/chirp/bin:$PATH +export PATH=/home/zzhang/ruby-1.8.7-p72/bin/bin:$PATH +PATH=/fuse/bin:/fuse/usr/bin:$PATH \ No newline at end of file Added: usertools/cio/bin/bcast.sh =================================================================== --- usertools/cio/bin/bcast.sh (rev 0) +++ usertools/cio/bin/bcast.sh 2009-03-31 02:41:22 UTC (rev 2788) @@ -0,0 +1,16 @@ +#!/fuse/bin/bash + +export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/zzhang/cio/lib +PATH=/fuse/bin:/fuse/usr/bin:$PATH + +IP=`/sbin/ifconfig | grep inet | tail -n 1 | cut -d ':' -f 2 |awk '{print $1}'` +#mkdir -p /dev/shm/share/common +#cp -r common/* /dev/shm/share/ +#chmod -R 755 /dev/shm/share/common +#exit 1 +# tree network path +DESTHOSTS=`seq 0 63 | sed "s/^/10.128.0./" | xargs` +echo ${DESTHOSTS/$IP/" "} +/home/zzhang/chirp/bin/chirp_distribute -a address -D 127.0.0.1 / ${DESTHOSTS/$IP/" "} + +exit 0 \ No newline at end of file Property changes on: usertools/cio/bin/bcast.sh ___________________________________________________________________ Name: svn:executable + * Added: usertools/cio/bin/broadcast.sh =================================================================== --- usertools/cio/bin/broadcast.sh (rev 0) +++ usertools/cio/bin/broadcast.sh 2009-03-31 02:41:22 UTC (rev 2788) @@ -0,0 +1,16 @@ +#!/fuse/bin/bash + +export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/zzhang/cio/lib +PATH=/fuse/bin:/fuse/usr/bin:$PATH + +IP=`/sbin/ifconfig | grep inet | tail -n 1 | cut -d ':' -f 2 |awk '{print $1}'` +#mkdir -p /dev/shm/share/common +cp -r common/* /dev/shm/share/ +#chmod -R 755 /dev/shm/share/common +#exit 1 +# tree network path +DESTHOSTS=`seq 0 63 | sed "s/^/10.128.0./" | xargs` +#DESTHOSTS1=`seq 0 255 | sed "s/^/10.128.1./" | xargs` +echo ${DESTHOSTS/$IP/" "} +echo start +/home/zzhang/chirp/bin/chirp_distribute -a address -D 127.0.0.1 / ${DESTHOSTS/$IP/" "} Property changes on: usertools/cio/bin/broadcast.sh ___________________________________________________________________ Name: svn:executable + * Added: usertools/cio/bin/chtclient.rb =================================================================== --- usertools/cio/bin/chtclient.rb (rev 0) +++ usertools/cio/bin/chtclient.rb 2009-03-31 02:41:22 UTC (rev 2788) @@ -0,0 +1,75 @@ +#!/home/espinosa/local/bin/ruby + +require 'drb' + +DRb.start_service +cht = DRbObject.new(nil, 'druby://192.168.1.64:9000') + +cht.put("/x/0", 0) +cht.put("/x/1", 1) +cht.put("/x/2", 2) +cht.put("/x/3", 3) +cht.put("/x/4", 4) +cht.put("/x/5", 5) +cht.put("/x/6", 6) +cht.put("/x/7", 7) +cht.put("/x/8", 8) +cht.put("/x/9", 9) +cht.put("/x/10", 10) +cht.put("/x/11", 11) +cht.put("/x/12", 12) +cht.put("/x/13", 13) +cht.put("/x/14", 14) +cht.put("/x/15", 15) +cht.put("/x/16", 16) +cht.put("/x/17", 17) +cht.put("/x/18", 18) +cht.put("/x/19", 19) +cht.put("/x/20", 20) +cht.put("/x/21", 21) +cht.put("/x/22", 22) +cht.put("/x/23", 23) +cht.put("/x/24", 24) +cht.put("/x/25", 25) +cht.put("/x/26", 26) +cht.put("/x/27", 27) +cht.put("/x/28", 28) +cht.put("/x/29", 29) +cht.put("/x/20", 20) +cht.put("/x/31", 31) +cht.put("/x/32", 32) +cht.put("/x/33", 33) +cht.put("/x/34", 34) +cht.put("/x/35", 35) +cht.put("/x/36", 36) +cht.put("/x/37", 37) +cht.put("/x/38", 38) +cht.put("/x/39", 39) +cht.put("/x/40", 40) +cht.put("/x/41", 41) +cht.put("/x/42", 42) +cht.put("/x/43", 43) +cht.put("/x/44", 44) +cht.put("/x/45", 45) +cht.put("/x/46", 46) +cht.put("/x/47", 47) +cht.put("/x/48", 48) +cht.put("/x/49", 49) +cht.put("/x/50", 50) +cht.put("/x/51", 51) +cht.put("/x/52", 52) +cht.put("/x/53", 53) +cht.put("/x/54", 54) +cht.put("/x/55", 55) +cht.put("/x/56", 56) +cht.put("/x/57", 57) +cht.put("/x/58", 58) +cht.put("/x/59", 59) +cht.put("/x/60", 60) +cht.put("/x/61", 61) +cht.put("/x/62", 62) +cht.put("/x/63", 63) +#ARGV.each do|a| +# puts "Argument: #{a}" +# puts cht.get"#{a}" +#end Property changes on: usertools/cio/bin/chtclient.rb ___________________________________________________________________ Name: svn:executable + * Added: usertools/cio/bin/chtget.rb =================================================================== --- usertools/cio/bin/chtget.rb (rev 0) +++ usertools/cio/bin/chtget.rb 2009-03-31 02:41:22 UTC (rev 2788) @@ -0,0 +1,16 @@ +#!/home/espinosa/local/bin/ruby + +require 'drb' + +DRb.start_service +cht = DRbObject.new(nil, 'druby://10.128.0.0:9000') + +#cht.put("/x/y", 0) +#cht.put("/x/z", 0) +#cht.put("/x/a", 3) +#cht.put("/x/b", 2) +#cht.put("/x/c", 2) +#cht.put("/x/d", 2) + + puts cht.get(ARGV[0]) + Property changes on: usertools/cio/bin/chtget.rb ___________________________________________________________________ Name: svn:executable + * Added: usertools/cio/bin/chtput.rb =================================================================== --- usertools/cio/bin/chtput.rb (rev 0) +++ usertools/cio/bin/chtput.rb 2009-03-31 02:41:22 UTC (rev 2788) @@ -0,0 +1,16 @@ +#!/home/espinosa/local/bin/ruby + +require 'drb' + +DRb.start_service +cht = DRbObject.new(nil, 'druby://10.128.0.0:9000') + +#cht.put("/x/y", 0) +#cht.put("/x/z", 0) +#cht.put("/x/a", 3) +#cht.put("/x/b", 2) +#cht.put("/x/c", 2) +puts ARGV[0] +puts ARGV[1] +cht.put(ARGV[0], ARGV[1]) + Property changes on: usertools/cio/bin/chtput.rb ___________________________________________________________________ Name: svn:executable + * Added: usertools/cio/bin/cn-ipfwd.2409 =================================================================== (Binary files differ) Property changes on: usertools/cio/bin/cn-ipfwd.2409 ___________________________________________________________________ Name: svn:executable + * Name: svn:mime-type + application/octet-stream Added: usertools/cio/bin/cnip-work.sh =================================================================== --- usertools/cio/bin/cnip-work.sh (rev 0) +++ usertools/cio/bin/cnip-work.sh 2009-03-31 02:41:22 UTC (rev 2788) @@ -0,0 +1,69 @@ +#!/fuse/bin/bash +LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib +PATH=/fuse/bin:/fuse/usr/bin:$PATH + +#Torus START +/home/iskra/ZeptoOS/packages/cnip/prebuilt/cn-ipfwd.2409 & + +while [ ! -f /tmp/ifconfig.cmd ]; do + sleep 1 +done + +. /tmp/ifconfig.cmd + +#IP Calculation +RANK=`echo $CONTROL_INIT | awk -F, '{print $4}'` +FIRST=`expr $RANK % 256` +SECOND=`expr $RANK / 256` + +BASE=`expr $FIRST / 64` +MODULO=`expr $FIRST % 64` +CHIRP_BASE=`expr $BASE \* 64` +CHIRP_1=`expr $CHIRP_BASE + 1` +CHIRP_2=`expr $CHIRP_BASE + 2` +CHIRP_3=`expr $CHIRP_BASE + 3` +CHIRP_4=`expr $CHIRP_BASE + 4` +export CHIRP_ADD=10.128.$SECOND.$CHIRP_1 +#Chirp START +if [ -d /dev/shm/share ]; + then + rm -rf /dev/shm/share +fi +mkdir /dev/shm/share +echo "address:192.168.1.* rwlda" >>/dev/shm/share/.__acl +echo "address:10.* rwlda" >> /dev/shm/share/.__acl +echo "address:127.0.0.1 rwlda" >> /dev/shm/share/.__acl + +/home/zzhang/chirp/bin/chirp_server -r /dev/shm/share & + +#MAIN +if [ "$RANK" = "0" ] +then + /home/zzhang/cio/bin/hashserver.rb & +elif [ "$MODULO" = "1" ] +then + cd /dev/shm/share + mkdir stripe + cd stripe + cp ../.__acl . + mkdir root + cp .__acl root/ + echo 10.128.$SECOND.$CHIRP_1 >> hosts + echo 10.128.$SECOND.$CHIRP_2 >> hosts + echo 10.128.$SECOND.$CHIRP_3 >> hosts + echo 10.128.$SECOND.$CHIRP_4 >> hosts + echo bigvolfiles > key + + mkdir /chirp + /home/zzhang/chirp/bin/chirp_fuse -a address /chirp + /home/zzhang/cio/bin/collector.sh & +elif [ "$MODULO" = "2" ] || [ "$MODULO" = "3" ] || [ "$MODULO" = "4" ] +then + sleep 3600 +else + mkdir /chirp + /home/zzhang/chirp/bin/chirp_fuse -a address /chirp + /home/zzhang/falkon/worker/run.worker-c-bgp.sh $1 $2 $3 $4 $5 $6 $7 +fi + +sleep 3600 Property changes on: usertools/cio/bin/cnip-work.sh ___________________________________________________________________ Name: svn:executable + * Added: usertools/cio/bin/cnip.sh =================================================================== --- usertools/cio/bin/cnip.sh (rev 0) +++ usertools/cio/bin/cnip.sh 2009-03-31 02:41:22 UTC (rev 2788) @@ -0,0 +1,34 @@ +#!/fuse/bin/bash +LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib +PATH=/fuse/bin:/fuse/usr/bin:$PATH + +/home/iskra/ZeptoOS/packages/cnip/prebuilt/cn-ipfwd.2409 & + +while [ ! -f /tmp/ifconfig.cmd ]; do + sleep 1 +done + +. /tmp/ifconfig.cmd + +RANK=`echo $CONTROL_INIT | awk -F, '{print $4}'` + +if [ -d /dev/shm/share ]; + then + rm -rf /dev/shm/share +fi +mkdir /dev/shm/share +echo "address:192.168.1.* rwlda" >>/dev/shm/share/.__acl +echo "address:10.* rwlda" >> /dev/shm/share/.__acl +echo "address:127.0.0.1 rwlda" >> /dev/shm/share/.__acl + +/home/zzhang/chirp/bin/chirp_server -r /dev/shm/share & + +if [ "$RANK" = "0" ]; +then + /home/zzhang/cio/bin/hashserver.rb & +else + /home/zzhang/falkon/worker/run.worker-c-bgp.sh $1 $2 $3 $4 $5 $6 $7 +fi + +echo $RANK > /dev/shm/rank +sleep 3600 Property changes on: usertools/cio/bin/cnip.sh ___________________________________________________________________ Name: svn:executable + * Added: usertools/cio/bin/collector.sh =================================================================== --- usertools/cio/bin/collector.sh (rev 0) +++ usertools/cio/bin/collector.sh 2009-03-31 02:41:22 UTC (rev 2788) @@ -0,0 +1,60 @@ +#!/fuse/bin/bash +LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib +PATH=/fuse/bin:/fuse/usr/bin:$PATH + +i=0 +RANK=`echo $CONTROL_INIT | awk -F, '{print $4}'` +IOPATH=`cat /fuse/tmp/iohost` +CHIRP_ADDR=`echo $CHIRP_ADD` +LOG=/dev/shm/collector +echo "CHIRP: $CHIRP_ADD" >> $LOG 2>&1 +echo "START RANK: $RANK" >> $LOG 2>&1 +function send() +{ + ext=$1 + cd /chirp/multi/${CHIRP_ADDR}@stripe/ + time tar cf /tmp/${RANK}_${i}_${ext}.tar `ls /dev/shm/share/stripe/root/` >> $LOG 2>&1 + DATE=`date` + echo "${DATE}: ${i}th tar done with exit code $?" >> $LOG 2>&1 + time dd if=/tmp/${RANK}_${i}_${ext}.tar of=${IOPATH}/${RANK}_${i}_${ext}.tar bs=128k >> $LOG 2>&1 + DATE=`date` + echo "${DATE}: ${i}th copy done with exit code $?" >> $LOG 2>&1 + rm /tmp/${MACH_ID}_${i}_${ext}.tar + i=$[$i+1] +} + +function loopminute() +{ + while true; do + sleep 60 + + num=`ls -l /dev/shm/share/stripe/root/ | wc -l` >> $LOG 2>&1 + DATE=`date` + echo "${DATE}: there are $num result files in output" >> $LOG 2>&1 + if [ "${num}" -gt "1" ]; then + send minute + fi + DATE=`date` + echo "${DATE}: loopminute over" >> $LOG 2>&1 + done +} + +function loopsecond() +{ + while true; do + sleep 1 + filesize=`ls -l /dev/shm/share/stripe/root/ | awk 'NR>1 {a[1]=a[1]+$5} END{print a[1]}'` + if [ "${filesize}" -gt "268000000" ]; then + DATE=`date` + echo "${DATE}: $filesize reaches limit" >> $LOG 2>&1 + mkdir ${FLAG} + send seconds + rmdir ${FLAG} + DATE=`date` + echo "${DATE}: $filesize reaches limit over" >> $LOG 2>&1 + fi + done +} + +#loopsecond & +loopminute \ No newline at end of file Property changes on: usertools/cio/bin/collector.sh ___________________________________________________________________ Name: svn:executable + * Added: usertools/cio/bin/hashserver.rb =================================================================== --- usertools/cio/bin/hashserver.rb (rev 0) +++ usertools/cio/bin/hashserver.rb 2009-03-31 02:41:22 UTC (rev 2788) @@ -0,0 +1,26 @@ +#!/home/espinosa/local/bin/ruby + +require 'drb' + +class HashServer + attr_reader :file + + def initialize + @file = {} + end + + def put(fname, rank) + @file[fname] = rank + end + + def get(fname) + @file[fname] + end + +end + +server = HashServer.new + +DRb.start_service('druby://*:9000', server) +DRb.thread.join + Property changes on: usertools/cio/bin/hashserver.rb ___________________________________________________________________ Name: svn:executable + * Added: usertools/cio/bin/ipcal =================================================================== (Binary files differ) Property changes on: usertools/cio/bin/ipcal ___________________________________________________________________ Name: svn:executable + * Name: svn:mime-type + application/octet-stream Added: usertools/cio/bin/torus-falkon-start-bgp-ram.sh =================================================================== --- usertools/cio/bin/torus-falkon-start-bgp-ram.sh (rev 0) +++ usertools/cio/bin/torus-falkon-start-bgp-ram.sh 2009-03-31 02:41:22 UTC (rev 2788) @@ -0,0 +1,174 @@ +#!/bin/bash + + + if [ -z "$3" ]; then + echo "usage: $0 " + echo "usage: $0 prod 1024 60" + echo "-- or --" + echo "usage: $0 " + echo "usage: $0 prod 1024 60 4" + exit 1 + fi + + +if [ -z "${FALKON_HOME}" ]; then + echo "ERROR: environment variable FALKON_HOME not defined" + exit 1 +fi + + +QUEUE_NAME=$1 +PROFILE_NAME="zeptocn" +NUM_NODES=$2 +let NUM_ION=NUM_NODES/64 +MAX_TIME_MIN=$3 +SERVICE_IP="192.168.1.254" +SERVICE_PORT1=55000 +SERVICE_PORT2=55001 +WORKERS_PER_NODE=4 +if [ ! -z $4 ];then + WORKERS_PER_NODE=$4 +fi + + +cp /home/zzhang/cio/bin/zoid-user-script.sh ${HOME}/zoid-user-script.sh +chmod +x ${HOME}/zoid-user-script.sh + +FALKON_JOB_ID=`falkon-id-get.sh N/A` +EXIT_CODE=$? + + + if [ "${EXIT_CODE}" -ne "0" ]; then + echo "Error in geting a unique falkon ID.. 'falkon-get-id.sh N/A'" + cqdel ${ZOID_JOB_ID} + exit ${EXIT_CODE} + fi + + + +#ZOID_JOB_ID=`cqsub -q ${QUEUE_NAME} -k ${PROFILE_NAME} -C ${HOME} -t ${MAX_TIME_MIN} -n ${NUM_NODES} -e LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib /bgsys/linux/1.2.020080512/bin/bash /home/zzhang/falkon/worker/run.worker-c-bgp.sh ${SERVICE_IP} ${SERVICE_PORT1} ${SERVICE_PORT2} ${WORKERS_PER_NODE} ${USER} ${FALKON_JOB_ID} ${FALKON_ROOT}` + +ZOID_JOB_ID=`cqsub -q ${QUEUE_NAME} -k ${PROFILE_NAME} -C ${HOME} -t ${MAX_TIME_MIN} -n ${NUM_NODES} -e LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib /bgsys/linux/1.2.020080512/bin/bash /home/zzhang/cio/bin/cnip-work.sh ${SERVICE_IP} ${SERVICE_PORT1} ${SERVICE_PORT2} ${WORKERS_PER_NODE} ${USER} ${FALKON_JOB_ID} ${FALKON_ROOT}` + +#ZOID_JOB_ID=`cqsub -q ${QUEUE_NAME} -k ${PROFILE_NAME} -C ${HOME} -t ${MAX_TIME_MIN} -n ${NUM_NODES} -e LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib /bin/sleep 1800` + +EXIT_CODE=$? + + if [ "${EXIT_CODE}" -ne "0" ]; then + echo "Error in submitting job to Cobalt.. 'cqsub -q ${QUEUE_NAME} -k ${PROFILE_NAME} -C ${HOME} -t ${MAX_TIME_MIN} -n ${NUM_NODES} -e LD_LIBRARY_PATH=/lib:/fuse/lib:/fuse/usr/lib /bgsys/linux/1.2.020080512/bin/bash /fuse/${FALKON_WORKER_HOME}/run.worker-c-bgp.sh ${SERVICE_IP} ${SERVICE_PORT1} ${SERVICE_PORT2} ${WORKERS_PER_NODE} ${USER} ${FALKON_JOB_ID} ${FALKON_ROOT}' " + exit ${EXIT_CODE} + fi + +falkon-id-update.sh ${FALKON_JOB_ID} ${ZOID_JOB_ID} + +EXIT_CODE=$? + + if [ "${EXIT_CODE}" -ne "0" ]; then + echo "Error in updating cobalt job info for falkon.. 'falkon-update-id.sh ${ZOID_JOB_ID}'" + cqdel ${ZOID_JOB_ID} + exit ${EXIT_CODE} + fi + + +FALKON_JOBID_HOME=${FALKON_ROOT}/users/${USER}/${FALKON_JOB_ID} + +echo "Submitted job ${ZOID_JOB_ID} to Cobalt, creating the job specific Falkon tree for logs and configuration in ${FALKON_JOBID_HOME}..." + +mkdir -p ${FALKON_JOBID_HOME} + +cp ${FALKON_HOME}/falkon.env.bgp* ${FALKON_JOBID_HOME}/ +cp -r ${FALKON_HOME}/config ${FALKON_JOBID_HOME}/ +cp ${FALKON_JOBID_HOME}/config/Client-service-URIs.config2 ${FALKON_JOBID_HOME}/config/Client-service-URIs.config +mkdir -p ${FALKON_JOBID_HOME}/logs/client ${FALKON_JOBID_HOME}/logs/service ${FALKON_JOBID_HOME}/logs/provisioner ${FALKON_JOBID_HOME}/logs/worker + + +DATE=`date +%s` +echo "$DATE: pre-creating log dirs for Falkon service..." +RACK_START=0 +RACK_END=48 +SEQUENCE_DIR=`seq -w ${RACK_START} ${RACK_END}` +PSET_START=1 +PSET_END=16 +for a in ${SEQUENCE_DIR} +do + for ((b=${PSET_START}; b <= ${PSET_END} ; b++)) # Double parentheses, and "LIMIT" with no "$". + do + DIR_NAME="ion-R${a}-${b}" + mkdir -p ${FALKON_JOBID_HOME}/logs/service/$DIR_NAME + done +done + +for ((b=${PSET_START}; b <= ${PSET_END} ; b++)) # Double parentheses, and "LIMIT" with no "$". +do + DIR_NAME="ion-${b}" + mkdir -p ${FALKON_JOBID_HOME}/logs/service/$DIR_NAME +done + + +DATE=`date +%s` +echo "$DATE: done creating log dirs for Falkon service!" + + + +DATE=`date +%s` +echo "$DATE: pre-creating data dirs for Falkon service..." +RACK_START=0 +RACK_END=48 +SEQUENCE_DIR=`seq -w ${RACK_START} ${RACK_END}` +PSET_START=1 +PSET_END=16 +for a in ${SEQUENCE_DIR} +do + for ((b=${PSET_START}; b <= ${PSET_END} ; b++)) # Double parentheses, and "LIMIT" with no "$". + do + DIR_NAME="ion-R${a}-${b}" + mkdir -p ${FALKON_JOBID_HOME}/data/$DIR_NAME + done +done + +for ((b=${PSET_START}; b <= ${PSET_END} ; b++)) # Double parentheses, and "LIMIT" with no "$". +do + DIR_NAME="ion-${b}" + mkdir -p ${FALKON_JOBID_HOME}/data/$DIR_NAME +done + + +DATE=`date +%s` +echo "$DATE: done creating data dirs for Falkon service!" + + + +FALKON_HOME_RAM=/tmp/${USER}/falkon + +ln -s ${FALKON_HOME}/apps ${FALKON_JOBID_HOME}/apps +ln -s ${FALKON_HOME_RAM}/container ${FALKON_JOBID_HOME}/container +ln -s ${FALKON_HOME}/service ${FALKON_JOBID_HOME}/service +ln -s ${FALKON_HOME}/worker ${FALKON_JOBID_HOME}/worker +ln -s ${FALKON_HOME}/AstroPortal ${FALKON_JOBID_HOME}/AstroPortal +ln -s ${FALKON_HOME}/client ${FALKON_JOBID_HOME}/client +ln -s ${FALKON_HOME}/monitor ${FALKON_JOBID_HOME}/monitor +ln -s ${FALKON_HOME}/bin ${FALKON_JOBID_HOME}/bin +ln -s ${FALKON_HOME}/config ${FALKON_JOBID_HOME}/config +ln -s ${FALKON_HOME}/ploticus ${FALKON_JOBID_HOME}/ploticus +ln -s ${FALKON_HOME}/webserver ${FALKON_JOBID_HOME}/webserver +ln -s ${FALKON_HOME}/workloads ${FALKON_JOBID_HOME}/workloads +ln -s ${FALKON_HOME}/id ${FALKON_JOBID_HOME}/id +ln -s ${FALKON_HOME}/apache-ant-1.7.0 ${FALKON_JOBID_HOME}/apache-ant-1.7.0 +ln -s ${FALKON_HOME}/ibm-java2-ppc64-50 ${FALKON_JOBID_HOME}/ibm-java2-ppc64-50 +ln -s ${FALKON_HOME_RAM}/ibm-java2-ppc-50 ${FALKON_JOBID_HOME}/ibm-java2-ppc-50 +ln -s ${FALKON_HOME}/falkon.tgz ${FALKON_JOBID_HOME}/falkon.tgz + + +if [ ! -d "${FALKON_JOBID_HOME}" ]; then + echo "ERROR: invalid path ${FALKON_JOBID_HOME}... exiting" + cqdel ${ZOID_JOB_ID} + exit 1 +fi + +echo "Succesfully submitted the job to Cobalt, and setup job specific Falkon tree!" +echo "To monitor the job status, type 'cqstat | grep ${USER}'; once it is in running state, you can use the Falkon specific command ...." +echo "To submit your Falkon-based workload, type: ....; you can do this any time, the falkon workload will wait for the resources to come online, and will only be submitted when everything is ready; the script is run in the background, so the workload will run even if the ssh session gets disconnected." +echo "" +echo "Remember, your job id is ${ZOID_JOB_ID}, and if you need to look through the logs manually for anything, remember that you can find them at ${HOME}/${ZOID_JOB_ID}.output, ${HOME}/${ZOID_JOB_ID}.error, and ${FALKON_JOBID_HOME}/logs/..." + + Property changes on: usertools/cio/bin/torus-falkon-start-bgp-ram.sh ___________________________________________________________________ Name: svn:executable + * Added: usertools/cio/bin/toruscp.sh =================================================================== --- usertools/cio/bin/toruscp.sh (rev 0) +++ usertools/cio/bin/toruscp.sh 2009-03-31 02:41:22 UTC (rev 2788) @@ -0,0 +1,18 @@ +#!/fuse/bin/bash + +if [ -z "$2" ]; then + echo "usage: $0 " + echo "usage: $0 hello /dev/shm" + exit 1 +fi + +FILENAME=$1 +FILEPATH=$2 +echo "Full name : $FILENAME" +FILE=`basename $FILENAME` +echo "FILE name: $FILENAME" +RANK=`/home/zzhang/cio/bin/chtget.rb $FILENAME` +echo "RANK: $RANK" +IP=`/home/zzhang/cio/bin/ipcal 10.128.0.0 $RANK` +echo "IP: $IP" +/home/zzhang/chirp/bin/chirp_get -a address $IP $FILE $FILEPATH/$FILE Property changes on: usertools/cio/bin/toruscp.sh ___________________________________________________________________ Name: svn:executable + * Added: usertools/cio/bin/wrapper.sh =================================================================== --- usertools/cio/bin/wrapper.sh (rev 0) +++ usertools/cio/bin/wrapper.sh 2009-03-31 02:41:22 UTC (rev 2788) @@ -0,0 +1,288 @@ +# this script must be invoked inside of bash, not plain sh +PATH=/fuse/bin:/fuse/usr/bin:$PATH +infosection() { + echo >& "$INFO" + echo "_____________________________________________________________________________" >& "$INFO" + echo >& "$INFO" + echo " $1" >& "$INFO" + echo "_____________________________________________________________________________" >& "$INFO" + echo >& "$INFO" +} + +info() { + infosection "uname -a" + uname -a 2>&1 >& "$INFO" + infosection "id" + id 2>&1 >& "$INFO" + infosection "env" + env 2>&1 >& "$INFO" + infosection "df" + df 2>&1 >& "$INFO" + infosection "/proc/cpuinfo" + cat /proc/cpuinfo 2>&1 >& "$INFO" + infosection "/proc/meminfo" + cat /proc/meminfo 2>&1 >& "$INFO" + infosection "command line" + echo $COMMANDLINE 2>&1 >& "$INFO" +} + +logstate() { + echo "Progress " `date +"%Y-%m-%d %H:%M:%S"` " $@" >& "$INFO" +} + +log() { + echo "$@" >& "$INFO" +} + +fail() { + EC=$1 + shift + log $@ + info + exit $EC +} + +checkError() { + if [ "$?" != "0" ]; then + fail $@ + fi +} + +checkEmpty() { + if [ "$1" == "" ]; then + shift + fail 254 $@ + fi +} + +getarg() { + NAME=$1 + shift + VALUE="" + SHIFTCOUNT=0 + if [ "$1" == "$NAME" ]; then + shift + let "SHIFTCOUNT=$SHIFTCOUNT+1" + while [ "${1:0:1}" != "-" ] && [ "$#" != "0" ]; do + VALUE="$VALUE $1" + shift + let "SHIFTCOUNT=$SHIFTCOUNT+1" + done + else + fail 254 "Missing $NAME argument" + fi + VALUE="${VALUE:1}" +} + +openinfo() { + exec 3<> $1 + INFO=3 +} + +closeinfo() { + exec 3>&- +} + +cioinput() { + INPUT=$1 + FILEPATH=`dirname $INPUT` + FILENAME=`basename $INPUT` + TYPE=${INPUT%%/*} + echo "INPUT_TYPE: $TYPE" >> /dev/shm/cio + if [ "$TYPE" == "common" ] && [ -e /dev/shm/share/$FILENAME ]; then + echo "cioinput(): link for common input $INPUT" >> /dev/shm/cio + ln -s "/dev/shm/share/$FILENAME" "$DIR/$L" + elif [ "$TYPE" == "_concurrent" ]; then + echo "cioinput(): toruscp for intermediate data $INPUT" >> /dev/shm/cio + mkdir -p $FILEPATH + echo "cioinput(): $DIR/$FILEPATH" >> /dev/shm/cio + /home/zzhang/cio/bin/toruscp.sh $FILENAME $DIR/$FILEPATH + else + echo "cioinput(): copy from GPFS $INPUT" >> /dev/shm/cio + cp "$PWD/shared/$L" "$DIR/$L" + fi +} +ciooutput() { + OUTPUT=$1 + FILEPATH=`dirname $OUTPUT` + FILENAME=`basename $OUTPUT` + TYPE=${OUTPUT%%/*} + echo "OUTPUT_TYPE: $TYPE" >> /dev/shm/cio + + if [ "$TYPE" == "_concurrent" ]; then + echo "ciooutput(): write intermediate data $OUTPUT" >> /dev/shm/cio + /home/zzhang/cio/bin/chtput.rb $FILENAME $RANK + cp $OUTPUT /dev/shm/share/ + else + echo "ciooutput(): write regular data $OUTPUT" >> /dev/shm/cio + #dd if="$OUTPUT" of="$WFDIR/shared/$OUTPUT" bs=128k + echo "$OUTPUT /chirp/multi/${CHIRP_ADD}@stripe/" >> /dev/shm/chirp_add + cp "$OUTPUT" /chirp/multi/${CHIRP_ADD}@stripe/ + fi +} +echo $@ >> /dev/shm/log +RANK=`echo $CONTROL_INIT | awk -F, '{print $4}'` +echo "RANK: $RANK" >> /dev/shm/log +COMMANDLINE=$@ +WFDIR=$PWD +ID=$1 +checkEmpty "$ID" "Missing job ID" + +shift + +getarg "-jobdir" "$@" +JOBDIR=$VALUE +shift $SHIFTCOUNT + +checkEmpty "$JOBDIR" "Missing job directory prefix" +mkdir -p /dev/shm/swift-info/$JOBDIR + +closeinfo +openinfo "/dev/shm/swift-info/$JOBDIR/${ID}-info" +#openinfo "/dev/null" + +logstate "LOG_START" + +getarg "-e" "$@" +EXEC=$VALUE +shift $SHIFTCOUNT + +getarg "-out" "$@" +STDOUT=$VALUE +shift $SHIFTCOUNT + +getarg "-err" "$@" +STDERR=$VALUE +shift $SHIFTCOUNT + +getarg "-i" "$@" +STDIN=$VALUE +shift $SHIFTCOUNT + +getarg "-d" "$@" +DIRS=$VALUE +shift $SHIFTCOUNT + +getarg "-if" "$@" +INF=$VALUE +shift $SHIFTCOUNT + +getarg "-of" "$@" +OUTF=$VALUE +shift $SHIFTCOUNT + +getarg "-k" "$@" +KICKSTART=$VALUE +shift $SHIFTCOUNT + +if [ "$1" == "-a" ]; then + shift +else + fail 254 "Missing arguments (-a option)" +fi + +if [ "X$SWIFT_JOBDIR_PATH" != "X" ]; then + DIR=${SWIFT_JOBDIR_PATH}/$JOBDIR/$ID + COPYNOTLINK=1 +else + DIR=/dev/shm/swift-work/$JOBDIR/$ID + COPYNOTLINK=0 +fi + +PATH=$PATH:/bin:/usr/bin + +if [ "$PATHPREFIX" != "" ]; then +export PATH=$PATHPREFIX:$PATH +fi + +IFS="^" + +logstate "CREATE_JOBDIR" +mkdir -p $DIR + +logstate "CREATE_INPUTDIR" + +for D in $DIRS ; do + mkdir -p "$DIR/$D" + checkError 254 "Failed to create input directory $D" +done + +#cd $DIR +logstate "LINK_INPUTS" +for L in $INF ; do + if [ $COPYNOTLINK = 1 ]; then + cp "$PWD/shared/$L" "$DIR/$L" + checkError 254 "Failed to copy input file $L" + else + cioinput $L + #cp "$PWD/shared/$L" "$DIR/$L" + checkError 254 "Failed to link input file $L" + fi +done + +logstate "EXECUTE" + +cd $DIR + +if [ "$KICKSTART" == "" ]; then + if [ "$STDIN" == "" ]; then + "$EXEC" "$@" 1>"$STDOUT" 2>"$STDERR" + else + "$EXEC" "$@" 1>"$STDOUT" 2>"$STDERR" <"$STDIN" + fi + checkError $? "Exit code $?" +else + if [ ! -f "$KICKSTART" ]; then + fail 254 "The Kickstart executable ($KICKSTART) was not found" + elif [ ! -x "$KICKSTART" ]; then + fail 254 "The Kickstart executable ($KICKSTART) does not have the executable bit set" + else + mkdir -p $WFDIR/kickstart/$JOBDIR + if [ "$STDIN" == "" ]; then + "$KICKSTART" -H -o "$STDOUT" -e "$STDERR" "$TMPEXEC" "$@" 1>kickstart.xml 2>"$STDERR" + else + "$KICKSTART" -H -o "$STDOUT" -i "$STDIN" -e "$STDERR" "$TMPEXEC" "$@" 1>kickstart.xml 2>"$STDERR" + fi + export APPEXIT=$? + mv -f kickstart.xml "$WFDIR/kickstart/$JOBDIR/$ID-kickstart.xml" 2>&1 >& "$INFO" + checkError 254 "Failed to copy Kickstart record to shared directory" + if [ "$APPEXIT" != "0" ]; then + fail $APPEXIT "Exit code $APPEXIT" + fi + fi +fi + +logstate "EXECUTE_DONE" + +MISSING= +for O in $OUTF ; do + if [ ! -f "$DIR/$O" ]; then + if [ "$MISSING" == "" ]; then + MISSING=$O + else + MISSING="$MISSING, $O" + fi + fi +done +if [ "$MISSING" != "" ]; then + fail 254 "The following output files were not created by the application: $MISSING" +fi + +logstate "COPYING_OUTPUTS" +for O in $OUTF ; do + #cp "$DIR/$O" "$WFDIR/shared/$O" 2>&1 >& "$INFO" + #cp "$DIR/$O" "$WFDIR/shared/$O" + #dd if="$DIR/$O" of="$WFDIR/shared/$JOBDIR/$O" bs=128k + #dd if="$DIR/$O" of="$WFDIR/shared/$O" bs=128k + ciooutput $O + checkError 254 "Failed to copy output file $O to shared directory" +done + +logstate "RM_JOBDIR" + +closeinfo +#rm -f "$WFDIR/info/$JOBDIR/${ID}-info" +#echo "$WFDIR/info/$JOBDIR/${ID}-info" >> /dev/shm/log +#mkdir -p "$WFDIR/info/$JOBDIR/" +#dd if=/dev/shm/swift-info/$JOBDIR/${ID}-info of="$WFDIR/info/$JOBDIR/${ID}-info" bs=128k +#dd if=/dev/shm/swift-info/$JOBDIR/${ID}-info of="/fuse/tmp/${ID}-info" bs=128k Added: usertools/cio/bin/zoid-user-script.sh =================================================================== --- usertools/cio/bin/zoid-user-script.sh (rev 0) +++ usertools/cio/bin/zoid-user-script.sh 2009-03-31 02:41:22 UTC (rev 2788) @@ -0,0 +1,90 @@ +#!/bin/bash + +#IP=`hostname -i` +#echo ${IP} >> /home/falkon/falkon/logs/provisioner/zoid-log.txt + +#startup + if [ "${1}" -eq "1" ]; then + +#ERROR_LOG=/home/falkon/users/error.log +ERROR_LOG=/dev/null + + echo "starting zoid-user-script.sh..." >> ${ERROR_LOG} + #echo "reviewing the command and arguements: $0 $1 $2 $3 $4 $5 $6 $7..." >> ${ERROR_LOG} + #tENV=`env` + #echo "reviewing the envirnoment: $tENV..." >> ${ERROR_LOG} + +original=$ZOID_JOB_ARGS +a0=${original%%:*}; rest=${original#*:} +a1=${rest%%:*}; rest=${rest#*:} +a2=${rest%%:*}; rest=${rest#*:} +a3=${rest%%:*}; rest=${rest#*:} +a4=${rest%%:*}; rest=${rest#*:} +a5=${rest%%:*}; rest=${rest#*:} +a6=${rest%%:*}; rest=${rest#*:} +a7=${rest%%:*}; + + + echo "reviewing the command and arguements: $a0 $a1 $a2 $a3 $a4 $a5 $a6 $a7..." >> ${ERROR_LOG} + + +#FALKON_JOBID_HOME=/home/falkon/users/${USER}/${ZOID_JOB_ID} +FALKON_JOBID_HOME=${a7}/users/${a5}/${a6} +FALKON_HOME=${a7}/users/${a5}/${a6} + echo "FALKON_JOBID_HOME: $FALKON_JOBID_HOME" >> ${ERROR_LOG} +MACH_ID=`uname -n` + #this should be done via some configuration parameter + +echo ${FALKON_JOBID_HOME}/data/${MACH_ID} >> /tmp/iohost +if [ ! -d "${FALKON_JOBID_HOME}" ]; then + echo "ERROR: invalid path ${FALKON_JOBID_HOME}... exiting" >> ${ERROR_LOG} + exit 1 +fi + + + + + cd ${FALKON_JOBID_HOME} + #source falkon.env.bgp-io + + echo "ION at ${MACH_ID} ==> ${FALKON_HOME}..." >> ${ERROR_LOG} + + LOG_FILE_DEBUG="${FALKON_HOME}/logs/provisioner/STATUS_ION_${MACH_ID}" + DATE=`date` + echo "${DATE} : falkon-ion-start : starting the I/O node Falkon startup in the background..." >> ${LOG_FILE_DEBUG} + + +if [ ! -e "${FALKON_HOME}/bin/falkon-ion-start-blocking-zeptocn.sh" ]; then + echo "ERROR: invalid script ${FALKON_HOME}/bin/falkon-ion-start-blocking-zeptocn.sh" >> ${ERROR_LOG} + exit 1 +fi + + #used to run from GPFS + #${FALKON_JOBID_HOME}/bin/falkon-ion-start-blocking-zeptocn.sh ${FALKON_JOBID_HOME} & + #used to run from RAM + ${FALKON_JOBID_HOME}/bin/falkon-ion-start-blocking-zeptocn-ram.sh ${FALKON_JOBID_HOME} & + + + EXIT_CODE=$? + DATE=`date` + + echo "${DATE} : falkon-ion-start : completed the I/O node Falkon startup in the background..." >> ${LOG_FILE_DEBUG} + + echo "ION at ${MACH_ID} finished OK!" >> ${ERROR_LOG} + + +exit ${EXIT_CODE} + + + fi + + + +#cleanup + if [ "${1}" -eq "0" ]; then + + killall -9 java + exit 0 + + fi + Property changes on: usertools/cio/bin/zoid-user-script.sh ___________________________________________________________________ Name: svn:executable + * From noreply at svn.ci.uchicago.edu Tue Mar 31 13:59:56 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 31 Mar 2009 13:59:56 -0500 (CDT) Subject: [Swift-commit] r2789 - usertools/cio/bin Message-ID: <20090331185956.78F6F22820E@www.ci.uchicago.edu> Author: aespinosa Date: 2009-03-31 13:59:55 -0500 (Tue, 31 Mar 2009) New Revision: 2789 Removed: usertools/cio/bin/toruscp.sh Modified: usertools/cio/bin/hashserver.rb Log: Modified hashserver to accept multiple values per key. HashServer.get now returns a random element node from that list Modified: usertools/cio/bin/hashserver.rb =================================================================== --- usertools/cio/bin/hashserver.rb 2009-03-31 02:41:22 UTC (rev 2788) +++ usertools/cio/bin/hashserver.rb 2009-03-31 18:59:55 UTC (rev 2789) @@ -6,15 +6,15 @@ attr_reader :file def initialize - @file = {} + @file = Hash.new([]) end def put(fname, rank) - @file[fname] = rank + @file[fname].push(rank) end def get(fname) - @file[fname] + @file[fname][ rand( @file[fname].size ) ] end end Deleted: usertools/cio/bin/toruscp.sh =================================================================== --- usertools/cio/bin/toruscp.sh 2009-03-31 02:41:22 UTC (rev 2788) +++ usertools/cio/bin/toruscp.sh 2009-03-31 18:59:55 UTC (rev 2789) @@ -1,18 +0,0 @@ -#!/fuse/bin/bash - -if [ -z "$2" ]; then - echo "usage: $0 " - echo "usage: $0 hello /dev/shm" - exit 1 -fi - -FILENAME=$1 -FILEPATH=$2 -echo "Full name : $FILENAME" -FILE=`basename $FILENAME` -echo "FILE name: $FILENAME" -RANK=`/home/zzhang/cio/bin/chtget.rb $FILENAME` -echo "RANK: $RANK" -IP=`/home/zzhang/cio/bin/ipcal 10.128.0.0 $RANK` -echo "IP: $IP" -/home/zzhang/chirp/bin/chirp_get -a address $IP $FILE $FILEPATH/$FILE From noreply at svn.ci.uchicago.edu Tue Mar 31 14:00:30 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 31 Mar 2009 14:00:30 -0500 (CDT) Subject: [Swift-commit] r2790 - usertools/cio/bin Message-ID: <20090331190030.C441C22820E@www.ci.uchicago.edu> Author: aespinosa Date: 2009-03-31 14:00:30 -0500 (Tue, 31 Mar 2009) New Revision: 2790 Added: usertools/cio/bin/ifscp.sh Log: rename toruscp to ifscp Added: usertools/cio/bin/ifscp.sh =================================================================== --- usertools/cio/bin/ifscp.sh (rev 0) +++ usertools/cio/bin/ifscp.sh 2009-03-31 19:00:30 UTC (rev 2790) @@ -0,0 +1,18 @@ +#!/fuse/bin/bash + +if [ -z "$2" ]; then + echo "usage: $0 " + echo "usage: $0 hello /dev/shm" + exit 1 +fi + +FILENAME=$1 +FILEPATH=$2 +echo "Full name : $FILENAME" +FILE=`basename $FILENAME` +echo "FILE name: $FILENAME" +RANK=`/home/zzhang/cio/bin/chtget.rb $FILENAME` +echo "RANK: $RANK" +IP=`/home/zzhang/cio/bin/ipcal 10.128.0.0 $RANK` +echo "IP: $IP" +/home/zzhang/chirp/bin/chirp_get -a address $IP $FILE $FILEPATH/$FILE Property changes on: usertools/cio/bin/ifscp.sh ___________________________________________________________________ Name: svn:executable + * From noreply at svn.ci.uchicago.edu Tue Mar 31 14:52:23 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 31 Mar 2009 14:52:23 -0500 (CDT) Subject: [Swift-commit] r2791 - trunk/src/org/griphyn/vdl/karajan Message-ID: <20090331195223.4C2B62281A1@www.ci.uchicago.edu> Author: hategan Date: 2009-03-31 14:52:22 -0500 (Tue, 31 Mar 2009) New Revision: 2791 Modified: trunk/src/org/griphyn/vdl/karajan/Loader.java Log: log max memory when starting up Modified: trunk/src/org/griphyn/vdl/karajan/Loader.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/Loader.java 2009-03-31 19:00:30 UTC (rev 2790) +++ trunk/src/org/griphyn/vdl/karajan/Loader.java 2009-03-31 19:52:22 UTC (rev 2791) @@ -40,6 +40,7 @@ import org.griphyn.vdl.engine.Karajan; import org.griphyn.vdl.karajan.functions.ConfigProperty; import org.griphyn.vdl.karajan.lib.Execute; +import org.griphyn.vdl.karajan.lib.Log; import org.griphyn.vdl.karajan.monitor.MonitorAppender; import org.griphyn.vdl.toolkit.VDLt2VDLx; import org.griphyn.vdl.toolkit.VDLt2VDLx.IncorrectInvocationException; @@ -72,6 +73,7 @@ public static void main(String[] argv) { logger.debug("Loader started"); + logger.info("Max heap: " + Runtime.getRuntime().maxMemory()); ArgumentParser ap = buildArgumentParser(); String project = null; try { @@ -436,6 +438,7 @@ if (ap.isPresent(ARG_TUI)) { ma = new MonitorAppender(projectName); Logger.getRootLogger().addAppender(ma); + Logger.getLogger(Log.class).setLevel(Level.DEBUG); Logger.getLogger(AbstractGridNode.class).setLevel(Level.DEBUG); Logger.getLogger(Execute.class).setLevel(Level.DEBUG); Logger.getLogger(VDL2ExecutionContext.class).setLevel(Level.INFO); From noreply at svn.ci.uchicago.edu Tue Mar 31 20:26:22 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 31 Mar 2009 20:26:22 -0500 (CDT) Subject: [Swift-commit] r2792 - in usertools/cio/tools: . strace Message-ID: <20090401012622.F142B228223@www.ci.uchicago.edu> Author: aespinosa Date: 2009-03-31 20:26:21 -0500 (Tue, 31 Mar 2009) New Revision: 2792 Added: usertools/cio/tools/strace/ usertools/cio/tools/strace/iostats.rb Log: Imported IO statistics summarizer ruby script Added: usertools/cio/tools/strace/iostats.rb =================================================================== --- usertools/cio/tools/strace/iostats.rb (rev 0) +++ usertools/cio/tools/strace/iostats.rb 2009-04-01 01:26:21 UTC (rev 2792) @@ -0,0 +1,51 @@ +#!/home/aespinosa/local/bin/ruby -na +# + +BEGIN { + READ = 'read$' + OPEN = 'open\(' + WRITE = 'write\(' + $desc = {} + $read = Hash.new(0) + $write = Hash.new(0) + $desc[0] = "STDIN" + $desc[1] = "STDOUT" + $desc[2] = "STDERR" + def tosec(time) + x=time.split(/:/) + x[0].to_f * 3600.0 + x[1].to_f * 60.0 + x[2].to_f + end +} + +now = $F[0] +retcode = $_.split("=")[1].to_i +if $_ =~ /\(.*$/ + args = $&.gsub( /^$|$$/, "" ).split ',' +end + +if $. == 1 + start = tosec(now) +end + +if $_ =~ /#{READ}/ + file = $desc[ args[0].to_i ] + $read[file] += retcode +elsif $_ =~ /#{WRITE}/ + file = $desc[ args[0].to_i ] + $write[file] += retcode +elsif $_ =~ /#{OPEN}/ and retcode > -1 + $desc[retcode] = args[0].gsub(/"/,"") +end + +prev = now + +END { + puts "Read summary\n------------" + $read.each { |file, size| + puts "#{file}: #{size} bytes" + } + puts "Write summary\n-----------" + $write.each { |file, size| + puts "#{file}: #{size} bytes" + } +} Property changes on: usertools/cio/tools/strace/iostats.rb ___________________________________________________________________ Name: svn:executable + * From noreply at svn.ci.uchicago.edu Tue Mar 31 20:26:56 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 31 Mar 2009 20:26:56 -0500 (CDT) Subject: [Swift-commit] r2793 - in usertools/cio: . science science/blast Message-ID: <20090401012656.D1990228223@www.ci.uchicago.edu> Author: aespinosa Date: 2009-03-31 20:26:56 -0500 (Tue, 31 Mar 2009) New Revision: 2793 Added: usertools/cio/science/ usertools/cio/science/blast/ usertools/cio/science/blast/mockblast.sh usertools/cio/science/blast/runblast.sh usertools/cio/science/blast/test.in Log: Created science directory. Initial tests for real blast and mock blast Added: usertools/cio/science/blast/mockblast.sh =================================================================== --- usertools/cio/science/blast/mockblast.sh (rev 0) +++ usertools/cio/science/blast/mockblast.sh 2009-04-01 01:26:56 UTC (rev 2793) @@ -0,0 +1,9 @@ +#!/bin/bash + + +DB=/disks/ci-gpfs/swift/blast/pir/UNIPROT_for_blast_14.0.seq.* + +for i in $DB; do + dd if=$i of=/dev/null bs=1M +done +sleep 18 Property changes on: usertools/cio/science/blast/mockblast.sh ___________________________________________________________________ Name: svn:executable + * Added: usertools/cio/science/blast/runblast.sh =================================================================== --- usertools/cio/science/blast/runblast.sh (rev 0) +++ usertools/cio/science/blast/runblast.sh 2009-04-01 01:26:56 UTC (rev 2793) @@ -0,0 +1,6 @@ + +DB=/disks/ci-gpfs/swift/blast/pir/UNIPROT_for_blast_14.0.seq +strace -tt -ff -o traceblast \ + /home/aespinosa/science/blast/ncbi.communicado/bin/blastall -p blastp -F F \ + -d $DB -i test.in -v 300 -b 300 -m8 -o test.out + Added: usertools/cio/science/blast/test.in =================================================================== --- usertools/cio/science/blast/test.in (rev 0) +++ usertools/cio/science/blast/test.in 2009-04-01 01:26:56 UTC (rev 2793) @@ -0,0 +1,5 @@ +>A0AQI4 +VWLRRCTHYLFIVVVAVNSTLLTINAGDYIFYTDWAWTSYTVFSISQTLMLIVGATYYLT +FTGVPGTATYYALIMTVYTWIAKAAWFSLGYPYDFIVTPVWLPSAMLLDLVYWATKKNKH +SLILFGGVLVGMSLPLSNMVNLITVADPLETAFKYPRPTLPPYMTPIEPQVGKFYNSPVA +LGAGAGAVLGCTFAALGCKLNTWTYR From noreply at svn.ci.uchicago.edu Tue Mar 31 21:46:26 2009 From: noreply at svn.ci.uchicago.edu (noreply at svn.ci.uchicago.edu) Date: Tue, 31 Mar 2009 21:46:26 -0500 (CDT) Subject: [Swift-commit] r2794 - trunk/libexec/log-processing Message-ID: <20090401024626.C82622281AD@www.ci.uchicago.edu> Author: benc Date: 2009-03-31 21:46:26 -0500 (Tue, 31 Mar 2009) New Revision: 2794 Modified: trunk/libexec/log-processing/kickstarts-to-event Log: Make kickstarts-to-event exit more gracefully in the absence of a kickstart directory. Modified: trunk/libexec/log-processing/kickstarts-to-event =================================================================== --- trunk/libexec/log-processing/kickstarts-to-event 2009-04-01 01:26:56 UTC (rev 2793) +++ trunk/libexec/log-processing/kickstarts-to-event 2009-04-01 02:46:26 UTC (rev 2794) @@ -1,5 +1,10 @@ #!/bin/bash +if [ "$KDIR" == "" ]; then + echo No kickstart record directory defined in environment variable KDIR - exiting + exit 0 +fi + while read t d id rest; do record=$(find ${KDIR} -name ${id}-kickstart.xml)

Research leading to Swift and SwiftScript

Karajan Technology used in Swift

Virtual Data Language and Virtual Data System - predecessors to Swift

Swift Tutorials

Swift User Guide

Log Processing tools