[Swift-commit] r7529 - in SwiftApps/swift-galaxy: . swift
ketan at ci.uchicago.edu
ketan at ci.uchicago.edu
Thu Jan 30 17:20:58 CST 2014
Author: ketan
Date: 2014-01-30 17:20:58 -0600 (Thu, 30 Jan 2014)
New Revision: 7529
Added:
SwiftApps/swift-galaxy/swift/postprocess.py
SwiftApps/swift-galaxy/swift/swift-catsn.sh
SwiftApps/swift-galaxy/swift/swift-catsn.xml
SwiftApps/swift-galaxy/swift/swift-skeleton.xml
Removed:
SwiftApps/swift-galaxy/swift/catsn.xml
SwiftApps/swift-galaxy/swift/swift-catsn
SwiftApps/swift-galaxy/swift/swiftforeachrangepp.py
Modified:
SwiftApps/swift-galaxy/README.txt
SwiftApps/swift-galaxy/swift/swiftforeachlist.sh
SwiftApps/swift-galaxy/swift/swiftforeachlist.xml
SwiftApps/swift-galaxy/swift/swiftforeachrange.sh
SwiftApps/swift-galaxy/swift/swiftforeachrange.xml
Log:
swift galaxy enhancements
Modified: SwiftApps/swift-galaxy/README.txt
===================================================================
--- SwiftApps/swift-galaxy/README.txt 2014-01-30 15:26:30 UTC (rev 7528)
+++ SwiftApps/swift-galaxy/README.txt 2014-01-30 23:20:58 UTC (rev 7529)
@@ -9,47 +9,31 @@
3. (Re)start Galaxy. You should see the tools in a toolset titled Swift.
-==An example (obsolete) ==
+==An example of integrating a Swift tool into Galaxy==
A simple Hello World Swift tool to run with Galaxy.
To set up with your Galaxy environment follow the steps below:
-0. Prerequisite: Sun/Oracle java. Should work with IBM java but not tested rigorously.
+0. Prerequisite: Sun/Oracle java. Should work with IBM java but not tested.
1. Download and install Swift where Galaxy server is running. From a command prompt the following wget will pull the latest release:
-wget http://www.ci.uchicago.edu/swift/packages/swift-0.94.1.tar.gz
+wget http://swiftlang.org/packages/swift-0.94.2-RC2.tar.gz
-Do untar at a suitable location:
+2. Create a swift-tool directory and copy the Swift tool skeleton files in this directory
-tar zxf swift-0.94.1.tar.gz
+cp swift/swift-skeleton* /location/of/galaxy-server/tools/
-Put the bin dir on system PATH:
+3. Edit the tool_conf.xml file present at the top-level of galaxy directory tree to add the following lines between the <toolbox> and </toolbox> xml tags:
-export PATH=$PATH:/location/to/swift-0.94.1/bin
-
-Edit above line to match the location of Swift and add it to your environment file (.bashrc, .bash_profile or the corresponding ones for other shells).
-
-Make sure Swift is available from a command prompt:
-
-which swift
-swift -version
-
-2. untar and copy the swift-galaxy.tgz to the tools directory of your Galaxy location:
-
-tar zxf swift-galaxy.tgz
-
-cp -r swift-galaxy /location/of/galaxy-server/tools/
-
-3. Edit the tool_conf.xml file present at the top-level of galaxy-server directory tree to add the following lines between the <toolbox> and </toolbox> xml tags:
-
-<section name="swift-hello" id="swift-hello">
- <tool file="swift-galaxy/helloswift.xml" />
+<section name="swift-galaxy" id="swift-galaxy">
+ <tool file="currdir/nameoftoolfile.xml" />
</section>
4. Restart the Galaxy server. The swift-galaxy tool should be available in the tool box of Galaxy GUI.
-5. Invoke it by entering any number in the dialog box or leaving it to default. A message "Hello World!" will be written to output files as many times and the files should be available for browsing from Galaxy GUI.
+5. Invoke it by entering any number in the dialog box or leaving it to default.
-Please note that the Cluster option is not implemented in this example.
+6. For examples of othe Galaxy-Swift tools, look into the swift directoty.
+
Deleted: SwiftApps/swift-galaxy/swift/catsn.xml
===================================================================
--- SwiftApps/swift-galaxy/swift/catsn.xml 2014-01-30 15:26:30 UTC (rev 7528)
+++ SwiftApps/swift-galaxy/swift/catsn.xml 2014-01-30 23:20:58 UTC (rev 7529)
@@ -1,22 +0,0 @@
-<tool id="catsn" name="Swift catsn">
- <description>A Swift tool to run the catsn workflow</description>
-
- <command interpreter="bash">
- swift-catsn "$site" "$data" "$n" "$html_file.files_path" "$log_file" "$html_file"
- </command>
-
- <inputs>
- <param name="site" type="select" label="Execution Location">
- <option value="localhost">Localhost</option>
- <option value="midway">Midway</option>
- <option value="uc3">UC3</option>
- <option value="stampede">Stampede</option>
- </param>
- <param name="n" type="text" value="1" label="Number of tasks to run "/>
- <param name="data" type="data" label="Input data file"/>
- </inputs>
- <outputs>
- <data format="txt" name="log_file" type="data" label="Swift output" />
- <data format="html" name="html_file" label="catsnoutput.html" />
- </outputs>
-</tool>
Copied: SwiftApps/swift-galaxy/swift/postprocess.py (from rev 7528, SwiftApps/swift-galaxy/swift/swiftforeachrangepp.py)
===================================================================
--- SwiftApps/swift-galaxy/swift/postprocess.py (rev 0)
+++ SwiftApps/swift-galaxy/swift/postprocess.py 2014-01-30 23:20:58 UTC (rev 7529)
@@ -0,0 +1,65 @@
+#post processing, set build for data and add additional data to history^M
+from galaxy import datatypes, config, jobs, tools
+from galaxy.model import HistoryDatasetAssociation
+import operator
+import shutil
+import os
+import glob
+from os.path import basename
+
+def createDsRef(outDataItems, param_dict, outputName, outputFormat, outputDsRefName, nameSpace):
+ dsrefXML="<?xml version=\"1.0\" ?>\n"
+ dsrefXML=dsrefXML+"<dataSetRefs xmlns=\"http://ci.uchicago.edu/galaxy-es/datasetrefs/"+nameSpace+"\" >\n"
+
+ dsrefFileName=""
+
+ for outDataName, outDataItem in outDataItems:
+ if outDataName==outputDsRefName:
+ dsrefFileName=outDataItem.get_file_name()
+ else:
+ if outDataName==outputName:
+ dsrefXML=dsrefXML+"<dataSetRef id=\"%s\" name=\"%s\" path=\"%s\" type=\"%s\"/>\n" % (outDataItem.id,outDataItem.name,outDataItem.get_file_name(),outputFormat)
+
+ print "dsrefFileName:"+dsrefFileName
+
+ primaryCollectedDatasets=param_dict["__collected_datasets__"]["primary"]
+
+ for primaryCollectedDataset in primaryCollectedDatasets:
+ primaryCollectedDatasetItems=primaryCollectedDatasets[primaryCollectedDataset]
+ for primaryCollectedDatasetItem in primaryCollectedDatasetItems:
+ outDataItem=primaryCollectedDatasetItems[primaryCollectedDatasetItem]
+ if outputFormat in outDataItem.extension:
+
+ dsrefXML=dsrefXML+"<dataSetRef id=\"%s\" name=\"%s\" path=\"%s\" type=\"%s\"/>\n" % (outDataItem.id,outDataItem.name,outDataItem.get_file_name(),outDataItem.extension)
+
+ dsrefXML=dsrefXML+"</dataSetRefs>\n"
+
+ return dsrefFileName, dsrefXML
+
+def writeDsRef(dsrefFileName, dsrefXML):
+ dsrefFile = open(dsrefFileName, "w", 0)
+ dsrefFile.write(dsrefXML)
+ dsrefFile.flush()
+ dsrefFile.close()
+
+def setSizeAndPeak(outDataItems,outputDsRefName):
+ for outDataName, outDataItem in outDataItems:
+ if outDataName==outputDsRefName:
+ outDataItem.dataset.file_size=None
+ outDataItem.set_size()
+ outDataItem.set_peek()
+
+def writeoutlist(app, inp_data, out_data, param_dict, tool, stdout, stderr):
+
+ outDataItems = out_data.items()
+ outDataItems = sorted(outDataItems, key=operator.itemgetter(0))
+
+ dsrefAcmoFileName, dsrefAcmoXML = createDsRef(outDataItems,param_dict,"acmo","acmo.csv","acmoDsRef","facit/acmo")
+
+ writeDsRef(dsrefAcmoFileName, dsrefAcmoXML)
+
+ setSizeAndPeak(outDataItems,"acmoDsRef")
+
+
+ app.model.context.flush()
+
Deleted: SwiftApps/swift-galaxy/swift/swift-catsn
===================================================================
--- SwiftApps/swift-galaxy/swift/swift-catsn 2014-01-30 15:26:30 UTC (rev 7528)
+++ SwiftApps/swift-galaxy/swift/swift-catsn 2014-01-30 23:20:58 UTC (rev 7529)
@@ -1,328 +0,0 @@
-#!/bin/bash
-
-#set -x
-set -e
-
-site=$1
-data=$2
-n=$3
-outdir=$4
-logfile=$5
-htmlfile=$6
-mkdir -p $outdir
-
-swift=`which swift`
-# use "here" document for configs
-
-cat << EOF > /tmp/catsn.swift
-type file;
-
-app (file o) cat (file i) {
- cat @i stdout=@o;
-}
-#file out[]<simple_mapper; location="outdir", prefix="f.",suffix=".out">;
-file out[]<simple_mapper; location="$outdir", prefix="f.",suffix=".txt">;
-
-string d=@arg("data");
-
-foreach j in [1:@toInt(@arg("n","1"))] {
- file data<single_file_mapper; file=d>;
- out[j] = cat(data);
-}
-EOF
-
-case "$site" in
-midway )
-cat << 'EOF' > /tmp/sites.xml
-<?xml version="1.0" encoding="UTF-8"?>
-<config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites">
- <pool handle="midway">
- <execution provider="coaster" jobmanager="local:slurm"/>
- <profile namespace="globus" key="queue">sandyb</profile>
- <profile namespace="globus" key="jobsPerNode">16</profile>
- <profile namespace="globus" key="maxWalltime">02:00:00</profile>
- <profile namespace="globus" key="maxTime">7500</profile>
- <profile namespace="globus" key="highOverAllocation">100</profile>
- <profile namespace="globus" key="lowOverAllocation">100</profile>
- <profile namespace="globus" key="slots">2</profile>
- <profile namespace="globus" key="maxNodes">1</profile>
- <profile namespace="globus" key="nodeGranularity">1</profile>
- <profile namespace="karajan" key="jobThrottle">.31</profile>
- <profile namespace="karajan" key="initialScore">10000</profile>
- <workdirectory>/tmp/swift.work</workdirectory>
- </pool>
-</config>
-EOF
-
-cat << 'EOF' > /tmp/tc
-midway echo echo
-midway cat cat
-EOF
-
-cat << 'EOF' > /tmp/cf
-use.provider.staging=true
-wrapperlog.always.transfer=true
-execution.retries=1
-provider.staging.pin.swiftfiles=false
-sitedir.keep=true
-lazy.errors=false
-EOF
-;;
-
-uc3 )
-cat << 'EOF' > /tmp/sites.xml
-<?xml version="1.0" encoding="UTF-8"?>
-<config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites">
- <pool handle="uc3">
- <execution provider="coaster" url="uc3-sub.uchicago.edu" jobmanager="ssh-cl:condor"/>
- <!--<execution provider="coaster" jobmanager="local:condor"/>-->
- <profile namespace="karajan" key="jobThrottle">.09</profile>
- <profile namespace="karajan" key="initialScore">1000</profile>
- <profile namespace="globus" key="jobsPerNode">1</profile>
- <profile namespace="globus" key="maxWalltime">3600</profile>
- <profile namespace="globus" key="nodeGranularity">1</profile>
- <profile namespace="globus" key="highOverAllocation">100</profile>
- <profile namespace="globus" key="lowOverAllocation">100</profile>
- <profile namespace="globus" key="slots">1</profile>
- <profile namespace="globus" key="condor.Requirements">UidDomain == "osg-gk.mwt2.org"</profile>
- <profile namespace="globus" key="maxNodes">1</profile>
- <profile namespace="globus" key="condor.+AccountingGroup">"group_friends.{env.USER}"</profile>
- <profile namespace="globus" key="jobType">nonshared</profile>
- <filesystem provider="local" url="none" />
- <!--<workdirectory>/home/maheshwari/test/catsn/swift.work</workdirectory>-->
- <workdirectory>/tmp/swift.work</workdirectory>
- </pool>
-</config>
-EOF
-
-cat << 'EOF' > /tmp/tc
-uc3 echo /bin/echo
-uc3 cat /bin/cat
-EOF
-
-cat << 'EOF' > /tmp/cf
-use.provider.staging=true
-wrapperlog.always.transfer=true
-execution.retries=1
-provider.staging.pin.swiftfiles=false
-sitedir.keep=true
-lazy.errors=false
-EOF
-;;
-
-stampede )
-cat << 'EOF' > /tmp/sites.xml
-<?xml version="1.0" encoding="UTF-8"?>
-<config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites">
-<pool handle="stampede">
- <execution provider="coaster" jobmanager="gt2:gt2:slurm" url="login5.stampede.tacc.utexas.edu:2119/jobmanager-slurm"/>
- <!--<execution provider="coaster" jobmanager="ssh-cl:slurm" url="stampede.tacc.utexas.edu"/>-->
- <!--<filesystem provider="gsiftp" url="gsiftp://gridftp.stampede.tacc.utexas.edu:2811"/>-->
- <filesystem provider="local"/>
- <profile namespace="globus" key="jobsPerNode">16</profile>
- <profile namespace="globus" key="ppn">16</profile>
- <profile namespace="globus" key="project">TG-STA110005S</profile>
- <profile namespace="globus" key="queue">normal</profile>
- <profile namespace="globus" key="slurm.mail-user">ketancmaheshwari at gmail.com</profile>
- <profile namespace="globus" key="slurm.mail-type">ALL</profile>
- <profile namespace="globus" key="maxWalltime">02:00:00</profile>
- <profile namespace="globus" key="maxTime">8000</profile>
- <profile namespace="globus" key="highOverAllocation">100</profile>
- <profile namespace="globus" key="lowOverAllocation">100</profile>
- <profile namespace="globus" key="slots">1</profile>
- <profile namespace="globus" key="maxNodes">1</profile>
- <profile namespace="globus" key="nodeGranularity">1</profile>
- <profile namespace="karajan" key="jobThrottle">.15</profile>
- <profile namespace="karajan" key="initialScore">10000</profile>
- <workdirectory>/tmp/swift.work</workdirectory>
-</pool>
-</config>
-EOF
-
-cat << 'EOF' > /tmp/tc
-stampede echo echo
-stampede cat cat
-EOF
-
-cat << 'EOF' > /tmp/cf
-use.provider.staging=true
-wrapperlog.always.transfer=true
-execution.retries=1
-provider.staging.pin.swiftfiles=false
-sitedir.keep=true
-lazy.errors=false
-EOF
-;;
-
-localhost )
-cat << 'EOF' > /tmp/sites.xml
-<?xml version="1.0" encoding="UTF-8"?>
-<config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites">
-<pool handle="localhost">
- <execution provider="local" jobmanager="local:local"/>
- <filesystem provider="local"/>
- <!--<workdirectory>/home/kcm92/powergridapps/swiftscripts/swift.work</workdirectory>-->
- <workdirectory>/tmp/swift.work</workdirectory>
- <profile namespace="karajan" key="jobThrottle">0.31</profile>
-</pool>
-</config>
-EOF
-
-cat << 'EOF' > /tmp/tc
-localhost echo echo
-localhost cat cat
-EOF
-
-cat << 'EOF' > /tmp/cf
-use.provider.staging=false
-wrapperlog.always.transfer=true
-execution.retries=1
-provider.staging.pin.swiftfiles=false
-sitedir.keep=true
-lazy.errors=false
-EOF
-;;
-
-* )
-echo "defaulting to localhost"
-
-cat << 'EOF' > /tmp/sites.xml
-<?xml version="1.0" encoding="UTF-8"?>
-<config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites">
-<pool handle="localhost">
- <execution provider="local" jobmanager="local:local"/>
- <filesystem provider="local"/>
- <!--<workdirectory>/home/kcm92/powergridapps/swiftscripts/swift.work</workdirectory>-->
- <workdirectory>/tmp/swift.work</workdirectory>
- <profile namespace="karajan" key="jobThrottle">0.31</profile>
- </pool>
-</config>
-EOF
-
-cat << 'EOF' > /tmp/tc
-localhost echo echo
-localhost cat cat
-EOF
-
-cat << 'EOF' > /tmp/cf
-use.provider.staging=false
-wrapperlog.always.transfer=true
-execution.retries=1
-provider.staging.pin.swiftfiles=false
-sitedir.keep=true
-lazy.errors=false
-EOF
-;;
-esac
-
-$swift -sites.file /tmp/sites.xml -tc.file /tmp/tc -config /tmp/cf /tmp/catsn.swift -n=$n -data=$data 2> std.err
-EXITCODE=$?
-
-swiftlog=`\ls -t *.log | head -1`
-
-#/tmp/plotswiftlogs $logfile 2>gnuplot.err
-cat << 'EOF' > /tmp/plotswiftlogs
-#!/bin/bash
-
-#usage: ./extract4plots <swift.logfile>
-
-SWIFTLOGFILE=$1
-#TMPDIR=`mktemp -d plotlog.XXX`
-grep -i ProgressTicker $SWIFTLOGFILE > swiftoutfile.out
-SWIFTOUTFILE=swiftoutfile.out
-#extract start time
-TMPDATE=`grep -i progress $SWIFTOUTFILE 2>/dev/null | head -n 1 | cut -f1-2 -d ' '`
-START_TIME=`date +%s -d "$TMPDATE"`
-
-#extract end time
-TMPDATE=`grep -i progress $SWIFTOUTFILE 2>/dev/null | tail -n 1 | cut -f1-2 -d ' '`
-END_TIME=`date +%s -d "$TMPDATE"`
-
-#duration
-DIFFTIME=$((END_TIME - START_TIME))
-#echo 0 > active.txt
-#echo 0 > cumulative.txt
-#extract active runs in a file
-(grep -o -i "Active:[0-9]*" $SWIFTOUTFILE 2>/dev/null | awk -F: '{print $2}' > active.txt)
-
-#extract successful completions in a file
-(grep -o -i "Successfully:[0-9]*" $SWIFTOUTFILE 2>/dev/null | awk -F: '{print $2}' > cumulative.txt)
-
-#prepare tics
-activelines=`wc -l active.txt | awk '{print $1}'`
-cumulines=`wc -l cumulative.txt | awk '{print $1}'`
-
-if [ $activelines -ne 0 ]
-then
- activelinespertic=`echo "scale=5 ; $DIFFTIME / $activelines" | bc`
-fi
-seq 0 $activelinespertic $DIFFTIME > activetics.txt
-
-if [ $cumulines -ne 0 ]
-then
- cumulinespertic=`echo "scale=5 ; $DIFFTIME / $cumulines" | bc`
-fi
-
-seq 0 $cumulinespertic $DIFFTIME > cumultics.txt
-
-#final plot data
-paste activetics.txt active.txt > plot_active.txt
-paste cumultics.txt cumulative.txt > plot_cumulative.txt
-
-cat << EOF1 > plotit.gp
-set terminal png enhanced
-set output "activeplot.png"
-set nokey
-set xlabel "Time in sec"
-set ylabel "number of active jobs"
-set title "Active jobs"
-plot "plot_active.txt" using 1:2 with line
-
-set output "cumulativeplot.png"
-set ylabel "number of completed jobs"
-set title "Cumulative jobs"
-plot "plot_cumulative.txt" using 1:2 with lines
-EOF1
-
-gnuplot plotit.gp 2>/dev/null
-
-#cp *.png ../
-echo `pwd`>/tmp/curdir
-echo `ls`>/tmp/dirlist
-#cd ..
-EOF
-
-sh /tmp/plotswiftlogs $swiftlog
-cp *.png $outdir/
-
-#for i in `find $outdir`
-#do
-# lastfile=$i
-#done
-#bname=`basename $lastfile`
-
-cat << EOF > $htmlfile
-<!DOCTYPE HTML>
-
-<html xmlns="http://www.w3.org/1999/xhtml">
-<head>
- <title>Catsn output</title>
-</head>
-<body>
-<h2>Swift run trace plots</h2>
-
-<a href="activeplot.png" alt="active plot">active plot</a>
-<a href="cumulativeplot.png" alt="cumulative plot">cumulative plot</a>
-
-<h2>Browse output of catsn application</h2>
-`for i in \`find $outdir -type f\`; do echo "<a href=\`basename $i\`>Output: \`basename $i\`</a><br/>"; done`
-</body>
-</html>
-EOF
-
-if [ "$EXITCODE" -ne "0" ]; then
- cat std.err >&2
-fi
-cp std.err $logfile
-rm std.err
-exit $EXITCODE
Copied: SwiftApps/swift-galaxy/swift/swift-catsn.sh (from rev 7523, SwiftApps/swift-galaxy/swift/swift-catsn)
===================================================================
--- SwiftApps/swift-galaxy/swift/swift-catsn.sh (rev 0)
+++ SwiftApps/swift-galaxy/swift/swift-catsn.sh 2014-01-30 23:20:58 UTC (rev 7529)
@@ -0,0 +1,276 @@
+#!/bin/bash
+
+#set -x
+set -e
+
+site=$1
+data=$2
+n=$3
+outdir=$4
+logfile=$5
+htmlfile=$6
+mkdir -p $outdir
+
+swift=`which swift`
+# use "here" document for configs
+
+cat << EOF > /tmp/catsn.swift
+type file;
+
+app (file o) cat (file i) {
+ cat @i stdout=@o;
+}
+#file out[]<simple_mapper; location="outdir", prefix="f.",suffix=".out">;
+file out[]<simple_mapper; location="$outdir", prefix="f.",suffix=".txt">;
+
+string d=@arg("data");
+
+foreach j in [1:@toInt(@arg("n","1"))] {
+ file data<single_file_mapper; file=d>;
+ out[j] = cat(data);
+}
+EOF
+
+case "$site" in
+midway )
+cat << 'EOF' > /tmp/sites.xml
+<?xml version="1.0" encoding="UTF-8"?>
+<config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites">
+ <pool handle="midway">
+ <execution provider="coaster" url="swift.rcc.uchicago.edu" jobmanager="ssh-cl:slurm"/>
+ <profile namespace="globus" key="queue">sandyb</profile>
+ <profile namespace="globus" key="jobsPerNode">16</profile>
+ <profile namespace="globus" key="maxWalltime">02:00:00</profile>
+ <profile namespace="globus" key="maxTime">7500</profile>
+ <profile namespace="globus" key="highOverAllocation">100</profile>
+ <profile namespace="globus" key="lowOverAllocation">100</profile>
+ <profile namespace="globus" key="slots">2</profile>
+ <profile namespace="globus" key="maxNodes">1</profile>
+ <profile namespace="globus" key="nodeGranularity">1</profile>
+ <profile namespace="karajan" key="jobThrottle">.31</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ <workdirectory>/tmp/swift.work</workdirectory>
+ </pool>
+</config>
+EOF
+;;
+
+uc3 )
+cat << 'EOF' > /tmp/sites.xml
+<?xml version="1.0" encoding="UTF-8"?>
+<config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites">
+ <pool handle="uc3">
+ <execution provider="coaster" url="uc3-sub.uchicago.edu" jobmanager="ssh-cl:condor"/>
+ <!--<execution provider="coaster" jobmanager="local:condor"/>-->
+ <profile namespace="karajan" key="jobThrottle">.09</profile>
+ <profile namespace="karajan" key="initialScore">1000</profile>
+ <profile namespace="globus" key="jobsPerNode">1</profile>
+ <profile namespace="globus" key="maxWalltime">3600</profile>
+ <profile namespace="globus" key="nodeGranularity">1</profile>
+ <profile namespace="globus" key="highOverAllocation">100</profile>
+ <profile namespace="globus" key="lowOverAllocation">100</profile>
+ <profile namespace="globus" key="slots">1</profile>
+ <profile namespace="globus" key="condor.Requirements">UidDomain == "osg-gk.mwt2.org"</profile>
+ <profile namespace="globus" key="maxNodes">1</profile>
+ <profile namespace="globus" key="condor.+AccountingGroup">"group_friends.{env.USER}"</profile>
+ <profile namespace="globus" key="jobType">nonshared</profile>
+ <filesystem provider="local" url="none" />
+ <!--<workdirectory>/home/maheshwari/test/catsn/swift.work</workdirectory>-->
+ <workdirectory>/tmp/swift.work</workdirectory>
+ </pool>
+</config>
+EOF
+;;
+
+stampede )
+cat << 'EOF' > /tmp/sites.xml
+<?xml version="1.0" encoding="UTF-8"?>
+<config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites">
+<pool handle="stampede">
+ <execution provider="coaster" jobmanager="ssh-cl:slurm" url="stampede.tacc.utexas.edu"/>
+ <filesystem provider="local"/>
+ <profile namespace="globus" key="jobsPerNode">16</profile>
+ <profile namespace="globus" key="ppn">16</profile>
+ <profile namespace="globus" key="project">TG-STA110005S</profile>
+ <profile namespace="globus" key="queue">normal</profile>
+ <profile namespace="globus" key="slurm.mail-user">ketancmaheshwari at gmail.com</profile>
+ <profile namespace="globus" key="slurm.mail-type">ALL</profile>
+ <profile namespace="globus" key="maxWalltime">02:00:00</profile>
+ <profile namespace="globus" key="maxTime">8000</profile>
+ <profile namespace="globus" key="highOverAllocation">100</profile>
+ <profile namespace="globus" key="lowOverAllocation">100</profile>
+ <profile namespace="globus" key="slots">1</profile>
+ <profile namespace="globus" key="maxNodes">1</profile>
+ <profile namespace="globus" key="nodeGranularity">1</profile>
+ <profile namespace="karajan" key="jobThrottle">.15</profile>
+ <profile namespace="karajan" key="initialScore">10000</profile>
+ <workdirectory>/tmp/swift.work</workdirectory>
+</pool>
+</config>
+EOF
+;;
+
+localhost )
+cat << 'EOF' > /tmp/sites.xml
+<?xml version="1.0" encoding="UTF-8"?>
+<config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites">
+<pool handle="localhost">
+ <execution provider="local" jobmanager="local:local"/>
+ <filesystem provider="local"/>
+ <!--<workdirectory>/home/kcm92/powergridapps/swiftscripts/swift.work</workdirectory>-->
+ <workdirectory>/tmp/swift.work</workdirectory>
+ <profile namespace="karajan" key="jobThrottle">0.31</profile>
+</pool>
+</config>
+EOF
+;;
+
+* )
+echo "defaulting to localhost"
+
+cat << 'EOF' > /tmp/sites.xml
+<?xml version="1.0" encoding="UTF-8"?>
+<config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites">
+<pool handle="localhost">
+ <execution provider="local" jobmanager="local:local"/>
+ <filesystem provider="local"/>
+ <!--<workdirectory>/home/kcm92/powergridapps/swiftscripts/swift.work</workdirectory>-->
+ <workdirectory>/tmp/swift.work</workdirectory>
+ <profile namespace="karajan" key="jobThrottle">0.31</profile>
+ </pool>
+</config>
+EOF
+;;
+esac
+
+cat << 'EOF' > /tmp/cf
+use.provider.staging=true
+wrapperlog.always.transfer=true
+execution.retries=1
+provider.staging.pin.swiftfiles=false
+sitedir.keep=true
+lazy.errors=false
+EOF
+
+cat << 'EOF' > /tmp/tc
+localhost echo echo
+localhost cat cat
+stampede echo echo
+stampede cat cat
+uc3 echo /bin/echo
+uc3 cat /bin/cat
+midway echo echo
+midway cat cat
+EOF
+
+$swift -sites.file /tmp/sites.xml -tc.file /tmp/tc -config /tmp/cf /tmp/catsn.swift -n=$n -data=$data 2> std.err
+EXITCODE=$?
+
+swiftlog=`\ls -t *.log | head -1`
+
+#/tmp/plotswiftlogs $logfile 2>gnuplot.err
+cat << 'EOF' > /tmp/plotswiftlogs
+#!/bin/bash
+
+#usage: ./extract4plots <swift.logfile>
+
+SWIFTLOGFILE=$1
+#TMPDIR=`mktemp -d plotlog.XXX`
+grep -i ProgressTicker $SWIFTLOGFILE > swiftoutfile.out
+SWIFTOUTFILE=swiftoutfile.out
+#extract start time
+TMPDATE=`grep -i progress $SWIFTOUTFILE 2>/dev/null | head -n 1 | cut -f1-2 -d ' '`
+START_TIME=`date +%s -d "$TMPDATE"`
+
+#extract end time
+TMPDATE=`grep -i progress $SWIFTOUTFILE 2>/dev/null | tail -n 1 | cut -f1-2 -d ' '`
+END_TIME=`date +%s -d "$TMPDATE"`
+
+#duration
+DIFFTIME=$((END_TIME - START_TIME))
+#echo 0 > active.txt
+#echo 0 > cumulative.txt
+#extract active runs in a file
+(grep -o -i "Active:[0-9]*" $SWIFTOUTFILE 2>/dev/null | awk -F: '{print $2}' > active.txt)
+
+#extract successful completions in a file
+(grep -o -i "Successfully:[0-9]*" $SWIFTOUTFILE 2>/dev/null | awk -F: '{print $2}' > cumulative.txt)
+
+#prepare tics
+activelines=`wc -l active.txt | awk '{print $1}'`
+cumulines=`wc -l cumulative.txt | awk '{print $1}'`
+
+if [ $activelines -ne 0 ]
+then
+ activelinespertic=`echo "scale=5 ; $DIFFTIME / $activelines" | bc`
+fi
+seq 0 $activelinespertic $DIFFTIME > activetics.txt
+
+if [ $cumulines -ne 0 ]
+then
+ cumulinespertic=`echo "scale=5 ; $DIFFTIME / $cumulines" | bc`
+fi
+
+seq 0 $cumulinespertic $DIFFTIME > cumultics.txt
+
+#final plot data
+paste activetics.txt active.txt > plot_active.txt
+paste cumultics.txt cumulative.txt > plot_cumulative.txt
+
+cat << EOF1 > plotit.gp
+set terminal png enhanced
+set output "activeplot.png"
+set nokey
+set xlabel "Time in sec"
+set ylabel "number of active jobs"
+set title "Active jobs"
+plot "plot_active.txt" using 1:2 with line
+
+set output "cumulativeplot.png"
+set ylabel "number of completed jobs"
+set title "Cumulative jobs"
+plot "plot_cumulative.txt" using 1:2 with lines
+EOF1
+
+gnuplot plotit.gp 2>/dev/null
+
+#cp *.png ../
+echo `pwd`>/tmp/curdir
+echo `ls`>/tmp/dirlist
+#cd ..
+EOF
+
+sh /tmp/plotswiftlogs $swiftlog
+cp *.png $outdir/
+
+#for i in `find $outdir`
+#do
+# lastfile=$i
+#done
+#bname=`basename $lastfile`
+
+cat << EOF > $htmlfile
+<!DOCTYPE HTML>
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+<head>
+ <title>Catsn output</title>
+</head>
+<body>
+<h2>Swift run trace plots</h2>
+
+<a href="activeplot.png" alt="active plot">active plot</a>
+<a href="cumulativeplot.png" alt="cumulative plot">cumulative plot</a>
+
+<h2>Browse output of catsn application</h2>
+`for i in \`find $outdir -type f\`; do echo "<a href=\`basename $i\`>Output: \`basename $i\`</a><br/>"; done`
+</body>
+</html>
+EOF
+
+if [ "$EXITCODE" -ne "0" ]; then
+ cat std.err >&2
+fi
+cp std.err $logfile
+rm std.err
+exit $EXITCODE
Copied: SwiftApps/swift-galaxy/swift/swift-catsn.xml (from rev 7523, SwiftApps/swift-galaxy/swift/catsn.xml)
===================================================================
--- SwiftApps/swift-galaxy/swift/swift-catsn.xml (rev 0)
+++ SwiftApps/swift-galaxy/swift/swift-catsn.xml 2014-01-30 23:20:58 UTC (rev 7529)
@@ -0,0 +1,22 @@
+<tool id="catsn" name="Swift catsn">
+ <description>A Swift tool to run the catsn workflow</description>
+
+ <command interpreter="bash">
+ swift-catsn "$site" "$data" "$n" "$html_file.files_path" "$log_file" "$html_file"
+ </command>
+
+ <inputs>
+ <param name="site" type="select" label="Execution Location">
+ <option value="localhost">Localhost</option>
+ <option value="midway">Midway</option>
+ <option value="uc3">UC3</option>
+ <option value="stampede">Stampede</option>
+ </param>
+ <param name="n" type="text" value="1" label="Number of tasks to run "/>
+ <param name="data" type="data" label="Input data file"/>
+ </inputs>
+ <outputs>
+ <data format="txt" name="log_file" type="data" label="Swift output" />
+ <data format="html" name="html_file" label="catsnoutput.html" />
+ </outputs>
+</tool>
Copied: SwiftApps/swift-galaxy/swift/swift-skeleton.xml (from rev 7523, SwiftApps/swift-galaxy/swift/catsn.xml)
===================================================================
--- SwiftApps/swift-galaxy/swift/swift-skeleton.xml (rev 0)
+++ SwiftApps/swift-galaxy/swift/swift-skeleton.xml 2014-01-30 23:20:58 UTC (rev 7529)
@@ -0,0 +1,22 @@
+<tool id="skeleton(replace with an id)" name="Swift Skeleton (replace with a name)">
+ <description>A description here</description>
+
+ <command interpreter="bash">
+ swift-skeleton.sh "$site" "$data" "$n" "$html_file.files_path" "$log_file" "$html_file"
+ </command>
+
+ <inputs>
+ <param name="site" type="select" label="Execution Location">
+ <option value="localhost">Localhost</option>
+ <option value="midway">Midway</option>
+ <option value="uc3">UC3</option>
+ <option value="stampede">Stampede</option>
+ </param>
+ <param name="n" type="text" value="1" label="Number of tasks to run "/>
+ <param name="data" type="data" label="Input data file"/>
+ </inputs>
+ <outputs>
+ <data format="txt" name="log_file" type="data" label="Standard out" />
+ <data format="html" name="html_file" label="catsnoutput.html" />
+ </outputs>
+</tool>
Modified: SwiftApps/swift-galaxy/swift/swiftforeachlist.sh
===================================================================
--- SwiftApps/swift-galaxy/swift/swiftforeachlist.sh 2014-01-30 15:26:30 UTC (rev 7528)
+++ SwiftApps/swift-galaxy/swift/swiftforeachlist.sh 2014-01-30 23:20:58 UTC (rev 7529)
@@ -16,6 +16,8 @@
listfile=$1
shift
logfile=$1
+shift
+outlistfile=$1
cat << EOF > /tmp/script.swift
type file;
@@ -60,12 +62,6 @@
</config>
EOF
-cat << EOF > /tmp/tc
-midway echo echo
-midway sh /bin/sh
-midway cat cat
-EOF
-
;;
uc3 )
@@ -93,12 +89,6 @@
</config>
EOF
-cat << EOF > /tmp/tc
-uc3 sh /bin/sh
-uc3 echo /bin/echo
-uc3 cat /bin/cat
-EOF
-
;;
stampede )
@@ -127,12 +117,6 @@
</config>
EOF
-cat << EOF > /tmp/tc
-stampede sh /bin/sh
-stampede echo echo
-stampede cat cat
-EOF
-
;;
localhost )
@@ -148,11 +132,6 @@
</config>
EOF
-cat << EOF > /tmp/tc
-localhost sh /bin/sh
-localhost echo echo
-localhost cat cat
-EOF
;;
* )
@@ -169,12 +148,6 @@
</config>
EOF
-cat << EOF > /tmp/tc
-localhost sh /bin/sh
-localhost echo echo
-localhost cat cat
-EOF
-
;;
esac
@@ -187,9 +160,29 @@
lazy.errors=false
EOF
+cat << EOF > /tmp/tc
+localhost sh /bin/sh
+localhost echo echo
+localhost cat cat
+stampede sh /bin/sh
+stampede echo echo
+stampede cat cat
+uc3 sh /bin/sh
+uc3 echo /bin/echo
+uc3 cat cat
+midway echo echo
+midway sh /bin/sh
+midway cat cat
+EOF
+
$swift -sites.file /tmp/sites.xml -tc.file /tmp/tc -config /tmp/cf /tmp/script.swift 2>$logfile
EXITCODE=$?
+#`for i in \`find $HOME/swift-sandbox -type f\`; do echo "\`basename $i\` $i"; done`
+cat << EOF > $outlistfile
+`for i in \`find $HOME/swift-sandbox -type f\`; do echo "$i"; done`
+EOF
+
if [ "$EXITCODE" -ne "0" ]; then
cat $logfile >&2
fi
Modified: SwiftApps/swift-galaxy/swift/swiftforeachlist.xml
===================================================================
--- SwiftApps/swift-galaxy/swift/swiftforeachlist.xml 2014-01-30 15:26:30 UTC (rev 7528)
+++ SwiftApps/swift-galaxy/swift/swiftforeachlist.xml 2014-01-30 23:20:58 UTC (rev 7529)
@@ -7,6 +7,7 @@
$executable
$listfile
$log_file
+ $outlist
</command>
<inputs>
@@ -46,9 +47,10 @@
<outputs>
<data format="txt" name="log_file" type="data" label="Swift output" />
+ <data format="txt" name="outlist" type="data" label="Output list" />
</outputs>
- <help>
+<help>
.. class:: warningmark
**TIP**. A generic Swift foreach to run arbitrary executables in parallel over a list of items listed in a file.
Modified: SwiftApps/swift-galaxy/swift/swiftforeachrange.sh
===================================================================
--- SwiftApps/swift-galaxy/swift/swiftforeachrange.sh 2014-01-30 15:26:30 UTC (rev 7528)
+++ SwiftApps/swift-galaxy/swift/swiftforeachrange.sh 2014-01-30 23:20:58 UTC (rev 7529)
@@ -20,6 +20,8 @@
stepsize=$1
shift
logfile=$1
+shift
+outlistfile=$1
cat << EOF > /tmp/script.swift
type file;
@@ -61,13 +63,6 @@
</pool>
</config>
EOF
-
-cat << EOF > /tmp/tc
-midway echo echo
-midway sh /bin/sh
-midway cat cat
-EOF
-
;;
uc3 )
@@ -94,13 +89,6 @@
</pool>
</config>
EOF
-
-cat << EOF > /tmp/tc
-uc3 sh /bin/sh
-uc3 echo /bin/echo
-uc3 cat /bin/cat
-EOF
-
;;
stampede )
@@ -128,13 +116,6 @@
</pool>
</config>
EOF
-
-cat << EOF > /tmp/tc
-stampede sh /bin/sh
-stampede echo echo
-stampede cat cat
-EOF
-
;;
localhost )
@@ -150,11 +131,6 @@
</config>
EOF
-cat << EOF > /tmp/tc
-localhost sh /bin/sh
-localhost echo echo
-localhost cat cat
-EOF
;;
* )
@@ -171,12 +147,6 @@
</config>
EOF
-cat << EOF > /tmp/tc
-localhost sh /bin/sh
-localhost echo echo
-localhost cat cat
-EOF
-
;;
esac
@@ -189,9 +159,30 @@
lazy.errors=false
EOF
+cat << EOF > /tmp/tc
+localhost sh /bin/sh
+localhost echo echo
+localhost cat cat
+uc3 sh /bin/sh
+uc3 echo /bin/echo
+uc3 cat /bin/cat
+stampede sh /bin/sh
+stampede echo echo
+stampede cat cat
+midway echo echo
+midway sh /bin/sh
+midway cat cat
+EOF
+
$swift -sites.file /tmp/sites.xml -tc.file /tmp/tc -config /tmp/cf /tmp/script.swift 2>$logfile
EXITCODE=$?
+#`for i in \`find $HOME/swift-sandbox -type f\`; do echo "\`basename $i\` $i"; done`
+cat << EOF > $outlistfile
+`for i in \`find $HOME/swift-sandbox -type f\`; do echo "$i"; done`
+EOF
+
+
if [ "$EXITCODE" -ne "0" ]; then
cat $logfile >&2
fi
Modified: SwiftApps/swift-galaxy/swift/swiftforeachrange.xml
===================================================================
--- SwiftApps/swift-galaxy/swift/swiftforeachrange.xml 2014-01-30 15:26:30 UTC (rev 7528)
+++ SwiftApps/swift-galaxy/swift/swiftforeachrange.xml 2014-01-30 23:20:58 UTC (rev 7529)
@@ -8,7 +8,8 @@
$rstart
$rend
$stepsize
- $log_file
+ $logfile
+ $outlist
</command>
<inputs>
@@ -49,12 +50,16 @@
</conditional>
</inputs>
<outputs>
- <data format="txt" name="log_file" type="data" label="Swift output" />
+ <data format="txt" name="logfile" type="data" label="Swift output" />
+ <data format="txt" name="outlist" type="data" label="Output list" />
</outputs>
- <code file="swiftforeachrangepp.py">
- <hook postprocess="postprocess" />
+
+ <!--
+ <code file="postprocess.py">
+ <hook postprocess="writeoutlist" />
</code>
- <help>
+ -->
+<help>
.. class:: warningmark
**TIP**. A generic Swift foreach to run arbitrary executables in parallel over a range of numbers.
@@ -69,13 +74,13 @@
**Example**
-Pick up an executable from the Executable list. Select the start, end and the step size values.
+Pick up an executable from the Executable list. Enter the start, end and the step size values.
If an executable is echo.sh:
echo $1
-Start, end and step is 0, 9 and 2, the executable will be invoked in parallel 5 times for n where n is 0 2 4 6 8:
+The start, end and step is 0, 9 and 2, the executable will be invoked in parallel 5 times for n where n is 0 2 4 6 8:
echo n
Deleted: SwiftApps/swift-galaxy/swift/swiftforeachrangepp.py
===================================================================
--- SwiftApps/swift-galaxy/swift/swiftforeachrangepp.py 2014-01-30 15:26:30 UTC (rev 7528)
+++ SwiftApps/swift-galaxy/swift/swiftforeachrangepp.py 2014-01-30 23:20:58 UTC (rev 7529)
@@ -1,67 +0,0 @@
-#post processing, set build for data and add additional data to history^M
-from galaxy import datatypes, config, jobs, tools
-from galaxy.model import HistoryDatasetAssociation
-import operator
-import shutil
-import os
-import glob
-from lxml import etree
-from os.path import basename
-
-def createDsRef(outDataItems, param_dict, outputName, outputFormat, outputDsRefName, nameSpace):
- dsrefXML="<?xml version=\"1.0\" ?>\n"
- dsrefXML=dsrefXML+"<dataSetRefs xmlns=\"http://ci.uchicago.edu/galaxy-es/datasetrefs/"+nameSpace+"\" >\n"
-
- dsrefFileName=""
-
- for outDataName, outDataItem in outDataItems:
- if outDataName==outputDsRefName:
- dsrefFileName=outDataItem.get_file_name()
- else:
- if outDataName==outputName:
- dsrefXML=dsrefXML+"<dataSetRef id=\"%s\" name=\"%s\" path=\"%s\" type=\"%s\"/>\n" % (outDataItem.id,outDataItem.name,outDataItem.get_file_name(),outputFormat)
-
- print "dsrefFileName:"+dsrefFileName
-
- primaryCollectedDatasets=param_dict["__collected_datasets__"]["primary"]
-
- for primaryCollectedDataset in primaryCollectedDatasets:
- primaryCollectedDatasetItems=primaryCollectedDatasets[primaryCollectedDataset]
- for primaryCollectedDatasetItem in primaryCollectedDatasetItems:
- outDataItem=primaryCollectedDatasetItems[primaryCollectedDatasetItem]
- if outputFormat in outDataItem.extension:
-
- dsrefXML=dsrefXML+"<dataSetRef id=\"%s\" name=\"%s\" path=\"%s\" type=\"%s\"/>\n" % (outDataItem.id,outDataItem.name,outDataItem.get_file_name(),outDataItem.extension)
-
-
- dsrefXML=dsrefXML+"</dataSetRefs>\n"
-
- return dsrefFileName, dsrefXML
-
-def writeDsRef(dsrefFileName, dsrefXML):
- dsrefFile = open(dsrefFileName, "w",0)
- dsrefFile.write(dsrefXML)
- dsrefFile.flush()
- dsrefFile.close()
-
-def setSizeAndPeak(outDataItems,outputDsRefName):
- for outDataName, outDataItem in outDataItems:
- if outDataName==outputDsRefName:
- outDataItem.dataset.file_size=None
- outDataItem.set_size()
- outDataItem.set_peek()
-
-def exec_after_process(app, inp_data, out_data, param_dict, tool, stdout, stderr):
-
- outDataItems = out_data.items()
- outDataItems = sorted(outDataItems, key=operator.itemgetter(0))
-
- dsrefAcmoFileName, dsrefAcmoXML = createDsRef(outDataItems,param_dict,"acmo","acmo.csv","acmoDsRef","facit/acmo")
-
- writeDsRef(dsrefAcmoFileName, dsrefAcmoXML)
-
- setSizeAndPeak(outDataItems,"acmoDsRef")
-
-
- app.model.context.flush()
-
More information about the Swift-commit
mailing list