[Swift-commit] r8328 - SwiftApps/dsp-swift/galaxy
ketan at ci.uchicago.edu
ketan at ci.uchicago.edu
Thu Dec 11 13:21:36 CST 2014
Author: ketan
Date: 2014-12-11 13:21:36 -0600 (Thu, 11 Dec 2014)
New Revision: 8328
Removed:
SwiftApps/dsp-swift/galaxy/README
SwiftApps/dsp-swift/galaxy/setup.py
SwiftApps/dsp-swift/galaxy/swiftjumpdsp.py
SwiftApps/dsp-swift/galaxy/swiftjumpdsp.xml
Log:
cleanup and refactor
Deleted: SwiftApps/dsp-swift/galaxy/README
===================================================================
--- SwiftApps/dsp-swift/galaxy/README 2014-12-11 19:04:26 UTC (rev 8327)
+++ SwiftApps/dsp-swift/galaxy/README 2014-12-11 19:21:36 UTC (rev 8328)
@@ -1,196 +0,0 @@
-The jump-DSP Swift Galaxy Integration
-======================================
-
-Introduction
--------------
-
-The Swift-Galaxy integration project aims to build tools which can be used to
-leverage the best features of http://swift-lang.org/main[Swift] and
-http://galaxyproject.org[Galaxy] combined. We start with building a generic
-Galaxy tool, called `foreach` which allows user to compose Galaxy workflows
-such that each computation stage in the workflow is executed by Swift. User can
-compose Galaxy tool and pick up arbitrary executables and dataset to run the
-executable over. User can also run the executable over a list of items such as
-string or file parameters.
-
-NOTE: This document can be found http://swift-lang.org/tutorials/galaxy/tutorial.html[here].
-
-Swift-Galaxy `foreach` Mechanism
----------------------------------
-
-The basic mechanism is that the tool user-interface will accept parameters and
-will pass them along to a backend Swift "generator". This generator program
-will run the parameters over Swift source and configuration templates. As a
-result, the right source and configuration files are produced in a temporary
-directory (`/tmp/tmpname`). This is then executed and the results collected.
-
-There are two ways the `foreach` Swift tool can be utilized. First, over a range
-of numbers. The tool is used to run any arbitrary executable via Swift `foreach`
-loop over a range of numbers. The executable will run with number in range as
-commandline argument, one at a time. Furthermore, user can add arbitrary string
-arguments in addition to the number argument to the call to executable.
-
-The second way runs the `foreach` loop over a list of items read from an input
-file. These items can be file references or simple strings. The tool will pass
-one item at a time to the executable as argument parameter and run them all in
-parallel. This is similar to the above described number range except that the
-iteration takes place over a list of items provided as a file.
-
-
-Installation
--------------
-This installation recipe assumes that you have Galaxy preinstalled. If not,
-installation instructions for Galaxy can be found
-https://wiki.galaxyproject.org/Admin/GetGalaxy[here].
-
-Step 0. Checkout the swift-galaxy SVN directory as follows:
-
-----
-svn co https://svn.ci.uchicago.edu/svn/vdl2/SwiftApps/swift-galaxy
-----
-
-Step 1. Create a symbolic link to the +swift-galaxy/swift+ directory from
-galaxy's +tools+ directory. For example, if Galaxy top level directory is
-+galaxy-dist+ and swift directory is $HOME/swift-galaxy/swift:
-
-----
-cd galaxy-dist/tools
-ln -s $HOME/swift-galaxy/swift swift
-----
-
-The advantage of symbolic links is that as users update the tool directory via
-`svn up`, they need not worry about reinstallation. Updates in installation
-gets automatically propagated to Galaxy.
-
-Step 2. Add the tool definitions of the tools contained in this Swift to
-Galaxy's tool_conf.xml. This can be done by copying the text in
-`<section>...</section>` from the tool_conf.xml in swift-galaxy directory and
-paste it to the tool_conf.xml of the Galaxy directory. In the current installation, the text to be copied is:
-
-----
-<section name="swift" id="swift">
- <tool file="swift/tryswift.xml" />
- <tool file="swift/genericswift.xml" />
- <tool file="swift/swiftforeach.xml" />
-</section>
-----
-
-Step 3. (Re)start Galaxy.
-
-----
-sh run.sh
-----
-
-In the Galaxy Web GUI, you should see the tools in a toolset titled Swift.
-
-Screenshots
------------
-
-The following screenshot shows overall look-and-feel of the Swift-Galaxy tool
-in the Galaxy web GUI.
-
-image::./screenshots/overall.png["swift-galaxy screenshot", width=600, align="center"]
-
-The rest of this section walks you through each of the components shown in the
-above screen and their utility. The following screenshot shows execution
-location selector. User can select one or more execution locations from the
-list as their target execution site.
-
-image::./screenshots/execloc.png["execution location selector screenshot", width=200, align="center"]
-
-The following screenshot shows a dialog box to add Swift-level commandline
-arguments to be passed to the script.
-
-image::./screenshots/swift-level-args.png["swift-args screenshot", width=200, align="center"]
-
-The following screenshot shows a selection dropdown list to chose the execution
-interpreter from. The interpreters currently supported and tested are shell,
-java and python. Upcoming are R and matlab.
-
-image::./screenshots/execution-interpreter.png["interpreter screenshot", width=150, align="center"]
-
-The following screenshot shows the selection for an executable. User can upload
-an executable and select it from the list.
-
-image::./screenshots/executable.png["executable screenshot", width=200, align="center"]
-
-The following screenshot shows a selection for range of numbers or list of
-items to chose from. User can select one and the corresponding dialog box will
-appear below.
-
-image::./screenshots/rangeorlist.png["select range or list screenshot", width=150, align="center"]
-
-The following screenshot shows a dialog for string or file type arguments. A
-user can add string and filetype arguments to their executable and the
-commandline will be built accordingly.
-
-image::./screenshots/string-or-fileargs.png["string or fileargs screenshot", width=250, align="center"]
-
-The following screenshot shows a dialog box to enter the directory location for
-results of execution. The Swift output files will be placed here:
-
-image::./screenshots/outlocation.png["output location screenshot", width=250, align="center"]
-
-An example
-----------
-
-A simple Hello World Swift tool to run with Galaxy.
-
-To set up with your Galaxy environment follow the steps below:
-Prerequisite: Sun/Oracle java. Should work with IBM java but not tested. To test your java version:
-
-----
-java -version
-----
-
-Typical output is:
-
-----
-java version "1.7.0_17"
-Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
-Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
-----
-
-Step 1. Download and install Swift where Galaxy server is running. From a
-command prompt the following wget will pull the latest release from
-http://swift-lang.org/downloads/index.php[here].
-
-Step 2. SVN checkout the swift-galaxy directory and copy the Swift tool directory to Galaxy's tools directory:
-
-----
-svn co https://svn.ci.uchicago.edu/svn/vdl2/SwiftApps/swift-galaxy
-cp -r swift-galaxy/swift /location/of/galaxy-server/tools/
-----
-
-Step 3. Edit the +tool_conf.xml+ file present at the top-level of galaxy directory tree to add the following lines between the +<toolbox>+ and +</toolbox>+ xml tags:
-
-----
-<section name="swift-galaxy" id="swift-galaxy">
- <tool file="currdir/nameoftoolfile.xml" />
-</section>
-----
-
-Step 4. Restart the Galaxy server. The swift-galaxy tool should be available in the tool box of Galaxy GUI.
-
-Step 5. Invoke it by entering any number in the dialog box or leaving it to default.
-
-Step 6. For examples of other Galaxy-Swift tools, look into the swift directoty.
-
-Troubleshoot
--------------
-
-Each tool generates a set of Swift source and configuration files. These files are placed in a unique directory with a random name generated on the fly. The location of the directory is in `/tmp` and it is prefixed with name `swift-gal.` . The suffix is a four letter random number. You can run Swift from this directory to reproduce possible issues as follows:
-
-----
-swift -tc.file tc -sites.file sites.xml -config cf script.swift
-----
-
-Feel free to contact Swift http://swift-lang.org/support/index.php[support] if issues persist.
-
-Upcoming Features
-------------------
-
-* Mixing the order of file and string arguments
-* Optionally allowing user to not use range number as arguments
-* Providing more Swift patterns, eg. `if-else` conditionals
-
Deleted: SwiftApps/dsp-swift/galaxy/setup.py
===================================================================
--- SwiftApps/dsp-swift/galaxy/setup.py 2014-12-11 19:04:26 UTC (rev 8327)
+++ SwiftApps/dsp-swift/galaxy/setup.py 2014-12-11 19:21:36 UTC (rev 8328)
@@ -1,40 +0,0 @@
-#!/usr/bin/env python
-
-"""
-Setup script to install swift-galaxy framework to your Galaxy installation.
-
-Usage: ./setup.py <path/to/galaxy>
-"""
-
-import xml.etree.ElementTree as ET
-import subprocess
-import sys
-import os
-import distutils.spawn
-import traceback
-import fnmatch
-import tarfile
-import urllib
-
-galaxyloc=sys.argv[1]
-
-#step 0. Download swift-galaxy.tgz and swift-rel.tgz untar and set path
-
-#step 1. Create symlink to the swift directory from the tools directory of galaxyloc if not already present
-if os.path.lexists(galaxyloc+"/tools/swift"):
- os.remove(galaxyloc+"/tools/swift")
-
-os.symlink ("/tmp/swift-galaxy/swift", galaxyloc+"/tools/swift");
-
-#step 2. update the tool_conf.xml
-tree = ET.parse(galaxyloc+"/config/tool_conf.xml.main")
-root = tree.getroot()
-
-if 'swift' not in root.attrib:
- root.append((ET.fromstring('<section name="swift" id="swift">\n <tool file="swift/tryswift.xml" />\n <tool file="swift/genericswift.xml" />\n <tool file="swift/swiftforeach.xml" />\n </section>\n ')))
-
-tree.write (galaxyloc+"/config/tool_conf.xml.main")
-
-#step 3. Print message
-print "Swift/Galaxy Installed."
-
Deleted: SwiftApps/dsp-swift/galaxy/swiftjumpdsp.py
===================================================================
--- SwiftApps/dsp-swift/galaxy/swiftjumpdsp.py 2014-12-11 19:04:26 UTC (rev 8327)
+++ SwiftApps/dsp-swift/galaxy/swiftjumpdsp.py 2014-12-11 19:21:36 UTC (rev 8328)
@@ -1,201 +0,0 @@
-#!/usr/bin/env python
-
-"""
-This is the Swift foreach driver for Galaxy. It receives inputs from the Swift
-"foreach" Galaxy tool and depending on the options specified, builds a
-canonical Swift/K or Swift/T script and runs over specified computational
-environment.
-
-swiftjumpdsp.py
- "${inlocation}"
- "${a}"
- "${n}"
- "${p}"
- "${I}"
- "${outlist}"
-
-"""
-
-import subprocess
-import sys
-import os
-import distutils.spawn
-import traceback
-import fnmatch
-import re
-
-def setwdir():
- return subprocess.check_output(["mktemp", "-d", "/tmp/swift-gal.XXXX"])
-
-def genargs(args):
- for a in args:
- yield a
-
-def buildcfg(wdir):
- f=open(wdir+"/cf", 'w')
- f.write("use.provider.staging=true\n")
- f.write("wrapperlog.always.transfer=true\n")
- f.write("execution.retries=0\n")
- f.write("provider.staging.pin.swiftfiles=false\n")
- f.write("sitedir.keep=true\n")
- f.write("lazy.errors=false\n")
- f.close()
-
-def main():
- myargs = genargs(sys.argv)
- try:
- this=next(myargs)
- inloc = next(myargs)
- a = next(myargs)
- n = next(myargs)
- p = next(myargs)
- I = next(myargs)
- outloc = next(myargs)
- outlistfile = next(myargs)
- logfile = next(myargs)
- except:
- traceback.print_exc()
- sys.exit(1)
-
- #which swift
- swift = distutils.spawn.find_executable("swift")
-
- BASEDIR = os.path.dirname(os.path.realpath(__file__))
-
- # workout the file args
- fileargs=[]
- for f in myargs:
- fileargs.append(f)
-
- #workout the file array only if user adds args else make it blank
- if fileargs:
- if k_or_t == "swiftK":
- filearrayexpr="file fileargs[] <fixed_array_mapper; files=\""+','.join(fileargs)+"\">;\n"
- else: #T
- filearrayexpr="file fileargs[]="+str(fileargs)+";\n"
- else:
- filearrayexpr="file fileargs[];\n"
-
- # create outloc dir
- if not os.path.exists(outloc):
- os.makedirs(outloc)
-
- # workout the stringargs
- if stringargs:
- stringarrayexpr="string stringargs[]="+re.sub('\'','"',str(stringargs.split()))+";\n"
- else:
- stringarrayexpr="string stringargs[];\n"
-
- wdir = setwdir().rstrip()
-
- #workout standard input file
- if stdin != "None":
- if k_or_t == "swiftK":
- stdinfilexpr = "file stdinfile<\""+stdin+"\">;\n"
- else:
- stdinfilexpr = "file stdinfile = input(\""+stdin+"\");\n"
- else:
- #print "no stdin"
- if k_or_t == "swiftK":
- stdinfilexpr = "file stdinfile<\"/dev/null\">;\n"
- else:
- stdinfilexpr = "file stdinfile = input(\"/dev/null\");\n"
-
-
- #Build Swift source code
- f = open(wdir+"/script.swift", "w");
-
- if k_or_t == "swiftK":
-
- f.write("type file; \n")
- f.write("app (file _out, file _err) anapp(file _exec, int _i, string _stringargs[], file _fileargs[], file _stdin){ \n")
- f.write(" "+interpreter+" @_exec _i _stringargs @_fileargs stdin=@_stdin stdout=@_out stderr=@_err; \n")
- f.write("}\n")
- f.write(stringarrayexpr)
- f.write(filearrayexpr)
- f.write(stdinfilexpr)
- f.write("file exec<\""+executable+"\">;\n")
-
- if range_or_list == "list":
-
- f.write("file listfile<\""+listfile+"\">;\n")
- f.write("string list[] = readData(listfile);\n");
- f.write("foreach l, i in list{\n")
- f.write(" file out <single_file_mapper; file=@strcat(\""+outloc+"\", \"/\", l, \".\", i, \".out\")>;\n")
- f.write(" file err <single_file_mapper; file=@strcat(\""+outloc+"\", \"/\", l, \".\", i, \".err\")>;\n")
-
- else:
-
- f.write("foreach i in ["+rstart+":"+rend+":"+stepsize+"]{\n")
- f.write(" file out <single_file_mapper; file=@strcat(\""+outloc+"\", \"/\", i, \".out\")>;\n")
- f.write(" file err <single_file_mapper; file=@strcat(\""+outloc+"\", \"/\", i, \".err\")>;\n")
-
- f.write(" (out,err) = anapp(exec, i, stringargs, fileargs, stdinfile);\n")
- f.write("}\n")
-
- #build site
- if os.path.dirname(sys.argv[0]):
- basedir=os.path.dirname(sys.argv[0])
- else:
- basedir='.'
- #print "%s %s %s" %(basedir, sites, wdir)
- subprocess.call([basedir+"/buildsite.sh", sites, wdir])
-
- #build config file
- buildcfg(wdir)
-
- #build tc file
- with open(wdir+"/tc", 'w') as tc:
- for site in sites.split():
- tc.write(site+' '+interpreter+' '+interpreter)
-
- else: # T
-
- f.write("import files;\n");
- f.write("import string;\n");
- f.write("import io;\n");
- f.write("\n");
- f.write(" app (file _out, file _err) anapp(file _exec, int _i, string _stringargs[], file _fileargs[], file _stdin){\n");
- f.write(" \"" +interpreter+"\" _exec _i _stringargs _fileargs @stdin=_stdin @stdout=_out @stderr=_err; \n");
- f.write("}\n");
- f.write("main{\n");
- f.write(stringarrayexpr + " \n");
- f.write(filearrayexpr + " \n");
- f.write(stdinfilexpr + " \n");
- f.write("file exec = input(\""+executable+"\");\n");
-
- if range_or_list == "list":
-
- f.write(" \n");
- f.write("file listfile=input(\""+listfile+"\");\n")
- f.write("string list[] = split(read(listfile),);\n");
- f.write("foreach l, i in list{\n")
- f.write(" file out <sprintf(\""+outloc+"/%s.%i.out\", l, i)>;\n")
- f.write(" file err <sprintf(\""+outloc+"/%s.%i.err\", l, i)>;\n")
-
- else:
-
- f.write("foreach i in ["+rstart+":"+rend+":"+stepsize+"]{ \n");
- f.write(" file out <sprintf(\""+outloc+"/%i.out\", i)>; \n");
- f.write(" file err <sprintf(\""+outloc+"/%i.err\", i)>; \n");
-
- f.write(" (out,err) = anapp(exec, i, stringargs, fileargs, stdinfile);\n")
- f.write("}\n");
- f.write("printf(\"%s\", \"Done!\");\n");
- f.write("}\n");
-
- f.close()
-
- #Invocation
- with open(logfile,"wb") as errfile:
- exitcode=subprocess.check_call([swift, "-sites.file", wdir+"/sites.xml", "-tc.file", wdir+"/tc", "-config", wdir+"/cf", wdir+"/script.swift", swiftargs],stderr=errfile)
- print exitcode
-
- #Populate output file
- outlist=subprocess.check_output(["find", outloc, "-type", "f", "-iname", "*.out"])
- f=open(outlistfile, "w")
- f.write(outlist);
- f.close()
-
-if __name__=='__main__':
- main()
Deleted: SwiftApps/dsp-swift/galaxy/swiftjumpdsp.xml
===================================================================
--- SwiftApps/dsp-swift/galaxy/swiftjumpdsp.xml 2014-12-11 19:04:26 UTC (rev 8327)
+++ SwiftApps/dsp-swift/galaxy/swiftjumpdsp.xml 2014-12-11 19:21:36 UTC (rev 8328)
@@ -1,59 +0,0 @@
-<tool id="swiftjumpdsp" name="jumpdsp">
- <!--
- ToDo:
- 1. Option if the counter should be an argument, first one or last one.
- 2. Option to add files (in Galaxy dataset format) as arguments. These files then be added to Swift app as arguments.
- 3. Allow the tool to select another tool and run it.
- 4. Allow user to add more than one resource to run a job for load balancing.
- 5. Add ability to run Swift/T.
- 6. Let the localhost option be selected by default
- -->
- <description>A Galaxy tool to run Dsp parameter sweeps via Swift foreach parallel construct</description>
-
- <command interpreter="python">
- swiftjumpdsp.py
- "${inlocation}"
- "${a}"
- "${n}"
- "${p}"
- "${I}"
- "${outlist}"
- </command>
- <!--
- #if $swiftargs.mode == "advanced"
- #for $sa in $swiftarg
- ${sa.swiftargname}
- #end for
- #end if
- -->
- <inputs>
- <param name="inlocation" size="60" type="text" label="inlocation" help="Enter the path of the directory containing input data, eg. /home/ketan/farmer" />
- <param name="a" size="60" type="text" label="aval" help="Enter the value of a"/>
- <param name="n" size="60" type="text" label="nval" help="Enter the value of n"/>
- <param name="p" size="60" type="text" label="pval" help="Enter the value of p"/>
- <param name="I" size="60" type="text" label="Ival" help="Enter the value of I"/>
- </inputs>
- <outputs>
- <data format="txt" name="logfile" type="data" label="Application Output" />
- <data format="txt" name="outlist" type="data" label="Output List" />
- </outputs>
-
- <!--
- <code file="postprocess.py">
- <hook postprocess="writeoutlist" />
- </code>
- -->
-
-<help>
-.. class:: warningmark
-
-**TIP**. Add args to provide additional arguments to your executable.
-
------
-
-**What it does**
-
-This is a tool to run dsp via Swift in parallel over files provided by user.
-
-</help>
-</tool>
More information about the Swift-commit
mailing list