Author: davidk
Date: 2013-08-12 16:33:18 -0500 (Mon, 12 Aug 2013)
New Revision: 6831
Modified:
trunk/docs/designs/swiftrun.txt
Log:
First few sections of a new swiftrun guide
Modified: trunk/docs/designs/swiftrun.txt
===================================================================
--- trunk/docs/designs/swiftrun.txt 2013-08-12 20:56:50 UTC (rev 6830)
+++ trunk/docs/designs/swiftrun.txt 2013-08-12 21:33:18 UTC (rev 6831)
@@ -3,342 +3,101 @@
:toc:
:icons:
-:numbered:
-Steps to implementation
------------------------
+Introduction
+------------
+Swift 0.95 introduces many changes to the method for configuring and running
+Swift. The goal of these changes is to make configuration easier for Swift
+users. This document will attempt to explain the reasons for these changes and
+document how these new configuration mechanisms work.
-1) Implement the .swift run directory, $HOME/.swiftrc, $PWD/.swiftrun
+Configuration in previous releases
+----------------------------------
+In Swift 0.94 and earlier versions, configuring Swift usually required setting
+up files called sites.xml, tc.data, and cf. These files typically needed to be
+specified on the command line. The following command line was pretty typical
+in previous Swift releases:
-NOTE: either simplify or provide some shortcuts and backwards
-compatibility.
-
-2) Implement the search path for site templates. Site templates are XML for now, but can eventually be in property-file format
-
-3) accept site overrides from property files
-
-4) Users should edit only proprty files unless they are making new templates.
-
-5) Generate app files automatically.
-
-
-New command line and configuration conventions for running Swift.
------------------------------------------------------------------
-
-This document describes a new method of running Swift scripts which
-aims to make Swift easier to use.
-
-Currently, Swift configuration settings are split up into multiple
-files. -sites.file (Sites.xml) defines sites, -tc.file (tc.data)
-defines applications, -config (swift.properties) defines other
-settings, and -cdm.file (cdm.data) defines data management rules.
-
-.Goals of reworking this configuration mechanism are
-
-1. Define all settings in a single configuration file format with more
-uniform conventions. Use a consistent format for all settings.
-
-2. Don't require users or site administrators to use XML.
-
-3. Make the settings in swift.properties simple to understand
-
-4. Build an effective run-directory mechanism into the swift
-command. Reduce the need for users to write their own runMyApp.sh
-wrappers, and reduce the complexity of such wrappers when necessary.
-
-5. Define effective locations for site-specific templates and defaults
-
-6. Enable users to specify default settings for their personal runs as
-well as for specific runs (ie in the current working
-directories where they are running swift)
-
-7. Improve the user-guide documentation of properties, in particular
-by separating out the common ones from the obscure ones and by
-clarifying how throttling actually works, especially for data management and
-coasters vs. single-level providers.
-
-8. Make it easier to use predefined site templates created by site
-Swift administrators or user groups
-
-9. Provide a simpler and more compact way of specifiying which apps
-should be invoked on which sites. (Includes: Don't force user to
-determine which sites to use by resticting the sites file or the tc
-list)
-
-10. Catch mis-spelled config options. Currently these errors are
-silently ignored.
-
-11. Enable simpler linkage from app() definitions in the .swift source
-to the executable to run for that app. This should support the new
-feature of inline multi-language app scripts/programs.
-
-12. Separate the selection of sites to use from the sets of sets
-defined in the configuration properties (do this w/o changes to the
-internals)
-
-13. Enable users to define groups of settings as "profiles" for
-various modes of operation.
-
-14. Sites templates and config files can be constructed with a
-graphical configurator (ideally one with comprehensive graphics to
-define the configuration... can envision wysiwyg and drag-and-drop)
-which creates the needed text files.
-
-15. For below: The parameters specified in a site.sitename.paramname=
-value statement will be checked for correctness against the xml
-schema.
-
-16. It should be possible to define an entire site entry from
-name=value statements. So templates could be specified that way, and
-XML could be used solely for the internal format and eventually
-removed altogether.
-
-17. site.beagle-fixed.maxnodes
-a. slots
-b. maxnodes
-c. nodegranularity
-d. high & low overallocation <= on/off or via one parameter (fixed/var)
-e. jobThrottle + scalefactor (?sp)
-f. maxtime => slotTime maxWallTime=>taskTime
-g. passthrough to lower level scheduler: site.LRM.param
-h. (special for aprun and ibrun, mpiexec...)
-i. site.sitename.param=\_REQUIRED_
-
-18. sites.xml values will be aliased to new, meaningful names. Since
-the providers will be retained in swift 2.0, we chould make these
-chainges internal, but still provide some period of backwards
-compatibility if thats affordable.
-
-19. Things to fix in sites.xml:
-
-a. jobsPer -> tasks? (in sync with a language spec that nails down
-runs, tasks, and jobs.)
-b. coresPerNode, slotsPerNode?
-c. params for dynamic coaster throttling?
-d. maxTime -> jobWallTime maxWallTime taskWallTime; time into
-consistent units: sec OR hh:mm:ss
-e. slots: are these used consistently between coaster-layered and single-level providers?
-f. it should be possible to define a site-set (eg: namd-sites)
-g. Enable the new mechanism to be implemented with little change to
-Swift internals. (Still use XML as the internal format used by swift
-and for backwards compatibility).
-h. Create a specification which can be retained in Swift 2.0
-i. The swift run log makes it clear where all property values came from:
- propert_name property_value property_source
-j. Related code improvements suggested by the config re-specification project:
-k. Dont assign jobs to sites w/o slots unless pre-staging (and even
-then allow an assignment to be abandoned if a better choice appears
-later) <== Mihael fix soon?
-l. Replication to work w/ provider staging
-m. clean up the internal Java structures for specifying parameters
-(properties, sites, tc). Make it much easier to add new
-parameters. Make it easy to add the pass-through parameters for LRMs.
-
-Below is a typical current swift command invocation:
-
-----
-swift -sites.file sites.xml -tc.file tc.data -config cf -cdm.file fs.data hello.swift
+$ swift -sites.file sites.xml -tc.file tc.data -config cf myscript.swift
-----
-We will retain this syntax and file separation for backward
-compatibility, and as the "internal" file representation of Swift
-configuration.
+sites.xml
+~~~~~~~~~
+The sites.xml file was an XML configuration file that defined site paramaters.
+It was used to determine how Swift should interact with the scheduler. Below
+is an example sites.xml for a campus cluster called midway
-
-Swift Properties
-----------------
-
-This new configuration file will be very similar to the current format
-of swift.properties. Settings that are currently defined in
-swift.properties will keep the same names and format.
-
+.sites.xml
-----
-execution.retries=0
-sitedir.keep=true
------
-
-Where will Swift look for swift.properties?
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Settings for swift.properties get set in the following order.
-
-1. Swift default values
-2. The etc/swift.properties file in the swift installation directory
-3. The swift.properties file located in the $SWIFT_CONFIG directory
-4. $HOME/.swift/swift.properties
-5. One or both of $PWD/swift.properties and $PWD/.swift/swift.properties (to
- be determined)
-6. Command line options. Swift will accept parameters on the command line, in
- the format of -Dexecution.retries=5. All parameters are unformly specifiable there.
-
-Settings are uniformly cumulative. Every swift.properties files that is found is read
-and values are set, with values from the most recent file taking precedence.
-
-Apps
-----
-Applications currently defined in tc.data will instead be defined in the
-new swift.properties file in the following format:
-
------
-app.beagle.wc = /usr/bin/wc
-app.midway.bash = /bin/bash
------
-
-Special cases
-~~~~~~~~~~~~~
-For most app definitions you need only the path to the application. You should
-also be able to handle special cases in a similar way. For example, to define
-that an app is WIN32, or to set an app specific walltime:
------
-app.beagle.wc.win32 = true
-app.beagle.wc.walltime = 00:05:00
------
-
-App wildcards
-~~~~~~~~~~~~~
-The existing tc.data format should be modified to allow wildcards. A * in your
-tc.data would allow you to run any application defined in your swift script on
-the site. It would search your PATH for the executable location.
-
------
-beagle * * null null null
------
-
-In the new tc format, this same expression would be:
------
-app.beagle.* = *
------
-
-NOTE: If the form of line in tc is so generic, it could also be considered a default/generate on the fly. Eg. pick site name from swift commandline and generate a tc internally on-the-fly. In other words, only if user has a special need to define specific path for a specific app, tc should appear otherwise, it should be auto-generated internally. --Ketan.
-
-Sites
------
-
-To specify which site a swift script should run on, use the -site
-command line argument.
-
------
-$ swift -site midway hello.swift
------
-
-NOTE: Is the -site switch needed? Could we say "swift -local hello.swift" or "swift -midway,beagle hello.swift" --Ketan.
-
-To run on multiple sites, separate site names by a comma.
-
------
-swift -site midway,beagle hello.swift
------
-
-NOTE: Perhaps we should also think about adding the ability to define "site
- groups". For example, the group "xsede" could define the sites for
- Trestles, Stampede, and Kraken. You could then also mix and match
- multiple groups.
-
-How does Swift know how to run on these sites?
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Swift uses XML to shape a specific run. These XML files define things that
-Swift needs to know about a particular site, for example, which scheduler
-should be used, what is the location of the work directory, and what degree
-of parallelism should be used.
-
-Swift will include several XML templates that allow a user to run on a
-variety of commonly used sites. These site templates could be found in
-multiple locations.
-
-. In your current working directory, Swift will search for a matching
- template in $PWD/.swift/sites.
-
-. Next, Swift will look in $HOME/.swift/sites.
-
-. Directories defined in $SWIFT_TEMPLATE_PATH. The format of this environment
- variable is similar to $PATH, /dir/location1:/dir/location2:/dir/location3.
-
-. Finally, the Swift installation directory etc/sites.
-
-How can I change the the values in these templates dynamically?
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Values defined in your Swift configuration file can override the
-default values stored in XML templates. Here is one example.
-
-Assume that you are looking at a site templates that looks like this:
------
- <pool handle="beagle">
- <execution provider="coaster" jobmanager="local:pbs"/>
- <profile namespace="globus" key="jobsPerNode">24</profile>
- <profile namespace="globus" key="lowOverAllocation">100</profile>
- <profile namespace="globus" key="highOverAllocation">100</profile>
- <profile namespace="globus" key="providerAttributes">pbs.aprun;pbs.mpp;depth=24</profile>
+<config>
+ <pool handle="midway-sandyb">
+ <execution provider="coaster" jobmanager="local:slurm"/>
+ <profile namespace="globus" key="jobsPerNode">16</profile>
<profile namespace="globus" key="maxWalltime">_WALLTIME_</profile>
- <profile namespace="globus" key="slots">50</profile>
- <profile namespace="globus" key="nodeGranularity">1</profile>
- <profile namespace="karajan" key="jobThrottle">12.00</profile>
+ <profile namespace="globus" key="highOverAllocation">100</profile>
+ <profile namespace="globus" key="lowOverAllocation">100</profile>
+ <profile namespace="globus" key="queue">sandyb</profile>
<profile namespace="karajan" key="initialScore">10000</profile>
<filesystem provider="local"/>
- <workdirectory>/lustre/beagle/{env.USER}/swiftwork</workdirectory>
+ <workdirectory>/scratch/midway/{env.USER}</workdirectory>
</pool>
+</config>
-----
-NOTE: This came up in one or more past discussions: sites file should be less cluttered. Some suggestions: 1) Default values for some attributes be used and should be only visible if a user explicitly wants a new value for them. eg. lowOverAllocation, highOverAllocation, initialscore. 2) One of the walltime and maxwalltime could be done away with. Ditto with depth/ppn and jobspernode. --Ketan
+tc.data
+~~~~~~~~
+The tc.data was a basic catalog that defined the location of applications on a
+given sites.
-By default in this template, jobsPerNode is set to 24. However, for
-our application we would like to set this to 12. To do this, we add
-this line to the Swift configuration file we are using:
-
+.tc.data
-----
-site.beagle.jobsPerNode=12
+localhost cat /bin/cat null null null
-----
-Can this be shortened? Initially assume not.
+cf
+~~
+The cf file (also called swift.properties) was a file that defined various
+swift configuration values, like retries and error handlinng.
-Can same attributes be applied to all sites? E.g.:
+.cf
+-----
+wrapperlog.always.transfer=true
+sitedir.keep=true
+file.gc.enabled=false
+status.mode=provider
+-----
-site.jobsNerNode=12
-
-or
-
-jobsPerNode=12
-
-OI: use name=value or name <space> value or both?
-
-Defining sites in swift.properties
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-We may also want to talk about the ability to define sites directly in
-swift.properties using the existing XML format. The program that
-parses the properties file can attempt to find pool entries there.
-
-NOTE: This came up in discussions, but I think we should try to avoid
-this one. We should be able to shape a sites file using only x.y.z=foo
-properties. In cases where a user wants to use XML directly, I think
-they should use the -sites.file option.
-
-Run directories and logs
+The new swift.properties
------------------------
+Previous versions of Swift required users to create multiple files, each in
+stored in a different format. In an attempt to make things easier, Swift 0.95
+merges these different configuration files into a single, common configuration
+file called swift.properties.
-Every time Swift runs, it should create a directory called
-$PWD/.swift/runNNN, where NNN is a numeric value that gets incremented
-with every run.
+The new swift.properties file is responsible for:
-NOTE: I think the runNNN directory should be created in a visible directory, not a hidden one. It could be simply named runs. --Ketan.
+1. Defining sites
+2. Defining applications
+3. Defining various swift settings
-These run directories contain:
+Here is an example of a new swift.properties file:
-* kml and swiftx files
-* Swift log files
-* Submit scripts
-* tc.data
-* sites.xml
-* swift.properties
-* cdm.data
+-----
+wrapperlog.always.transfer=true
+sitedir.keep=true
+file.gc.enabled=false
+status.mode=provider
+site=westmere
+site.westmere.provider=local:slurm
+site.westmere.jobsPerNode=12
+site.westmere.maxWalltime=00:05:00
+site.westmere.queue=westmere
+site.westmere.initialScore=10000
+site.westmere.filesystem=local
+site.westmere.workdirectory=/scratch/midway/davidkelly999
+app.westmere.cat=/bin/cat
+-----
-What changes need to be made to make this work?
------------------------------------------------
-
-The vast majority of this can done with changes to the swift shell
-script and a new perl script.
-
-The swift shell script will check the command line parameters it
-receives. If the newer style of running is specified, where no
-configuration files are specified, swift and swiftrun will generate
-sites.xml, tc.data, swift.properties, cdm.data as described in this
-document. They will be created in $PWD/.swift/runNNN and referenced.
+This single swift.properties file works identically to using the sites.xml,
+tc.data, and cf files listed in the previous section.