[Swift-commit] r6274 - in trunk/docs: . cookbook merged merged/userguide userguide
ketan at ci.uchicago.edu
ketan at ci.uchicago.edu
Thu Feb 14 21:27:36 CST 2013
Author: ketan
Date: 2013-02-14 21:27:36 -0600 (Thu, 14 Feb 2013)
New Revision: 6274
Added:
trunk/docs/merged/
trunk/docs/merged/build_docs.sh
trunk/docs/merged/refmanual/
trunk/docs/merged/userguide/
trunk/docs/merged/userguide/coasters
Removed:
trunk/docs/cookbook/coasters
Modified:
trunk/docs/cookbook/cookbook.txt
trunk/docs/userguide/commands
Log:
starting merging docs
Deleted: trunk/docs/cookbook/coasters
===================================================================
--- trunk/docs/cookbook/coasters 2013-02-15 01:27:42 UTC (rev 6273)
+++ trunk/docs/cookbook/coasters 2013-02-15 03:27:36 UTC (rev 6274)
@@ -1,110 +0,0 @@
-Coasters
---------
-Coasters were introduced in Swift v0.6 as an experimental feature. In many
-applications, Swift performance can be greatly enhanced by the use of
-coasters. Coasters provide a low-overhead job submission and file transfer
-mechanism suited for the execution of short jobs (on the order of a few
-seconds). A detailed information on coasters can be found at http://www.ci.uchicago.edu/swift/guides/userguide.php#coasters.
-//**Include neat diagrams.**
-
-Following is a coasters setup case-study for a PBS underlying provider where sites.xml coaster settings were:
-
-[xml]
-source~~~~
-<execution provider="coaster" jobmanager="local:pbs"/>
-<profile namespace="globus" key="project">CI-CCR000013</profile>
-
-<!-- Note that the following is going to be defunct in the new version (0.93+) and replaced by
-"ProviderAttributes" key and may not work in the future Swift versions-->
-
-<!--<profile namespace="globus" key="ppn">24:cray:pack</profile>-->
-
-<profile namespace="globus" key="providerAttributes">
-pbs.aprun
-pbs.mpp=true
-</profile>
-
-<profile namespace="globus" key="workersPerNode">24</profile>
-<profile namespace="globus" key="maxTime">100000</profile>
-
-<profile namespace="globus" key="lowOverallocation">100</profile>
-<profile namespace="globus" key="highOverallocation">100</profile>
-
-<profile namespace="globus" key="slots">20</profile>
-<profile namespace="globus" key="nodeGranularity">5</profile>
-<profile namespace="globus" key="maxNodes">5</profile>
-<profile namespace="karajan" key="jobThrottle">20.00</profile>
-<profile namespace="karajan" key="initialScore">10000</profile>
-source~~~~
-
-The following table briefly describes the elements on the coasters setup:
-[width="70%", cols="^3,10", options="header"]
-|=============================================================================================
-|profile key | brief description
-|slots | How many maximum LRM jobs/worker blocks are allowed
-|workersPerNode | How many coaster workers to run per execution node
-|nodeGranularity | Each worker block uses a number of nodes that is a multiple of this number
-|lowOverallocation | How many times larger than the job walltime should a block's walltime be if all jobs are 1s long
-|highOverallocation | How many times larger than the job walltime should a block's walltime be if all jobs are infinitely long
-|workersPerNode | How many coaster workers to run per execution node reserve How many seconds to reserve in a block's walltime for starting/shutdown operations
-|maxnodes | The maximum number of nodes allowed in a block
-|maxtime | The maximum number of walltime allowed for a block coaster service
-|jobThrottle |the number of concurrent jobs allowed on a site
-|==============================================================================================
-
-//
-//For Beginners
-//~~~~~~~~~~~~~~
-//Coasters for beginners. Usage of existing, prebuilt templates.
-//
-//For Intermediate Users
-//~~~~~~~~~~~~~~~~~~~~~~~
-//Coasters for intermediate users.
-//
-//Using gensites
-//^^^^^^^^^^^^^^^
-//Usage of gensites to generate your own sites
-//configurations.
-//
-//
-//
-
-For Advanced Users
-~~~~~~~~~~~~~~~~~~
-
-//Coasters for advanced users. Getting your hands dirty.
-
-One of the main reason that one would initially deviate from coaster
-defaults into more complex pool entries is to force jobs to fit into some
-site-imposed constraint. For instance a typical submission to the experimental
-queue requires a user to request upto 3 nodes for under 1 hour. This setup
-could be achieved with a careful tuning of coaters parameters.
-
-//How to run Swift under different Coasters configurations
-//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-//
-//Manual
-//^^^^^^^
-//Todo
-//
-//Passive
-//^^^^^^^^
-//Todo
-//
-//Persistent
-//^^^^^^^^^^^
-//Todo
-//
-//Summary of Differences Between different Coaster types
-//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-//A tabular representations of highlights of different coaster setups
-//
-//
-//Data Management
-//
-
-Coaster providers: local, ssh, pbs
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Settings and examples for different coaster providers mechanisms.
-
Modified: trunk/docs/cookbook/cookbook.txt
===================================================================
--- trunk/docs/cookbook/cookbook.txt 2013-02-15 01:27:42 UTC (rev 6273)
+++ trunk/docs/cookbook/cookbook.txt 2013-02-15 03:27:36 UTC (rev 6274)
@@ -11,8 +11,6 @@
include::swift_basics[]
-include::coasters[]
-
include::infrastructures[]
include::debugging[]
Added: trunk/docs/merged/build_docs.sh
===================================================================
--- trunk/docs/merged/build_docs.sh (rev 0)
+++ trunk/docs/merged/build_docs.sh 2013-02-15 03:27:36 UTC (rev 6274)
@@ -0,0 +1,115 @@
+#!/bin/bash
+
+# Usage: See usage() for usage
+
+# crash: Report a problem and exit
+crash()
+{
+ MSG=$1
+ echo ${MSG} >&2
+ exit 1
+}
+
+# Output directory mode
+CHMOD_DIRECTORY_MODE="775"
+# Output file mode
+CHMOD_FILE_MODE="664"
+# Output group
+GROUP="vdl2-svn"
+# Make PDFs iff MAKE_PDF=1
+MAKE_PDF=1
+
+# See getopts loop below for options
+usage()
+{
+ echo "Usage: ./build_docs.sh <opts> <installation directory>"
+}
+
+while getopts "dh" OPTION
+do
+ case ${OPTION} in
+ d)
+ MAKE_PDF=0
+ shift
+ ;;
+ h)
+ usage
+ exit 0
+ ;;
+ esac
+done
+
+INSTALLATION_DIRECTORY=$1
+
+if [[ $INSTALLATION_DIRECTORY == "" ]]
+then
+ echo "Not given: installation directory"
+ usage
+ exit 1
+fi
+
+INSTALLATION_DIRECTORY=$1
+
+# Create installation directory if needed
+if [ ! -d "$INSTALLATION_DIRECTORY" ]; then
+ mkdir $INSTALLATION_DIRECTORY || crash "Unable to create directory $INSTALLATION_DIRECTORY"
+ chgrp $GROUP $INSTALLATION_DIRECTORY > /dev/null 2>&1
+ chmod $CHMOD_DIRECTORY_MODE $INSTALLATION_DIRECTORY > /dev/null 2>&1
+fi
+
+unamestr=`\uname`
+if [[ "$unamestr" == 'Linux' ]]; then
+ pushd $(dirname $(readlink -f $0)) > /dev/null 2>&1
+else
+ pushd $(dirname $(greadlink -f $0)) > /dev/null 2>&1
+fi
+
+# Gather version information
+pushd .. > /dev/null 2>&1
+VERSION=`svn info |grep URL|awk -F / '{print $NF}'`
+popd > /dev/null 2>&1
+echo Installing docs into $INSTALLATION_DIRECTORY
+
+# Convert files
+DIRECTORIES=`ls -d */ 2>/dev/null`
+for directory in $DIRECTORIES
+do
+ pushd $directory > /dev/null 2>&1
+ FILES=`ls -1 *.txt 2>/dev/null`
+ for file in $FILES
+ do
+ echo Converting $directory"$file" to HTML
+ asciidoc -a toc -a toclevels=2 \
+ -a max-width=750px \
+ -a textwidth=80 \
+ -a stylesheet=$(pwd)/../stylesheets/asciidoc.css \
+ $file
+ if (( MAKE_PDF ))
+ then
+ echo Converting $directory"$file" to PDF
+ a2x --format=pdf --no-xmllint $file
+ fi
+ done
+
+ if [ ! -d "$INSTALLATION_DIRECTORY/$VERSION" ]; then
+ mkdir $INSTALLATION_DIRECTORY/$VERSION || crash "Unable to create directory $INSTALLATION_DIRECTORY/$VERSION"
+ fi
+
+ if [ ! -d "$INSTALLATION_DIRECTORY/$VERSION/$directory" ]; then
+ mkdir $INSTALLATION_DIRECTORY/$VERSION/$directory || crash "Unable to create directory $INSTALLATION_DIRECTORY/$VERSION/$directory"
+ fi
+
+ # Copy all files to destination (may include graphics, etc)
+ for copyfile in `find -L . -type f 2>/dev/null |grep -v .svn`
+ do
+ DN=`dirname $copyfile`
+ mkdir -p $INSTALLATION_DIRECTORY/$VERSION/$directory/$DN > /dev/null 2>&1
+ cp $copyfile $INSTALLATION_DIRECTORY/$VERSION/$directory/$DN || crash "Unable to copy $copyfile to $INSTALLATION_DIRECTORY/$VERSION/$directory"
+ done
+
+ popd > /dev/null 2>&1
+done
+popd > /dev/null 2>&1
+
+find $INSTALLATION_DIRECTORY/$VERSION -type f -exec chgrp $GROUP {} \; -exec chmod $CHMOD_FILE_MODE {} \; > /dev/null 2>&1
+find $INSTALLATION_DIRECTORY/$VERSION -type d -exec chgrp $GROUP {} \; -exec chmod $CHMOD_DIRECTORY_MODE {} \; > /dev/null 2>&1
Property changes on: trunk/docs/merged/build_docs.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: trunk/docs/merged/userguide/coasters
===================================================================
--- trunk/docs/merged/userguide/coasters (rev 0)
+++ trunk/docs/merged/userguide/coasters 2013-02-15 03:27:36 UTC (rev 6274)
@@ -0,0 +1,202 @@
+Coasters
+--------
+
+Introduction
+~~~~~~~~~~~~
+Coasters are the Swift's implementation of _pilot job abstraction_.
+
+In many applications, Swift performance can be greatly enhanced by the use of
+coasters. Coasters provide a low-overhead job submission and file transfer
+mechanism suited for the execution of jobs and the transfer of files for which
+other grid protocols such as GRAM and GridFTP are poorly suited.
+
+Benefits
+~~~~~~~~
+Much of the overhead associated with other grid protocols (such as
+authentication and authorization, and allocation of worker nodes by the site's
+local resource manager) is reduced, because that overhead is associated with
+the allocation of a coaster pilot or coaster worker, rather than with every
+Swift-level procedure invocation; potentially hundreds or thousands of
+Swift-level procedure invocations can be run through a single worker. Coasters
+can be configured for two purposes: job execution and file staging. In
+practice, the Swift script remains the same while working with coasters. A
+detailed description of coaster mechanism is explained in the next section.
+
+Mechanism
+~~~~~~~~~
+Coasters run at the task management layer logically under the Swift script. The
+jobs and data movement requirements resulting after the interpretation of a
+Swift script are handled by the coasters. The coaster mechanism submits a pilot
+job using some other execution mechanism such as GRAM, SGE or PBS scheduler,
+and for each worker node that will be used in a remote cluster, it submits a
+worker job, again using some other execution mechanism such as GRAM. Details on
+the design of the coaster mechanism can be found here:
+<http://wiki.cogkit.org/wiki/Coasters>. The pilot job manages file transfers
+and the dispatch of execution jobs to workers.
+
+Coasters How-to
+~~~~~~~~~~~~~~~
+To use for job execution, specify a sites.xml execution element like this:
+
+----
+<execution provider="coaster" jobmanager="gt2:gt2:pbs" url="grid.myhost.org">
+----
+
+The jobmanager string contains more detail than with other providers. It
+contains either two or three colon separated fields: 1:the provider to
+be use to execute the coaster pilot job - this provider will submit from
+the Swift client side environment. Commonly this will be one of the GRAM
+providers; 2: the provider to be used to execute coaster worker jobs.
+This provider will be used to submit from the coaster pilot job
+environment, so a local scheduler provider can sometimes be used instead
+of GRAM. 3: optionally, the jobmanager to be used when submitting worker
+job using the provider specified in field 2.
+
+To use for file transfer, specify a sites.xml filesystem element like this:
+
+----
+<filesystem provider="coaster" url="gt2://grid.myhost.org" />
+----
+
+The url parameter should be a pseudo-URI formed with the URI scheme
+being the name of the provider to use to submit the coaster pilot job,
+and the hostname portion being the hostname to be used to execute the
+coaster pilot job. Note that this provider and hostname will be used for
+execution of a coaster pilot job, not for file transfer; so for example,
+a GRAM endpoint should be specified here rather than a GridFTP endpoint.
+
+Coasters are affected by the following profile settings, which are
+documented in the Globus namespace profile section:
+
+[options="header, autowidth"]
+|=================
+|Profile key|Brief description
+|slots|How many maximum LRM jobs/worker blocks are allowed
+|jobsPerNode|How many coaster workers to run per execution node
+|nodeGranularity|Each worker block uses a number of nodes that is a
+multiple of this number
+|lowOverallocation|How many times larger than the job walltime should a
+block's walltime be if all jobs are 1s long
+|highOverallocation|How many times larger than the job walltime should a
+block's walltime be if all jobs are infinitely long
+|overallocationDecayFactor|How quickly should the overallocation curve
+tend towards the highOverallocation as job walltimes get larger
+|spread|By how much should worker blocks vary in worker size
+|jobsPerNode|How many coaster workers to run per execution node
+|reserve|How many seconds to reserve in a block's walltime for
+starting/shutdown operations
+|maxnodes|The maximum number of nodes allowed in a block
+|maxtime|The maximum walltime allowed for a block, in integer seconds
+|remoteMonitorEnabled|If true, show a graphical display of the status of
+the coaster service
+|==================
+Coasters
+--------
+Coasters were introduced in Swift v0.6 as an experimental feature. In many
+applications, Swift performance can be greatly enhanced by the use of
+coasters. Coasters provide a low-overhead job submission and file transfer
+mechanism suited for the execution of short jobs (on the order of a few
+seconds). A detailed information on coasters can be found at http://www.ci.uchicago.edu/swift/guides/userguide.php#coasters.
+//**Include neat diagrams.**
+
+Following is a coasters setup case-study for a PBS underlying provider where sites.xml coaster settings were:
+
+[xml]
+source~~~~
+<execution provider="coaster" jobmanager="local:pbs"/>
+<profile namespace="globus" key="project">CI-CCR000013</profile>
+
+<!-- Note that the following is going to be defunct in the new version (0.93+) and replaced by
+"ProviderAttributes" key and may not work in the future Swift versions-->
+
+<!--<profile namespace="globus" key="ppn">24:cray:pack</profile>-->
+
+<profile namespace="globus" key="providerAttributes">
+pbs.aprun
+pbs.mpp=true
+</profile>
+
+<profile namespace="globus" key="workersPerNode">24</profile>
+<profile namespace="globus" key="maxTime">100000</profile>
+
+<profile namespace="globus" key="lowOverallocation">100</profile>
+<profile namespace="globus" key="highOverallocation">100</profile>
+
+<profile namespace="globus" key="slots">20</profile>
+<profile namespace="globus" key="nodeGranularity">5</profile>
+<profile namespace="globus" key="maxNodes">5</profile>
+<profile namespace="karajan" key="jobThrottle">20.00</profile>
+<profile namespace="karajan" key="initialScore">10000</profile>
+source~~~~
+
+The following table briefly describes the elements on the coasters setup:
+[width="70%", cols="^3,10", options="header"]
+|=============================================================================================
+|profile key | brief description
+|slots | How many maximum LRM jobs/worker blocks are allowed
+|workersPerNode | How many coaster workers to run per execution node
+|nodeGranularity | Each worker block uses a number of nodes that is a multiple of this number
+|lowOverallocation | How many times larger than the job walltime should a block's walltime be if all jobs are 1s long
+|highOverallocation | How many times larger than the job walltime should a block's walltime be if all jobs are infinitely long
+|workersPerNode | How many coaster workers to run per execution node reserve How many seconds to reserve in a block's walltime for starting/shutdown operations
+|maxnodes | The maximum number of nodes allowed in a block
+|maxtime | The maximum number of walltime allowed for a block coaster service
+|jobThrottle |the number of concurrent jobs allowed on a site
+|==============================================================================================
+
+//
+//For Beginners
+//~~~~~~~~~~~~~~
+//Coasters for beginners. Usage of existing, prebuilt templates.
+//
+//For Intermediate Users
+//~~~~~~~~~~~~~~~~~~~~~~~
+//Coasters for intermediate users.
+//
+//Using gensites
+//^^^^^^^^^^^^^^^
+//Usage of gensites to generate your own sites
+//configurations.
+//
+//
+//
+
+For Advanced Users
+~~~~~~~~~~~~~~~~~~
+
+//Coasters for advanced users. Getting your hands dirty.
+
+One of the main reason that one would initially deviate from coaster
+defaults into more complex pool entries is to force jobs to fit into some
+site-imposed constraint. For instance a typical submission to the experimental
+queue requires a user to request upto 3 nodes for under 1 hour. This setup
+could be achieved with a careful tuning of coaters parameters.
+
+//How to run Swift under different Coasters configurations
+//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+//
+//Manual
+//^^^^^^^
+//Todo
+//
+//Passive
+//^^^^^^^^
+//Todo
+//
+//Persistent
+//^^^^^^^^^^^
+//Todo
+//
+//Summary of Differences Between different Coaster types
+//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+//A tabular representations of highlights of different coaster setups
+//
+//
+//Data Management
+//
+
+Coaster providers: local, ssh, pbs
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Settings and examples for different coaster providers mechanisms.
+
Modified: trunk/docs/userguide/commands
===================================================================
--- trunk/docs/userguide/commands 2013-02-15 01:27:42 UTC (rev 6273)
+++ trunk/docs/userguide/commands 2013-02-15 03:27:36 UTC (rev 6274)
@@ -39,7 +39,7 @@
-resume file
- Resumes the execution using a log file
+ Resumes the execution using a resume-log file .rlog
-config file
More information about the Swift-commit
mailing list