[Swift-commit] r7910 - in SwiftTutorials/swift-localhost-tutorial: . app bin doc doc/figs doc/images doc/images/icons part01 part01/run001 part01/run002 part01/run003 part01/run004 part01/run005 part02 part03 part04
yadunandb at ci.uchicago.edu
yadunandb at ci.uchicago.edu
Mon Jun 9 17:36:21 CDT 2014
Author: yadunandb
Date: 2014-06-09 17:37:36 -0500 (Mon, 09 Jun 2014)
New Revision: 7910
Added:
SwiftTutorials/swift-localhost-tutorial/README.asc
SwiftTutorials/swift-localhost-tutorial/app/
SwiftTutorials/swift-localhost-tutorial/app/simulate
SwiftTutorials/swift-localhost-tutorial/app/simulate.py
SwiftTutorials/swift-localhost-tutorial/app/simulate.sh
SwiftTutorials/swift-localhost-tutorial/app/stats
SwiftTutorials/swift-localhost-tutorial/app/stats.py
SwiftTutorials/swift-localhost-tutorial/app/stats.sh
SwiftTutorials/swift-localhost-tutorial/bin/
SwiftTutorials/swift-localhost-tutorial/bin/cleanup
SwiftTutorials/swift-localhost-tutorial/bin/hosts
SwiftTutorials/swift-localhost-tutorial/bin/package_tutorial.sh
SwiftTutorials/swift-localhost-tutorial/bin/plot.sh
SwiftTutorials/swift-localhost-tutorial/doc/
SwiftTutorials/swift-localhost-tutorial/doc/README
SwiftTutorials/swift-localhost-tutorial/doc/TODO
SwiftTutorials/swift-localhost-tutorial/doc/activeplot.png
SwiftTutorials/swift-localhost-tutorial/doc/asciidoc.css
SwiftTutorials/swift-localhost-tutorial/doc/build_docs.sh
SwiftTutorials/swift-localhost-tutorial/doc/cumulativeplot.png
SwiftTutorials/swift-localhost-tutorial/doc/figs/
SwiftTutorials/swift-localhost-tutorial/doc/figs/modis.dia
SwiftTutorials/swift-localhost-tutorial/doc/figs/modis.png
SwiftTutorials/swift-localhost-tutorial/doc/images/
SwiftTutorials/swift-localhost-tutorial/doc/images/icons/
SwiftTutorials/swift-localhost-tutorial/doc/images/icons/caution.png
SwiftTutorials/swift-localhost-tutorial/doc/images/icons/example.png
SwiftTutorials/swift-localhost-tutorial/doc/images/icons/home.png
SwiftTutorials/swift-localhost-tutorial/doc/images/icons/important.png
SwiftTutorials/swift-localhost-tutorial/doc/images/icons/next.png
SwiftTutorials/swift-localhost-tutorial/doc/images/icons/note.png
SwiftTutorials/swift-localhost-tutorial/doc/images/icons/prev.png
SwiftTutorials/swift-localhost-tutorial/doc/images/icons/tip.png
SwiftTutorials/swift-localhost-tutorial/doc/images/icons/up.png
SwiftTutorials/swift-localhost-tutorial/doc/images/icons/warning.png
SwiftTutorials/swift-localhost-tutorial/doc/part01.png
SwiftTutorials/swift-localhost-tutorial/doc/part02.png
SwiftTutorials/swift-localhost-tutorial/doc/part03.png
SwiftTutorials/swift-localhost-tutorial/doc/part04.png
SwiftTutorials/swift-localhost-tutorial/doc/part05.png
SwiftTutorials/swift-localhost-tutorial/doc/part06.png
SwiftTutorials/swift-localhost-tutorial/doc/push.sh
SwiftTutorials/swift-localhost-tutorial/doc/raven.1000.progress.out
SwiftTutorials/swift-localhost-tutorial/doc/raven.10000.progress.out
SwiftTutorials/swift-localhost-tutorial/doc/tutorial.html
SwiftTutorials/swift-localhost-tutorial/part01/
SwiftTutorials/swift-localhost-tutorial/part01/p1.swift
SwiftTutorials/swift-localhost-tutorial/part01/run001/
SwiftTutorials/swift-localhost-tutorial/part01/run001/apps
SwiftTutorials/swift-localhost-tutorial/part01/run001/cf
SwiftTutorials/swift-localhost-tutorial/part01/run001/run001.log
SwiftTutorials/swift-localhost-tutorial/part01/run001/sites.xml
SwiftTutorials/swift-localhost-tutorial/part01/run001/swift.out
SwiftTutorials/swift-localhost-tutorial/part01/run002/
SwiftTutorials/swift-localhost-tutorial/part01/run002/apps
SwiftTutorials/swift-localhost-tutorial/part01/run002/cf
SwiftTutorials/swift-localhost-tutorial/part01/run002/run002.log
SwiftTutorials/swift-localhost-tutorial/part01/run002/sites.xml
SwiftTutorials/swift-localhost-tutorial/part01/run002/swift.out
SwiftTutorials/swift-localhost-tutorial/part01/run003/
SwiftTutorials/swift-localhost-tutorial/part01/run003/sites.xml
SwiftTutorials/swift-localhost-tutorial/part01/run003/swift.out
SwiftTutorials/swift-localhost-tutorial/part01/run004/
SwiftTutorials/swift-localhost-tutorial/part01/run004/sites.xml
SwiftTutorials/swift-localhost-tutorial/part01/run004/swift.out
SwiftTutorials/swift-localhost-tutorial/part01/run005/
SwiftTutorials/swift-localhost-tutorial/part01/run005/run005.log
SwiftTutorials/swift-localhost-tutorial/part01/run005/sites.xml
SwiftTutorials/swift-localhost-tutorial/part01/run005/swift.out
SwiftTutorials/swift-localhost-tutorial/part01/sim.out
SwiftTutorials/swift-localhost-tutorial/part01/swift.properties
SwiftTutorials/swift-localhost-tutorial/part02/
SwiftTutorials/swift-localhost-tutorial/part02/p2.swift
SwiftTutorials/swift-localhost-tutorial/part02/swift.properties
SwiftTutorials/swift-localhost-tutorial/part03/
SwiftTutorials/swift-localhost-tutorial/part03/p3.swift
SwiftTutorials/swift-localhost-tutorial/part03/swift.properties
SwiftTutorials/swift-localhost-tutorial/part04/
SwiftTutorials/swift-localhost-tutorial/part04/p4.swift
SwiftTutorials/swift-localhost-tutorial/part04/simulate.sh
SwiftTutorials/swift-localhost-tutorial/part04/stats.sh
SwiftTutorials/swift-localhost-tutorial/part04/swift.properties
SwiftTutorials/swift-localhost-tutorial/setup.sh
SwiftTutorials/swift-localhost-tutorial/test_all.sh
Log:
Committing localhost tutorial. Live copy on git:
https://github.com/yadudoc/swift-on-cloud/tree/master/swift-localhost-tutorial
Added: SwiftTutorials/swift-localhost-tutorial/README.asc
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/README.asc (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/README.asc 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1 @@
+link doc/README
\ No newline at end of file
Property changes on: SwiftTutorials/swift-localhost-tutorial/README.asc
___________________________________________________________________
Added: svn:special
+ *
Added: SwiftTutorials/swift-localhost-tutorial/app/simulate
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/app/simulate (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/app/simulate 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1 @@
+link simulate.sh
\ No newline at end of file
Property changes on: SwiftTutorials/swift-localhost-tutorial/app/simulate
___________________________________________________________________
Added: svn:special
+ *
Added: SwiftTutorials/swift-localhost-tutorial/app/simulate.py
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/app/simulate.py (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/app/simulate.py 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,125 @@
+#!/usr/bin/env python
+import sys
+import os
+import socket
+import time
+import math
+import random
+import getpass
+
+def parse():
+ from optparse import OptionParser
+ parser=OptionParser()
+ parser.add_option("-b", "--bias", type="int", dest="bias", default=0,
+ help="offset bias: add this integer to all results");
+
+ parser.add_option("-B", "--biasfile", type="string", dest="biasfile",
+ default="none", help="file of integer biases to add to results" );
+
+ parser.add_option("-l", "--log", type="string", dest="log", default="yes",
+ help="generate a log in stderr if not nul");
+
+ parser.add_option("-n", "--nvalues", type="int", dest="nvalues",default=1,
+ help="print this many values per simulation" );
+
+ parser.add_option("-s", "--seed", type="int", dest="initseed", default=0,
+ help="use this integer [0..32767] as a seed");
+
+ parser.add_option("-S", "--seedfile", type="string", dest="seedfile",
+ default="none", help="use this file (containing integer seeds [0..32767]) one per line" );
+
+ parser.add_option("-t", "--timesteps", type="int", dest="timesteps",
+ default=0, help='number of simulated "timesteps" in seconds (determines runtime)' );
+
+ parser.add_option("-r", "--range", type="int", dest="range", default=100,
+ help="range (limit) of generated results" );
+
+ parser.add_option("-w", "--width", type="int", dest="width", default=8,
+ help="Width ?" );
+
+ parser.add_option("-x", "--scale", type="int", dest="scale", default=1,
+ help="scale the results by this integer" );
+ # Not implemented yet
+ parser.add_option("-p", "--paramfile", type="string", dest="paramfile", default="none",
+ help="Not implemented yet" );
+ return (parser.parse_args());
+
+def log():
+ import datetime
+ print >> sys.stderr, "Called as: ", str(sys.argv)
+ print >> sys.stderr, "Start time: ", datetime.datetime.now()
+ print >> sys.stderr, "Running on node: ", socket.gethostname()
+ print >> sys.stderr, "Running as user: ", getpass.getuser()
+ print >> sys.stderr, "\nEnvironment:\n\n"
+ print >> sys.stderr, os.environ
+
+def printparams(options):
+ print 'bias =',options.bias
+ print 'biasfile = ',options.biasfile
+ print 'log = ', options.log
+ print 'nvalues = ',options.nvalues
+ print 'seed = ', options.initseed
+ print 'seedfile = ', options.seedfile
+ print 'timesteps = ', options.timesteps
+ print 'range = ', options.range
+ print 'width = ', options.width
+ print 'scale = ', options.scale
+ print 'paramfile = ', options.paramfile
+
+def simulate(options):
+ time.sleep(options.timesteps)
+ bias=[];
+ if (options.biasfile != "none"):
+ try:
+ with open(options.biasfile) as biasfile:
+ lines = biasfile.read().splitlines()
+ for line in lines:
+ bias.append(int(line))
+ except IOError:
+ print "Error accessing content from file: ", options.biasfile
+ bias_count = len(bias)
+
+ for i in range(options.nvalues):
+ value = (random.random() +
+ random.random()*math.pow(2,16) +
+ random.random()*math.pow(2,32) +
+ random.random()*math.pow(2,48))
+ value=( (int(value)%options.range) * options.scale + options.bias)
+ if ( i < bias_count ):
+ value = value + bias[i]
+ elif ( bias_count > 0 ):
+ value = value + bias[bias_count-1]
+
+ print '{num:{fill}{width}}'.format(num=value, fill=' ', width=options.width)
+
+
+def seed(options):
+ if (options.initseed != 0 ):
+ random.seed(options.initseed)
+ if (options.seedfile != "none"):
+ try:
+ with open(options.seedfile) as seedfile:
+ lines=seedfile.read().splitlines()
+ seed = 0
+ for line in lines:
+ seed = seed + int(line)
+ random.seed(seed)
+ except IOError:
+ print "Could not open file: ", options.seedfile
+
+
+if __name__ == "__main__":
+ (options, args) = parse()
+ printparams(options)
+ seed(options)
+ simulate(options)
+ log()
+
+
+
+
+
+
+
+
+
Property changes on: SwiftTutorials/swift-localhost-tutorial/app/simulate.py
___________________________________________________________________
Added: svn:executable
+ *
Added: SwiftTutorials/swift-localhost-tutorial/app/simulate.sh
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/app/simulate.sh (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/app/simulate.sh 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,135 @@
+#! /bin/bash
+
+printparams()
+{
+ printf "\nSimulation parameters:\n\n"
+ echo bias=$bias
+ echo biasfile=$biasfile
+ echo initseed=$initseed
+ echo log=$log
+ echo paramfile=$paramfile
+ echo range=$range
+ echo scale=$scale
+ echo seedfile=$seedfile
+ echo timesteps=$timesteps
+ echo output width=$width
+}
+
+log() {
+ printf "\nCalled as: $0: $cmdargs\n\n"
+ printf "Start time: "; /bin/date
+ printf "Running as user: "; /usr/bin/id
+ printf "Running on node: "; /bin/hostname
+ printf "Node IP address: "; /bin/hostname -I
+ printparams
+ printf "\nEnvironment:\n\n"
+ printenv | sort
+}
+
+addsims() {
+ while read f1 ; do
+ read -u 3 f2
+ if [ _$f1 = _ ]; then f1=$lastf1; fi
+ if [ _$f2 = _ ]; then f2=$lastf2; fi
+ printf "%${width}d\n" $(($f1+$f2))
+ lastf1=$f1
+ lastf2=$f2
+ done <$1 3<$2
+}
+
+# set defaults
+
+bias=0
+biasfile=none
+initseed=none
+log=yes
+paramfile=none
+range=100
+scale=1
+seedfile=none
+timesteps=0
+nvalues=1
+width=8
+cmdargs="$*"
+
+usage()
+{
+ echo $0: usage:
+ cat <<END
+ -b|--bias offset bias: add this integer to all results
+ -B|--biasfile file of integer biases to add to results
+ -l|--log generate a log in stderr if not null
+ -n|--nvalues print this many values per simulation
+ -r|--range range (limit) of generated results
+ -s|--seed use this integer [0..32767] as a seed
+ -S|--seedfile use this file (containing integer seeds [0..32767]) one per line
+ -t|--timesteps number of simulated "timesteps" in seconds (determines runtime)
+ -x|--scale scale the results by this integer
+ -h|-?|?|--help print this help
+END
+}
+
+# FIXME: NOT YET IMPLEMENTED:
+# -p|--paramfile take these parameters (in form param=value) from this file
+# -p|--paramfile) paramfile=$2 ;;
+
+while [ $# -gt 0 ]; do
+ case $1 in
+ -b|--bias) bias=$2 ;;
+ -B|--biasfile) biasfile=$2 ;;
+ -l|--log) log=$2 ;;
+ -n|--nvalues) nvalues=$2 ;;
+ -s|--seed) initseed=$2 ;;
+ -S|--seedfile) seedfile=$2 ;;
+ -t|--timesteps) timesteps=$2 ;;
+ -r|--range) range=$2 ;;
+ -w|--width) width=$2 ;;
+ -x|--scale) scale=$2 ;;
+ -h|-?|--help|*) usage; exit ;;
+ esac
+ shift 2
+done
+
+# process initial seed
+
+if [ $initseed != none ]; then
+ RANDOM=$initseed
+fi
+
+# process file of seeds
+
+if [ $seedfile != none ]; then
+ seed=0
+ while read $seedfile s; do
+ seed=$(($seed+$s))
+ done <$seedfile
+ RANDOM=$seed
+fi
+
+# run for some number of "timesteps"
+
+sleep $timesteps
+
+# emit N (nvalues) "simulation results" scaled and biased by argument values
+
+simout=$(mktemp simout.XXXX)
+for ((i=0;i<$nvalues;i++)); do
+ # value=$(( (($RANDOM)*(2**16))+$RANDOM ))
+ value=$(( (($RANDOM)*(2**48)) + (($RANDOM)*(2**32)) + (($RANDOM)*(2**16)) + $RANDOM ))
+ printf "%${width}d\n" $(( ($value%range)*scale+bias))
+done >$simout
+
+# process file of biases
+
+if [ $biasfile != none ]; then
+ addsims $simout $biasfile
+else
+ cat $simout
+fi
+rm $simout
+
+# log environmental data
+
+if [ $log != off ]; then
+ log 1>&2
+fi
Property changes on: SwiftTutorials/swift-localhost-tutorial/app/simulate.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: SwiftTutorials/swift-localhost-tutorial/app/stats
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/app/stats (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/app/stats 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1 @@
+link stats.sh
\ No newline at end of file
Property changes on: SwiftTutorials/swift-localhost-tutorial/app/stats
___________________________________________________________________
Added: svn:special
+ *
Added: SwiftTutorials/swift-localhost-tutorial/app/stats.py
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/app/stats.py (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/app/stats.py 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,31 @@
+#!/usr/bin/env python
+import socket
+import sys
+import os
+
+def log():
+ import datetime
+ print >> sys.stderr, "Called as: ", str(sys.argv)
+ print >> sys.stderr, "Start time: ", datetime.datetime.now()
+ print >> sys.stderr, "Running on node: ", socket.gethostname()
+ print >> sys.stderr, "Running as user: ",os.getlogin()
+ print >> sys.stderr, "\nEnvironment:\n\n"
+ print >> sys.stderr, os.environ
+
+def stat():
+ total = 0
+ count = 0
+ for f in sys.argv[1:]:
+ try:
+ with open(f) as ifile:
+ lines = ifile.read().splitlines()
+ for line in lines:
+ total += int(line)
+ count += 1
+ except IOError:
+ print "Error accessing content from file: ", options.biasfile
+ print total/count
+
+if __name__ == "__main__":
+ log()
+ stat()
Property changes on: SwiftTutorials/swift-localhost-tutorial/app/stats.py
___________________________________________________________________
Added: svn:executable
+ *
Added: SwiftTutorials/swift-localhost-tutorial/app/stats.sh
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/app/stats.sh (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/app/stats.sh 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,19 @@
+#! /bin/sh
+
+log() {
+ printf "\nCalled as: $0: $cmdargs\n\n"
+ printf "Start time: "; /bin/date
+ printf "Running as user: "; /usr/bin/id
+ printf "Running on node: "; /bin/hostname
+ printf "Node IP address: "; /bin/hostname -I
+ printf "\nEnvironment:\n\n"
+ printenv | sort
+}
+
+awk '
+
+{ sum += $1}
+
+END { printf("%d\n",sum/NR) }
+' $*
+log 1>&2
Property changes on: SwiftTutorials/swift-localhost-tutorial/app/stats.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: SwiftTutorials/swift-localhost-tutorial/bin/cleanup
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/bin/cleanup (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/bin/cleanup 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,2 @@
+rm -rf *.log *.rlog *.d p?-*-*-* *.kml *.swiftx *.out output outdir logs hi.* _concurrent .swift/tmp run???
+
Property changes on: SwiftTutorials/swift-localhost-tutorial/bin/cleanup
___________________________________________________________________
Added: svn:executable
+ *
Added: SwiftTutorials/swift-localhost-tutorial/bin/hosts
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/bin/hosts (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/bin/hosts 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1 @@
+for h in $(cat hosts.txt); do ssh $h hostname -f; done
Property changes on: SwiftTutorials/swift-localhost-tutorial/bin/hosts
___________________________________________________________________
Added: svn:executable
+ *
Added: SwiftTutorials/swift-localhost-tutorial/bin/package_tutorial.sh
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/bin/package_tutorial.sh (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/bin/package_tutorial.sh 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,15 @@
+#! /bin/sh
+
+BIN=$(cd $(dirname $0); pwd)
+BASE=$(cd $BIN/..; pwd)
+LOC=$(cd $BASE/..; pwd)
+
+echo executing $0 from BIN=$BIN
+echo making tutorial package of BASE=$BASE
+echo placing tutorial package in LOC=$LOC
+
+(
+ cd $LOC
+ tar zcf swift-cray-tutorial.tgz --exclude-vcs swift-cray-tutorial
+ scp swift-cray-tutorial.tgz p01532 at raven.cray.com:
+)
Property changes on: SwiftTutorials/swift-localhost-tutorial/bin/package_tutorial.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: SwiftTutorials/swift-localhost-tutorial/bin/plot.sh
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/bin/plot.sh (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/bin/plot.sh 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,68 @@
+#!/bin/bash
+
+#usage: ./plotswiftlogs.ketan <swift-logfile.log>
+
+SWIFTLOGFILE=$1
+
+#TMPDIR=`mktemp -d plotlog.XXX`
+
+grep -i ProgressTicker $SWIFTLOGFILE > swiftoutfile.out
+
+SWIFTOUTFILE=swiftoutfile.out
+
+#extract start time
+TMPDATE=`grep -i progress $SWIFTOUTFILE 2>/dev/null | head -n 1 | cut -f1-2 -d ' '`
+START_TIME=`date +%s -d "$TMPDATE"`
+
+#extract end time
+TMPDATE=`grep -i progress $SWIFTOUTFILE 2>/dev/null | tail -n 1 | cut -f1-2 -d ' '`
+END_TIME=`date +%s -d "$TMPDATE"`
+
+#duration
+DIFFTIME=$((END_TIME - START_TIME))
+
+#extract active runs in a file
+(grep -o -i "Active:[0-9]*" $SWIFTOUTFILE 2>/dev/null | awk -F: '{print $2}' >active.txt)
+
+#extract successful completions in a file
+(grep -o -i "Successfully:[0-9]*" $SWIFTOUTFILE 2>/dev/null | awk -F: '{print $2}' > cumulative.txt)
+
+#prepare tics
+activelines=`wc -l active.txt | awk '{print $1}'`
+cumulines=`wc -l cumulative.txt | awk '{print $1}'`
+
+if [ $activelines -ne 0 ]
+then
+ activelinespertic=`echo "scale=5 ; $DIFFTIME / $activelines" | bc`
+fi
+
+seq 0 $activelinespertic $DIFFTIME > activetics.txt
+
+if [ $cumulines -ne 0 ]
+then
+ cumulinespertic=`echo "scale=5 ; $DIFFTIME / $cumulines" | bc`
+fi
+
+seq 0 $cumulinespertic $DIFFTIME > cumultics.txt
+
+#final plot data
+paste activetics.txt active.txt > plot_active.txt
+paste cumultics.txt cumulative.txt > plot_cumulative.txt
+
+cat << EOF1 > plotit.gp
+set terminal png enhanced
+set nokey
+set output "cumulativeplot.png"
+set xlabel "Time in sec"
+set ylabel "number of completed apps"
+set title "Cumulative Apps Completed"
+plot "plot_cumulative.txt" using 1:2 with lines
+set output "activeplot.png"
+set xlabel "Time in sec"
+set ylabel "number of active apps"
+set title "Active Apps"
+plot "plot_active.txt" using 1:2 with line
+EOF1
+
+gnuplot plotit.gp 2>/dev/null
+rm swiftoutfile.out plotit.gp active.txt cumulative.txt activetics.txt plot_active.txt plot_cumulative.txt cumultics.txt
Property changes on: SwiftTutorials/swift-localhost-tutorial/bin/plot.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: SwiftTutorials/swift-localhost-tutorial/doc/README
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/doc/README (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/doc/README 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,619 @@
+Swift Tutorial for running on localhost
+=======================================
+
+This tutorial is viewable at:
+http://swift-lang.org/tutorials/localhost/tutorial.html
+
+
+Introduction: Why Parallel Scripting?
+------------------------------------
+
+Swift is a simple scripting language for executing many instances of
+ordinary application programs on distributed parallel resources.
+Swift scripts run many copies of ordinary programs concurrently, using
+statements like this:
+-----
+file out[];
+foreach param,i in paramList {
+ out[i] = simulate(param);
+}
+-----
+
+Swift acts like a structured "shell" language. It runs programs
+concurrently as soon as their inputs are available, reducing the need
+for complex parallel programming. Swift expresses your workflow in a
+portable fashion: The same script runs on multicore computers,
+clusters, clouds, grids, and supercomputers.
+
+In this tutorial, you'll be able to first try a few Swift examples
+on your local machine, to get a sense of the language.
+
+////
+Then in parts 4-6 you'll run similar workflows on cloud
+compute nodes, and see how more complex workflows can be expressed
+with Swift scripts.
+////
+
+Swift installation
+------------------
+1. Download the file from http://swiftlang.org/packages/swift-0.95-RC6.tar.gz
+2. Extract by running "tar xfz swift-0.95-RC6.tar.gz"
+3. Add to PATH by running "export PATH=$PATH:/path/to/swift-0.95-RC6/bin"
+
+Swift tutorial installation
+---------------------------
+Run the following commands to extract these tutorial scripts.
+
+-----
+$ cd $HOME
+$ wget http://swiftlang.org/tutorials/localhost/swift-localhost-tutorial.tar.gz
+$ tar xvfz swift-localhost-tutorial.tar.gz
+$ cd swift-localhost-tutorial
+-----
+
+Run the tutorial setup script
+-----
+$ source setup.sh # You must run this with "source" !
+-----
+
+To check out the tutorial scripts from SVN
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you later want to get the most recent version of this tutorial from
+the Swift Subversion repository, do:
+
+-----
+$ svn co https://svn.ci.uchicago.edu/svn/vdl2/SwiftTutorials/swift-localhost-tutorial <TODO: Fix link>
+-----
+
+This will create a directory called "swift-localhost-tutorial" which contains all of the
+files used in this tutorial.
+
+
+Simple "science applications" for the workflow tutorial
+-------------------------------------------------------
+
+This tutorial is based on two intentionally trivial example programs,
+`simulation.sh` and `stats.sh`, (implemented as bash shell scripts)
+that serve as easy-to-understand proxies for real science
+applications. These "programs" behave as follows.
+
+simulate.sh
+~~~~~~~~~~~
+
+The simulation.sh script serves as a trivial proxy for any more
+complex scientific simulation application. It generates and prints a
+set of one or more random integers in the range [0-2^62) as controlled
+by its command line arguments, which are:
+
+-----
+$ ./app/simulate.sh --help
+./app/simulate.sh: usage:
+ -b|--bias offset bias: add this integer to all results [0]
+ -B|--biasfile file of integer biases to add to results [none]
+ -l|--log generate a log in stderr if not null [y]
+ -n|--nvalues print this many values per simulation [1]
+ -r|--range range (limit) of generated results [100]
+ -s|--seed use this integer [0..32767] as a seed [none]
+ -S|--seedfile use this file (containing integer seeds [0..32767]) one per line [none]
+ -t|--timesteps number of simulated "timesteps" in seconds (determines runtime) [1]
+ -x|--scale scale the results by this integer [1]
+ -h|-?|?|--help print this help
+$
+-----
+
+All of thess arguments are optional, with default values indicated above as `[n]`.
+
+////
+.simulation.sh arguments
+[width="80%",cols="^2,10",options="header"]
+
+|=======================
+|Argument|Short|Description
+|1 |runtime: sets run time of simulation.sh in seconds
+|2 |range: limits generated values to the range [0,range-1]
+|3 |biasfile: add the integer contained in this file to each value generated
+|4 |scale: multiplies each generated value by this integer
+|5 |count: number of values to generate in the simulation
+|=======================
+////
+
+With no arguments, simulate.sh prints 1 number in the range of
+1-100. Otherwise it generates n numbers of the form (R*scale)+bias
+where R is a random integer. By default it logs information about its
+execution environment to stderr. Here's some examples of its usage:
+
+-----
+$ simulate.sh 2>log
+ 5
+$ head -4 log
+
+Called as: /home/wilde/swift/tut/CIC_2013-08-09/app/simulate.sh:
+Start time: Thu Aug 22 12:40:24 CDT 2013
+Running on node: login01.osgconnect.net
+
+$ simulate.sh -n 4 -r 1000000 2>log
+ 239454
+ 386702
+ 13849
+ 873526
+
+$ simulate.sh -n 3 -r 1000000 -x 100 2>log
+ 6643700
+62182300
+ 5230600
+
+$ simulate.sh -n 2 -r 1000 -x 1000 2>log
+ 565000
+ 636000
+
+$ time simulate.sh -n 2 -r 1000 -x 1000 -t 3 2>log
+ 336000
+ 320000
+real 0m3.012s
+user 0m0.005s
+sys 0m0.006s
+-----
+
+stats.sh
+~~~~~~~~
+
+The stats.sh script serves as a trivial model of an "analysis"
+program. It reads N files each containing M integers and simply prints
+the\ average of all those numbers to stdout. Similarly to simulate.sh
+it logs environmental information to the stderr.
+
+-----
+$ ls f*
+f1 f2 f3 f4
+
+$ cat f*
+25
+60
+40
+75
+
+$ stats.sh f* 2>log
+50
+-----
+
+
+Basics of the Swift language with local execution
+-------------------------------------------------
+
+A Summary of Swift in a nutshell
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Swift scripts are text files ending in `.swift` The `swift` command
+runs on any host, and executes these scripts. `swift` is a Java
+application, which you can install almost anywhere. On Linux, just
+unpack the distribution `tar` file and add its `bin/` directory to
+your `PATH`.
+
+* Swift scripts run ordinary applications, just like shell scripts
+do. Swift makes it easy to run these applications on parallel and
+remote computers (from laptops to supercomputers). If you can `ssh` to
+the system, Swift can likely run applications there.
+
+* The details of where to run applications and how to get files back
+and forth are described in configuration files separate from your
+program. Swift speaks ssh, PBS, Condor, SLURM, LSF, SGE, Cobalt, and
+Globus to run applications, and scp, http, ftp, and GridFTP to move
+data.
+
+* The Swift language has 5 main data types: `boolean`, `int`,
+`string`, `float`, and `file`. Collections of these are dynamic,
+sparse arrays of arbitrary dimension and structures of scalars and/or
+arrays defined by the `type` declaration.
+
+* Swift file variables are "mapped" to external files. Swift sends
+files to and from remote systems for you automatically.
+
+* Swift variables are "single assignment": once you set them you can't
+change them (in a given block of code). This makes Swift a natural,
+"parallel data flow" language. This programming model keeps your
+workflow scripts simple and easy to write and understand.
+
+* Swift lets you define functions to "wrap" application programs, and
+to cleanly structure more complex scripts. Swift `app` functions take
+files and parameters as inputs and return files as outputs.
+
+* A compact set of built-in functions for string and file
+manipulation, type conversions, high level IO, etc. is provided.
+Swift's equivalent of `printf()` is `tracef()`, with limited and
+slightly different format codes.
+
+* Swift's `foreach {}` statement is the main parallel workhorse of the
+language, and executes all iterations of the loop concurrently. The
+actual number of parallel tasks executed is based on available
+resources and settable "throttles".
+
+* In fact, Swift conceptually executes *all* the statements,
+expressions and function calls in your program in parallel, based on
+data flow. These are similarly throttled based on available resources
+and settings.
+
+* Swift also has `if` and `switch` statements for conditional
+execution. These are seldom needed in simple workflows but they enable
+very dynamic workflow patterns to be specified.
+
+We'll see many of these points in action in the examples below. Lets
+get started!
+
+Part 1: Run a single application under Swift
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The first swift script, p1.swift, runs simulate.sh to generate a
+single random number. It writes the number to a file.
+
+image::part01.png["p1 workflow",align="center"]
+
+.p1.swift
+-----
+sys::[cat ../part01/p1.swift]
+-----
+
+
+To run this script, run the following command:
+-----
+$ cd part01
+$ swift p1.swift
+Swift 0.95 Branch SVN swift-r7903 cog-r3908
+RunID: run001
+Progress: Tue, 03 Jun 2014 15:01:28-0500
+Final status:Tue, 03 Jun 2014 15:01:29-0500 Finished successfully:1
+$ cat sim.out
+ 84
+$ swift p1.swift
+$ cat sim.out
+ 36
+-----
+
+To cleanup the directory and remove all outputs (including the log
+files and directories that Swift generates), run the cleanup script
+which is located in the tutorial PATH:
+
+-----
+$ cleanup
+------
+
+NOTE: You'll also find a Swift configuration files in each `partNN`
+directory of this tutorial. These specify the environment-specific
+details of where to find application programs and where to run them .
+These files will be explained in more detail in parts 4-6, and can
+be ignored for now.
+
+////
+It defines
+things like the work directory, the scheduler to use, and how to
+control parallelism. The sites.xml file below will tell Swift to run
+on the local machine only, and run just 1 task at a time.
+
+.swift.properties
+-----
+sys::[cat ../part01/swift.properties]
+-----
+
+ In this case, it
+indicates that the app "simulate" (the first token in the command line
+declaration of the function `simulation`, at line NNN) is located in the file
+simulate.sh and (since the path `simulate.sh` is specified with no
+directory components) Swift expects that the `simulate.sh` executable
+will be available in your $PATH.
+
+////
+
+Part 2: Running an ensemble of many apps in parallel with a "foreach" loop
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The `p2.swift` script introduces the `foreach` parallel iteration
+construct to run many concurrent simulations.
+
+image::part02.png[align="center"]
+
+.p2.swift
+-----
+sys::[cat ../part02/p2.swift]
+-----
+
+The script also shows an
+example of naming the output files of an ensemble run. In this case, the output files will be named
+`output/sim_N.out`.
+
+In part 2, we update the swift.properties file.
+
+Instead of using shell script (simulate is a link to simulate.sh), we use the equivalent
+python version (simulate.py). Swift does not need to know anything about the language an
+application is written in. The application can be written in Perl, Python, Java, Fortran,
+or any other language.
+
+The swift.properties file also defines properties that control the parallelism involved
+in the swift execution. By setting the tasksPerWorker to the desired number of parallel
+application invocations it is possible to take advantage of multiple cores in the
+processor. The taskThrottle variable is used to limit number of tasks sent to
+workers for execution. Both tasksPerWorker and taskThrottle are set to 2 in the tutorial
+and could be set to the number of cores available on your machine.
+
+-----
+sys::[cat ../part02/swift.properties]
+-----
+
+
+To run the script and view the output:
+-----
+$ cd ../part02
+$ swift p2.swift
+$ ls output
+sim_0.out sim_1.out sim_2.out sim_3.out sim_4.out sim_5.out sim_6.out sim_7.out sim_8.out sim_9.out
+$ more output/*
+::::::::::::::
+output/sim_0.out
+::::::::::::::
+ 44
+::::::::::::::
+output/sim_1.out
+::::::::::::::
+ 55
+...
+::::::::::::::
+output/sim_9.out
+::::::::::::::
+ 82
+-----
+
+Part 3: Analyzing results of a parallel ensemble
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+After all the parallel simulations in an ensemble run have completed,
+its typically necessary to gather and analyze their results with some
+kind of post-processing analysis program or script. p3.swift
+introduces such a postprocessing step. In this case, the files created
+by all of the parallel runs of `simulation.sh` will be averaged by by
+the trivial "analysis application" `stats.sh`:
+
+image::part03.png[align="center"]
+
+.p3.swift
+----
+sys::[cat ../part03/p3.swift]
+----
+
+To run:
+----
+$ cd part03
+$ swift p3.swift
+----
+
+Note that in `p3.swift` we expose more of the capabilities of the
+`simulate.sh` application to the `simulation()` app function:
+
+-----
+app (file o) simulation (int sim_steps, int sim_range, int sim_values)
+{
+ simulate "--timesteps" sim_steps "--range" sim_range "--nvalues" sim_values stdout=filename(o);
+}
+-----
+
+`p3.swift` also shows how to fetch application-specific values from
+the `swift` command line in a Swift script using `arg()` which
+accepts a keyword-style argument and its default value:
+
+-----
+int nsim = toInt(arg("nsim","10"));
+int steps = toInt(arg("steps","1"));
+int range = toInt(arg("range","100"));
+int values = toInt(arg("values","5"));
+-----
+
+Now we can specify that more runs should be performed and that each should run for more timesteps, and produce more that one value each, within a specified range, using command line arguments placed after the Swift script name in the form `-parameterName=value`:
+
+-----
+$ swift p3.swift -nsim=3 -steps=10 -values=4 -range=1000000
+Swift 0.95 Branch SVN swift-r7903 cog-r3908
+RunID: run001
+Progress: Tue, 03 Jun 2014 17:06:58-0500
+Progress: Tue, 03 Jun 2014 17:06:59-0500 Submitted:2 Active:1
+Progress: Tue, 03 Jun 2014 17:07:00-0500 Submitted:1 Active:2
+Progress: Tue, 03 Jun 2014 17:07:01-0500 Active:3
+Progress: Tue, 03 Jun 2014 17:07:09-0500 Active:2 Finished successfully:1
+Progress: Tue, 03 Jun 2014 17:07:10-0500 Active:1 Finished successfully:2
+Final status:Tue, 03 Jun 2014 17:07:11-0500 Finished successfully:4
+
+$ ls output/
+average.out sim_0.out sim_1.out sim_2.out
+$ more output/*
+::::::::::::::
+output/average.out
+::::::::::::::
+651368
+::::::::::::::
+output/sim_0.out
+::::::::::::::
+ 735700
+ 886206
+ 997391
+ 982970
+::::::::::::::
+output/sim_1.out
+::::::::::::::
+ 260071
+ 264195
+ 869198
+ 933537
+::::::::::::::
+output/sim_2.out
+::::::::::::::
+ 201806
+ 213540
+ 527576
+ 944233
+-----
+
+Now try running (`-nsim=`) 100 simulations of (`-steps=`) 1 second each:
+
+-----
+$ swift p3.swift -nsim=100 -steps=1
+Swift 0.95 Branch SVN swift-r7903 cog-r3908
+RunID: run002
+Progress: Tue, 03 Jun 2014 17:08:05-0500
+Progress: Tue, 03 Jun 2014 17:08:06-0500 Selecting site:79 Submitted:20 Active:1
+Progress: Tue, 03 Jun 2014 17:08:07-0500 Selecting site:78 Submitted:18 Active:3 Finished successfully:1
+Progress: Tue, 03 Jun 2014 17:08:08-0500 Selecting site:75 Submitted:17 Active:4 Finished successfully:4
+Progress: Tue, 03 Jun 2014 17:08:09-0500 Selecting site:71 Submitted:16 Active:5 Finished successfully:8
+Progress: Tue, 03 Jun 2014 17:08:11-0500 Selecting site:66 Submitted:15 Active:6 Finished successfully:13
+Progress: Tue, 03 Jun 2014 17:08:12-0500 Selecting site:60 Submitted:14 Active:7 Finished successfully:19
+Progress: Tue, 03 Jun 2014 17:08:13-0500 Selecting site:53 Submitted:13 Active:8 Finished successfully:26
+Progress: Tue, 03 Jun 2014 17:08:14-0500 Selecting site:45 Submitted:12 Active:9 Finished successfully:34
+Progress: Tue, 03 Jun 2014 17:08:15-0500 Selecting site:36 Submitted:11 Active:10 Finished successfully:43
+Progress: Tue, 03 Jun 2014 17:08:16-0500 Selecting site:26 Submitted:10 Active:11 Finished successfully:53
+Progress: Tue, 03 Jun 2014 17:08:17-0500 Selecting site:15 Active:21 Finished successfully:64
+Progress: Tue, 03 Jun 2014 17:08:18-0500 Active:16 Finished successfully:84
+Progress: Tue, 03 Jun 2014 17:08:19-0500 Active:11 Finished successfully:89
+Final status:Tue, 03 Jun 2014 17:08:19-0500 Finished successfully:101
+-----
+
+Part 4: Specifying more complex workflow patterns
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+p4.swift expands the workflow pattern of p3.swift to add additional
+stages to the workflow. Here, we generate a dynamic seed value that
+will be used by all of the simulations, and for each simulation, we
+run an pre-processing application to generate a unique "bias
+file". This pattern is shown below, followed by the Swift script.
+
+image::part06.png[align="center"]
+
+.p4.swift
+----
+sys::[cat ../part04/p4.swift]
+----
+
+Note that the workflow is based on data flow dependencies: each simulation depends on the seed value, calculated in this statement:
+-----
+seedfile = genseed(1);
+-----
+and on the bias file, computed and then consumed in these two dependent statements:
+-----
+ biasfile = genbias(1000, 20, simulate_script);
+ (simout,simlog) = simulation(steps, range, biasfile, 1000000, values, simulate_script, seedfile);
+-----
+
+To run:
+----
+$ cd ../part04
+$ swift p4.swift
+----
+
+The default parameters result in the following execution log:
+
+-----
+$ swift p4.swift
+Swift 0.95 Branch SVN swift-r7903 cog-r3908
+RunID: run001
+Progress: Thu, 05 Jun 2014 06:01:36-0500
+
+*** Script parameters: nsim=10 range=100 num values=10
+
+Progress: Thu, 05 Jun 2014 06:01:37-0500 Submitted:8 Active:2 Finished successfully:11
+Progress: Thu, 05 Jun 2014 06:01:38-0500 Submitted:6 Active:2 Finished successfully:13
+Progress: Thu, 05 Jun 2014 06:01:39-0500 Submitted:4 Active:2 Finished successfully:15
+Progress: Thu, 05 Jun 2014 06:01:40-0500 Submitted:2 Active:2 Finished successfully:17
+Progress: Thu, 05 Jun 2014 06:01:41-0500 Active:2 Finished successfully:19
+Final status:Thu, 05 Jun 2014 06:01:41-0500 Finished successfully:22
+-----
+
+which produces the following output:
+-----
+$ ls -lrt output
+total 264
+-rw-r--r-- 1 p01532 61532 9 Aug 27 19:17 seed.dat
+-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_9.dat
+-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_8.dat
+-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_7.dat
+-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_6.dat
+-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_5.dat
+-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_4.dat
+-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_3.dat
+-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_2.dat
+-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_1.dat
+-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_0.dat
+-rw-r--r-- 1 p01532 61532 90 Aug 27 19:17 sim_9.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_9.log
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_8.log
+-rw-r--r-- 1 p01532 61532 90 Aug 27 19:17 sim_7.out
+-rw-r--r-- 1 p01532 61532 90 Aug 27 19:17 sim_6.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_6.log
+-rw-r--r-- 1 p01532 61532 90 Aug 27 19:17 sim_5.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_5.log
+-rw-r--r-- 1 p01532 61532 90 Aug 27 19:17 sim_4.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_4.log
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_1.log
+-rw-r--r-- 1 p01532 61532 90 Aug 27 19:18 sim_8.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:18 sim_7.log
+-rw-r--r-- 1 p01532 61532 90 Aug 27 19:18 sim_3.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:18 sim_3.log
+-rw-r--r-- 1 p01532 61532 90 Aug 27 19:18 sim_2.out
+-rw-r--r-- 1 p01532 61532 14898 Aug 27 19:18 sim_2.log
+-rw-r--r-- 1 p01532 61532 90 Aug 27 19:18 sim_1.out
+-rw-r--r-- 1 p01532 61532 90 Aug 27 19:18 sim_0.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:18 sim_0.log
+-rw-r--r-- 1 p01532 61532 9 Aug 27 19:18 average.out
+-rw-r--r-- 1 p01532 61532 14675 Aug 27 19:18 average.log
+-----
+
+Each sim_N.out file is the sum of its bias file plus newly "simulated" random output scaled by 1,000,000:
+
+-----
+$ cat output/bias_0.dat
+ 302
+ 489
+ 81
+ 582
+ 664
+ 290
+ 839
+ 258
+ 506
+ 310
+ 293
+ 508
+ 88
+ 261
+ 453
+ 187
+ 26
+ 198
+ 402
+ 555
+
+$ cat output/sim_0.out
+64000302
+38000489
+32000081
+12000582
+46000664
+36000290
+35000839
+22000258
+49000506
+75000310
+-----
+
+We produce 20 values in each bias file. Simulations of less than that
+number of values ignore the unneeded number, while simualtions of more
+than 20 will use the last bias number for all remoaining values past
+20. As an exercise, adjust the code to produce the same number of
+bias values as is needed for each simulation. As a further exercise,
+modify the script to generate a unique seed value for each simulation,
+which is a common practice in ensemble computations.
+
+Performing larger Swift runs
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To test with larger runs, provide suitable command line arguments.
+The example below will run 1000 simulations with each simulation
+taking 5 seconds.
+
+-----
+$ swift p4.swift -steps=5 -nsim=1000
+-----
Added: SwiftTutorials/swift-localhost-tutorial/doc/TODO
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/doc/TODO (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/doc/TODO 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,137 @@
+PLAN
+
+basics
+cloud
+uc3
+multisystems
+(modis)
+
+get hosts.txt for cloud dynamically
+
+Show variety of UC3 activity and what ran where
+
+get hostname and app_resources into provider staging (hostname into all _swiftwrap)
+
+Show gridftp and go connect to process the modis dataset
+
+Show SwiftR for R users, SwiftPy for Enthought PY?
+
+fetching apps vs sending apps vs parrot/cvmfs/oasis etc ????
+
+Demo script-aps, bin apps, building apps, caching apps. etc.
+
+Do a Swift BLAST workflow -- with viz !!! (spice it up with help from Dina)
+
+OSG page from Mats: "Is your job HTC-ready"
+
+Use new Trunk config in tutorial?
+
+
+Add viz and status display; add node probing to tutorial.
+
+Debug failure with pinfiles
+
+
+What "exercises to suggest?"
+
+- run stats app locally
+- installed vs transferred apps
+- transferred programs vs transferred scripts
+- summarize things about the env from the .log files, ala stats.
+
+Y-style dag: 2 sims and average
+
+Show how to find temp files at runtime...
+
+Show how to find hung processes at runtime... (or get other trace data...)
+
+make your owb versions of the programs; verify that you can substitute them
+
+Add a layer to isolate shell version v installed version
+
+Show how to handle a paramfile
+
+Test if OSGCOnnect works and account string (PorjectName) works OK for both!
+
+SHow how to handle long lists of files (eg writeData and other swift.properties)
+
+Expand arith to arb prec using bc
+
+
+ISSUES
+
+error in pin files? debug and turn back on
+PATH issues for various providers
+
+bin dir?
+
+setup.csh and .csh testing
+
+IMPROVEMENTS
+
+1. Cleanup.sh mentioned in the tutorial
+[fixed]
+
+2. cd ../part02
+[fixed]
+
+3. support for zsh, csh
+[pending]
+
+4. What results are created and where did the outputs go ?
+and pointers to what the log files mean
+[pending]
+
+5. Part4,
+- cd ../part04 is missing. Cleanup the README
+[fixed]
+
+6. Part5
+- cd ../part05 is missing. Cleanup
+[fixed]
+
+7. Part7
+-Change from running on Tukey analysis cluster compute nodes
+[fixed]
+
+8. Part6
+command line args for nsim and steps, tell user the different behavior
+that can be expected with different arg options
+[pending]
+
+9. Draw workflow for the modis demos
+[pending]
+
+10. Link to more technical swift-lang references
+[pending]
+
+11. Links to detailed language constructs, when explaining
+apps, link to the section on apps in the userguide.
+[pending]
+
+
+
+---
+
+From Yadu:
+
+I've committed all the changes that I made on Friday and you can get them
+from -> https://svn.ci.uchicago.edu/svn/vdl2/SwiftTutorials/CIC_2013-08-09
+
+I'm trying to make this tutorial easily work on several sites so that, we
+do not have to tweak things for every single tutorial. Once this is done,
+I can also add it to the test-battery as well. I think this has value
+considering that we spend atleast 1-2 days for every demo.
+
+Now, the major pending item is the Modis demo, specifically the issue of
+bringing data to any site and scripts. Slight tweaks to allow for sending
+the scripts as args to apps would also be needed.
+
+So the major changes are:
+1. Source the setup.sh script with the target site name
+eg. source setups.sh <cloud|uc3|midway|beagle>
+2. No separate cloud folder
+3. [pending] fix modis data and scripts
+4. [pending] support for zsh and other shells?
+
+---
Added: SwiftTutorials/swift-localhost-tutorial/doc/activeplot.png
===================================================================
(Binary files differ)
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/activeplot.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: SwiftTutorials/swift-localhost-tutorial/doc/asciidoc.css
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/doc/asciidoc.css (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/doc/asciidoc.css 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,26 @@
+a:link { color:navy; }
+a:visited { color:navy; }
+
+.monospaced, code, pre {
+ font-family: "Courier New", Courier, monospace;
+ font-size: medium; /* inherit; */
+ color: black;
+ padding: 0;
+ margin: 0;
+}
+
+/*
+ background: #f8f8f8;
+ border: 1px solid #dddddd;
+ border-left: 5px solid #f0f0f0;
+ margin-right: 10%;
+*/
+
+div.listingblock > div.content {
+ padding: 0.5em;
+ background: none;
+ border: none;
+ border-left: none;
+ margin-right: none;
+}
+
Added: SwiftTutorials/swift-localhost-tutorial/doc/build_docs.sh
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/doc/build_docs.sh (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/doc/build_docs.sh 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,4 @@
+#!/bin/bash -e
+
+asciidoc -a icons -a toc -a toplevels=2 -a stylesheet=$PWD/asciidoc.css -a max-width=800px -o tutorial.html README
+
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/build_docs.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: SwiftTutorials/swift-localhost-tutorial/doc/cumulativeplot.png
===================================================================
(Binary files differ)
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/cumulativeplot.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: SwiftTutorials/swift-localhost-tutorial/doc/figs/modis.dia
===================================================================
(Binary files differ)
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/figs/modis.dia
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: SwiftTutorials/swift-localhost-tutorial/doc/figs/modis.png
===================================================================
(Binary files differ)
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/figs/modis.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/caution.png
===================================================================
(Binary files differ)
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/caution.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/example.png
===================================================================
(Binary files differ)
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/example.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/home.png
===================================================================
(Binary files differ)
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/home.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/important.png
===================================================================
(Binary files differ)
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/important.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/next.png
===================================================================
(Binary files differ)
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/next.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/note.png
===================================================================
(Binary files differ)
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/note.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/prev.png
===================================================================
(Binary files differ)
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/prev.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/tip.png
===================================================================
(Binary files differ)
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/tip.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/up.png
===================================================================
(Binary files differ)
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/up.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/warning.png
===================================================================
(Binary files differ)
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/warning.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: SwiftTutorials/swift-localhost-tutorial/doc/part01.png
===================================================================
(Binary files differ)
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/part01.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: SwiftTutorials/swift-localhost-tutorial/doc/part02.png
===================================================================
(Binary files differ)
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/part02.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: SwiftTutorials/swift-localhost-tutorial/doc/part03.png
===================================================================
(Binary files differ)
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/part03.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: SwiftTutorials/swift-localhost-tutorial/doc/part04.png
===================================================================
(Binary files differ)
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/part04.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: SwiftTutorials/swift-localhost-tutorial/doc/part05.png
===================================================================
(Binary files differ)
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/part05.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: SwiftTutorials/swift-localhost-tutorial/doc/part06.png
===================================================================
(Binary files differ)
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/part06.png
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: SwiftTutorials/swift-localhost-tutorial/doc/push.sh
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/doc/push.sh (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/doc/push.sh 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,11 @@
+#! /bin/sh
+
+# scp -r swift-cray-tutorial.html images *png login.ci.uchicago.edu:/ci/www/projects/swift/tutorials/cray
+
+pushd .
+cd ../../
+tar -cvzf swift-localhost-tutorial.tar.gz swift-localhost-tutorial
+mv swift-localhost-tutorial.tar.gz swift-localhost-tutorial/doc/
+popd
+
+tar zcf - --exclude-vcs *tar.gz *html *png images | ssh login.ci.uchicago.edu "cd /ci/www/projects/swift/tutorials/localhost; tar zxf -"
Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/push.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: SwiftTutorials/swift-localhost-tutorial/doc/raven.1000.progress.out
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/doc/raven.1000.progress.out (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/doc/raven.1000.progress.out 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,34 @@
+p01532 at raven:~/swift-cray-tutorial/part04> swift p4.swift -nsim=1000 -steps=5
+Swift 0.94.1 RC2 swift-r6895 cog-r3765
+
+RunID: 20130824-2131-g43l2g51
+Progress: time: Sat, 24 Aug 2013 21:31:42 -0500
+
+*** Script parameters: nsim=1000 steps=5 range=100
+
+Progress: time: Sat, 24 Aug 2013 21:31:43 -0500 Initializing:998 Selecting site:2
+Progress: time: Sat, 24 Aug 2013 21:31:45 -0500 Submitting:999 Submitted:1
+Progress: time: Sat, 24 Aug 2013 21:31:46 -0500 Stage in:1 Submitted:999
+Progress: time: Sat, 24 Aug 2013 21:31:52 -0500 Submitted:712 Active:287 Stage out:1
+Progress: time: Sat, 24 Aug 2013 21:31:53 -0500 Submitted:682 Active:278 Stage out:8 Finished successfully:32
+Progress: time: Sat, 24 Aug 2013 21:31:54 -0500 Submitted:605 Active:252 Stage out:32 Finished successfully:111
+Progress: time: Sat, 24 Aug 2013 21:31:55 -0500 Submitted:480 Active:264 Stage out:23 Finished successfully:233
+Progress: time: Sat, 24 Aug 2013 21:31:57 -0500 Submitted:442 Active:287 Stage out:1 Finished successfully:270
+Progress: time: Sat, 24 Aug 2013 21:31:58 -0500 Submitted:432 Active:287 Stage out:1 Finished successfully:280
+Progress: time: Sat, 24 Aug 2013 21:31:59 -0500 Submitted:411 Active:287 Stage out:1 Finished successfully:301
+Progress: time: Sat, 24 Aug 2013 21:32:00 -0500 Submitted:317 Active:287 Finished successfully:396
+Progress: time: Sat, 24 Aug 2013 21:32:01 -0500 Submitted:244 Active:263 Stage out:25 Finished successfully:468
+Progress: time: Sat, 24 Aug 2013 21:32:02 -0500 Submitted:213 Active:233 Stage out:1 Finished successfully:553
+Progress: time: Sat, 24 Aug 2013 21:32:03 -0500 Submitted:210 Active:221 Stage out:1 Finished successfully:568
+Progress: time: Sat, 24 Aug 2013 21:32:04 -0500 Submitted:191 Active:221 Finished successfully:588
+Progress: time: Sat, 24 Aug 2013 21:32:05 -0500 Submitted:122 Active:213 Stage out:7 Finished successfully:658
+Progress: time: Sat, 24 Aug 2013 21:32:06 -0500 Submitted:117 Active:129 Stage out:5 Finished successfully:749
+Progress: time: Sat, 24 Aug 2013 21:32:09 -0500 Submitted:117 Active:98 Stage out:1 Finished successfully:784
+Progress: time: Sat, 24 Aug 2013 21:32:10 -0500 Submitted:96 Active:95 Stage out:1 Finished successfully:808
+Progress: time: Sat, 24 Aug 2013 21:32:11 -0500 Stage in:1 Submitted:33 Active:91 Stage out:4 Finished successfully:871
+Progress: time: Sat, 24 Aug 2013 21:32:12 -0500 Submitted:21 Active:96 Finished successfully:883
+Progress: time: Sat, 24 Aug 2013 21:32:14 -0500 Submitted:21 Active:95 Stage out:1 Finished successfully:883
+Progress: time: Sat, 24 Aug 2013 21:32:15 -0500 Active:93 Stage out:1 Finished successfully:906
+Progress: time: Sat, 24 Aug 2013 21:32:16 -0500 Active:42 Stage out:2 Finished successfully:956
+Progress: time: Sat, 24 Aug 2013 21:32:20 -0500 Active:20 Stage out:1 Finished successfully:979
+Final status: Sat, 24 Aug 2013 21:32:20 -0500 Finished successfully:1000
Added: SwiftTutorials/swift-localhost-tutorial/doc/raven.10000.progress.out
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/doc/raven.10000.progress.out (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/doc/raven.10000.progress.out 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,34 @@
+p01532 at raven:~/swift-cray-tutorial/part04> swift p4.swift -nsim=10000 -steps=2
+Swift 0.94.1 RC2 swift-r6895 cog-r3765
+
+RunID: 20130824-2132-n4z3dxk1
+Progress: time: Sat, 24 Aug 2013 21:32:43 -0500
+
+*** Script parameters: nsim=10000 steps=2 range=100
+
+Progress: time: Sat, 24 Aug 2013 21:32:45 -0500 Initializing:3
+Progress: time: Sat, 24 Aug 2013 21:32:46 -0500 Initializing:33
+Progress: time: Sat, 24 Aug 2013 21:32:47 -0500 Initializing:1613
+Progress: time: Sat, 24 Aug 2013 21:32:50 -0500 Selecting site:9999 Submitting:1
+Progress: time: Sat, 24 Aug 2013 21:32:52 -0500 Selecting site:8998 Stage in:1 Submitted:1001
+Progress: time: Sat, 24 Aug 2013 21:32:55 -0500 Selecting site:8998 Submitted:714 Active:287 Stage out:1
+Progress: time: Sat, 24 Aug 2013 21:32:57 -0500 Selecting site:8950 Submitted:714 Active:287 Stage out:1 Finished successfully:48
+Progress: time: Sat, 24 Aug 2013 21:32:58 -0500 Selecting site:8921 Submitting:1 Submitted:713 Active:257 Stage out:31 Finished successfully:77
+Progress: time: Sat, 24 Aug 2013 21:32:59 -0500 Selecting site:8858 Stage in:1 Submitting:5 Submitted:708 Active:178 Stage out:110 Finished successfully:140
+Progress: time: Sat, 24 Aug 2013 21:33:00 -0500 Selecting site:8658 Stage in:1 Submitting:1 Submitted:713 Active:287 Finished successfully:340
+.....
+Progress: time: Sat, 24 Aug 2013 21:34:39 -0500 Submitted:555 Active:287 Finished successfully:9158
+Progress: time: Sat, 24 Aug 2013 21:34:40 -0500 Submitted:454 Active:283 Stage out:4 Finished successfully:9259
+Progress: time: Sat, 24 Aug 2013 21:34:41 -0500 Submitted:361 Active:285 Stage out:3 Finished successfully:9351
+Progress: time: Sat, 24 Aug 2013 21:34:42 -0500 Submitted:265 Active:287 Stage out:1 Finished successfully:9447
+Progress: time: Sat, 24 Aug 2013 21:34:43 -0500 Submitted:206 Active:224 Stage out:6 Finished successfully:9564
+Progress: time: Sat, 24 Aug 2013 21:34:44 -0500 Submitted:185 Active:155 Stage out:10 Finished successfully:9650
+Progress: time: Sat, 24 Aug 2013 21:34:45 -0500 Submitted:181 Active:124 Stage out:39 Finished successfully:9656
+Progress: time: Sat, 24 Aug 2013 21:34:46 -0500 Submitted:169 Active:56 Stage out:94 Finished successfully:9681
+Progress: time: Sat, 24 Aug 2013 21:34:47 -0500 Submitted:76 Active:95 Stage out:1 Finished successfully:9828
+Progress: time: Sat, 24 Aug 2013 21:34:48 -0500 Submitted:67 Active:95 Stage out:1 Finished successfully:9837
+Progress: time: Sat, 24 Aug 2013 21:34:49 -0500 Active:81 Stage out:1 Finished successfully:9918
+Progress: time: Sat, 24 Aug 2013 21:34:50 -0500 Active:71 Stage out:1 Finished successfully:9928
+Progress: time: Sat, 24 Aug 2013 21:34:51 -0500 Active:6 Stage out:2 Finished successfully:9992
+Final status: Sat, 24 Aug 2013 21:34:51 -0500 Finished successfully:10000
+p01532 at raven:~/swift-cray-tutorial/part04>
Added: SwiftTutorials/swift-localhost-tutorial/doc/tutorial.html
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/doc/tutorial.html (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/doc/tutorial.html 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,1518 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
+ "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
+<head>
+<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" />
+<meta name="generator" content="AsciiDoc 8.6.9" />
+<title>Swift Tutorial for running on localhost</title>
+<style type="text/css">
+/* Shared CSS for AsciiDoc xhtml11 and html5 backends */
+
+/* Default font. */
+body {
+ font-family: Georgia,serif;
+}
+
+/* Title font. */
+h1, h2, h3, h4, h5, h6,
+div.title, caption.title,
+thead, p.table.header,
+#toctitle,
+#author, #revnumber, #revdate, #revremark,
+#footer {
+ font-family: Arial,Helvetica,sans-serif;
+}
+
+body {
+ margin: 1em 5% 1em 5%;
+}
+
+a {
+ color: blue;
+ text-decoration: underline;
+}
+a:visited {
+ color: fuchsia;
+}
+
+em {
+ font-style: italic;
+ color: navy;
+}
+
+strong {
+ font-weight: bold;
+ color: #083194;
+}
+
+h1, h2, h3, h4, h5, h6 {
+ color: #527bbd;
+ margin-top: 1.2em;
+ margin-bottom: 0.5em;
+ line-height: 1.3;
+}
+
+h1, h2, h3 {
+ border-bottom: 2px solid silver;
+}
+h2 {
+ padding-top: 0.5em;
+}
+h3 {
+ float: left;
+}
+h3 + * {
+ clear: left;
+}
+h5 {
+ font-size: 1.0em;
+}
+
+div.sectionbody {
+ margin-left: 0;
+}
+
+hr {
+ border: 1px solid silver;
+}
+
+p {
+ margin-top: 0.5em;
+ margin-bottom: 0.5em;
+}
+
+ul, ol, li > p {
+ margin-top: 0;
+}
+ul > li { color: #aaa; }
+ul > li > * { color: black; }
+
+.monospaced, code, pre {
+ font-family: "Courier New", Courier, monospace;
+ font-size: inherit;
+ color: navy;
+ padding: 0;
+ margin: 0;
+}
+pre {
+ white-space: pre-wrap;
+}
+
+#author {
+ color: #527bbd;
+ font-weight: bold;
+ font-size: 1.1em;
+}
+#email {
+}
+#revnumber, #revdate, #revremark {
+}
+
+#footer {
+ font-size: small;
+ border-top: 2px solid silver;
+ padding-top: 0.5em;
+ margin-top: 4.0em;
+}
+#footer-text {
+ float: left;
+ padding-bottom: 0.5em;
+}
+#footer-badges {
+ float: right;
+ padding-bottom: 0.5em;
+}
+
+#preamble {
+ margin-top: 1.5em;
+ margin-bottom: 1.5em;
+}
+div.imageblock, div.exampleblock, div.verseblock,
+div.quoteblock, div.literalblock, div.listingblock, div.sidebarblock,
+div.admonitionblock {
+ margin-top: 1.0em;
+ margin-bottom: 1.5em;
+}
+div.admonitionblock {
+ margin-top: 2.0em;
+ margin-bottom: 2.0em;
+ margin-right: 10%;
+ color: #606060;
+}
+
+div.content { /* Block element content. */
+ padding: 0;
+}
+
+/* Block element titles. */
+div.title, caption.title {
+ color: #527bbd;
+ font-weight: bold;
+ text-align: left;
+ margin-top: 1.0em;
+ margin-bottom: 0.5em;
+}
+div.title + * {
+ margin-top: 0;
+}
+
+td div.title:first-child {
+ margin-top: 0.0em;
+}
+div.content div.title:first-child {
+ margin-top: 0.0em;
+}
+div.content + div.title {
+ margin-top: 0.0em;
+}
+
+div.sidebarblock > div.content {
+ background: #ffffee;
+ border: 1px solid #dddddd;
+ border-left: 4px solid #f0f0f0;
+ padding: 0.5em;
+}
+
+div.listingblock > div.content {
+ border: 1px solid #dddddd;
+ border-left: 5px solid #f0f0f0;
+ background: #f8f8f8;
+ padding: 0.5em;
+}
+
+div.quoteblock, div.verseblock {
+ padding-left: 1.0em;
+ margin-left: 1.0em;
+ margin-right: 10%;
+ border-left: 5px solid #f0f0f0;
+ color: #888;
+}
+
+div.quoteblock > div.attribution {
+ padding-top: 0.5em;
+ text-align: right;
+}
+
+div.verseblock > pre.content {
+ font-family: inherit;
+ font-size: inherit;
+}
+div.verseblock > div.attribution {
+ padding-top: 0.75em;
+ text-align: left;
+}
+/* DEPRECATED: Pre version 8.2.7 verse style literal block. */
+div.verseblock + div.attribution {
+ text-align: left;
+}
+
+div.admonitionblock .icon {
+ vertical-align: top;
+ font-size: 1.1em;
+ font-weight: bold;
+ text-decoration: underline;
+ color: #527bbd;
+ padding-right: 0.5em;
+}
+div.admonitionblock td.content {
+ padding-left: 0.5em;
+ border-left: 3px solid #dddddd;
+}
+
+div.exampleblock > div.content {
+ border-left: 3px solid #dddddd;
+ padding-left: 0.5em;
+}
+
+div.imageblock div.content { padding-left: 0; }
+span.image img { border-style: none; vertical-align: text-bottom; }
+a.image:visited { color: white; }
+
+dl {
+ margin-top: 0.8em;
+ margin-bottom: 0.8em;
+}
+dt {
+ margin-top: 0.5em;
+ margin-bottom: 0;
+ font-style: normal;
+ color: navy;
+}
+dd > *:first-child {
+ margin-top: 0.1em;
+}
+
+ul, ol {
+ list-style-position: outside;
+}
+ol.arabic {
+ list-style-type: decimal;
+}
+ol.loweralpha {
+ list-style-type: lower-alpha;
+}
+ol.upperalpha {
+ list-style-type: upper-alpha;
+}
+ol.lowerroman {
+ list-style-type: lower-roman;
+}
+ol.upperroman {
+ list-style-type: upper-roman;
+}
+
+div.compact ul, div.compact ol,
+div.compact p, div.compact p,
+div.compact div, div.compact div {
+ margin-top: 0.1em;
+ margin-bottom: 0.1em;
+}
+
+tfoot {
+ font-weight: bold;
+}
+td > div.verse {
+ white-space: pre;
+}
+
+div.hdlist {
+ margin-top: 0.8em;
+ margin-bottom: 0.8em;
+}
+div.hdlist tr {
+ padding-bottom: 15px;
+}
+dt.hdlist1.strong, td.hdlist1.strong {
+ font-weight: bold;
+}
+td.hdlist1 {
+ vertical-align: top;
+ font-style: normal;
+ padding-right: 0.8em;
+ color: navy;
+}
+td.hdlist2 {
+ vertical-align: top;
+}
+div.hdlist.compact tr {
+ margin: 0;
+ padding-bottom: 0;
+}
+
+.comment {
+ background: yellow;
+}
+
+.footnote, .footnoteref {
+ font-size: 0.8em;
+}
+
+span.footnote, span.footnoteref {
+ vertical-align: super;
+}
+
+#footnotes {
+ margin: 20px 0 20px 0;
+ padding: 7px 0 0 0;
+}
+
+#footnotes div.footnote {
+ margin: 0 0 5px 0;
+}
+
+#footnotes hr {
+ border: none;
+ border-top: 1px solid silver;
+ height: 1px;
+ text-align: left;
+ margin-left: 0;
+ width: 20%;
+ min-width: 100px;
+}
+
+div.colist td {
+ padding-right: 0.5em;
+ padding-bottom: 0.3em;
+ vertical-align: top;
+}
+div.colist td img {
+ margin-top: 0.3em;
+}
+
+ at media print {
+ #footer-badges { display: none; }
+}
+
+#toc {
+ margin-bottom: 2.5em;
+}
+
+#toctitle {
+ color: #527bbd;
+ font-size: 1.1em;
+ font-weight: bold;
+ margin-top: 1.0em;
+ margin-bottom: 0.1em;
+}
+
+div.toclevel0, div.toclevel1, div.toclevel2, div.toclevel3, div.toclevel4 {
+ margin-top: 0;
+ margin-bottom: 0;
+}
+div.toclevel2 {
+ margin-left: 2em;
+ font-size: 0.9em;
+}
+div.toclevel3 {
+ margin-left: 4em;
+ font-size: 0.9em;
+}
+div.toclevel4 {
+ margin-left: 6em;
+ font-size: 0.9em;
+}
+
+span.aqua { color: aqua; }
+span.black { color: black; }
+span.blue { color: blue; }
+span.fuchsia { color: fuchsia; }
+span.gray { color: gray; }
+span.green { color: green; }
+span.lime { color: lime; }
+span.maroon { color: maroon; }
+span.navy { color: navy; }
+span.olive { color: olive; }
+span.purple { color: purple; }
+span.red { color: red; }
+span.silver { color: silver; }
+span.teal { color: teal; }
+span.white { color: white; }
+span.yellow { color: yellow; }
+
+span.aqua-background { background: aqua; }
+span.black-background { background: black; }
+span.blue-background { background: blue; }
+span.fuchsia-background { background: fuchsia; }
+span.gray-background { background: gray; }
+span.green-background { background: green; }
+span.lime-background { background: lime; }
+span.maroon-background { background: maroon; }
+span.navy-background { background: navy; }
+span.olive-background { background: olive; }
+span.purple-background { background: purple; }
+span.red-background { background: red; }
+span.silver-background { background: silver; }
+span.teal-background { background: teal; }
+span.white-background { background: white; }
+span.yellow-background { background: yellow; }
+
+span.big { font-size: 2em; }
+span.small { font-size: 0.6em; }
+
+span.underline { text-decoration: underline; }
+span.overline { text-decoration: overline; }
+span.line-through { text-decoration: line-through; }
+
+div.unbreakable { page-break-inside: avoid; }
+
+
+/*
+ * xhtml11 specific
+ *
+ * */
+
+div.tableblock {
+ margin-top: 1.0em;
+ margin-bottom: 1.5em;
+}
+div.tableblock > table {
+ border: 3px solid #527bbd;
+}
+thead, p.table.header {
+ font-weight: bold;
+ color: #527bbd;
+}
+p.table {
+ margin-top: 0;
+}
+/* Because the table frame attribute is overriden by CSS in most browsers. */
+div.tableblock > table[frame="void"] {
+ border-style: none;
+}
+div.tableblock > table[frame="hsides"] {
+ border-left-style: none;
+ border-right-style: none;
+}
+div.tableblock > table[frame="vsides"] {
+ border-top-style: none;
+ border-bottom-style: none;
+}
+
+
+/*
+ * html5 specific
+ *
+ * */
+
+table.tableblock {
+ margin-top: 1.0em;
+ margin-bottom: 1.5em;
+}
+thead, p.tableblock.header {
+ font-weight: bold;
+ color: #527bbd;
+}
+p.tableblock {
+ margin-top: 0;
+}
+table.tableblock {
+ border-width: 3px;
+ border-spacing: 0px;
+ border-style: solid;
+ border-color: #527bbd;
+ border-collapse: collapse;
+}
+th.tableblock, td.tableblock {
+ border-width: 1px;
+ padding: 4px;
+ border-style: solid;
+ border-color: #527bbd;
+}
+
+table.tableblock.frame-topbot {
+ border-left-style: hidden;
+ border-right-style: hidden;
+}
+table.tableblock.frame-sides {
+ border-top-style: hidden;
+ border-bottom-style: hidden;
+}
+table.tableblock.frame-none {
+ border-style: hidden;
+}
+
+th.tableblock.halign-left, td.tableblock.halign-left {
+ text-align: left;
+}
+th.tableblock.halign-center, td.tableblock.halign-center {
+ text-align: center;
+}
+th.tableblock.halign-right, td.tableblock.halign-right {
+ text-align: right;
+}
+
+th.tableblock.valign-top, td.tableblock.valign-top {
+ vertical-align: top;
+}
+th.tableblock.valign-middle, td.tableblock.valign-middle {
+ vertical-align: middle;
+}
+th.tableblock.valign-bottom, td.tableblock.valign-bottom {
+ vertical-align: bottom;
+}
+
+
+/*
+ * manpage specific
+ *
+ * */
+
+body.manpage h1 {
+ padding-top: 0.5em;
+ padding-bottom: 0.5em;
+ border-top: 2px solid silver;
+ border-bottom: 2px solid silver;
+}
+body.manpage h2 {
+ border-style: none;
+}
+body.manpage div.sectionbody {
+ margin-left: 3em;
+}
+
+ at media print {
+ body.manpage div#toc { display: none; }
+}
+
+
+a:link { color:navy; }
+a:visited { color:navy; }
+
+.monospaced, code, pre {
+ font-family: "Courier New", Courier, monospace;
+ font-size: medium; /* inherit; */
+ color: black;
+ padding: 0;
+ margin: 0;
+}
+
+/*
+ background: #f8f8f8;
+ border: 1px solid #dddddd;
+ border-left: 5px solid #f0f0f0;
+ margin-right: 10%;
+*/
+
+div.listingblock > div.content {
+ padding: 0.5em;
+ background: none;
+ border: none;
+ border-left: none;
+ margin-right: none;
+}
+
+</style>
+<script type="text/javascript">
+/*<+'])');
+ // Function that scans the DOM tree for header elements (the DOM2
+ // nodeIterator API would be a better technique but not supported by all
+ // browsers).
+ var iterate = function (el) {
+ for (var i = el.firstChild; i != null; i = i.nextSibling) {
+ if (i.nodeType == 1 /* Node.ELEMENT_NODE */) {
+ var mo = re.exec(i.tagName);
+ if (mo && (i.getAttribute("class") || i.getAttribute("className")) != "float") {
+ result[result.length] = new TocEntry(i, getText(i), mo[1]-1);
+ }
+ iterate(i);
+ }
+ }
+ }
+ iterate(el);
+ return result;
+ }
+
+ var toc = document.getElementById("toc");
+ if (!toc) {
+ return;
+ }
+
+ // Delete existing TOC entries in case we're reloading the TOC.
+ var tocEntriesToRemove = [];
+ var i;
+ for (i = 0; i < toc.childNodes.length; i++) {
+ var entry = toc.childNodes[i];
+ if (entry.nodeName.toLowerCase() == 'div'
+ && entry.getAttribute("class")
+ && entry.getAttribute("class").match(/^toclevel/))
+ tocEntriesToRemove.push(entry);
+ }
+ for (i = 0; i < tocEntriesToRemove.length; i++) {
+ toc.removeChild(tocEntriesToRemove[i]);
+ }
+
+ // Rebuild TOC entries.
+ var entries = tocEntries(document.getElementById("content"), toclevels);
+ for (var i = 0; i < entries.length; ++i) {
+ var entry = entries[i];
+ if (entry.element.id == "")
+ entry.element.id = "_toc_" + i;
+ var a = document.createElement("a");
+ a.href = "#" + entry.element.id;
+ a.appendChild(document.createTextNode(entry.text));
+ var div = document.createElement("div");
+ div.appendChild(a);
+ div.className = "toclevel" + entry.toclevel;
+ toc.appendChild(div);
+ }
+ if (entries.length == 0)
+ toc.parentNode.removeChild(toc);
+},
+
+
+/////////////////////////////////////////////////////////////////////
+// Footnotes generator
+/////////////////////////////////////////////////////////////////////
+
+/* Based on footnote generation code from:
+ * http://www.brandspankingnew.net/archive/2005/07/format_footnote.html
+ */
+
+footnotes: function () {
+ // Delete existing footnote entries in case we're reloading the footnodes.
+ var i;
+ var noteholder = document.getElementById("footnotes");
+ if (!noteholder) {
+ return;
+ }
+ var entriesToRemove = [];
+ for (i = 0; i < noteholder.childNodes.length; i++) {
+ var entry = noteholder.childNodes[i];
+ if (entry.nodeName.toLowerCase() == 'div' && entry.getAttribute("class") == "footnote")
+ entriesToRemove.push(entry);
+ }
+ for (i = 0; i < entriesToRemove.length; i++) {
+ noteholder.removeChild(entriesToRemove[i]);
+ }
+
+ // Rebuild footnote entries.
+ var cont = document.getElementById("content");
+ var spans = cont.getElementsByTagName("span");
+ var refs = {};
+ var n = 0;
+ for (i=0; i<spans.length; i++) {
+ if (spans[i].className == "footnote") {
+ n++;
+ var note = spans[i].getAttribute("data-note");
+ if (!note) {
+ // Use [\s\S] in place of . so multi-line matches work.
+ // Because JavaScript has no s (dotall) regex flag.
+ note = spans[i].innerHTML.match(/\s*\[([\s\S]*)]\s*/)[1];
+ spans[i].innerHTML =
+ "[<a id='_footnoteref_" + n + "' href='#_footnote_" + n +
+ "' title='View footnote' class='footnote'>" + n + "</a>]";
+ spans[i].setAttribute("data-note", note);
+ }
+ noteholder.innerHTML +=
+ "<div class='footnote' id='_footnote_" + n + "'>" +
+ "<a href='#_footnoteref_" + n + "' title='Return to text'>" +
+ n + "</a>. " + note + "</div>";
+ var id =spans[i].getAttribute("id");
+ if (id != null) refs["#"+id] = n;
+ }
+ }
+ if (n == 0)
+ noteholder.parentNode.removeChild(noteholder);
+ else {
+ // Process footnoterefs.
+ for (i=0; i<spans.length; i++) {
+ if (spans[i].className == "footnoteref") {
+ var href = spans[i].getElementsByTagName("a")[0].getAttribute("href");
+ href = href.match(/#.*/)[0]; // Because IE return full URL.
+ n = refs[href];
+ spans[i].innerHTML =
+ "[<a href='#_footnote_" + n +
+ "' title='View footnote' class='footnote'>" + n + "</a>]";
+ }
+ }
+ }
+},
+
+install: function(toclevels) {
+ var timerId;
+
+ function reinstall() {
+ asciidoc.footnotes();
+ if (toclevels) {
+ asciidoc.toc(toclevels);
+ }
+ }
+
+ function reinstallAndRemoveTimer() {
+ clearInterval(timerId);
+ reinstall();
+ }
+
+ timerId = setInterval(reinstall, 500);
+ if (document.addEventListener)
+ document.addEventListener("DOMContentLoaded", reinstallAndRemoveTimer, false);
+ else
+ window.onload = reinstallAndRemoveTimer;
+}
+
+}
+asciidoc.install(2);
+/*]]>*/
+</script>
+</head>
+<body class="article" style="max-width:800px">
+<div id="header">
+<h1>Swift Tutorial for running on localhost</h1>
+<div id="toc">
+ <div id="toctitle">Table of Contents</div>
+ <noscript><p><b>JavaScript must be enabled in your browser to display the table of contents.</b></p></noscript>
+</div>
+</div>
+<div id="content">
+<div id="preamble">
+<div class="sectionbody">
+<div class="paragraph"><p>This tutorial is viewable at:
+<a href="http://swift-lang.org/tutorials/localhost/tutorial.html">http://swift-lang.org/tutorials/localhost/tutorial.html</a></p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_introduction_why_parallel_scripting">Introduction: Why Parallel Scripting?</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>Swift is a simple scripting language for executing many instances of
+ordinary application programs on distributed parallel resources.
+Swift scripts run many copies of ordinary programs concurrently, using
+statements like this:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>file out[];
+foreach param,i in paramList {
+ out[i] = simulate(param);
+}</code></pre>
+</div></div>
+<div class="paragraph"><p>Swift acts like a structured "shell" language. It runs programs
+concurrently as soon as their inputs are available, reducing the need
+for complex parallel programming. Swift expresses your workflow in a
+portable fashion: The same script runs on multicore computers,
+clusters, clouds, grids, and supercomputers.</p></div>
+<div class="paragraph"><p>In this tutorial, you’ll be able to first try a few Swift examples
+on your local machine, to get a sense of the language.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_swift_installation">Swift installation</h2>
+<div class="sectionbody">
+<div class="olist arabic"><ol class="arabic">
+<li>
+<p>
+Download the file from <a href="http://swiftlang.org/packages/swift-0.95-RC6.tar.gz">http://swiftlang.org/packages/swift-0.95-RC6.tar.gz</a>
+</p>
+</li>
+<li>
+<p>
+Extract by running "tar xfz swift-0.95-RC6.tar.gz"
+</p>
+</li>
+<li>
+<p>
+Add to PATH by running "export PATH=$PATH:/path/to/swift-0.95-RC6/bin"
+</p>
+</li>
+</ol></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_swift_tutorial_installation">Swift tutorial installation</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>Run the following commands to extract these tutorial scripts.</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>$ cd $HOME
+$ wget http://swiftlang.org/tutorials/localhost/swift-localhost-tutorial.tar.gz
+$ tar xvfz swift-localhost-tutorial.tar.gz
+$ cd swift-localhost-tutorial</code></pre>
+</div></div>
+<div class="paragraph"><p>Run the tutorial setup script</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>$ source setup.sh # You must run this with "source" !</code></pre>
+</div></div>
+<div class="sect2">
+<h3 id="_to_check_out_the_tutorial_scripts_from_svn">To check out the tutorial scripts from SVN</h3>
+<div class="paragraph"><p>If you later want to get the most recent version of this tutorial from
+the Swift Subversion repository, do:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>$ svn co https://svn.ci.uchicago.edu/svn/vdl2/SwiftTutorials/swift-localhost-tutorial <TODO: Fix link></code></pre>
+</div></div>
+<div class="paragraph"><p>This will create a directory called "swift-localhost-tutorial" which contains all of the
+files used in this tutorial.</p></div>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_simple_science_applications_for_the_workflow_tutorial">Simple "science applications" for the workflow tutorial</h2>
+<div class="sectionbody">
+<div class="paragraph"><p>This tutorial is based on two intentionally trivial example programs,
+<code>simulation.sh</code> and <code>stats.sh</code>, (implemented as bash shell scripts)
+that serve as easy-to-understand proxies for real science
+applications. These "programs" behave as follows.</p></div>
+<div class="sect2">
+<h3 id="_simulate_sh">simulate.sh</h3>
+<div class="paragraph"><p>The simulation.sh script serves as a trivial proxy for any more
+complex scientific simulation application. It generates and prints a
+set of one or more random integers in the range [0-2^62) as controlled
+by its command line arguments, which are:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>$ ./app/simulate.sh --help
+./app/simulate.sh: usage:
+ -b|--bias offset bias: add this integer to all results [0]
+ -B|--biasfile file of integer biases to add to results [none]
+ -l|--log generate a log in stderr if not null [y]
+ -n|--nvalues print this many values per simulation [1]
+ -r|--range range (limit) of generated results [100]
+ -s|--seed use this integer [0..32767] as a seed [none]
+ -S|--seedfile use this file (containing integer seeds [0..32767]) one per line [none]
+ -t|--timesteps number of simulated "timesteps" in seconds (determines runtime) [1]
+ -x|--scale scale the results by this integer [1]
+ -h|-?|?|--help print this help
+$</code></pre>
+</div></div>
+<div class="paragraph"><p>All of thess arguments are optional, with default values indicated above as <code>[n]</code>.</p></div>
+<div class="paragraph"><p>With no arguments, simulate.sh prints 1 number in the range of
+1-100. Otherwise it generates n numbers of the form (R*scale)+bias
+where R is a random integer. By default it logs information about its
+execution environment to stderr. Here’s some examples of its usage:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>$ simulate.sh 2>log
+ 5
+$ head -4 log
+
+Called as: /home/wilde/swift/tut/CIC_2013-08-09/app/simulate.sh:
+Start time: Thu Aug 22 12:40:24 CDT 2013
+Running on node: login01.osgconnect.net
+
+$ simulate.sh -n 4 -r 1000000 2>log
+ 239454
+ 386702
+ 13849
+ 873526
+
+$ simulate.sh -n 3 -r 1000000 -x 100 2>log
+ 6643700
+62182300
+ 5230600
+
+$ simulate.sh -n 2 -r 1000 -x 1000 2>log
+ 565000
+ 636000
+
+$ time simulate.sh -n 2 -r 1000 -x 1000 -t 3 2>log
+ 336000
+ 320000
+real 0m3.012s
+user 0m0.005s
+sys 0m0.006s</code></pre>
+</div></div>
+</div>
+<div class="sect2">
+<h3 id="_stats_sh">stats.sh</h3>
+<div class="paragraph"><p>The stats.sh script serves as a trivial model of an "analysis"
+program. It reads N files each containing M integers and simply prints
+the\ average of all those numbers to stdout. Similarly to simulate.sh
+it logs environmental information to the stderr.</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>$ ls f*
+f1 f2 f3 f4
+
+$ cat f*
+25
+60
+40
+75
+
+$ stats.sh f* 2>log
+50</code></pre>
+</div></div>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_basics_of_the_swift_language_with_local_execution">Basics of the Swift language with local execution</h2>
+<div class="sectionbody">
+<div class="sect2">
+<h3 id="_a_summary_of_swift_in_a_nutshell">A Summary of Swift in a nutshell</h3>
+<div class="ulist"><ul>
+<li>
+<p>
+Swift scripts are text files ending in <code>.swift</code> The <code>swift</code> command
+runs on any host, and executes these scripts. <code>swift</code> is a Java
+application, which you can install almost anywhere. On Linux, just
+unpack the distribution <code>tar</code> file and add its <code>bin/</code> directory to
+your <code>PATH</code>.
+</p>
+</li>
+<li>
+<p>
+Swift scripts run ordinary applications, just like shell scripts
+do. Swift makes it easy to run these applications on parallel and
+remote computers (from laptops to supercomputers). If you can <code>ssh</code> to
+the system, Swift can likely run applications there.
+</p>
+</li>
+<li>
+<p>
+The details of where to run applications and how to get files back
+and forth are described in configuration files separate from your
+program. Swift speaks ssh, PBS, Condor, SLURM, LSF, SGE, Cobalt, and
+Globus to run applications, and scp, http, ftp, and GridFTP to move
+data.
+</p>
+</li>
+<li>
+<p>
+The Swift language has 5 main data types: <code>boolean</code>, <code>int</code>,
+<code>string</code>, <code>float</code>, and <code>file</code>. Collections of these are dynamic,
+sparse arrays of arbitrary dimension and structures of scalars and/or
+arrays defined by the <code>type</code> declaration.
+</p>
+</li>
+<li>
+<p>
+Swift file variables are "mapped" to external files. Swift sends
+files to and from remote systems for you automatically.
+</p>
+</li>
+<li>
+<p>
+Swift variables are "single assignment": once you set them you can’t
+change them (in a given block of code). This makes Swift a natural,
+"parallel data flow" language. This programming model keeps your
+workflow scripts simple and easy to write and understand.
+</p>
+</li>
+<li>
+<p>
+Swift lets you define functions to "wrap" application programs, and
+to cleanly structure more complex scripts. Swift <code>app</code> functions take
+files and parameters as inputs and return files as outputs.
+</p>
+</li>
+<li>
+<p>
+A compact set of built-in functions for string and file
+manipulation, type conversions, high level IO, etc. is provided.
+Swift’s equivalent of <code>printf()</code> is <code>tracef()</code>, with limited and
+slightly different format codes.
+</p>
+</li>
+<li>
+<p>
+Swift’s <code>foreach {}</code> statement is the main parallel workhorse of the
+language, and executes all iterations of the loop concurrently. The
+actual number of parallel tasks executed is based on available
+resources and settable "throttles".
+</p>
+</li>
+<li>
+<p>
+In fact, Swift conceptually executes <strong>all</strong> the statements,
+expressions and function calls in your program in parallel, based on
+data flow. These are similarly throttled based on available resources
+and settings.
+</p>
+</li>
+<li>
+<p>
+Swift also has <code>if</code> and <code>switch</code> statements for conditional
+execution. These are seldom needed in simple workflows but they enable
+very dynamic workflow patterns to be specified.
+</p>
+</li>
+</ul></div>
+<div class="paragraph"><p>We’ll see many of these points in action in the examples below. Lets
+get started!</p></div>
+</div>
+<div class="sect2">
+<h3 id="_part_1_run_a_single_application_under_swift">Part 1: Run a single application under Swift</h3>
+<div class="paragraph"><p>The first swift script, p1.swift, runs simulate.sh to generate a
+single random number. It writes the number to a file.</p></div>
+<div class="imageblock" style="text-align:center;">
+<div class="content">
+<img src="part01.png" alt="p1 workflow" />
+</div>
+</div>
+<div class="listingblock">
+<div class="title">p1.swift</div>
+<div class="content">
+<pre><code>type file;
+
+app (file o) simulation ()
+{
+ simulate stdout=filename(o);
+}
+
+file f <"sim.out">;
+f = simulation();</code></pre>
+</div></div>
+<div class="paragraph"><p>To run this script, run the following command:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>$ cd part01
+$ swift p1.swift
+Swift 0.95 Branch SVN swift-r7903 cog-r3908
+RunID: run001
+Progress: Tue, 03 Jun 2014 15:01:28-0500
+Final status:Tue, 03 Jun 2014 15:01:29-0500 Finished successfully:1
+$ cat sim.out
+ 84
+$ swift p1.swift
+$ cat sim.out
+ 36</code></pre>
+</div></div>
+<div class="paragraph"><p>To cleanup the directory and remove all outputs (including the log
+files and directories that Swift generates), run the cleanup script
+which is located in the tutorial PATH:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>$ cleanup</code></pre>
+</div></div>
+<div class="admonitionblock">
+<table><tr>
+<td class="icon">
+<img src="./images/icons/note.png" alt="Note" />
+</td>
+<td class="content">You’ll also find a Swift configuration files in each <code>partNN</code>
+directory of this tutorial. These specify the environment-specific
+details of where to find application programs and where to run them .
+These files will be explained in more detail in parts 4-6, and can
+be ignored for now.</td>
+</tr></table>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_part_2_running_an_ensemble_of_many_apps_in_parallel_with_a_foreach_loop">Part 2: Running an ensemble of many apps in parallel with a "foreach" loop</h3>
+<div class="paragraph"><p>The <code>p2.swift</code> script introduces the <code>foreach</code> parallel iteration
+construct to run many concurrent simulations.</p></div>
+<div class="imageblock" style="text-align:center;">
+<div class="content">
+<img src="part02.png" alt="part02.png" />
+</div>
+</div>
+<div class="listingblock">
+<div class="title">p2.swift</div>
+<div class="content">
+<pre><code>type file;
+
+app (file o) simulation ()
+{
+ simulate stdout=filename(o);
+}
+
+foreach i in [0:9] {
+ file f <single_file_mapper; file=strcat("output/sim_",i,".out")>;
+ f = simulation();
+}</code></pre>
+</div></div>
+<div class="paragraph"><p>The script also shows an
+example of naming the output files of an ensemble run. In this case, the output files will be named
+<code>output/sim_N.out</code>.</p></div>
+<div class="paragraph"><p>In part 2, we update the swift.properties file.</p></div>
+<div class="paragraph"><p>Instead of using shell script (simulate is a link to simulate.sh), we use the equivalent
+python version (simulate.py). Swift does not need to know anything about the language an
+application is written in. The application can be written in Perl, Python, Java, Fortran,
+or any other language.</p></div>
+<div class="paragraph"><p>The swift.properties file also defines properties that control the parallelism involved
+in the swift execution. By setting the tasksPerWorker to the desired number of parallel
+application invocations it is possible to take advantage of multiple cores in the
+processor. The taskThrottle variable is used to limit number of tasks sent to
+workers for execution. Both tasksPerWorker and taskThrottle are set to 2 in the tutorial
+and could be set to the number of cores available on your machine.</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>site=localhost
+use.provider.staging=true
+execution.retries=2
+
+site.localhost {
+ jobmanager=local
+ filesystem=local
+ initialscore=10000
+ taskWalltime=00:02:00
+ workdir=/tmp/swiftwork
+ tasksPerWorker=2
+ taskThrottle=2
+}
+
+app.localhost.simulate=simulate.py
+app.localhost.stats=stats.py</code></pre>
+</div></div>
+<div class="paragraph"><p>To run the script and view the output:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>$ cd ../part02
+$ swift p2.swift
+$ ls output
+sim_0.out sim_1.out sim_2.out sim_3.out sim_4.out sim_5.out sim_6.out sim_7.out sim_8.out sim_9.out
+$ more output/*
+::::::::::::::
+output/sim_0.out
+::::::::::::::
+ 44
+::::::::::::::
+output/sim_1.out
+::::::::::::::
+ 55
+...
+::::::::::::::
+output/sim_9.out
+::::::::::::::
+ 82</code></pre>
+</div></div>
+</div>
+<div class="sect2">
+<h3 id="_part_3_analyzing_results_of_a_parallel_ensemble">Part 3: Analyzing results of a parallel ensemble</h3>
+<div class="paragraph"><p>After all the parallel simulations in an ensemble run have completed,
+its typically necessary to gather and analyze their results with some
+kind of post-processing analysis program or script. p3.swift
+introduces such a postprocessing step. In this case, the files created
+by all of the parallel runs of <code>simulation.sh</code> will be averaged by by
+the trivial "analysis application" <code>stats.sh</code>:</p></div>
+<div class="imageblock" style="text-align:center;">
+<div class="content">
+<img src="part03.png" alt="part03.png" />
+</div>
+</div>
+<div class="listingblock">
+<div class="title">p3.swift</div>
+<div class="content">
+<pre><code>type file;
+
+app (file o) simulation (int sim_steps, int sim_range, int sim_values)
+{
+ simulate "--timesteps" sim_steps "--range" sim_range "--nvalues" sim_values stdout=filename(o);
+}
+
+app (file o) analyze (file s[])
+{
+ stats filenames(s) stdout=filename(o);
+}
+
+int nsim = toInt(arg("nsim","10"));
+int steps = toInt(arg("steps","1"));
+int range = toInt(arg("range","100"));
+int values = toInt(arg("values","5"));
+
+file sims[];
+
+foreach i in [0:nsim-1] {
+ file simout <single_file_mapper; file=strcat("output/sim_",i,".out")>;
+ simout = simulation(steps,range,values);
+ sims[i] = simout;
+}
+
+file stats<"output/average.out">;
+stats = analyze(sims);</code></pre>
+</div></div>
+<div class="paragraph"><p>To run:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>$ cd part03
+$ swift p3.swift</code></pre>
+</div></div>
+<div class="paragraph"><p>Note that in <code>p3.swift</code> we expose more of the capabilities of the
+<code>simulate.sh</code> application to the <code>simulation()</code> app function:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>app (file o) simulation (int sim_steps, int sim_range, int sim_values)
+{
+ simulate "--timesteps" sim_steps "--range" sim_range "--nvalues" sim_values stdout=filename(o);
+}</code></pre>
+</div></div>
+<div class="paragraph"><p><code>p3.swift</code> also shows how to fetch application-specific values from
+the <code>swift</code> command line in a Swift script using <code>arg()</code> which
+accepts a keyword-style argument and its default value:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>int nsim = toInt(arg("nsim","10"));
+int steps = toInt(arg("steps","1"));
+int range = toInt(arg("range","100"));
+int values = toInt(arg("values","5"));</code></pre>
+</div></div>
+<div class="paragraph"><p>Now we can specify that more runs should be performed and that each should run for more timesteps, and produce more that one value each, within a specified range, using command line arguments placed after the Swift script name in the form <code>-parameterName=value</code>:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>$ swift p3.swift -nsim=3 -steps=10 -values=4 -range=1000000
+Swift 0.95 Branch SVN swift-r7903 cog-r3908
+RunID: run001
+Progress: Tue, 03 Jun 2014 17:06:58-0500
+Progress: Tue, 03 Jun 2014 17:06:59-0500 Submitted:2 Active:1
+Progress: Tue, 03 Jun 2014 17:07:00-0500 Submitted:1 Active:2
+Progress: Tue, 03 Jun 2014 17:07:01-0500 Active:3
+Progress: Tue, 03 Jun 2014 17:07:09-0500 Active:2 Finished successfully:1
+Progress: Tue, 03 Jun 2014 17:07:10-0500 Active:1 Finished successfully:2
+Final status:Tue, 03 Jun 2014 17:07:11-0500 Finished successfully:4
+
+$ ls output/
+average.out sim_0.out sim_1.out sim_2.out
+$ more output/*
+::::::::::::::
+output/average.out
+::::::::::::::
+651368
+::::::::::::::
+output/sim_0.out
+::::::::::::::
+ 735700
+ 886206
+ 997391
+ 982970
+::::::::::::::
+output/sim_1.out
+::::::::::::::
+ 260071
+ 264195
+ 869198
+ 933537
+::::::::::::::
+output/sim_2.out
+::::::::::::::
+ 201806
+ 213540
+ 527576
+ 944233</code></pre>
+</div></div>
+<div class="paragraph"><p>Now try running (<code>-nsim=</code>) 100 simulations of (<code>-steps=</code>) 1 second each:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>$ swift p3.swift -nsim=100 -steps=1
+Swift 0.95 Branch SVN swift-r7903 cog-r3908
+RunID: run002
+Progress: Tue, 03 Jun 2014 17:08:05-0500
+Progress: Tue, 03 Jun 2014 17:08:06-0500 Selecting site:79 Submitted:20 Active:1
+Progress: Tue, 03 Jun 2014 17:08:07-0500 Selecting site:78 Submitted:18 Active:3 Finished successfully:1
+Progress: Tue, 03 Jun 2014 17:08:08-0500 Selecting site:75 Submitted:17 Active:4 Finished successfully:4
+Progress: Tue, 03 Jun 2014 17:08:09-0500 Selecting site:71 Submitted:16 Active:5 Finished successfully:8
+Progress: Tue, 03 Jun 2014 17:08:11-0500 Selecting site:66 Submitted:15 Active:6 Finished successfully:13
+Progress: Tue, 03 Jun 2014 17:08:12-0500 Selecting site:60 Submitted:14 Active:7 Finished successfully:19
+Progress: Tue, 03 Jun 2014 17:08:13-0500 Selecting site:53 Submitted:13 Active:8 Finished successfully:26
+Progress: Tue, 03 Jun 2014 17:08:14-0500 Selecting site:45 Submitted:12 Active:9 Finished successfully:34
+Progress: Tue, 03 Jun 2014 17:08:15-0500 Selecting site:36 Submitted:11 Active:10 Finished successfully:43
+Progress: Tue, 03 Jun 2014 17:08:16-0500 Selecting site:26 Submitted:10 Active:11 Finished successfully:53
+Progress: Tue, 03 Jun 2014 17:08:17-0500 Selecting site:15 Active:21 Finished successfully:64
+Progress: Tue, 03 Jun 2014 17:08:18-0500 Active:16 Finished successfully:84
+Progress: Tue, 03 Jun 2014 17:08:19-0500 Active:11 Finished successfully:89
+Final status:Tue, 03 Jun 2014 17:08:19-0500 Finished successfully:101</code></pre>
+</div></div>
+</div>
+<div class="sect2">
+<h3 id="_part_4_specifying_more_complex_workflow_patterns">Part 4: Specifying more complex workflow patterns</h3>
+<div class="paragraph"><p>p4.swift expands the workflow pattern of p3.swift to add additional
+stages to the workflow. Here, we generate a dynamic seed value that
+will be used by all of the simulations, and for each simulation, we
+run an pre-processing application to generate a unique "bias
+file". This pattern is shown below, followed by the Swift script.</p></div>
+<div class="imageblock" style="text-align:center;">
+<div class="content">
+<img src="part06.png" alt="part06.png" />
+</div>
+</div>
+<div class="listingblock">
+<div class="title">p4.swift</div>
+<div class="content">
+<pre><code>type file;
+
+# app() functions for application programs to be called:
+
+app (file out) genseed (int nseeds, file seed_script)
+{
+ bash "simulate.sh" "-r" 2000000 "-n" nseeds stdout=@out;
+}
+
+app (file out) genbias (int bias_range, int nvalues, file bias_script)
+{
+ bash "simulate.sh" "-r" bias_range "-n" nvalues stdout=@out;
+}
+
+app (file out, file log) simulation (int timesteps, int sim_range,
+ file bias_file, int scale, int sim_count,
+ file sim_script, file seed_file)
+{
+ bash "simulate.sh" "-t" timesteps "-r" sim_range "-B" @bias_file "-x" scale
+ "-n" sim_count "-S" @seed_file stdout=@out stderr=@log;
+}
+
+app (file out, file log) analyze (file s[], file stat_script)
+{
+ bash "stats.sh" filenames(s) stdout=@out stderr=@log;
+}
+
+# Command line arguments
+
+int nsim = toInt(arg("nsim", "10")); # number of simulation programs to run
+int steps = toInt(arg("steps", "1")); # number of timesteps (seconds) per simulation
+int range = toInt(arg("range", "100")); # range of the generated random numbers
+int values = toInt(arg("values", "10")); # number of values generated per simulation
+
+# Main script and data
+
+file simulate_script <"simulate.sh">;
+file stats_script <"stats.sh">;
+file seedfile <"output/seed.dat">; # Dynamically generated bias for simulation ensemble
+
+tracef("\n*** Script parameters: nsim=%i range=%i num values=%i\n\n", nsim, range, values);
+seedfile = genseed(1,simulate_script);
+
+file sims[]; # Array of files to hold each simulation output
+
+foreach i in [0:nsim-1] {
+ file biasfile <single_file_mapper; file=strcat("output/bias_",i,".dat")>;
+ file simout <single_file_mapper; file=strcat("output/sim_",i,".out")>;
+ file simlog <single_file_mapper; file=strcat("output/sim_",i,".log")>;
+ biasfile = genbias(1000, 20, simulate_script);
+ (simout,simlog) = simulation(steps, range, biasfile, 1000000, values, simulate_script, seedfile);
+ sims[i] = simout;
+}
+
+file stats_out<"output/average.out">;
+file stats_log<"output/average.log">;
+(stats_out,stats_log) = analyze(sims, stats_script);</code></pre>
+</div></div>
+<div class="paragraph"><p>Note that the workflow is based on data flow dependencies: each simulation depends on the seed value, calculated in this statement:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>seedfile = genseed(1);</code></pre>
+</div></div>
+<div class="paragraph"><p>and on the bias file, computed and then consumed in these two dependent statements:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code> biasfile = genbias(1000, 20, simulate_script);
+ (simout,simlog) = simulation(steps, range, biasfile, 1000000, values, simulate_script, seedfile);</code></pre>
+</div></div>
+<div class="paragraph"><p>To run:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>$ cd ../part04
+$ swift p4.swift</code></pre>
+</div></div>
+<div class="paragraph"><p>The default parameters result in the following execution log:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>$ swift p4.swift
+Swift 0.95 Branch SVN swift-r7903 cog-r3908
+RunID: run001
+Progress: Thu, 05 Jun 2014 06:01:36-0500
+
+*** Script parameters: nsim=10 range=100 num values=10
+
+Progress: Thu, 05 Jun 2014 06:01:37-0500 Submitted:8 Active:2 Finished successfully:11
+Progress: Thu, 05 Jun 2014 06:01:38-0500 Submitted:6 Active:2 Finished successfully:13
+Progress: Thu, 05 Jun 2014 06:01:39-0500 Submitted:4 Active:2 Finished successfully:15
+Progress: Thu, 05 Jun 2014 06:01:40-0500 Submitted:2 Active:2 Finished successfully:17
+Progress: Thu, 05 Jun 2014 06:01:41-0500 Active:2 Finished successfully:19
+Final status:Thu, 05 Jun 2014 06:01:41-0500 Finished successfully:22</code></pre>
+</div></div>
+<div class="paragraph"><p>which produces the following output:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>$ ls -lrt output
+total 264
+-rw-r--r-- 1 p01532 61532 9 Aug 27 19:17 seed.dat
+-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_9.dat
+-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_8.dat
+-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_7.dat
+-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_6.dat
+-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_5.dat
+-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_4.dat
+-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_3.dat
+-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_2.dat
+-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_1.dat
+-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_0.dat
+-rw-r--r-- 1 p01532 61532 90 Aug 27 19:17 sim_9.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_9.log
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_8.log
+-rw-r--r-- 1 p01532 61532 90 Aug 27 19:17 sim_7.out
+-rw-r--r-- 1 p01532 61532 90 Aug 27 19:17 sim_6.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_6.log
+-rw-r--r-- 1 p01532 61532 90 Aug 27 19:17 sim_5.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_5.log
+-rw-r--r-- 1 p01532 61532 90 Aug 27 19:17 sim_4.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_4.log
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_1.log
+-rw-r--r-- 1 p01532 61532 90 Aug 27 19:18 sim_8.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:18 sim_7.log
+-rw-r--r-- 1 p01532 61532 90 Aug 27 19:18 sim_3.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:18 sim_3.log
+-rw-r--r-- 1 p01532 61532 90 Aug 27 19:18 sim_2.out
+-rw-r--r-- 1 p01532 61532 14898 Aug 27 19:18 sim_2.log
+-rw-r--r-- 1 p01532 61532 90 Aug 27 19:18 sim_1.out
+-rw-r--r-- 1 p01532 61532 90 Aug 27 19:18 sim_0.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:18 sim_0.log
+-rw-r--r-- 1 p01532 61532 9 Aug 27 19:18 average.out
+-rw-r--r-- 1 p01532 61532 14675 Aug 27 19:18 average.log</code></pre>
+</div></div>
+<div class="paragraph"><p>Each sim_N.out file is the sum of its bias file plus newly "simulated" random output scaled by 1,000,000:</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>$ cat output/bias_0.dat
+ 302
+ 489
+ 81
+ 582
+ 664
+ 290
+ 839
+ 258
+ 506
+ 310
+ 293
+ 508
+ 88
+ 261
+ 453
+ 187
+ 26
+ 198
+ 402
+ 555
+
+$ cat output/sim_0.out
+64000302
+38000489
+32000081
+12000582
+46000664
+36000290
+35000839
+22000258
+49000506
+75000310</code></pre>
+</div></div>
+<div class="paragraph"><p>We produce 20 values in each bias file. Simulations of less than that
+number of values ignore the unneeded number, while simualtions of more
+than 20 will use the last bias number for all remoaining values past
+20. As an exercise, adjust the code to produce the same number of
+bias values as is needed for each simulation. As a further exercise,
+modify the script to generate a unique seed value for each simulation,
+which is a common practice in ensemble computations.</p></div>
+</div>
+<div class="sect2">
+<h3 id="_performing_larger_swift_runs">Performing larger Swift runs</h3>
+<div class="paragraph"><p>To test with larger runs, provide suitable command line arguments.
+The example below will run 1000 simulations with each simulation
+taking 5 seconds.</p></div>
+<div class="listingblock">
+<div class="content">
+<pre><code>$ swift p4.swift -steps=5 -nsim=1000</code></pre>
+</div></div>
+</div>
+</div>
+</div>
+</div>
+<div id="footnotes"><hr /></div>
+<div id="footer">
+<div id="footer-text">
+Last updated 2014-06-09 17:28:17 CDT
+</div>
+</div>
+</body>
+</html>
Added: SwiftTutorials/swift-localhost-tutorial/part01/p1.swift
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part01/p1.swift (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part01/p1.swift 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,9 @@
+type file;
+
+app (file o) simulation ()
+{
+ simulate stdout=filename(o);
+}
+
+file f <"sim.out">;
+f = simulation();
Added: SwiftTutorials/swift-localhost-tutorial/part01/run001/apps
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part01/run001/apps (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part01/run001/apps 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1 @@
+* * * INSTALLED INTEL32::LINUX null
Added: SwiftTutorials/swift-localhost-tutorial/part01/run001/cf
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part01/run001/cf (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part01/run001/cf 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,29 @@
+kickstart.enabled=maybe
+replication.limit=3
+clustering.queue.delay=4
+caching.algorithm=LRU
+wrapperlog.always.transfer=false
+throttle.transfers=4
+throttle.host.submit=2
+execution.retries=2
+provenance.log=false
+use.wrapper.staging=false
+pgraph.node.options=color="seagreen", style="filled"
+sites.file=${swift.home}/etc/sites.xml
+throttle.file.operations=8
+clustering.min.time=60
+foreach.max.threads=16384
+pgraph.graph.options=splines="compound", rankdir="TB"
+tracing.enabled=true
+pgraph=false
+sitedir.keep=false
+throttle.score.job.factor=0.2
+provider.staging.pin.swiftfiles=false
+throttle.submit=4
+kickstart.always.transfer=false
+replication.min.queue.time=60
+replication.enabled=false
+tc.file=${swift.home}/etc/tc.data
+use.provider.staging=true
+clustering.enabled=false
+lazy.errors=false
Added: SwiftTutorials/swift-localhost-tutorial/part01/run001/run001.log
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part01/run001/run001.log (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part01/run001/run001.log 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,219 @@
+2014-06-05 17:35:15,062-0500 DEBUG Loader setting: sites.file to: /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/sites.xml
+2014-06-05 17:35:15,063-0500 DEBUG Loader setting: tc.file to: /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/apps
+2014-06-05 17:35:15,063-0500 INFO Loader VERSION Swift 0.95 Branch SVN swift-r7903 cog-r3908
+2014-06-05 17:35:15,063-0500 INFO Loader RUN_ID run001
+2014-06-05 17:35:15,064-0500 INFO Loader ARGUMENTS [-runid, run001, -logfile, /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/run001.log, -sites.file, /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/sites.xml, -tc.file, /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/apps, -config, /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/cf, p1.swift]
+2014-06-05 17:35:15,064-0500 INFO Loader MAX_HEAP 954466304
+2014-06-05 17:35:15,064-0500 INFO Loader GLOBUS_HOSTNAME borgmachine
+2014-06-05 17:35:15,064-0500 INFO Loader CWD /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/.
+2014-06-05 17:35:15,064-0500 DEBUG Loader SWIFT_CONFIGURATION Swift configuration ([/home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/cf]): {pgraph.node.options=color="seagreen", style="filled", clustering.min.time=60, clustering.queue.delay=4, sites.file=/home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/sites.xml, ticker.prefix=Progress: , provenance.log=false, kickstart.enabled=maybe, tc.file=/home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/apps, foreach.max.threads=16384, lazy.errors=false, ticker.date.format=, sitedir.keep=false, throttle.score.job.factor=0.2, kickstart.always.transfer=false, throttle.host.submit=2, replication.limit=3, clustering.enabled=false, pgraph=false, file.gc.enabled=true, replication.min.queue.time=60, wrapper.parameter.mode=args, replication.enabled=false, throttle.transfers=4, wrapperlog.always.transfer=false, throttle.submit=4, pgraph.graph.options=splines="compound", rankdir="TB", cac
hing.algorithm=LRU, use.wrapper.staging=false, provider.staging.pin.swiftfiles=false, tracing.enabled=true, wrapper.invocation.mode=absolute, mapping.checker=on, execution.retries=2, cdm.broadcast.mode=file, status.mode=files, use.provider.staging=true, throttle.file.operations=8}
+2014-06-05 17:35:15,064-0500 INFO Loader SITES_FILE /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/sites.xml
+2014-06-05 17:35:15,065-0500 DEBUG textfiles BEGIN SITES:
+<?xml version="1.0" encoding="UTF-8"?>
+<config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites">
+
+<pool handle="localhost">
+<profile namespace="karajan" key="initialScore">10000</profile>
+<filesystem provider="local"/>
+<workdirectory>/tmp/swiftwork</workdirectory>
+<execution provider="coaster" jobmanager="local:local"/>
+<profile namespace="globus" key="maxwalltime">00:02:00</profile>
+</pool>
+
+</config>
+
+
+2014-06-05 17:35:15,065-0500 DEBUG textfiles END SITES:
+2014-06-05 17:35:15,065-0500 INFO Loader TC_FILE /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/apps
+2014-06-05 17:35:15,065-0500 DEBUG textfiles BEGIN TC:
+* * * INSTALLED INTEL32::LINUX null
+
+
+2014-06-05 17:35:15,065-0500 DEBUG textfiles END TC:
+2014-06-05 17:35:15,065-0500 DEBUG textfiles BEGIN SWIFTSCRIPT:
+type file;
+
+app (file o) simulation ()
+{
+ simulate stdout=filename(o);
+}
+
+file f <"sim.out">;
+f = simulation();
+
+
+2014-06-05 17:35:15,065-0500 DEBUG textfiles END SWIFTSCRIPT:
+2014-06-05 17:35:15,065-0500 INFO Loader p1.swift: source file is new. Recompiling.
+2014-06-05 17:35:15,871-0500 INFO Loader RUN_START
+2014-06-05 17:35:15,873-0500 INFO SwiftExecutor swift.home = /home/yadu/src/swift-0.95/cog/modules/swift/dist/swift-svn
+2014-06-05 17:35:15,880-0500 INFO swift Using sites file: /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/sites.xml
+2014-06-05 17:35:15,898-0500 INFO swift Using tc.data: /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/apps
+2014-06-05 17:35:15,975-0500 INFO AbstractScheduler Setting resources to: {localhost=localhost}
+2014-06-05 17:35:16,002-0500 INFO TRACE VARIABLE, line 8, thread R, f INITIALIZED <SingleFileMapper; , file = file://localhost/sim.out>
+2014-06-05 17:35:16,003-0500 INFO TRACE Variable declaration, line 8, thread R, f <SingleFileMapper; , file = file://localhost/sim.out>
+2014-06-05 17:35:16,010-0500 INFO RuntimeStats$ProgressTicker
+2014-06-05 17:35:16,010-0500 INFO RuntimeStats$ProgressTicker HeapMax: 954466304, CrtHeap: 156893184, UsedHeap: 10912672
+2014-06-05 17:35:16,010-0500 INFO TRACE APPCALL, line 3, thread R, simulation()
+2014-06-05 17:35:16,010-0500 DEBUG swift PROCEDURE line=3 thread=R name=simulation
+2014-06-05 17:35:16,015-0500 DEBUG swift THREAD_ASSOCIATION jobid=simulate-lieirnrl thread=R host=localhost replicationGroup=null
+2014-06-05 17:35:16,016-0500 DEBUG swift JOB_START jobid=simulate-lieirnrl tr=simulate arguments=[] tmpdir=p1-run001/jobs/l/simulate-lieirnrl host=localhost
+2014-06-05 17:35:16,024-0500 INFO GlobalSubmitQueue No global submit throttle set. Using default (1024)
+2014-06-05 17:35:16,049-0500 INFO LocalService Started local service: 127.0.1.1:58430
+2014-06-05 17:35:16,052-0500 INFO BootstrapService Socket bound. URL is http://borgmachine:57712
+2014-06-05 17:35:16,078-0500 INFO Settings Local contacts: [http://128.135.224.51:60251]
+2014-06-05 17:35:16,080-0500 INFO CoasterService Started local service: http://127.0.1.1:60251
+2014-06-05 17:35:16,080-0500 INFO CoasterService Reserving channel for registration
+2014-06-05 17:35:16,091-0500 INFO MetaChannel MetaChannel [context: cpipe, boundTo: null] binding to cpipe://1
+2014-06-05 17:35:16,091-0500 INFO MetaChannel MetaChannel [context: spipe, boundTo: null] binding to spipe://1
+2014-06-05 17:35:16,091-0500 INFO CoasterService Sending registration
+2014-06-05 17:35:16,093-0500 INFO MetaChannel Trying to re-bind current channel
+2014-06-05 17:35:16,094-0500 INFO RequestHandler Handler(tag: 1, REGISTER) unregistering (send)
+2014-06-05 17:35:16,094-0500 INFO CoasterService Registration complete
+2014-06-05 17:35:16,094-0500 INFO CoasterService Started coaster service: http://127.0.1.1:51304
+2014-06-05 17:35:16,095-0500 INFO RequestHandler Handler(tag: 1, CONFIGSERVICE) unregistering (send)
+2014-06-05 17:35:16,103-0500 INFO BlockQueueProcessor Starting... id=0605-3505160
+2014-06-05 17:35:16,103-0500 INFO RequestHandler Handler(tag: 2, SUBMITJOB) unregistering (send)
+2014-06-05 17:35:16,107-0500 INFO BlockQueueProcessor
+Settings {
+ slots = 20
+ jobsPerNode = 1
+ workersPerNode = 1
+ nodeGranularity = 1
+ allocationStepSize = 0.1
+ maxNodes = 2097151
+ lowOverallocation = 10.0
+ highOverallocation = 1.0
+ overallocationDecayFactor = 0.001
+ spread = 0.9
+ reserve = 60.000s
+ maxtime = 2147483647
+ remoteMonitorEnabled = false
+ internalHostname = 128.135.224.51
+ hookClass = null
+ workerManager = block
+ workerLoggingLevel = NONE
+ workerLoggingDirectory = DEFAULT
+ ldLibraryPath = null
+ workerCopies = null
+ directory = null
+ useHashBang = null
+ parallelism = 0.01
+ coresPerNode = 1
+ perfTraceWorker = false
+ perfTraceInterval = -1
+ attributes = {}
+ callbackURIs = [http://128.135.224.51:60251]
+}
+
+2014-06-05 17:35:16,107-0500 INFO BlockQueueProcessor Jobs in holding queue: 1
+2014-06-05 17:35:16,107-0500 INFO BlockQueueProcessor Time estimate for holding queue (seconds): 1
+2014-06-05 17:35:16,108-0500 INFO BlockQueueProcessor Allocating blocks for a total walltime of: 1s
+2014-06-05 17:35:16,108-0500 INFO BlockQueueProcessor Considering: Job(id:0 120.000s)
+2014-06-05 17:35:16,108-0500 INFO BlockQueueProcessor Max Walltime (seconds): 120
+2014-06-05 17:35:16,108-0500 INFO BlockQueueProcessor Time estimate (seconds): 1080
+2014-06-05 17:35:16,108-0500 INFO BlockQueueProcessor Total for this new Block (est. seconds): 0
+2014-06-05 17:35:16,110-0500 INFO BlockQueueProcessor index: 0, last: 0, holding.size(): 1
+2014-06-05 17:35:16,110-0500 INFO BlockQueueProcessor Queued: 1 jobs to new Block
+2014-06-05 17:35:16,110-0500 INFO BlockQueueProcessor index: 0, last: 0, ii: 1, holding.size(): 1
+2014-06-05 17:35:16,110-0500 INFO Block Starting block: workers=1, walltime=1080.000s
+2014-06-05 17:35:16,111-0500 INFO RemoteLogHandler BLOCK_REQUESTED id=0605-3505160-000000, cores=1, coresPerWorker=1, walltime=1080
+2014-06-05 17:35:16,111-0500 INFO RequestHandler Handler(tag: 2, RLOG) unregistering (send)
+2014-06-05 17:35:16,112-0500 INFO BlockTaskSubmitter Queuing block Block 0605-3505160-000000 (1x1080.000s) for submission
+2014-06-05 17:35:16,112-0500 INFO BlockQueueProcessor Added 1 jobs to new blocks
+2014-06-05 17:35:16,112-0500 INFO BlockTaskSubmitter Submitting block Block 0605-3505160-000000 (1x1080.000s)
+2014-06-05 17:35:16,112-0500 INFO ExecutionTaskHandler provider=local
+2014-06-05 17:35:16,114-0500 INFO Block Block task status changed: Submitting
+2014-06-05 17:35:16,114-0500 INFO JobSubmissionTaskHandler Submit: in: / command: /usr/bin/perl /home/yadu/.globus/coasters/cscript2254547756894002938.pl http://128.135.224.51:60251 0605-3505160-000000 NOLOGGING
+2014-06-05 17:35:16,115-0500 INFO Block Block task status changed: Submitted
+2014-06-05 17:35:16,116-0500 INFO Block Block task status changed: Active
+2014-06-05 17:35:16,116-0500 INFO RemoteLogHandler BLOCK_ACTIVE id=0605-3505160-000000
+2014-06-05 17:35:16,116-0500 INFO RequestHandler Handler(tag: 3, RLOG) unregistering (send)
+2014-06-05 17:35:16,174-0500 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
+2014-06-05 17:35:16,177-0500 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 128.135.224.51:41290]
+2014-06-05 17:35:16,177-0500 INFO LocalTCPService Received registration: blockid = 0605-3505160-000000, url = borgmachine
+2014-06-05 17:35:16,178-0500 INFO MetaChannel MetaChannel [context: worker-1, boundTo: null] binding to TCPChannel [type: server, contact: 128.135.224.51:41290]
+2014-06-05 17:35:16,178-0500 INFO RemoteLogHandler WORKER_ACTIVE blockid=0605-3505160-000000
+2014-06-05 17:35:16,179-0500 INFO RequestHandler Handler(tag: 4, RLOG) unregistering (send)
+2014-06-05 17:35:16,180-0500 DEBUG Cpu worker started: block=0605-3505160-000000 host=borgmachine id=0
+2014-06-05 17:35:16,180-0500 DEBUG Cpu ready for work: block=0605-3505160-000000 id=0
+2014-06-05 17:35:16,180-0500 INFO Block Started CPU 0:1402007716s
+2014-06-05 17:35:16,181-0500 INFO Block Started worker 0605-3505160-000000:000000
+2014-06-05 17:35:16,181-0500 DEBUG Cpu requesting work: block=0605-3505160-000000 id=0 Cpus sleeping: 1
+2014-06-05 17:35:16,181-0500 INFO AbstractStreamCoasterChannel Using NIO sender for TCPChannel [type: server, contact: 0605-3505160-000000-000000]
+2014-06-05 17:35:16,181-0500 INFO Cpu 0605-3505160-000000:0 (quality: 1.0) submitting urn:1402007715887-1402007715890-1402007715891: /bin/bash _swiftwrap.staging -e simulate -out sim.out -err stderr.txt -i -d -if -of sim.out -k -cdmfile -status provider -a
+2014-06-05 17:35:16,183-0500 INFO RequestHandler Handler(tag: 5, JOBSTATUS) unregistering (send)
+2014-06-05 17:35:16,187-0500 INFO RequestHandler Handler(tag: 7345, REGISTER) unregistering (send)
+2014-06-05 17:35:16,188-0500 INFO RequestHandler Handler(tag: 6, JOBSTATUS) unregistering (send)
+2014-06-05 17:35:16,188-0500 DEBUG Cpu Stagein workerid=000000
+2014-06-05 17:35:16,188-0500 INFO RequestHandler Handler(tag: 7, JOBSTATUS) unregistering (send)
+2014-06-05 17:35:16,188-0500 INFO RequestHandler Handler(tag: 7346, JOBSTATUS) unregistering (send)
+2014-06-05 17:35:16,189-0500 INFO GetFileHandler Handler(tag: 7347, GET) request complete; src=proxy://u-794c6b3b-1466e2c00f8--7fff-u-794c6b3b-1466e2c00f8--8000S//home/yadu/src/swift-0.95/cog/modules/swift/dist/swift-svn/libexec/_swiftwrap.staging
+2014-06-05 17:35:16,197-0500 INFO GetFileHandler Handler(tag: 7347, GET) reader: PR proxy://u-794c6b3b-1466e2c00f8--7fff-u-794c6b3b-1466e2c00f8--8000S//home/yadu/src/swift-0.95/cog/modules/swift/dist/swift-svn/libexec/_swiftwrap.staging
+2014-06-05 17:35:16,198-0500 INFO GetFileHandler Handler(tag: 7347, GET) -> 8
+2014-06-05 17:35:16,198-0500 INFO GetFileHandler Handler(tag: 8, GET) request complete; src=file://localhost//home/yadu/src/swift-0.95/cog/modules/swift/dist/swift-svn/libexec/_swiftwrap.staging
+2014-06-05 17:35:16,199-0500 INFO GetFileHandler Handler(tag: 8, GET) reader: LR-Handler(tag: 8, GET)
+2014-06-05 17:35:16,199-0500 INFO GetFileHandler Handler(tag: 8, GET) sending length: 6078, 162788727
+2014-06-05 17:35:16,199-0500 INFO GetFileHandler Handler(tag: 7347, GET) sending length: 6078, 95968411
+2014-06-05 17:35:16,199-0500 INFO ReadBuffer RB-LR-Handler(tag: 8, GET) will ask for 1 buffers for a size of 6078
+2014-06-05 17:35:16,200-0500 INFO ReadBuffer RB-LR-Handler(tag: 8, GET) actual allocated buffers 1(0 queued)
+2014-06-05 17:35:16,200-0500 INFO LocalIOProvider LR-Handler(tag: 8, GET) rbuf: RB-LR-Handler(tag: 8, GET)
+2014-06-05 17:35:16,200-0500 INFO GetFileHandler Handler(tag: 8, GET) unregistering (last)
+2014-06-05 17:35:16,200-0500 INFO ReadBuffer RB-LR-Handler(tag: 8, GET) de-allocating 1 buffers
+2014-06-05 17:35:16,200-0500 INFO GetFileHandler Handler(tag: 7347, GET) unregistering (last)
+2014-06-05 17:35:16,202-0500 DEBUG Cpu Active workerid=0605-3505160-000000:000000
+2014-06-05 17:35:16,202-0500 INFO RequestHandler Handler(tag: 7348, JOBSTATUS) unregistering (send)
+2014-06-05 17:35:16,202-0500 INFO RequestHandler Handler(tag: 9, JOBSTATUS) unregistering (send)
+2014-06-05 17:35:16,240-0500 DEBUG Cpu Stageout workerid=000000
+2014-06-05 17:35:16,240-0500 INFO RequestHandler Handler(tag: 7349, JOBSTATUS) unregistering (send)
+2014-06-05 17:35:16,241-0500 INFO RequestHandler Handler(tag: 10, JOBSTATUS) unregistering (send)
+2014-06-05 17:35:16,241-0500 INFO PutFileHandler Handler(tag: 7350, PUT) source: /tmp/swiftwork/p1-run001/jobs/l/simulate-lieirnrl/sim.out
+2014-06-05 17:35:16,242-0500 INFO PutFileHandler Handler(tag: 7350, PUT) destination: proxy://u-794c6b3b-1466e2c00f8--7fff-u-794c6b3b-1466e2c00f8--8000S//home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/./sim.out
+2014-06-05 17:35:16,244-0500 INFO PutFileCommand Sending Command(tag: 11, PUT) (t) on cpipe://1
+2014-06-05 17:35:16,245-0500 INFO PutFileCommand Command(tag: 11, PUT) (t) sending data
+2014-06-05 17:35:16,245-0500 INFO PutFileHandler Handler(tag: 7350, PUT) -> 11
+2014-06-05 17:35:16,245-0500 INFO PutFileHandler Handler(tag: 11, PUT) source: /tmp/swiftwork/p1-run001/jobs/l/simulate-lieirnrl/sim.out
+2014-06-05 17:35:16,245-0500 INFO PutFileHandler Handler(tag: 11, PUT) destination: file://localhost//home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/sim.out
+2014-06-05 17:35:16,246-0500 INFO PutFileHandler Handler(tag: 11, PUT) Transfer done
+2014-06-05 17:35:16,246-0500 INFO RequestHandler Handler(tag: 11, PUT) unregistering (send)
+2014-06-05 17:35:16,246-0500 INFO PutFileHandler Handler(tag: 7350, PUT) Transfer done
+2014-06-05 17:35:16,246-0500 INFO RequestHandler Handler(tag: 7350, PUT) unregistering (send)
+2014-06-05 17:35:16,247-0500 DEBUG Cpu Completed
+2014-06-05 17:35:16,247-0500 INFO Cpu 0605-3505160-000000:0 jobTerminated
+2014-06-05 17:35:16,247-0500 DEBUG Cpu ready for work: block=0605-3505160-000000 id=0
+2014-06-05 17:35:16,247-0500 DEBUG Cpu requesting work: block=0605-3505160-000000 id=0 Cpus sleeping: 1
+2014-06-05 17:35:16,247-0500 INFO RequestHandler Handler(tag: 7351, JOBSTATUS) unregistering (send)
+2014-06-05 17:35:16,248-0500 DEBUG Cpu 0605-3505160-000000:0 sleeping
+2014-06-05 17:35:16,248-0500 INFO LateBindingScheduler jobs queued: 0
+2014-06-05 17:35:16,248-0500 INFO RequestHandler Handler(tag: 12, JOBSTATUS) unregistering (send)
+2014-06-05 17:35:16,248-0500 DEBUG swift JOB_END jobid=simulate-lieirnrl
+2014-06-05 17:35:16,248-0500 INFO TRACE VARIABLE, line 8, thread R, f INITIALIZED <SingleFileMapper; , file = file://localhost/sim.out>
+2014-06-05 17:35:16,248-0500 INFO swift END_SUCCESS thread=R tr=simulate
+2014-06-05 17:35:16,249-0500 DEBUG swift PROCEDURE_END line=3 thread=R name=simulation
+2014-06-05 17:35:16,250-0500 INFO RuntimeStats$ProgressTicker Finished successfully:1
+2014-06-05 17:35:16,250-0500 INFO RuntimeStats$ProgressTicker HeapMax: 954466304, CrtHeap: 156893184, UsedHeap: 29221392
+2014-06-05 17:35:16,259-0500 DEBUG swift Starting cleanups
+2014-06-05 17:35:16,259-0500 INFO swift START cleanups=[]
+2014-06-05 17:35:16,260-0500 DEBUG swift Ending cleanups
+2014-06-05 17:35:16,260-0500 INFO SwiftFunction Cleaning f:file = java.lang.Object at 67d3caf - Closed
+2014-06-05 17:35:16,263-0500 INFO Loader Swift finished with no errors
+2014-06-05 17:35:16,263-0500 INFO ServiceManager Cleaning up...
+2014-06-05 17:35:16,264-0500 INFO ServiceManager Shutting down service at https://127.0.1.1:51304
+2014-06-05 17:35:16,264-0500 INFO RequestHandler Handler(tag: 3, SHUTDOWNSERVICE) unregistering (send)
+2014-06-05 17:35:16,366-0500 INFO RequestHandler Handler(tag: 13, UNREGISTER) unregistering (send)
+2014-06-05 17:35:16,371-0500 INFO BlockQueueProcessor Shutting down blocks
+2014-06-05 17:35:16,371-0500 INFO Block Shutting down block Block 0605-3505160-000000 (1x1080.000s)
+2014-06-05 17:35:16,372-0500 INFO RemoteLogHandler BLOCK_SHUTDOWN id=0605-3505160-000000
+2014-06-05 17:35:16,372-0500 INFO RequestHandler Handler(tag: 14, RLOG) unregistering (send)
+2014-06-05 17:35:16,373-0500 INFO Block Adding short shutdown watchdog: count = 1, workers = 1, now = true
+2014-06-05 17:35:16,373-0500 INFO Block Average utilization: 98.5%
+2014-06-05 17:35:16,374-0500 INFO CoasterService Shutdown sequence completed
+2014-06-05 17:35:16,374-0500 INFO RemoteLogHandler BLOCK_UTILIZATION id=0605-3505160-000000, u=98.5
+2014-06-05 17:35:16,374-0500 INFO RequestHandler Handler(tag: 15, RLOG) unregistering (send)
+2014-06-05 17:35:16,374-0500 INFO Node Node [borgmachine] 0 shut down successfully
+2014-06-05 17:35:16,374-0500 INFO AbstractTCPChannel 0605-3505160-000000-000000: Channel shut down
+2014-06-05 17:35:16,375-0500 INFO MetaChannel MetaChannel [context: worker-1, boundTo: TCPChannel [type: server, contact: 0605-3505160-000000-000000]] binding to NullChannel
+2014-06-05 17:35:16,376-0500 INFO AbstractStreamCoasterChannel Using threaded sender for TCPChannel [type: server, contact: 0605-3505160-000000-000000]
+2014-06-05 17:35:16,376-0500 INFO MetaChannel MetaChannel [context: worker-1, boundTo: TCPChannel [type: server, contact: 0605-3505160-000000-000000]] binding to NullChannel
+2014-06-05 17:35:16,377-0500 INFO AbstractTCPChannel 0605-3505160-000000-000000: Channel terminated
+2014-06-05 17:35:16,377-0500 INFO RemoteLogHandler WORKER_SHUTDOWN blockid=0605-3505160-000000
+2014-06-05 17:35:16,377-0500 INFO RequestHandler Handler(tag: 16, RLOG) unregistering (send)
Added: SwiftTutorials/swift-localhost-tutorial/part01/run001/sites.xml
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part01/run001/sites.xml (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part01/run001/sites.xml 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,12 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites">
+
+<pool handle="localhost">
+<profile namespace="karajan" key="initialScore">10000</profile>
+<filesystem provider="local"/>
+<workdirectory>/tmp/swiftwork</workdirectory>
+<execution provider="coaster" jobmanager="local:local"/>
+<profile namespace="globus" key="maxwalltime">00:02:00</profile>
+</pool>
+
+</config>
Added: SwiftTutorials/swift-localhost-tutorial/part01/run001/swift.out
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part01/run001/swift.out (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part01/run001/swift.out 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,4 @@
+Swift 0.95 Branch SVN swift-r7903 cog-r3908
+RunID: run001
+Progress: Thu, 05 Jun 2014 17:35:16-0500
+Final status:Thu, 05 Jun 2014 17:35:16-0500 Finished successfully:1
Added: SwiftTutorials/swift-localhost-tutorial/part01/run002/apps
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part01/run002/apps (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part01/run002/apps 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1 @@
+* * * INSTALLED INTEL32::LINUX null
Added: SwiftTutorials/swift-localhost-tutorial/part01/run002/cf
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part01/run002/cf (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part01/run002/cf 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,29 @@
+clustering.enabled=false
+replication.min.queue.time=60
+kickstart.enabled=maybe
+pgraph.node.options=color="seagreen", style="filled"
+clustering.queue.delay=4
+pgraph.graph.options=splines="compound", rankdir="TB"
+use.wrapper.staging=false
+replication.limit=3
+clustering.min.time=60
+throttle.submit=4
+replication.enabled=false
+sites.file=${swift.home}/etc/sites.xml
+foreach.max.threads=16384
+tc.file=${swift.home}/etc/tc.data
+kickstart.always.transfer=false
+throttle.file.operations=8
+lazy.errors=false
+caching.algorithm=LRU
+throttle.transfers=4
+throttle.host.submit=2
+throttle.score.job.factor=0.2
+wrapperlog.always.transfer=false
+tracing.enabled=true
+execution.retries=2
+pgraph=false
+provenance.log=false
+provider.staging.pin.swiftfiles=false
+sitedir.keep=false
+use.provider.staging=true
Added: SwiftTutorials/swift-localhost-tutorial/part01/run002/run002.log
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part01/run002/run002.log (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part01/run002/run002.log 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,220 @@
+2014-06-05 17:39:03,925-0500 DEBUG Loader setting: sites.file to: /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/sites.xml
+2014-06-05 17:39:03,928-0500 DEBUG Loader setting: tc.file to: /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/apps
+2014-06-05 17:39:03,928-0500 INFO Loader VERSION Swift 0.95 RC6 swift-r7900 cog-r3908
+2014-06-05 17:39:03,928-0500 INFO Loader RUN_ID run002
+2014-06-05 17:39:03,928-0500 INFO Loader ARGUMENTS [-runid, run002, -logfile, /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/run002.log, -sites.file, /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/sites.xml, -tc.file, /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/apps, -config, /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/cf, p1.swift]
+2014-06-05 17:39:03,928-0500 INFO Loader MAX_HEAP 954466304
+2014-06-05 17:39:03,928-0500 INFO Loader GLOBUS_HOSTNAME borgmachine
+2014-06-05 17:39:03,928-0500 INFO Loader CWD /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/.
+2014-06-05 17:39:03,928-0500 DEBUG Loader SWIFT_CONFIGURATION Swift configuration ([/home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/cf]): {pgraph.node.options=color="seagreen", style="filled", clustering.min.time=60, clustering.queue.delay=4, sites.file=/home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/sites.xml, ticker.prefix=Progress: , provenance.log=false, kickstart.enabled=maybe, tc.file=/home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/apps, foreach.max.threads=16384, lazy.errors=false, ticker.date.format=, sitedir.keep=false, throttle.score.job.factor=0.2, kickstart.always.transfer=false, throttle.host.submit=2, replication.limit=3, clustering.enabled=false, pgraph=false, file.gc.enabled=true, replication.min.queue.time=60, wrapper.parameter.mode=args, replication.enabled=false, throttle.transfers=4, wrapperlog.always.transfer=false, throttle.submit=4, pgraph.graph.options=splines="compound", rankdir="TB", cac
hing.algorithm=LRU, use.wrapper.staging=false, provider.staging.pin.swiftfiles=false, tracing.enabled=true, wrapper.invocation.mode=absolute, mapping.checker=on, execution.retries=2, cdm.broadcast.mode=file, status.mode=files, use.provider.staging=true, throttle.file.operations=8}
+2014-06-05 17:39:03,928-0500 INFO Loader SITES_FILE /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/sites.xml
+2014-06-05 17:39:03,929-0500 DEBUG textfiles BEGIN SITES:
+<?xml version="1.0" encoding="UTF-8"?>
+<config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites">
+
+<pool handle="localhost">
+<workdirectory>/tmp/swiftwork</workdirectory>
+<filesystem provider="local"/>
+<execution provider="coaster" jobmanager="local:local"/>
+<profile namespace="globus" key="maxwalltime">00:02:00</profile>
+<profile namespace="karajan" key="initialScore">10000</profile>
+</pool>
+
+</config>
+
+
+2014-06-05 17:39:03,929-0500 DEBUG textfiles END SITES:
+2014-06-05 17:39:03,929-0500 INFO Loader TC_FILE /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/apps
+2014-06-05 17:39:03,929-0500 DEBUG textfiles BEGIN TC:
+* * * INSTALLED INTEL32::LINUX null
+
+
+2014-06-05 17:39:03,929-0500 DEBUG textfiles END TC:
+2014-06-05 17:39:03,929-0500 DEBUG textfiles BEGIN SWIFTSCRIPT:
+type file;
+
+app (file o) simulation ()
+{
+ simulate stdout=filename(o);
+}
+
+file f <"sim.out">;
+f = simulation();
+
+
+2014-06-05 17:39:03,929-0500 DEBUG textfiles END SWIFTSCRIPT:
+2014-06-05 17:39:03,929-0500 INFO Loader p1.swift: source file is new. Recompiling.
+2014-06-05 17:39:04,806-0500 INFO Loader RUN_START
+2014-06-05 17:39:04,808-0500 INFO SwiftExecutor swift.home = /home/yadu/src/swift-on-cloud/swift-0.95-RC6
+2014-06-05 17:39:04,814-0500 INFO swift Using sites file: /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/sites.xml
+2014-06-05 17:39:04,832-0500 INFO swift Using tc.data: /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/apps
+2014-06-05 17:39:04,906-0500 INFO AbstractScheduler Setting resources to: {localhost=localhost}
+2014-06-05 17:39:04,944-0500 INFO TRACE VARIABLE, line 8, thread R, f INITIALIZED <SingleFileMapper; , file = file://localhost/sim.out>
+2014-06-05 17:39:04,944-0500 INFO TRACE VARIABLE, line 8, thread R, f MAPPING $, file://localhost/sim.out
+2014-06-05 17:39:04,944-0500 INFO TRACE Variable declaration, line 8, thread R, f <SingleFileMapper; , file = file://localhost/sim.out>
+2014-06-05 17:39:04,951-0500 INFO RuntimeStats$ProgressTicker
+2014-06-05 17:39:04,951-0500 INFO RuntimeStats$ProgressTicker HeapMax: 954466304, CrtHeap: 156893184, UsedHeap: 9549896
+2014-06-05 17:39:04,951-0500 INFO TRACE APPCALL, line 3, thread R, simulation()
+2014-06-05 17:39:04,951-0500 DEBUG swift PROCEDURE line=3 thread=R name=simulation
+2014-06-05 17:39:04,957-0500 DEBUG swift THREAD_ASSOCIATION jobid=simulate-2eqnrnrl thread=R host=localhost replicationGroup=null
+2014-06-05 17:39:04,957-0500 DEBUG swift JOB_START jobid=simulate-2eqnrnrl tr=simulate arguments=[] tmpdir=p1-run002/jobs/2/simulate-2eqnrnrl host=localhost
+2014-06-05 17:39:04,965-0500 INFO GlobalSubmitQueue No global submit throttle set. Using default (1024)
+2014-06-05 17:39:04,994-0500 INFO LocalService Started local service: 127.0.1.1:40133
+2014-06-05 17:39:04,998-0500 INFO BootstrapService Socket bound. URL is http://borgmachine:42442
+2014-06-05 17:39:05,022-0500 INFO Settings Local contacts: [http://128.135.224.51:37413]
+2014-06-05 17:39:05,024-0500 INFO CoasterService Started local service: http://127.0.1.1:37413
+2014-06-05 17:39:05,024-0500 INFO CoasterService Reserving channel for registration
+2014-06-05 17:39:05,035-0500 INFO MetaChannel MetaChannel [context: cpipe, boundTo: null] binding to cpipe://1
+2014-06-05 17:39:05,035-0500 INFO MetaChannel MetaChannel [context: spipe, boundTo: null] binding to spipe://1
+2014-06-05 17:39:05,035-0500 INFO CoasterService Sending registration
+2014-06-05 17:39:05,037-0500 INFO MetaChannel Trying to re-bind current channel
+2014-06-05 17:39:05,038-0500 INFO RequestHandler Handler(tag: 1, REGISTER) unregistering (send)
+2014-06-05 17:39:05,038-0500 INFO CoasterService Registration complete
+2014-06-05 17:39:05,038-0500 INFO CoasterService Started coaster service: http://127.0.1.1:45430
+2014-06-05 17:39:05,038-0500 INFO RequestHandler Handler(tag: 1, CONFIGSERVICE) unregistering (send)
+2014-06-05 17:39:05,046-0500 INFO BlockQueueProcessor Starting... id=0605-3905050
+2014-06-05 17:39:05,047-0500 INFO RequestHandler Handler(tag: 2, SUBMITJOB) unregistering (send)
+2014-06-05 17:39:05,051-0500 INFO BlockQueueProcessor
+Settings {
+ slots = 20
+ jobsPerNode = 1
+ workersPerNode = 1
+ nodeGranularity = 1
+ allocationStepSize = 0.1
+ maxNodes = 2097151
+ lowOverallocation = 10.0
+ highOverallocation = 1.0
+ overallocationDecayFactor = 0.001
+ spread = 0.9
+ reserve = 60.000s
+ maxtime = 2147483647
+ remoteMonitorEnabled = false
+ internalHostname = 128.135.224.51
+ hookClass = null
+ workerManager = block
+ workerLoggingLevel = NONE
+ workerLoggingDirectory = DEFAULT
+ ldLibraryPath = null
+ workerCopies = null
+ directory = null
+ useHashBang = null
+ parallelism = 0.01
+ coresPerNode = 1
+ perfTraceWorker = false
+ perfTraceInterval = -1
+ attributes = {}
+ callbackURIs = [http://128.135.224.51:37413]
+}
+
+2014-06-05 17:39:05,051-0500 INFO BlockQueueProcessor Jobs in holding queue: 1
+2014-06-05 17:39:05,051-0500 INFO BlockQueueProcessor Time estimate for holding queue (seconds): 1
+2014-06-05 17:39:05,051-0500 INFO BlockQueueProcessor Allocating blocks for a total walltime of: 1s
+2014-06-05 17:39:05,051-0500 INFO BlockQueueProcessor Considering: Job(id:0 120.000s)
+2014-06-05 17:39:05,051-0500 INFO BlockQueueProcessor Max Walltime (seconds): 120
+2014-06-05 17:39:05,051-0500 INFO BlockQueueProcessor Time estimate (seconds): 1080
+2014-06-05 17:39:05,051-0500 INFO BlockQueueProcessor Total for this new Block (est. seconds): 0
+2014-06-05 17:39:05,053-0500 INFO BlockQueueProcessor index: 0, last: 0, holding.size(): 1
+2014-06-05 17:39:05,053-0500 INFO BlockQueueProcessor Queued: 1 jobs to new Block
+2014-06-05 17:39:05,053-0500 INFO BlockQueueProcessor index: 0, last: 0, ii: 1, holding.size(): 1
+2014-06-05 17:39:05,053-0500 INFO Block Starting block: workers=1, walltime=1080.000s
+2014-06-05 17:39:05,055-0500 INFO RemoteLogHandler BLOCK_REQUESTED id=0605-3905050-000000, cores=1, coresPerWorker=1, walltime=1080
+2014-06-05 17:39:05,055-0500 INFO RequestHandler Handler(tag: 2, RLOG) unregistering (send)
+2014-06-05 17:39:05,055-0500 INFO BlockTaskSubmitter Queuing block Block 0605-3905050-000000 (1x1080.000s) for submission
+2014-06-05 17:39:05,055-0500 INFO BlockQueueProcessor Added 1 jobs to new blocks
+2014-06-05 17:39:05,056-0500 INFO BlockTaskSubmitter Submitting block Block 0605-3905050-000000 (1x1080.000s)
+2014-06-05 17:39:05,056-0500 INFO ExecutionTaskHandler provider=local
+2014-06-05 17:39:05,058-0500 INFO Block Block task status changed: Submitting
+2014-06-05 17:39:05,058-0500 INFO JobSubmissionTaskHandler Submit: in: / command: /usr/bin/perl /home/yadu/.globus/coasters/cscript4102113601101877218.pl http://128.135.224.51:37413 0605-3905050-000000 NOLOGGING
+2014-06-05 17:39:05,059-0500 INFO Block Block task status changed: Submitted
+2014-06-05 17:39:05,060-0500 INFO Block Block task status changed: Active
+2014-06-05 17:39:05,060-0500 INFO RemoteLogHandler BLOCK_ACTIVE id=0605-3905050-000000
+2014-06-05 17:39:05,060-0500 INFO RequestHandler Handler(tag: 3, RLOG) unregistering (send)
+2014-06-05 17:39:05,102-0500 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
+2014-06-05 17:39:05,105-0500 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 128.135.224.51:44286]
+2014-06-05 17:39:05,106-0500 INFO LocalTCPService Received registration: blockid = 0605-3905050-000000, url = borgmachine
+2014-06-05 17:39:05,106-0500 INFO MetaChannel MetaChannel [context: worker-1, boundTo: null] binding to TCPChannel [type: server, contact: 128.135.224.51:44286]
+2014-06-05 17:39:05,107-0500 INFO RemoteLogHandler WORKER_ACTIVE blockid=0605-3905050-000000
+2014-06-05 17:39:05,107-0500 INFO RequestHandler Handler(tag: 4, RLOG) unregistering (send)
+2014-06-05 17:39:05,108-0500 DEBUG Cpu worker started: block=0605-3905050-000000 host=borgmachine id=0
+2014-06-05 17:39:05,108-0500 DEBUG Cpu ready for work: block=0605-3905050-000000 id=0
+2014-06-05 17:39:05,109-0500 INFO Block Started CPU 0:1402007945s
+2014-06-05 17:39:05,109-0500 INFO Block Started worker 0605-3905050-000000:000000
+2014-06-05 17:39:05,109-0500 DEBUG Cpu requesting work: block=0605-3905050-000000 id=0 Cpus sleeping: 1
+2014-06-05 17:39:05,109-0500 INFO AbstractStreamCoasterChannel Using NIO sender for TCPChannel [type: server, contact: 0605-3905050-000000-000000]
+2014-06-05 17:39:05,110-0500 INFO Cpu 0605-3905050-000000:0 (quality: 1.0) submitting urn:1402007944821-1402007944824-1402007944825: /bin/bash _swiftwrap.staging -e simulate -out sim.out -err stderr.txt -i -d -if -of sim.out -k -cdmfile -status provider -a
+2014-06-05 17:39:05,111-0500 INFO RequestHandler Handler(tag: 5, JOBSTATUS) unregistering (send)
+2014-06-05 17:39:05,112-0500 INFO RequestHandler Handler(tag: 2416, REGISTER) unregistering (send)
+2014-06-05 17:39:05,113-0500 INFO RequestHandler Handler(tag: 6, JOBSTATUS) unregistering (send)
+2014-06-05 17:39:05,114-0500 DEBUG Cpu Stagein workerid=000000
+2014-06-05 17:39:05,114-0500 INFO RequestHandler Handler(tag: 7, JOBSTATUS) unregistering (send)
+2014-06-05 17:39:05,114-0500 INFO RequestHandler Handler(tag: 2417, JOBSTATUS) unregistering (send)
+2014-06-05 17:39:05,115-0500 INFO GetFileHandler Handler(tag: 2418, GET) request complete; src=proxy://u-1e4d6cef-1466e2f7f48--7fff-u-1e4d6cef-1466e2f7f48--8000S//home/yadu/src/swift-on-cloud/swift-0.95-RC6/libexec/_swiftwrap.staging
+2014-06-05 17:39:05,123-0500 INFO GetFileHandler Handler(tag: 2418, GET) reader: PR proxy://u-1e4d6cef-1466e2f7f48--7fff-u-1e4d6cef-1466e2f7f48--8000S//home/yadu/src/swift-on-cloud/swift-0.95-RC6/libexec/_swiftwrap.staging
+2014-06-05 17:39:05,123-0500 INFO GetFileHandler Handler(tag: 2418, GET) -> 8
+2014-06-05 17:39:05,123-0500 INFO GetFileHandler Handler(tag: 8, GET) request complete; src=file://localhost//home/yadu/src/swift-on-cloud/swift-0.95-RC6/libexec/_swiftwrap.staging
+2014-06-05 17:39:05,124-0500 INFO GetFileHandler Handler(tag: 8, GET) reader: LR-Handler(tag: 8, GET)
+2014-06-05 17:39:05,124-0500 INFO GetFileHandler Handler(tag: 8, GET) sending length: 6078, 1878503554
+2014-06-05 17:39:05,124-0500 INFO GetFileHandler Handler(tag: 2418, GET) sending length: 6078, 1852137531
+2014-06-05 17:39:05,124-0500 INFO ReadBuffer RB-LR-Handler(tag: 8, GET) will ask for 1 buffers for a size of 6078
+2014-06-05 17:39:05,125-0500 INFO ReadBuffer RB-LR-Handler(tag: 8, GET) actual allocated buffers 1(0 queued)
+2014-06-05 17:39:05,125-0500 INFO LocalIOProvider LR-Handler(tag: 8, GET) rbuf: RB-LR-Handler(tag: 8, GET)
+2014-06-05 17:39:05,125-0500 INFO GetFileHandler Handler(tag: 8, GET) unregistering (last)
+2014-06-05 17:39:05,125-0500 INFO ReadBuffer RB-LR-Handler(tag: 8, GET) de-allocating 1 buffers
+2014-06-05 17:39:05,125-0500 INFO GetFileHandler Handler(tag: 2418, GET) unregistering (last)
+2014-06-05 17:39:05,126-0500 DEBUG Cpu Active workerid=0605-3905050-000000:000000
+2014-06-05 17:39:05,126-0500 INFO RequestHandler Handler(tag: 2419, JOBSTATUS) unregistering (send)
+2014-06-05 17:39:05,126-0500 INFO RequestHandler Handler(tag: 9, JOBSTATUS) unregistering (send)
+2014-06-05 17:39:05,158-0500 DEBUG Cpu Stageout workerid=000000
+2014-06-05 17:39:05,158-0500 INFO RequestHandler Handler(tag: 2420, JOBSTATUS) unregistering (send)
+2014-06-05 17:39:05,158-0500 INFO RequestHandler Handler(tag: 10, JOBSTATUS) unregistering (send)
+2014-06-05 17:39:05,160-0500 INFO PutFileHandler Handler(tag: 2421, PUT) source: /tmp/swiftwork/p1-run002/jobs/2/simulate-2eqnrnrl/sim.out
+2014-06-05 17:39:05,160-0500 INFO PutFileHandler Handler(tag: 2421, PUT) destination: proxy://u-1e4d6cef-1466e2f7f48--7fff-u-1e4d6cef-1466e2f7f48--8000S//home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/./sim.out
+2014-06-05 17:39:05,164-0500 INFO PutFileCommand Sending Command(tag: 11, PUT) (t) on cpipe://1
+2014-06-05 17:39:05,164-0500 INFO PutFileCommand Command(tag: 11, PUT) (t) sending data
+2014-06-05 17:39:05,164-0500 INFO PutFileHandler Handler(tag: 2421, PUT) -> 11
+2014-06-05 17:39:05,164-0500 INFO PutFileHandler Handler(tag: 11, PUT) source: /tmp/swiftwork/p1-run002/jobs/2/simulate-2eqnrnrl/sim.out
+2014-06-05 17:39:05,164-0500 INFO PutFileHandler Handler(tag: 11, PUT) destination: file://localhost//home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/sim.out
+2014-06-05 17:39:05,166-0500 INFO PutFileHandler Handler(tag: 11, PUT) Transfer done
+2014-06-05 17:39:05,166-0500 INFO RequestHandler Handler(tag: 11, PUT) unregistering (send)
+2014-06-05 17:39:05,166-0500 INFO PutFileHandler Handler(tag: 2421, PUT) Transfer done
+2014-06-05 17:39:05,166-0500 INFO RequestHandler Handler(tag: 2421, PUT) unregistering (send)
+2014-06-05 17:39:05,167-0500 DEBUG Cpu Completed
+2014-06-05 17:39:05,167-0500 INFO Cpu 0605-3905050-000000:0 jobTerminated
+2014-06-05 17:39:05,167-0500 DEBUG Cpu ready for work: block=0605-3905050-000000 id=0
+2014-06-05 17:39:05,167-0500 DEBUG Cpu requesting work: block=0605-3905050-000000 id=0 Cpus sleeping: 1
+2014-06-05 17:39:05,167-0500 INFO RequestHandler Handler(tag: 2422, JOBSTATUS) unregistering (send)
+2014-06-05 17:39:05,167-0500 DEBUG Cpu 0605-3905050-000000:0 sleeping
+2014-06-05 17:39:05,167-0500 INFO LateBindingScheduler jobs queued: 0
+2014-06-05 17:39:05,167-0500 INFO RequestHandler Handler(tag: 12, JOBSTATUS) unregistering (send)
+2014-06-05 17:39:05,168-0500 DEBUG swift JOB_END jobid=simulate-2eqnrnrl
+2014-06-05 17:39:05,168-0500 INFO TRACE VARIABLE, line 8, thread R, f INITIALIZED <SingleFileMapper; , file = file://localhost/sim.out>
+2014-06-05 17:39:05,168-0500 INFO swift END_SUCCESS thread=R tr=simulate
+2014-06-05 17:39:05,168-0500 DEBUG swift PROCEDURE_END line=3 thread=R name=simulation
+2014-06-05 17:39:05,169-0500 INFO RuntimeStats$ProgressTicker Finished successfully:1
+2014-06-05 17:39:05,169-0500 INFO RuntimeStats$ProgressTicker HeapMax: 954466304, CrtHeap: 156893184, UsedHeap: 27726864
+2014-06-05 17:39:05,179-0500 DEBUG swift Starting cleanups
+2014-06-05 17:39:05,180-0500 INFO swift START cleanups=[]
+2014-06-05 17:39:05,180-0500 DEBUG swift Ending cleanups
+2014-06-05 17:39:05,180-0500 INFO SwiftFunction Cleaning f:file = java.lang.Object at 61b8ebe3 - Closed
+2014-06-05 17:39:05,182-0500 INFO Loader Swift finished with no errors
+2014-06-05 17:39:05,183-0500 INFO ServiceManager Cleaning up...
+2014-06-05 17:39:05,183-0500 INFO ServiceManager Shutting down service at https://127.0.1.1:45430
+2014-06-05 17:39:05,183-0500 INFO RequestHandler Handler(tag: 3, SHUTDOWNSERVICE) unregistering (send)
+2014-06-05 17:39:05,286-0500 INFO RequestHandler Handler(tag: 13, UNREGISTER) unregistering (send)
+2014-06-05 17:39:05,291-0500 INFO BlockQueueProcessor Shutting down blocks
+2014-06-05 17:39:05,291-0500 INFO Block Shutting down block Block 0605-3905050-000000 (1x1080.000s)
+2014-06-05 17:39:05,292-0500 INFO RemoteLogHandler BLOCK_SHUTDOWN id=0605-3905050-000000
+2014-06-05 17:39:05,292-0500 INFO RequestHandler Handler(tag: 14, RLOG) unregistering (send)
+2014-06-05 17:39:05,292-0500 INFO Block Adding short shutdown watchdog: count = 1, workers = 1, now = true
+2014-06-05 17:39:05,293-0500 INFO Block Average utilization: 98.3%
+2014-06-05 17:39:05,293-0500 INFO CoasterService Shutdown sequence completed
+2014-06-05 17:39:05,293-0500 INFO RemoteLogHandler BLOCK_UTILIZATION id=0605-3905050-000000, u=98.3
+2014-06-05 17:39:05,293-0500 INFO RequestHandler Handler(tag: 15, RLOG) unregistering (send)
+2014-06-05 17:39:05,294-0500 INFO Node Node [borgmachine] 0 shut down successfully
+2014-06-05 17:39:05,294-0500 INFO AbstractTCPChannel 0605-3905050-000000-000000: Channel shut down
+2014-06-05 17:39:05,295-0500 INFO MetaChannel MetaChannel [context: worker-1, boundTo: TCPChannel [type: server, contact: 0605-3905050-000000-000000]] binding to NullChannel
+2014-06-05 17:39:05,295-0500 INFO AbstractStreamCoasterChannel Using threaded sender for TCPChannel [type: server, contact: 0605-3905050-000000-000000]
+2014-06-05 17:39:05,296-0500 INFO MetaChannel MetaChannel [context: worker-1, boundTo: TCPChannel [type: server, contact: 0605-3905050-000000-000000]] binding to NullChannel
+2014-06-05 17:39:05,296-0500 INFO AbstractTCPChannel 0605-3905050-000000-000000: Channel terminated
+2014-06-05 17:39:05,297-0500 INFO RemoteLogHandler WORKER_SHUTDOWN blockid=0605-3905050-000000
+2014-06-05 17:39:05,297-0500 INFO RequestHandler Handler(tag: 16, RLOG) unregistering (send)
Added: SwiftTutorials/swift-localhost-tutorial/part01/run002/sites.xml
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part01/run002/sites.xml (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part01/run002/sites.xml 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,12 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites">
+
+<pool handle="localhost">
+<workdirectory>/tmp/swiftwork</workdirectory>
+<filesystem provider="local"/>
+<execution provider="coaster" jobmanager="local:local"/>
+<profile namespace="globus" key="maxwalltime">00:02:00</profile>
+<profile namespace="karajan" key="initialScore">10000</profile>
+</pool>
+
+</config>
Added: SwiftTutorials/swift-localhost-tutorial/part01/run002/swift.out
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part01/run002/swift.out (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part01/run002/swift.out 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,4 @@
+Swift 0.95 RC6 swift-r7900 cog-r3908
+RunID: run002
+Progress: Thu, 05 Jun 2014 17:39:04-0500
+Final status:Thu, 05 Jun 2014 17:39:05-0500 Finished successfully:1
Added: SwiftTutorials/swift-localhost-tutorial/part01/run003/sites.xml
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part01/run003/sites.xml (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part01/run003/sites.xml 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,12 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites">
+
+<pool handle="localhost">
+<filesystem provider="local"/>
+<profile namespace="karajan" key="initialScore">10000</profile>
+<workdirectory>/tmp/swiftwork</workdirectory>
+<profile namespace="globus" key="maxwalltime">00:02:00</profile>
+<execution provider="coaster" jobmanager="local:local"/>
+</pool>
+
+</config>
Added: SwiftTutorials/swift-localhost-tutorial/part01/run003/swift.out
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part01/run003/swift.out (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part01/run003/swift.out 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,4 @@
+SwiftScript program does not exist: option_siteslist=
+
+For usage information: swift -help
+
Added: SwiftTutorials/swift-localhost-tutorial/part01/run004/sites.xml
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part01/run004/sites.xml (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part01/run004/sites.xml 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,12 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites">
+
+<pool handle="localhost">
+<execution provider="coaster" jobmanager="local:local"/>
+<filesystem provider="local"/>
+<workdirectory>/tmp/swiftwork</workdirectory>
+<profile namespace="karajan" key="initialScore">10000</profile>
+<profile namespace="globus" key="maxwalltime">00:02:00</profile>
+</pool>
+
+</config>
Added: SwiftTutorials/swift-localhost-tutorial/part01/run004/swift.out
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part01/run004/swift.out (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part01/run004/swift.out 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,4 @@
+SwiftScript program does not exist: in
+
+For usage information: swift -help
+
Added: SwiftTutorials/swift-localhost-tutorial/part01/run005/run005.log
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part01/run005/run005.log (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part01/run005/run005.log 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,220 @@
+2014-06-05 19:00:03,520-0500 DEBUG Loader setting: sites.file to: run001/sites.xml
+2014-06-05 19:00:03,522-0500 DEBUG Loader setting: tc.file to: run001/apps
+2014-06-05 19:00:03,522-0500 INFO Loader VERSION Swift 0.95 Branch SVN swift-r7903 cog-r3908
+2014-06-05 19:00:03,522-0500 INFO Loader RUN_ID run005
+2014-06-05 19:00:03,522-0500 INFO Loader ARGUMENTS [-runid, run005, -logfile, /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run005/run005.log, -sites.file, /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run005/sites.xml, -sites.file, run001/sites.xml, -config, run001/cf, -tc.file, run001/apps, p1.swift]
+2014-06-05 19:00:03,522-0500 INFO Loader MAX_HEAP 954466304
+2014-06-05 19:00:03,522-0500 INFO Loader GLOBUS_HOSTNAME borgmachine
+2014-06-05 19:00:03,522-0500 INFO Loader CWD /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/.
+2014-06-05 19:00:03,523-0500 DEBUG Loader SWIFT_CONFIGURATION Swift configuration ([run001/cf]): {pgraph.node.options=color="seagreen", style="filled", clustering.min.time=60, clustering.queue.delay=4, sites.file=run001/sites.xml, ticker.prefix=Progress: , provenance.log=false, kickstart.enabled=maybe, tc.file=run001/apps, foreach.max.threads=16384, lazy.errors=false, ticker.date.format=, sitedir.keep=false, throttle.score.job.factor=0.2, kickstart.always.transfer=false, throttle.host.submit=2, replication.limit=3, clustering.enabled=false, pgraph=false, file.gc.enabled=true, replication.min.queue.time=60, wrapper.parameter.mode=args, replication.enabled=false, throttle.transfers=4, wrapperlog.always.transfer=false, throttle.submit=4, pgraph.graph.options=splines="compound", rankdir="TB", caching.algorithm=LRU, use.wrapper.staging=false, provider.staging.pin.swiftfiles=false, tracing.enabled=true, wrapper.invocation.mode=absolute, mapping.checker=on, execution.retries=2, cdm
.broadcast.mode=file, status.mode=files, use.provider.staging=true, throttle.file.operations=8}
+2014-06-05 19:00:03,523-0500 INFO Loader SITES_FILE run001/sites.xml
+2014-06-05 19:00:03,523-0500 DEBUG textfiles BEGIN SITES:
+<?xml version="1.0" encoding="UTF-8"?>
+<config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites">
+
+<pool handle="localhost">
+<profile namespace="karajan" key="initialScore">10000</profile>
+<filesystem provider="local"/>
+<workdirectory>/tmp/swiftwork</workdirectory>
+<execution provider="coaster" jobmanager="local:local"/>
+<profile namespace="globus" key="maxwalltime">00:02:00</profile>
+</pool>
+
+</config>
+
+
+2014-06-05 19:00:03,523-0500 DEBUG textfiles END SITES:
+2014-06-05 19:00:03,523-0500 INFO Loader TC_FILE run001/apps
+2014-06-05 19:00:03,523-0500 DEBUG textfiles BEGIN TC:
+* * * INSTALLED INTEL32::LINUX null
+
+
+2014-06-05 19:00:03,523-0500 DEBUG textfiles END TC:
+2014-06-05 19:00:03,524-0500 DEBUG textfiles BEGIN SWIFTSCRIPT:
+type file;
+
+app (file o) simulation ()
+{
+ simulate stdout=filename(o);
+}
+
+file f <"sim.out">;
+f = simulation();
+
+
+2014-06-05 19:00:03,524-0500 DEBUG textfiles END SWIFTSCRIPT:
+2014-06-05 19:00:03,524-0500 INFO Loader p1.swift: source file is new. Recompiling.
+2014-06-05 19:00:04,290-0500 INFO Loader RUN_START
+2014-06-05 19:00:04,292-0500 INFO SwiftExecutor swift.home = /home/yadu/src/swift-0.95/cog/modules/swift/dist/swift-svn
+2014-06-05 19:00:04,298-0500 INFO swift Using sites file: run001/sites.xml
+2014-06-05 19:00:04,316-0500 INFO swift Using tc.data: run001/apps
+2014-06-05 19:00:04,394-0500 INFO AbstractScheduler Setting resources to: {localhost=localhost}
+2014-06-05 19:00:04,425-0500 INFO TRACE VARIABLE, line 8, thread R, f INITIALIZED <SingleFileMapper; , file = file://localhost/sim.out>
+2014-06-05 19:00:04,425-0500 INFO TRACE VARIABLE, line 8, thread R, f MAPPING $, file://localhost/sim.out
+2014-06-05 19:00:04,426-0500 INFO TRACE Variable declaration, line 8, thread R, f <SingleFileMapper; , file = file://localhost/sim.out>
+2014-06-05 19:00:04,433-0500 INFO RuntimeStats$ProgressTicker
+2014-06-05 19:00:04,433-0500 INFO TRACE APPCALL, line 3, thread R, simulation()
+2014-06-05 19:00:04,433-0500 INFO RuntimeStats$ProgressTicker HeapMax: 954466304, CrtHeap: 156893184, UsedHeap: 11015880
+2014-06-05 19:00:04,433-0500 DEBUG swift PROCEDURE line=3 thread=R name=simulation
+2014-06-05 19:00:04,439-0500 DEBUG swift THREAD_ASSOCIATION jobid=simulate-2c3xunrl thread=R host=localhost replicationGroup=null
+2014-06-05 19:00:04,440-0500 DEBUG swift JOB_START jobid=simulate-2c3xunrl tr=simulate arguments=[] tmpdir=p1-run005/jobs/2/simulate-2c3xunrl host=localhost
+2014-06-05 19:00:04,448-0500 INFO GlobalSubmitQueue No global submit throttle set. Using default (1024)
+2014-06-05 19:00:04,472-0500 INFO LocalService Started local service: 127.0.1.1:49694
+2014-06-05 19:00:04,476-0500 INFO BootstrapService Socket bound. URL is http://borgmachine:35322
+2014-06-05 19:00:04,501-0500 INFO Settings Local contacts: [http://128.135.224.51:59460]
+2014-06-05 19:00:04,503-0500 INFO CoasterService Started local service: http://127.0.1.1:59460
+2014-06-05 19:00:04,503-0500 INFO CoasterService Reserving channel for registration
+2014-06-05 19:00:04,514-0500 INFO MetaChannel MetaChannel [context: cpipe, boundTo: null] binding to cpipe://1
+2014-06-05 19:00:04,514-0500 INFO MetaChannel MetaChannel [context: spipe, boundTo: null] binding to spipe://1
+2014-06-05 19:00:04,514-0500 INFO CoasterService Sending registration
+2014-06-05 19:00:04,517-0500 INFO MetaChannel Trying to re-bind current channel
+2014-06-05 19:00:04,517-0500 INFO RequestHandler Handler(tag: 1, REGISTER) unregistering (send)
+2014-06-05 19:00:04,518-0500 INFO CoasterService Registration complete
+2014-06-05 19:00:04,518-0500 INFO CoasterService Started coaster service: http://127.0.1.1:43560
+2014-06-05 19:00:04,518-0500 INFO RequestHandler Handler(tag: 1, CONFIGSERVICE) unregistering (send)
+2014-06-05 19:00:04,526-0500 INFO BlockQueueProcessor Starting... id=0605-0007040
+2014-06-05 19:00:04,526-0500 INFO RequestHandler Handler(tag: 2, SUBMITJOB) unregistering (send)
+2014-06-05 19:00:04,531-0500 INFO BlockQueueProcessor
+Settings {
+ slots = 20
+ jobsPerNode = 1
+ workersPerNode = 1
+ nodeGranularity = 1
+ allocationStepSize = 0.1
+ maxNodes = 2097151
+ lowOverallocation = 10.0
+ highOverallocation = 1.0
+ overallocationDecayFactor = 0.001
+ spread = 0.9
+ reserve = 60.000s
+ maxtime = 2147483647
+ remoteMonitorEnabled = false
+ internalHostname = 128.135.224.51
+ hookClass = null
+ workerManager = block
+ workerLoggingLevel = NONE
+ workerLoggingDirectory = DEFAULT
+ ldLibraryPath = null
+ workerCopies = null
+ directory = null
+ useHashBang = null
+ parallelism = 0.01
+ coresPerNode = 1
+ perfTraceWorker = false
+ perfTraceInterval = -1
+ attributes = {}
+ callbackURIs = [http://128.135.224.51:59460]
+}
+
+2014-06-05 19:00:04,531-0500 INFO BlockQueueProcessor Jobs in holding queue: 1
+2014-06-05 19:00:04,531-0500 INFO BlockQueueProcessor Time estimate for holding queue (seconds): 1
+2014-06-05 19:00:04,531-0500 INFO BlockQueueProcessor Allocating blocks for a total walltime of: 1s
+2014-06-05 19:00:04,531-0500 INFO BlockQueueProcessor Considering: Job(id:0 120.000s)
+2014-06-05 19:00:04,531-0500 INFO BlockQueueProcessor Max Walltime (seconds): 120
+2014-06-05 19:00:04,531-0500 INFO BlockQueueProcessor Time estimate (seconds): 1080
+2014-06-05 19:00:04,531-0500 INFO BlockQueueProcessor Total for this new Block (est. seconds): 0
+2014-06-05 19:00:04,533-0500 INFO BlockQueueProcessor index: 0, last: 0, holding.size(): 1
+2014-06-05 19:00:04,533-0500 INFO BlockQueueProcessor Queued: 1 jobs to new Block
+2014-06-05 19:00:04,533-0500 INFO BlockQueueProcessor index: 0, last: 0, ii: 1, holding.size(): 1
+2014-06-05 19:00:04,533-0500 INFO Block Starting block: workers=1, walltime=1080.000s
+2014-06-05 19:00:04,534-0500 INFO RemoteLogHandler BLOCK_REQUESTED id=0605-0007040-000000, cores=1, coresPerWorker=1, walltime=1080
+2014-06-05 19:00:04,534-0500 INFO RequestHandler Handler(tag: 2, RLOG) unregistering (send)
+2014-06-05 19:00:04,535-0500 INFO BlockTaskSubmitter Queuing block Block 0605-0007040-000000 (1x1080.000s) for submission
+2014-06-05 19:00:04,535-0500 INFO BlockQueueProcessor Added 1 jobs to new blocks
+2014-06-05 19:00:04,535-0500 INFO BlockTaskSubmitter Submitting block Block 0605-0007040-000000 (1x1080.000s)
+2014-06-05 19:00:04,535-0500 INFO ExecutionTaskHandler provider=local
+2014-06-05 19:00:04,538-0500 INFO Block Block task status changed: Submitting
+2014-06-05 19:00:04,538-0500 INFO JobSubmissionTaskHandler Submit: in: / command: /usr/bin/perl /home/yadu/.globus/coasters/cscript2332216852508998008.pl http://128.135.224.51:59460 0605-0007040-000000 NOLOGGING
+2014-06-05 19:00:04,539-0500 INFO Block Block task status changed: Submitted
+2014-06-05 19:00:04,542-0500 INFO Block Block task status changed: Active
+2014-06-05 19:00:04,542-0500 INFO RemoteLogHandler BLOCK_ACTIVE id=0605-0007040-000000
+2014-06-05 19:00:04,542-0500 INFO RequestHandler Handler(tag: 3, RLOG) unregistering (send)
+2014-06-05 19:00:04,584-0500 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
+2014-06-05 19:00:04,587-0500 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 128.135.224.51:35162]
+2014-06-05 19:00:04,588-0500 INFO LocalTCPService Received registration: blockid = 0605-0007040-000000, url = borgmachine
+2014-06-05 19:00:04,588-0500 INFO MetaChannel MetaChannel [context: worker-1, boundTo: null] binding to TCPChannel [type: server, contact: 128.135.224.51:35162]
+2014-06-05 19:00:04,589-0500 INFO RemoteLogHandler WORKER_ACTIVE blockid=0605-0007040-000000
+2014-06-05 19:00:04,589-0500 INFO RequestHandler Handler(tag: 4, RLOG) unregistering (send)
+2014-06-05 19:00:04,590-0500 DEBUG Cpu worker started: block=0605-0007040-000000 host=borgmachine id=0
+2014-06-05 19:00:04,590-0500 DEBUG Cpu ready for work: block=0605-0007040-000000 id=0
+2014-06-05 19:00:04,591-0500 INFO Block Started CPU 0:1402012804s
+2014-06-05 19:00:04,591-0500 INFO Block Started worker 0605-0007040-000000:000000
+2014-06-05 19:00:04,591-0500 DEBUG Cpu requesting work: block=0605-0007040-000000 id=0 Cpus sleeping: 1
+2014-06-05 19:00:04,591-0500 INFO AbstractStreamCoasterChannel Using NIO sender for TCPChannel [type: server, contact: 0605-0007040-000000-000000]
+2014-06-05 19:00:04,591-0500 INFO Cpu 0605-0007040-000000:0 (quality: 1.0) submitting urn:1402012804305-1402012804308-1402012804309: /bin/bash _swiftwrap.staging -e simulate -out sim.out -err stderr.txt -i -d -if -of sim.out -k -cdmfile -status provider -a
+2014-06-05 19:00:04,592-0500 INFO RequestHandler Handler(tag: 5, JOBSTATUS) unregistering (send)
+2014-06-05 19:00:04,594-0500 INFO RequestHandler Handler(tag: 5897, REGISTER) unregistering (send)
+2014-06-05 19:00:04,595-0500 INFO RequestHandler Handler(tag: 6, JOBSTATUS) unregistering (send)
+2014-06-05 19:00:04,596-0500 DEBUG Cpu Stagein workerid=000000
+2014-06-05 19:00:04,596-0500 INFO RequestHandler Handler(tag: 7, JOBSTATUS) unregistering (send)
+2014-06-05 19:00:04,596-0500 INFO RequestHandler Handler(tag: 5898, JOBSTATUS) unregistering (send)
+2014-06-05 19:00:04,596-0500 INFO GetFileHandler Handler(tag: 5899, GET) request complete; src=proxy://u-5cab3f23-1466e79a59f--7fff-u-5cab3f23-1466e79a59f--8000S//home/yadu/src/swift-0.95/cog/modules/swift/dist/swift-svn/libexec/_swiftwrap.staging
+2014-06-05 19:00:04,605-0500 INFO GetFileHandler Handler(tag: 5899, GET) reader: PR proxy://u-5cab3f23-1466e79a59f--7fff-u-5cab3f23-1466e79a59f--8000S//home/yadu/src/swift-0.95/cog/modules/swift/dist/swift-svn/libexec/_swiftwrap.staging
+2014-06-05 19:00:04,605-0500 INFO GetFileHandler Handler(tag: 5899, GET) -> 8
+2014-06-05 19:00:04,605-0500 INFO GetFileHandler Handler(tag: 8, GET) request complete; src=file://localhost//home/yadu/src/swift-0.95/cog/modules/swift/dist/swift-svn/libexec/_swiftwrap.staging
+2014-06-05 19:00:04,606-0500 INFO GetFileHandler Handler(tag: 8, GET) reader: LR-Handler(tag: 8, GET)
+2014-06-05 19:00:04,606-0500 INFO GetFileHandler Handler(tag: 8, GET) sending length: 6078, 1878503554
+2014-06-05 19:00:04,606-0500 INFO GetFileHandler Handler(tag: 5899, GET) sending length: 6078, 1852137531
+2014-06-05 19:00:04,607-0500 INFO ReadBuffer RB-LR-Handler(tag: 8, GET) will ask for 1 buffers for a size of 6078
+2014-06-05 19:00:04,608-0500 INFO ReadBuffer RB-LR-Handler(tag: 8, GET) actual allocated buffers 1(0 queued)
+2014-06-05 19:00:04,608-0500 INFO LocalIOProvider LR-Handler(tag: 8, GET) rbuf: RB-LR-Handler(tag: 8, GET)
+2014-06-05 19:00:04,609-0500 INFO GetFileHandler Handler(tag: 8, GET) unregistering (last)
+2014-06-05 19:00:04,609-0500 INFO ReadBuffer RB-LR-Handler(tag: 8, GET) de-allocating 1 buffers
+2014-06-05 19:00:04,609-0500 INFO GetFileHandler Handler(tag: 5899, GET) unregistering (last)
+2014-06-05 19:00:04,612-0500 DEBUG Cpu Active workerid=0605-0007040-000000:000000
+2014-06-05 19:00:04,612-0500 INFO RequestHandler Handler(tag: 5900, JOBSTATUS) unregistering (send)
+2014-06-05 19:00:04,612-0500 INFO RequestHandler Handler(tag: 9, JOBSTATUS) unregistering (send)
+2014-06-05 19:00:04,645-0500 DEBUG Cpu Stageout workerid=000000
+2014-06-05 19:00:04,645-0500 INFO RequestHandler Handler(tag: 5901, JOBSTATUS) unregistering (send)
+2014-06-05 19:00:04,645-0500 INFO RequestHandler Handler(tag: 10, JOBSTATUS) unregistering (send)
+2014-06-05 19:00:04,646-0500 INFO PutFileHandler Handler(tag: 5902, PUT) source: /tmp/swiftwork/p1-run005/jobs/2/simulate-2c3xunrl/sim.out
+2014-06-05 19:00:04,646-0500 INFO PutFileHandler Handler(tag: 5902, PUT) destination: proxy://u-5cab3f23-1466e79a59f--7fff-u-5cab3f23-1466e79a59f--8000S//home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/./sim.out
+2014-06-05 19:00:04,649-0500 INFO PutFileCommand Sending Command(tag: 11, PUT) (t) on cpipe://1
+2014-06-05 19:00:04,649-0500 INFO PutFileCommand Command(tag: 11, PUT) (t) sending data
+2014-06-05 19:00:04,649-0500 INFO PutFileHandler Handler(tag: 11, PUT) source: /tmp/swiftwork/p1-run005/jobs/2/simulate-2c3xunrl/sim.out
+2014-06-05 19:00:04,649-0500 INFO PutFileHandler Handler(tag: 5902, PUT) -> 11
+2014-06-05 19:00:04,649-0500 INFO PutFileHandler Handler(tag: 11, PUT) destination: file://localhost//home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/sim.out
+2014-06-05 19:00:04,651-0500 INFO PutFileHandler Handler(tag: 11, PUT) Transfer done
+2014-06-05 19:00:04,651-0500 INFO RequestHandler Handler(tag: 11, PUT) unregistering (send)
+2014-06-05 19:00:04,651-0500 INFO PutFileHandler Handler(tag: 5902, PUT) Transfer done
+2014-06-05 19:00:04,651-0500 INFO RequestHandler Handler(tag: 5902, PUT) unregistering (send)
+2014-06-05 19:00:04,652-0500 DEBUG Cpu Completed
+2014-06-05 19:00:04,652-0500 INFO Cpu 0605-0007040-000000:0 jobTerminated
+2014-06-05 19:00:04,652-0500 DEBUG Cpu ready for work: block=0605-0007040-000000 id=0
+2014-06-05 19:00:04,652-0500 DEBUG Cpu requesting work: block=0605-0007040-000000 id=0 Cpus sleeping: 1
+2014-06-05 19:00:04,652-0500 INFO LateBindingScheduler jobs queued: 0
+2014-06-05 19:00:04,652-0500 INFO RequestHandler Handler(tag: 5903, JOBSTATUS) unregistering (send)
+2014-06-05 19:00:04,652-0500 DEBUG Cpu 0605-0007040-000000:0 sleeping
+2014-06-05 19:00:04,653-0500 INFO RequestHandler Handler(tag: 12, JOBSTATUS) unregistering (send)
+2014-06-05 19:00:04,653-0500 DEBUG swift JOB_END jobid=simulate-2c3xunrl
+2014-06-05 19:00:04,653-0500 INFO TRACE VARIABLE, line 8, thread R, f INITIALIZED <SingleFileMapper; , file = file://localhost/sim.out>
+2014-06-05 19:00:04,653-0500 INFO swift END_SUCCESS thread=R tr=simulate
+2014-06-05 19:00:04,654-0500 DEBUG swift PROCEDURE_END line=3 thread=R name=simulation
+2014-06-05 19:00:04,655-0500 INFO RuntimeStats$ProgressTicker Finished successfully:1
+2014-06-05 19:00:04,655-0500 INFO RuntimeStats$ProgressTicker HeapMax: 954466304, CrtHeap: 156893184, UsedHeap: 29329432
+2014-06-05 19:00:04,664-0500 DEBUG swift Starting cleanups
+2014-06-05 19:00:04,665-0500 INFO swift START cleanups=[]
+2014-06-05 19:00:04,665-0500 DEBUG swift Ending cleanups
+2014-06-05 19:00:04,665-0500 INFO SwiftFunction Cleaning f:file = java.lang.Object at 61b8ebe3 - Closed
+2014-06-05 19:00:04,667-0500 INFO Loader Swift finished with no errors
+2014-06-05 19:00:04,667-0500 INFO ServiceManager Cleaning up...
+2014-06-05 19:00:04,667-0500 INFO ServiceManager Shutting down service at https://127.0.1.1:43560
+2014-06-05 19:00:04,668-0500 INFO RequestHandler Handler(tag: 3, SHUTDOWNSERVICE) unregistering (send)
+2014-06-05 19:00:04,769-0500 INFO RequestHandler Handler(tag: 13, UNREGISTER) unregistering (send)
+2014-06-05 19:00:04,770-0500 INFO BlockQueueProcessor Shutting down blocks
+2014-06-05 19:00:04,770-0500 INFO Block Shutting down block Block 0605-0007040-000000 (1x1080.000s)
+2014-06-05 19:00:04,771-0500 INFO RemoteLogHandler BLOCK_SHUTDOWN id=0605-0007040-000000
+2014-06-05 19:00:04,771-0500 INFO RequestHandler Handler(tag: 14, RLOG) unregistering (send)
+2014-06-05 19:00:04,771-0500 INFO Block Adding short shutdown watchdog: count = 1, workers = 1, now = true
+2014-06-05 19:00:04,771-0500 INFO Block Average utilization: 98.38%
+2014-06-05 19:00:04,771-0500 INFO CoasterService Shutdown sequence completed
+2014-06-05 19:00:04,771-0500 INFO RemoteLogHandler BLOCK_UTILIZATION id=0605-0007040-000000, u=98.38
+2014-06-05 19:00:04,771-0500 INFO RequestHandler Handler(tag: 15, RLOG) unregistering (send)
+2014-06-05 19:00:04,771-0500 INFO Node Node [borgmachine] 0 shut down successfully
+2014-06-05 19:00:04,771-0500 INFO AbstractTCPChannel 0605-0007040-000000-000000: Channel shut down
+2014-06-05 19:00:04,772-0500 INFO MetaChannel MetaChannel [context: worker-1, boundTo: TCPChannel [type: server, contact: 0605-0007040-000000-000000]] binding to NullChannel
+2014-06-05 19:00:04,772-0500 INFO AbstractStreamCoasterChannel Using threaded sender for TCPChannel [type: server, contact: 0605-0007040-000000-000000]
+2014-06-05 19:00:04,772-0500 INFO MetaChannel MetaChannel [context: worker-1, boundTo: TCPChannel [type: server, contact: 0605-0007040-000000-000000]] binding to NullChannel
+2014-06-05 19:00:04,772-0500 INFO AbstractTCPChannel 0605-0007040-000000-000000: Channel terminated
+2014-06-05 19:00:04,772-0500 INFO RemoteLogHandler WORKER_SHUTDOWN blockid=0605-0007040-000000
+2014-06-05 19:00:04,772-0500 INFO RequestHandler Handler(tag: 16, RLOG) unregistering (send)
Added: SwiftTutorials/swift-localhost-tutorial/part01/run005/sites.xml
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part01/run005/sites.xml (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part01/run005/sites.xml 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,12 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites">
+
+<pool handle="localhost">
+<workdirectory>/tmp/swiftwork</workdirectory>
+<profile namespace="globus" key="maxwalltime">00:02:00</profile>
+<profile namespace="karajan" key="initialScore">10000</profile>
+<filesystem provider="local"/>
+<execution provider="coaster" jobmanager="local:local"/>
+</pool>
+
+</config>
Added: SwiftTutorials/swift-localhost-tutorial/part01/run005/swift.out
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part01/run005/swift.out (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part01/run005/swift.out 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,4 @@
+Swift 0.95 Branch SVN swift-r7903 cog-r3908
+RunID: run005
+Progress: Thu, 05 Jun 2014 19:00:04-0500
+Final status:Thu, 05 Jun 2014 19:00:04-0500 Finished successfully:1
Added: SwiftTutorials/swift-localhost-tutorial/part01/sim.out
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part01/sim.out (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part01/sim.out 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1 @@
+ 93
Added: SwiftTutorials/swift-localhost-tutorial/part01/swift.properties
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part01/swift.properties (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part01/swift.properties 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,11 @@
+site=localhost
+use.provider.staging=true
+execution.retries=2
+
+site.localhost {
+ jobmanager=local
+ filesystem=local
+ initialscore=10000
+ taskWalltime=00:02:00
+ workdir=/tmp/swiftwork
+}
Added: SwiftTutorials/swift-localhost-tutorial/part02/p2.swift
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part02/p2.swift (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part02/p2.swift 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,11 @@
+type file;
+
+app (file o) simulation ()
+{
+ simulate stdout=filename(o);
+}
+
+foreach i in [0:9] {
+ file f <single_file_mapper; file=strcat("output/sim_",i,".out")>;
+ f = simulation();
+}
Added: SwiftTutorials/swift-localhost-tutorial/part02/swift.properties
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part02/swift.properties (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part02/swift.properties 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,16 @@
+site=localhost
+use.provider.staging=true
+execution.retries=2
+
+site.localhost {
+ jobmanager=local
+ filesystem=local
+ initialscore=10000
+ taskWalltime=00:02:00
+ workdir=/tmp/swiftwork
+ tasksPerWorker=2
+ taskThrottle=2
+}
+
+app.localhost.simulate=simulate.py
+app.localhost.stats=stats.py
\ No newline at end of file
Added: SwiftTutorials/swift-localhost-tutorial/part03/p3.swift
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part03/p3.swift (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part03/p3.swift 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,27 @@
+type file;
+
+app (file o) simulation (int sim_steps, int sim_range, int sim_values)
+{
+ simulate "--timesteps" sim_steps "--range" sim_range "--nvalues" sim_values stdout=filename(o);
+}
+
+app (file o) analyze (file s[])
+{
+ stats filenames(s) stdout=filename(o);
+}
+
+int nsim = toInt(arg("nsim","10"));
+int steps = toInt(arg("steps","1"));
+int range = toInt(arg("range","100"));
+int values = toInt(arg("values","5"));
+
+file sims[];
+
+foreach i in [0:nsim-1] {
+ file simout <single_file_mapper; file=strcat("output/sim_",i,".out")>;
+ simout = simulation(steps,range,values);
+ sims[i] = simout;
+}
+
+file stats<"output/average.out">;
+stats = analyze(sims);
Added: SwiftTutorials/swift-localhost-tutorial/part03/swift.properties
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part03/swift.properties (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part03/swift.properties 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,13 @@
+site=localhost
+use.provider.staging=true
+execution.retries=2
+
+site.localhost {
+ jobmanager=local
+ filesystem=local
+ initialscore=10000
+ taskWalltime=00:02:00
+ workdir=/tmp/swiftwork
+ tasksPerWorker=2
+ taskThrottle=2
+}
Added: SwiftTutorials/swift-localhost-tutorial/part04/p4.swift
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part04/p4.swift (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part04/p4.swift 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,57 @@
+type file;
+
+# app() functions for application programs to be called:
+
+app (file out) genseed (int nseeds, file seed_script)
+{
+ bash "simulate.sh" "-r" 2000000 "-n" nseeds stdout=@out;
+}
+
+app (file out) genbias (int bias_range, int nvalues, file bias_script)
+{
+ bash "simulate.sh" "-r" bias_range "-n" nvalues stdout=@out;
+}
+
+app (file out, file log) simulation (int timesteps, int sim_range,
+ file bias_file, int scale, int sim_count,
+ file sim_script, file seed_file)
+{
+ bash "simulate.sh" "-t" timesteps "-r" sim_range "-B" @bias_file "-x" scale
+ "-n" sim_count "-S" @seed_file stdout=@out stderr=@log;
+}
+
+app (file out, file log) analyze (file s[], file stat_script)
+{
+ bash "stats.sh" filenames(s) stdout=@out stderr=@log;
+}
+
+# Command line arguments
+
+int nsim = toInt(arg("nsim", "10")); # number of simulation programs to run
+int steps = toInt(arg("steps", "1")); # number of timesteps (seconds) per simulation
+int range = toInt(arg("range", "100")); # range of the generated random numbers
+int values = toInt(arg("values", "10")); # number of values generated per simulation
+
+# Main script and data
+
+file simulate_script <"simulate.sh">;
+file stats_script <"stats.sh">;
+file seedfile <"output/seed.dat">; # Dynamically generated bias for simulation ensemble
+
+tracef("\n*** Script parameters: nsim=%i range=%i num values=%i\n\n", nsim, range, values);
+seedfile = genseed(1,simulate_script);
+
+file sims[]; # Array of files to hold each simulation output
+
+foreach i in [0:nsim-1] {
+ file biasfile <single_file_mapper; file=strcat("output/bias_",i,".dat")>;
+ file simout <single_file_mapper; file=strcat("output/sim_",i,".out")>;
+ file simlog <single_file_mapper; file=strcat("output/sim_",i,".log")>;
+ biasfile = genbias(1000, 20, simulate_script);
+ (simout,simlog) = simulation(steps, range, biasfile, 1000000, values, simulate_script, seedfile);
+ sims[i] = simout;
+}
+
+file stats_out<"output/average.out">;
+file stats_log<"output/average.log">;
+(stats_out,stats_log) = analyze(sims, stats_script);
Added: SwiftTutorials/swift-localhost-tutorial/part04/simulate.sh
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part04/simulate.sh (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part04/simulate.sh 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1 @@
+link ../app/simulate
\ No newline at end of file
Property changes on: SwiftTutorials/swift-localhost-tutorial/part04/simulate.sh
___________________________________________________________________
Added: svn:special
+ *
Added: SwiftTutorials/swift-localhost-tutorial/part04/stats.sh
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part04/stats.sh (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part04/stats.sh 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1 @@
+link ../app/stats
\ No newline at end of file
Property changes on: SwiftTutorials/swift-localhost-tutorial/part04/stats.sh
___________________________________________________________________
Added: svn:special
+ *
Added: SwiftTutorials/swift-localhost-tutorial/part04/swift.properties
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/part04/swift.properties (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/part04/swift.properties 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,13 @@
+site=localhost
+use.provider.staging=true
+execution.retries=2
+
+site.localhost {
+ jobmanager=local
+ filesystem=local
+ initialscore=10000
+ taskWalltime=00:02:00
+ workdir=/tmp/swiftwork
+ tasksPerWorker=40
+ taskThrottle=40
+}
Added: SwiftTutorials/swift-localhost-tutorial/setup.sh
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/setup.sh (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/setup.sh 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,34 @@
+# ensure that this script is being sourced
+if [ ${BASH_VERSINFO[0]} -gt 2 -a "${BASH_SOURCE[0]}" = "${0}" ] ; then
+ echo ERROR: script ${BASH_SOURCE[0]} must be executed as: source ${BASH_SOURCE[0]}
+ exit 1
+fi
+
+if [ -f $HOME/.swift/swift.properties ]; then
+ echo "WARNING: Found swift.properties config file in $HOME/.swift/swift.properties"
+ echo "Recommend removing the $HOME/.swift/swift.properties as older config file may"
+ echo "result in unknown behavior"
+fi
+
+
+# Setting scripts folder to the PATH env var.
+TUTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+if [ _$(which cleanup 2>/dev/null) != _$TUTDIR/bin/cleanup ]; then
+ echo Adding $TUTDIR/bin:$TUTDIR/app: to front of PATH
+ PATH=$TUTDIR/bin:$TUTDIR/app:$PATH
+else
+ echo Assuming $TUTDIR/bin:$TUTDIR/app: is already at front of PATH
+fi
+
+if [[ $HOSTNAME == headnode* ]]
+then
+ export JAVA=/usr/local/bin/jdk1.7.0_51/bin
+ export SWIFT=/usr/local/bin/swift-0.95/bin
+ export PATH=$JAVA:$SWIFT:$PATH
+fi
+
+echo Swift version is $(swift -version)
+
+return
+
Added: SwiftTutorials/swift-localhost-tutorial/test_all.sh
===================================================================
--- SwiftTutorials/swift-localhost-tutorial/test_all.sh (rev 0)
+++ SwiftTutorials/swift-localhost-tutorial/test_all.sh 2014-06-09 22:37:36 UTC (rev 7910)
@@ -0,0 +1,16 @@
+#!/bin/bash
+
+for i in $(seq 1 1 5)
+do
+ pushd .
+ echo "============================TESTING part0$i==========================="
+ cd part0$i
+ swift p$i.swift
+ if [[ $? == 0 ]]
+ then
+ echo "Cleaning up!"
+ cleanup
+ fi
+ echo -e "\n\n"
+ popd
+done
Property changes on: SwiftTutorials/swift-localhost-tutorial/test_all.sh
___________________________________________________________________
Added: svn:executable
+ *
More information about the Swift-commit
mailing list