#where dir is a tutorial script directory
e.g.
- ./addcloud.sh ../p01 #will add cloud resources to p01 sites pool
+ ./addcloud.sh ../p01 #will create a new site pool "both.xml"
- Finally, to clean up the log files, kill agent and shutdown the coaster service:
./cleanme
Modified: SwiftTutorials/ATPESC_2013-08-06/cloud/addcloud.sh
===================================================================
--- SwiftTutorials/ATPESC_2013-08-06/cloud/addcloud.sh 2013-08-06 20:29:27 UTC (rev 6794)
+++ SwiftTutorials/ATPESC_2013-08-06/cloud/addcloud.sh 2013-08-06 21:16:16 UTC (rev 6795)
@@ -17,6 +17,11 @@
cloudpool=$(cat poolentry.xml)
-sed -i "/<\/config>/i\
-$(echo $cloudpool | sed -e 's/\\/\\\\/g' -e 's/\//\\\//g')" $1/sites.xml
+#escape some characters and add pool entry to sites file
+sed "/<\/config>/i\
+$(echo $cloudpool | sed -e 's/\\/\\\\/g' -e 's/\//\\\//g')" $1/sites.xml > $1/both.xml
+echo "Done."
+echo "To run with the new setup, cd into $1 and use '-sites.file both.xml' option with Swift commandline"
+echo "eg. swift .swift> -sites.file both.xml"
+
Modified: SwiftTutorials/ATPESC_2013-08-06/cloud/hosts.txt
===================================================================
--- SwiftTutorials/ATPESC_2013-08-06/cloud/hosts.txt 2013-08-06 20:29:27 UTC (rev 6794)
+++ SwiftTutorials/ATPESC_2013-08-06/cloud/hosts.txt 2013-08-06 21:16:16 UTC (rev 6795)
@@ -1,3 +1,3 @@
-54.214.98.22
-54.212.124.129
-54.218.221.157
+54.212.170.216
+54.244.171.51
+54.212.140.138
Modified: SwiftTutorials/ATPESC_2013-08-06/cloud/sites.xml
===================================================================
--- SwiftTutorials/ATPESC_2013-08-06/cloud/sites.xml 2013-08-06 20:29:27 UTC (rev 6794)
+++ SwiftTutorials/ATPESC_2013-08-06/cloud/sites.xml 2013-08-06 21:16:16 UTC (rev 6795)
@@ -1,7 +1,7 @@
passive
8
Modified: SwiftTutorials/ATPESC_2013-08-06/cloud/tc.data
===================================================================
--- SwiftTutorials/ATPESC_2013-08-06/cloud/tc.data 2013-08-06 20:29:27 UTC (rev 6794)
+++ SwiftTutorials/ATPESC_2013-08-06/cloud/tc.data 2013-08-06 21:16:16 UTC (rev 6795)
@@ -85,3 +85,9 @@
persistent-coasters cat /bin/cat null null null
persistent-coasters bash /bin/bash null null null
persistent-coasters echo /bin/echo null null null
+persistent-coasters cat /bin/cat null null null
+persistent-coasters bash /bin/bash null null null
+persistent-coasters echo /bin/echo null null null
+persistent-coasters cat /bin/cat null null null
+persistent-coasters bash /bin/bash null null null
+persistent-coasters echo /bin/echo null null null
From ketan at ci.uchicago.edu Tue Aug 6 16:27:44 2013
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Tue, 6 Aug 2013 16:27:44 -0500 (CDT)
Subject: [Swift-commit] r6796 - SwiftTutorials/ATPESC_2013-08-06
Message-ID: <20130806212744.0EE589CC99@svn.ci.uchicago.edu>
Author: ketan
Date: 2013-08-06 16:27:43 -0500 (Tue, 06 Aug 2013)
New Revision: 6796
Modified:
SwiftTutorials/ATPESC_2013-08-06/README
Log:
updated README with addcloud
Modified: SwiftTutorials/ATPESC_2013-08-06/README
===================================================================
--- SwiftTutorials/ATPESC_2013-08-06/README 2013-08-06 21:16:16 UTC (rev 6795)
+++ SwiftTutorials/ATPESC_2013-08-06/README 2013-08-06 21:27:43 UTC (rev 6796)
@@ -473,6 +473,15 @@
swift -sites.file sites.xml -config cf -tc.file tc p8.swift
swift -sites.file sites.xml -config cf -tc.file tc p9.swift
-----
+
+* Add cloud resources to existing examples:
+-----
+ ./addcloud.sh #where dir is a tutorial script directory
+-----
+ e.g.
+-----
+ ./addcloud.sh ../p01 #will create a new site pool "both.xml"
+-----
* Finally, to clean up the log files, kill agent and shutdown the coaster service:
-----
./cleanme
From ketan at ci.uchicago.edu Tue Aug 6 16:36:13 2013
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Tue, 6 Aug 2013 16:36:13 -0500 (CDT)
Subject: [Swift-commit] r6797 - SwiftTutorials/ATPESC_2013-08-06
Message-ID: <20130806213613.058DE9CC99@svn.ci.uchicago.edu>
Author: ketan
Date: 2013-08-06 16:36:12 -0500 (Tue, 06 Aug 2013)
New Revision: 6797
Modified:
SwiftTutorials/ATPESC_2013-08-06/README
Log:
formatting to README
Modified: SwiftTutorials/ATPESC_2013-08-06/README
===================================================================
--- SwiftTutorials/ATPESC_2013-08-06/README 2013-08-06 21:27:43 UTC (rev 6796)
+++ SwiftTutorials/ATPESC_2013-08-06/README 2013-08-06 21:36:12 UTC (rev 6797)
@@ -477,11 +477,10 @@
* Add cloud resources to existing examples:
-----
./addcloud.sh #where dir is a tutorial script directory
------
e.g.
+ ./addcloud.sh ../p01 #will create a new site pool "both.xml" in ../p01
-----
- ./addcloud.sh ../p01 #will create a new site pool "both.xml"
------
+
* Finally, to clean up the log files, kill agent and shutdown the coaster service:
-----
./cleanme
From ketan at ci.uchicago.edu Tue Aug 6 16:41:08 2013
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Tue, 6 Aug 2013 16:41:08 -0500 (CDT)
Subject: [Swift-commit] r6798 - SwiftTutorials/ATPESC_2013-08-06
Message-ID: <20130806214108.51D129CC99@svn.ci.uchicago.edu>
Author: ketan
Date: 2013-08-06 16:41:08 -0500 (Tue, 06 Aug 2013)
New Revision: 6798
Modified:
SwiftTutorials/ATPESC_2013-08-06/README
Log:
Modified: SwiftTutorials/ATPESC_2013-08-06/README
===================================================================
--- SwiftTutorials/ATPESC_2013-08-06/README 2013-08-06 21:36:12 UTC (rev 6797)
+++ SwiftTutorials/ATPESC_2013-08-06/README 2013-08-06 21:41:08 UTC (rev 6798)
@@ -478,7 +478,7 @@
-----
./addcloud.sh #where dir is a tutorial script directory
e.g.
- ./addcloud.sh ../p01 #will create a new site pool "both.xml" in ../p01
+ ./addcloud.sh ../part01 #will create a new site pool "both.xml" in ../part01
-----
* Finally, to clean up the log files, kill agent and shutdown the coaster service:
From ketan at ci.uchicago.edu Thu Aug 8 10:24:55 2013
From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu)
Date: Thu, 8 Aug 2013 10:24:55 -0500 (CDT)
Subject: [Swift-commit] r6799 - trunk/libexec/log-processing
Message-ID: <20130808152455.CF1F3178884@svn.ci.uchicago.edu>
Author: ketan
Date: 2013-08-08 10:24:54 -0500 (Thu, 08 Aug 2013)
New Revision: 6799
Added:
trunk/libexec/log-processing/plotswiftlogs.ketan
Log:
plotting script version ketan
Added: trunk/libexec/log-processing/plotswiftlogs.ketan
===================================================================
--- trunk/libexec/log-processing/plotswiftlogs.ketan (rev 0)
+++ trunk/libexec/log-processing/plotswiftlogs.ketan 2013-08-08 15:24:54 UTC (rev 6799)
@@ -0,0 +1,67 @@
+#!/bin/bash
+
+#usage: ./extract4plots
+
+SWIFTLOGFILE=$1
+
+#TMPDIR=`mktemp -d plotlog.XXX`
+
+grep -i ProgressTicker $SWIFTLOGFILE > swiftoutfile.out
+
+SWIFTOUTFILE=swiftoutfile.out
+
+#extract start time
+TMPDATE=`grep -i progress $SWIFTOUTFILE 2>/dev/null | head -n 1 | cut -f1-2 -d ' '`
+START_TIME=`date +%s -d "$TMPDATE"`
+
+#extract end time
+TMPDATE=`grep -i progress $SWIFTOUTFILE 2>/dev/null | tail -n 1 | cut -f1-2 -d ' '`
+END_TIME=`date +%s -d "$TMPDATE"`
+
+#duration
+DIFFTIME=$((END_TIME - START_TIME))
+
+#extract active runs in a file
+(grep -o -i "Active:[0-9]*" $SWIFTOUTFILE 2>/dev/null | awk -F: '{print $2}' >active.txt)
+
+#extract successful completions in a file
+(grep -o -i "Successfully:[0-9]*" $SWIFTOUTFILE 2>/dev/null | awk -F: '{print $2}' > cumulative.txt)
+
+#prepare tics
+activelines=`wc -l active.txt | awk '{print $1}'`
+cumulines=`wc -l cumulative.txt | awk '{print $1}'`
+
+if [ $activelines -ne 0 ]
+then
+ activelinespertic=`echo "scale=5 ; $DIFFTIME / $activelines" | bc`
+fi
+
+seq 0 $activelinespertic $DIFFTIME > activetics.txt
+
+if [ $cumulines -ne 0 ]
+then
+ cumulinespertic=`echo "scale=5 ; $DIFFTIME / $cumulines" | bc`
+fi
+
+seq 0 $cumulinespertic $DIFFTIME > cumultics.txt
+
+#final plot data
+paste activetics.txt active.txt > plot_active.txt
+paste cumultics.txt cumulative.txt > plot_cumulative.txt
+
+cat << EOF1 > plotit.gp
+set terminal png enhanced
+set output "activeplot.png"
+set nokey
+set xlabel "Time in sec"
+set ylabel "number of active jobs"
+set title "Active jobs"
+plot "plot_active.txt" using 1:2 with line
+set output "cumulativeplot.png"
+set ylabel "number of completed jobs"
+set title "Cumulative jobs"
+plot "plot_cumulative.txt" using 1:2 with lines
+EOF1
+
+gnuplot plotit.gp 2>/dev/null
+
Property changes on: trunk/libexec/log-processing/plotswiftlogs.ketan
___________________________________________________________________
Added: svn:executable
+ *
From davidk at ci.uchicago.edu Thu Aug 8 10:50:11 2013
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Thu, 8 Aug 2013 10:50:11 -0500 (CDT)
Subject: [Swift-commit] r6800 - SwiftApps/Scattering/paintgrid
Message-ID: <20130808155011.494CD178884@svn.ci.uchicago.edu>
Author: davidk
Date: 2013-08-08 10:50:11 -0500 (Thu, 08 Aug 2013)
New Revision: 6800
Added:
SwiftApps/Scattering/paintgrid/beagle.xml
SwiftApps/Scattering/paintgrid/clean.sh
SwiftApps/Scattering/paintgrid/orthros.xml
SwiftApps/Scattering/paintgrid/setup.sh
Removed:
SwiftApps/Scattering/paintgrid/cleanup
SwiftApps/Scattering/paintgrid/pecos.xml
SwiftApps/Scattering/paintgrid/swift.properties
Modified:
SwiftApps/Scattering/paintgrid/README
SwiftApps/Scattering/paintgrid/apps
SwiftApps/Scattering/paintgrid/sites.xml
Log:
Add setup.sh for adding swift and setting up swift.properties
Split up sites files
Modified: SwiftApps/Scattering/paintgrid/README
===================================================================
--- SwiftApps/Scattering/paintgrid/README 2013-08-08 15:24:54 UTC (rev 6799)
+++ SwiftApps/Scattering/paintgrid/README 2013-08-08 15:50:11 UTC (rev 6800)
@@ -1,6 +1,6 @@
Demo / tutorial application to mimic the needs of the PaintGrid application
+===========================================================================
-
Setup:
$ PATH=/clhome/WILDE/swift/rev/swift-0.94.1/bin:$PATH
Modified: SwiftApps/Scattering/paintgrid/apps
===================================================================
--- SwiftApps/Scattering/paintgrid/apps 2013-08-08 15:24:54 UTC (rev 6799)
+++ SwiftApps/Scattering/paintgrid/apps 2013-08-08 15:50:11 UTC (rev 6800)
@@ -1,4 +1,3 @@
localhost local_python python
-beagle python python
-#orthros python python
-
+orthros python python
+beagle python python
Added: SwiftApps/Scattering/paintgrid/beagle.xml
===================================================================
--- SwiftApps/Scattering/paintgrid/beagle.xml (rev 0)
+++ SwiftApps/Scattering/paintgrid/beagle.xml 2013-08-08 15:50:11 UTC (rev 6800)
@@ -0,0 +1,19 @@
+
+
+
+ /lustre/beagle/{env.USER}
+ 24
+ 100
+ 100
+ pbs.aprun;pbs.mpp;depth=24
+ 3600
+ 00:05:00
+ 5
+ 1
+ 1
+ 1.20
+ 10000
+ /lustre/beagle/{env.USER}/swiftwork
+
+
+
Copied: SwiftApps/Scattering/paintgrid/clean.sh (from rev 6798, SwiftApps/Scattering/paintgrid/cleanup)
===================================================================
--- SwiftApps/Scattering/paintgrid/clean.sh (rev 0)
+++ SwiftApps/Scattering/paintgrid/clean.sh 2013-08-08 15:50:11 UTC (rev 6800)
@@ -0,0 +1,6 @@
+#! /bin/sh
+
+rm -rf out _concurrent *swiftx *kml *~ paintgrid-* python-* swift.log
+
+
+
Deleted: SwiftApps/Scattering/paintgrid/cleanup
===================================================================
--- SwiftApps/Scattering/paintgrid/cleanup 2013-08-08 15:24:54 UTC (rev 6799)
+++ SwiftApps/Scattering/paintgrid/cleanup 2013-08-08 15:50:11 UTC (rev 6800)
@@ -1,6 +0,0 @@
-#! /bin/sh
-
-rm -rf out _concurrent *swiftx *kml *~ paintgrid-* python-* swift.log
-
-
-
Added: SwiftApps/Scattering/paintgrid/orthros.xml
===================================================================
--- SwiftApps/Scattering/paintgrid/orthros.xml (rev 0)
+++ SwiftApps/Scattering/paintgrid/orthros.xml 2013-08-08 15:50:11 UTC (rev 6800)
@@ -0,0 +1,17 @@
+
+
+
+ 1
+ 00:15:00
+ 100
+ 100
+ 10
+ 10000
+ sec1all.q
+ sec1_all
+ 1200
+ 10.00
+
+ {env.HOME}/swiftwork
+
+
Deleted: SwiftApps/Scattering/paintgrid/pecos.xml
===================================================================
--- SwiftApps/Scattering/paintgrid/pecos.xml 2013-08-08 15:24:54 UTC (rev 6799)
+++ SwiftApps/Scattering/paintgrid/pecos.xml 2013-08-08 15:50:11 UTC (rev 6800)
@@ -1,59 +0,0 @@
-
-
-
-
-
- /tmp/wilde/swiftwork
- local
-
-
-
-
-
-
- sec1all.q -->
- sec1_all -->
- 1
-
-
-
- 1
- 4
- 4
- 64
- 3.20
-
-
- 00:15:00
- 1800
-
-
-
- /clhome/WILDE/swiftwork
-
-
-
- 100
- 100
- 10000
-
-
-
-
-
- /lustre/beagle/wilde/swift.scripts
- 24
- 100
- 100
- pbs.aprun;pbs.mpp;depth=24
- 3600
- 00:05:00
- 5
- 1
- 1
- 4.80
- 10000
- /lustre/beagle/wilde/swiftwork
-
-
-
Added: SwiftApps/Scattering/paintgrid/setup.sh
===================================================================
--- SwiftApps/Scattering/paintgrid/setup.sh (rev 0)
+++ SwiftApps/Scattering/paintgrid/setup.sh 2013-08-08 15:50:11 UTC (rev 6800)
@@ -0,0 +1,59 @@
+# Add Swift and java to PATH
+export SWIFT_HEAP_MAX=4G
+export PATH=/clhome/WILDE/swift/rev/swift-0.94.1/bin:$PATH
+
+echo -e "\\nUsing swift release from: \\c"
+which swift
+swift -version
+
+echo -e "Using java from: \\c"
+which java
+java -version
+echo
+
+# Cleanup swift.log
+if [ -f "swift.log" ]; then
+ rm swift.log > /dev/null 2>&1
+fi
+
+# Create data files
+if [ ! -f "data.0001.tif" ]; then
+ echo Creating data.0001.tif
+ dd bs=8M count=1 if=/dev/zero of=data.0001.tif
+ echo
+fi
+
+if [ ! -f "data.0001.tiny" ]; then
+ echo Creating data.0001.tiny
+ dd bs=800 count=1 if=/dev/zero of=data.0001.tiny
+ echo
+fi
+
+if [ -e $HOME/.swift ]; then
+ savedotswift=$(mktemp -d $HOME/.swift.save.XXXX)
+ echo -e "Saving $HOME/.swift in $savedotswift \\n"
+ mv $HOME/.swift/* $savedotswift
+else
+ mkdir -p $HOME/.swift
+fi
+
+cat >>$HOME/.swift/swift.properties <
-
-
-
-
- /tmp/wilde/swiftwork
- local
-
-
-
-
-
-
- sec1all.q -->
- sec1_all -->
- 1
-
-
-
- 1
- 4
- 4
- 64
- 2.56
-
-
- 00:15:00
- 1800
-
-
-
- /clhome/WILDE/swiftwork
-
-
-
- 100
- 100
- 10000
-
-
-
-
-
- /lustre/beagle/wilde/swift.scripts
- 24
- 100
- 100
- pbs.aprun;pbs.mpp;depth=24
- 3600
- 00:05:00
- 5
- 1
- 1
- 1.20
- 10000
- /lustre/beagle/wilde/swiftwork
-
-
+
+
+
+ 0
+ 10000
+
+ {env.HOME}/swiftwork
+
Deleted: SwiftApps/Scattering/paintgrid/swift.properties
===================================================================
--- SwiftApps/Scattering/paintgrid/swift.properties 2013-08-08 15:24:54 UTC (rev 6799)
+++ SwiftApps/Scattering/paintgrid/swift.properties 2013-08-08 15:50:11 UTC (rev 6800)
@@ -1,430 +0,0 @@
-
-sites.file=sites.xml
-tc.file=apps
-
-status.mode=provider
-use.provider.staging=true
-use.wrapper.staging=false
-wrapperlog.always.transfer=true
-execution.retries=0
-lazy.errors=false
-provider.staging.pin.swiftfiles=true
-sitedir.keep=true
-file.gc.enabled=false
-#tcp.port.range=50000,51000
-
-###########################################################################
-# Throttling options #
-###########################################################################
-#
-# For the throttling parameters, valid values are either a positive integer
-# or "off" (without the quotes).
-#
-
-#
-# Limits the number of concurrent submissions for a workflow instance. This
-# throttle only limits the number of concurrent tasks (jobs) that are being
-# sent to sites, not the total number of concurrent jobs that can be run.
-# The submission stage in GRAM is one of the most CPU expensive stages (due
-# mostly to the mutual authentication and delegation). Having too many
-# concurrent submissions can overload either or both the submit host CPU
-# and the remote host/head node causing degraded performance.
-#
-# Default: 4
-#
-
-throttle.submit=4
-#throttle.submit=off
-
-#
-# Limits the number of concurrent submissions for any of the sites Swift will
-# try to send jobs to. In other words it guarantees that no more than the
-# value of this throttle jobs sent to any site will be concurrently in a state
-# of being submitted.
-#
-# Default: 2
-#
-
-### throttle.host.submit=2
-#throttle.host.submit=off
-
-#
-# The Swift scheduler has the ability to limit the number of concurrent jobs
-# allowed on a site based on the performance history of that site. Each site
-# is assigned a score (initially 1), which can increase or decrease based
-# on whether the site yields successful or faulty job runs. The score for a
-# site can take values in the (0.1, 100) interval. The number of allowed jobs
-# is calculated using the following formula:
-# 2 + score*throttle.score.job.factor
-# This means a site will always be allowed at least two concurrent jobs and
-# at most 2 + 100*throttle.score.job.factor. With a default of 4 this means
-# at least 2 jobs and at most 402.
-#
-# Default: 4
-#
-
-### throttle.score.job.factor=0.2
-#throttle.score.job.factor=off
-
-
-#
-# Limits the total number of concurrent file transfers that can happen at any
-# given time. File transfers consume bandwidth. Too many concurrent transfers
-# can cause the network to be overloaded preventing various other signalling
-# traffic from flowing properly.
-#
-# Default: 4
-#
-
-throttle.transfers=1
-#throttle.transfers=off
-
-# Limits the total number of concurrent file operations that can happen at any
-# given time. File operations (like transfers) require an exclusive connection
-# to a site. These connections can be expensive to establish. A large number
-# of concurrent file operations may cause Swift to attempt to establish many
-# such expensive connections to various sites. Limiting the number of concurrent
-# file operations causes Swift to use a small number of cached connections and
-# achieve better overall performance.
-#
-# Default: 8
-#
-
-throttle.file.operations=1
-#throttle.file.operations=off
-
-# Swift runtime properties for CMTS Cyber Tutorial
-
-# Get sites and apps lists from these files in cwd:
-
-sites.file=sites.xml
-tc.file=apps
-
-# Data and status staging modes:
-
-use.provider.staging=false
-provider.staging.pin.swiftfiles=false
-use.wrapper.staging=false
-status.mode=provider
-
-# For better debugging:
-
-wrapperlog.always.transfer=true
-execution.retries=0
-lazy.errors=false
-sitedir.keep=true
-file.gc.enabled=false
-file.gc.enabled=false
-
-
-# Swift runtime properties for CMTS Cyber Tutorial
-
-# Get sites and apps lists from these files in cwd:
-
-sites.file=sites.xml
-tc.file=apps
-
-# Data and status staging modes:
-
-use.provider.staging=false
-provider.staging.pin.swiftfiles=false
-use.wrapper.staging=false
-status.mode=provider
-
-# For better debugging:
-
-wrapperlog.always.transfer=true
-execution.retries=0
-lazy.errors=false
-sitedir.keep=true
-file.gc.enabled=false
-file.gc.enabled=false
-
-
-# Swift runtime properties for CMTS Cyber Tutorial
-
-# Get sites and apps lists from these files in cwd:
-
-sites.file=sites.xml
-tc.file=apps
-
-# Data and status staging modes:
-
-use.provider.staging=false
-provider.staging.pin.swiftfiles=false
-use.wrapper.staging=false
-status.mode=provider
-
-# For better debugging:
-
-wrapperlog.always.transfer=true
-execution.retries=0
-lazy.errors=false
-sitedir.keep=true
-file.gc.enabled=false
-file.gc.enabled=false
-
-
-# Swift runtime properties for CMTS Cyber Tutorial
-
-# Get sites and apps lists from these files in cwd:
-
-sites.file=sites.xml
-tc.file=apps
-
-# Data and status staging modes:
-
-use.provider.staging=false
-provider.staging.pin.swiftfiles=false
-use.wrapper.staging=false
-status.mode=provider
-
-# For better debugging:
-
-wrapperlog.always.transfer=true
-execution.retries=0
-lazy.errors=false
-sitedir.keep=true
-file.gc.enabled=false
-file.gc.enabled=false
-
-
-# Swift runtime properties for CMTS Cyber Tutorial
-
-# Get sites and apps lists from these files in cwd:
-
-sites.file=sites.xml
-tc.file=apps
-
-# Data and status staging modes:
-
-use.provider.staging=false
-provider.staging.pin.swiftfiles=false
-use.wrapper.staging=false
-status.mode=provider
-
-# For better debugging:
-
-wrapperlog.always.transfer=true
-execution.retries=0
-lazy.errors=false
-sitedir.keep=true
-file.gc.enabled=false
-file.gc.enabled=false
-
-
-# Swift runtime properties for CMTS Cyber Tutorial
-
-# Get sites and apps lists from these files in cwd:
-
-sites.file=sites.xml
-tc.file=apps
-
-# Data and status staging modes:
-
-use.provider.staging=false
-provider.staging.pin.swiftfiles=false
-use.wrapper.staging=false
-status.mode=provider
-
-# For better debugging:
-
-wrapperlog.always.transfer=true
-execution.retries=0
-lazy.errors=false
-sitedir.keep=true
-file.gc.enabled=false
-file.gc.enabled=false
-
-
-# Swift runtime properties for CMTS Cyber Tutorial
-
-# Get sites and apps lists from these files in cwd:
-
-sites.file=sites.xml
-tc.file=apps
-
-# Data and status staging modes:
-
-use.provider.staging=false
-provider.staging.pin.swiftfiles=false
-use.wrapper.staging=false
-status.mode=provider
-
-# For better debugging:
-
-wrapperlog.always.transfer=true
-execution.retries=0
-lazy.errors=false
-sitedir.keep=true
-file.gc.enabled=false
-file.gc.enabled=false
-
-
-# Swift runtime properties for CMTS Cyber Tutorial
-
-# Get sites and apps lists from these files in cwd:
-
-sites.file=sites.xml
-tc.file=apps
-
-# Data and status staging modes:
-
-use.provider.staging=false
-provider.staging.pin.swiftfiles=false
-use.wrapper.staging=false
-status.mode=provider
-
-# For better debugging:
-
-wrapperlog.always.transfer=true
-execution.retries=0
-lazy.errors=false
-sitedir.keep=true
-file.gc.enabled=false
-file.gc.enabled=false
-
-
-# Swift runtime properties for CMTS Cyber Tutorial
-
-# Get sites and apps lists from these files in cwd:
-
-sites.file=sites.xml
-tc.file=apps
-
-# Data and status staging modes:
-
-use.provider.staging=false
-provider.staging.pin.swiftfiles=false
-use.wrapper.staging=false
-status.mode=provider
-
-# For better debugging:
-
-wrapperlog.always.transfer=true
-execution.retries=0
-lazy.errors=false
-sitedir.keep=true
-file.gc.enabled=false
-file.gc.enabled=false
-
-
-# Swift runtime properties for CMTS Cyber Tutorial
-
-# Get sites and apps lists from these files in cwd:
-
-sites.file=sites.xml
-tc.file=apps
-
-# Data and status staging modes:
-
-use.provider.staging=false
-provider.staging.pin.swiftfiles=false
-use.wrapper.staging=false
-status.mode=provider
-
-# For better debugging:
-
-wrapperlog.always.transfer=true
-execution.retries=0
-lazy.errors=false
-sitedir.keep=true
-file.gc.enabled=false
-file.gc.enabled=false
-
-
-# Swift runtime properties for CMTS Cyber Tutorial
-
-# Get sites and apps lists from these files in cwd:
-
-sites.file=sites.xml
-tc.file=apps
-
-# Data and status staging modes:
-
-use.provider.staging=false
-provider.staging.pin.swiftfiles=false
-use.wrapper.staging=false
-status.mode=provider
-
-# For better debugging:
-
-wrapperlog.always.transfer=true
-execution.retries=0
-lazy.errors=false
-sitedir.keep=true
-file.gc.enabled=false
-file.gc.enabled=false
-
-
-# Swift runtime properties for CMTS Cyber Tutorial
-
-# Get sites and apps lists from these files in cwd:
-
-sites.file=sites.xml
-tc.file=apps
-
-# Data and status staging modes:
-
-use.provider.staging=false
-provider.staging.pin.swiftfiles=false
-use.wrapper.staging=false
-status.mode=provider
-
-# For better debugging:
-
-wrapperlog.always.transfer=true
-execution.retries=0
-lazy.errors=false
-sitedir.keep=true
-file.gc.enabled=false
-file.gc.enabled=false
-
-
-# Swift runtime properties for CMTS Cyber Tutorial
-
-# Get sites and apps lists from these files in cwd:
-
-sites.file=sites.xml
-tc.file=apps
-
-# Data and status staging modes:
-
-use.provider.staging=false
-provider.staging.pin.swiftfiles=false
-use.wrapper.staging=false
-status.mode=provider
-
-# For better debugging:
-
-wrapperlog.always.transfer=true
-execution.retries=0
-lazy.errors=false
-sitedir.keep=true
-file.gc.enabled=false
-file.gc.enabled=false
-
-
-# Swift runtime properties for CMTS Cyber Tutorial
-
-# Get sites and apps lists from these files in cwd:
-
-sites.file=sites.xml
-tc.file=apps
-
-# Data and status staging modes:
-
-use.provider.staging=false
-provider.staging.pin.swiftfiles=false
-use.wrapper.staging=false
-status.mode=provider
-
-# For better debugging:
-
-wrapperlog.always.transfer=true
-execution.retries=0
-lazy.errors=false
-sitedir.keep=true
-file.gc.enabled=false
-file.gc.enabled=false
-
From davidk at ci.uchicago.edu Thu Aug 8 11:59:59 2013
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Thu, 8 Aug 2013 11:59:59 -0500 (CDT)
Subject: [Swift-commit] r6801 - SwiftApps/Scattering/paintgrid
Message-ID: <20130808165959.E30B5178884@svn.ci.uchicago.edu>
Author: davidk
Date: 2013-08-08 11:59:59 -0500 (Thu, 08 Aug 2013)
New Revision: 6801
Modified:
SwiftApps/Scattering/paintgrid/beagle.xml
SwiftApps/Scattering/paintgrid/clean.sh
SwiftApps/Scattering/paintgrid/setup.sh
SwiftApps/Scattering/paintgrid/start-beagle
Log:
Add provider staging to swift properties
Use persistent coasters with beagle
Modified: SwiftApps/Scattering/paintgrid/beagle.xml
===================================================================
--- SwiftApps/Scattering/paintgrid/beagle.xml 2013-08-08 15:50:11 UTC (rev 6800)
+++ SwiftApps/Scattering/paintgrid/beagle.xml 2013-08-08 16:59:59 UTC (rev 6801)
@@ -1,19 +1,29 @@
-
-
- /lustre/beagle/{env.USER}
- 24
- 100
- 100
- pbs.aprun;pbs.mpp;depth=24
- 3600
- 00:05:00
- 5
- 1
- 1
- 1.20
- 10000
- /lustre/beagle/{env.USER}/swiftwork
-
+
+
+
+ 0
+ 10000
+
+ {env.HOME}/swiftwork
+
+
+
+
+ /lustre/beagle/{env.USER}/swift.scripts
+ 24
+ 100
+ 100
+ pbs.aprun;pbs.mpp;depth=24
+ 3600
+ 00:05:00
+ 5
+ 1
+ 1
+ 4.80
+ 10000
+ /lustre/beagle/{env.USER}/swiftwork
+
+
Modified: SwiftApps/Scattering/paintgrid/clean.sh
===================================================================
--- SwiftApps/Scattering/paintgrid/clean.sh 2013-08-08 15:50:11 UTC (rev 6800)
+++ SwiftApps/Scattering/paintgrid/clean.sh 2013-08-08 16:59:59 UTC (rev 6801)
@@ -1,6 +1,3 @@
#! /bin/sh
-rm -rf out _concurrent *swiftx *kml *~ paintgrid-* python-* swift.log
-
-
-
+rm -rf out _concurrent *swiftx *kml *~ paintgrid-* python-* swift.log beagle-pid beagle-log
Modified: SwiftApps/Scattering/paintgrid/setup.sh
===================================================================
--- SwiftApps/Scattering/paintgrid/setup.sh 2013-08-08 15:50:11 UTC (rev 6800)
+++ SwiftApps/Scattering/paintgrid/setup.sh 2013-08-08 16:59:59 UTC (rev 6801)
@@ -37,14 +37,16 @@
mkdir -p $HOME/.swift
fi
+# Try to make things easier by assuming username are the same on orthros and beagle
+sed -i s/davidk/$USER/g start-beagle
+
cat >>$HOME/.swift/swift.properties <&2 & echo $!; sleep 99999' 1>beagle-pid 2>beagle-log &
From davidk at ci.uchicago.edu Thu Aug 8 12:03:00 2013
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Thu, 8 Aug 2013 12:03:00 -0500 (CDT)
Subject: [Swift-commit] r6802 - SwiftApps/Scattering/paintgrid
Message-ID: <20130808170300.BAD25178884@svn.ci.uchicago.edu>
Author: davidk
Date: 2013-08-08 12:03:00 -0500 (Thu, 08 Aug 2013)
New Revision: 6802
Modified:
SwiftApps/Scattering/paintgrid/clean.sh
SwiftApps/Scattering/paintgrid/stop-beagle
Log:
Move cleanup of beagle files to stop-beagle
Modified: SwiftApps/Scattering/paintgrid/clean.sh
===================================================================
--- SwiftApps/Scattering/paintgrid/clean.sh 2013-08-08 16:59:59 UTC (rev 6801)
+++ SwiftApps/Scattering/paintgrid/clean.sh 2013-08-08 17:03:00 UTC (rev 6802)
@@ -1,3 +1,3 @@
#! /bin/sh
-rm -rf out _concurrent *swiftx *kml *~ paintgrid-* python-* swift.log beagle-pid beagle-log
+rm -rf out _concurrent *swiftx *kml *~ paintgrid-* python-* swift.log
Modified: SwiftApps/Scattering/paintgrid/stop-beagle
===================================================================
--- SwiftApps/Scattering/paintgrid/stop-beagle 2013-08-08 16:59:59 UTC (rev 6801)
+++ SwiftApps/Scattering/paintgrid/stop-beagle 2013-08-08 17:03:00 UTC (rev 6802)
@@ -1 +1,2 @@
ssh -n login1.beagle.ci.uchicago.edu "echo killing coaster service: ; ps -j $(cat beagle-pid); /bin/kill 15 -\$(ps -j $(cat beagle-pid)| tail -1 | awk '{print \$2}' ) "
+rm beagle-pid beagle-log > /dev/null 2>&1
From davidk at ci.uchicago.edu Thu Aug 8 13:08:53 2013
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Thu, 8 Aug 2013 13:08:53 -0500 (CDT)
Subject: [Swift-commit] r6803 - in SwiftApps/Scattering/paintgrid: . docs
Message-ID: <20130808180853.5E2D1178884@svn.ci.uchicago.edu>
Author: davidk
Date: 2013-08-08 13:08:53 -0500 (Thu, 08 Aug 2013)
New Revision: 6803
Added:
SwiftApps/Scattering/paintgrid/docs/
SwiftApps/Scattering/paintgrid/docs/README
SwiftApps/Scattering/paintgrid/docs/README.old
SwiftApps/Scattering/paintgrid/docs/build_docs.sh
Removed:
SwiftApps/Scattering/paintgrid/README
SwiftApps/Scattering/paintgrid/run-paintgrid
SwiftApps/Scattering/paintgrid/run-remote-persist.sh
SwiftApps/Scattering/paintgrid/sites-used
SwiftApps/Scattering/paintgrid/start-tunnels
SwiftApps/Scattering/paintgrid/tssh
Log:
Start of new doc structure and some cleanup
Deleted: SwiftApps/Scattering/paintgrid/README
===================================================================
--- SwiftApps/Scattering/paintgrid/README 2013-08-08 17:03:00 UTC (rev 6802)
+++ SwiftApps/Scattering/paintgrid/README 2013-08-08 18:08:53 UTC (rev 6803)
@@ -1,50 +0,0 @@
-Demo / tutorial application to mimic the needs of the PaintGrid application
-===========================================================================
-
-Setup:
-
- $ PATH=/clhome/WILDE/swift/rev/swift-0.94.1/bin:$PATH
-
-Create a test data file:
-
- $ dd bs=8M count=1 if=/dev/zero of=data.0001.tif
- $ dd bs=800 count=1 if=/dev/zero of=data.0001.tiny
-
-Run the Swift script:
-
- $ mkdir ~/swift
- $ cp swift.properties ~/.swift
-
- $ swift -tc.file apps -sites.file amd.xml paintgrid.swift -runDir=$PWD -params=genpoints.params -image=data.0001.tiny
-
-
-
-TODO:
-
-Execution scenarios:
-
- run.local.sh
-
- run.beagle.sh
-
- run.midway.sh
-
- run.midway+beagle.sh
-
- run.blues.sh
-
- run.orthros.sh
-
-
-Data transfer models:
-
-- provider staging without caching
-
-- provider staging with ad-hoc caching of big common files
-
-- gridftp staging
-
-- ssh (scp) staging
-
-- wrapper staging with scp and caching
-
Copied: SwiftApps/Scattering/paintgrid/docs/README (from rev 6800, SwiftApps/Scattering/paintgrid/README)
===================================================================
--- SwiftApps/Scattering/paintgrid/docs/README (rev 0)
+++ SwiftApps/Scattering/paintgrid/docs/README 2013-08-08 18:08:53 UTC (rev 6803)
@@ -0,0 +1,50 @@
+Demo / tutorial application to mimic the needs of the PaintGrid application
+===========================================================================
+
+Setup:
+
+ $ PATH=/clhome/WILDE/swift/rev/swift-0.94.1/bin:$PATH
+
+Create a test data file:
+
+ $ dd bs=8M count=1 if=/dev/zero of=data.0001.tif
+ $ dd bs=800 count=1 if=/dev/zero of=data.0001.tiny
+
+Run the Swift script:
+
+ $ mkdir ~/swift
+ $ cp swift.properties ~/.swift
+
+ $ swift -tc.file apps -sites.file amd.xml paintgrid.swift -runDir=$PWD -params=genpoints.params -image=data.0001.tiny
+
+
+
+TODO:
+
+Execution scenarios:
+
+ run.local.sh
+
+ run.beagle.sh
+
+ run.midway.sh
+
+ run.midway+beagle.sh
+
+ run.blues.sh
+
+ run.orthros.sh
+
+
+Data transfer models:
+
+- provider staging without caching
+
+- provider staging with ad-hoc caching of big common files
+
+- gridftp staging
+
+- ssh (scp) staging
+
+- wrapper staging with scp and caching
+
Copied: SwiftApps/Scattering/paintgrid/docs/README.old (from rev 6800, SwiftApps/Scattering/paintgrid/README)
===================================================================
--- SwiftApps/Scattering/paintgrid/docs/README.old (rev 0)
+++ SwiftApps/Scattering/paintgrid/docs/README.old 2013-08-08 18:08:53 UTC (rev 6803)
@@ -0,0 +1,50 @@
+Demo / tutorial application to mimic the needs of the PaintGrid application
+===========================================================================
+
+Setup:
+
+ $ PATH=/clhome/WILDE/swift/rev/swift-0.94.1/bin:$PATH
+
+Create a test data file:
+
+ $ dd bs=8M count=1 if=/dev/zero of=data.0001.tif
+ $ dd bs=800 count=1 if=/dev/zero of=data.0001.tiny
+
+Run the Swift script:
+
+ $ mkdir ~/swift
+ $ cp swift.properties ~/.swift
+
+ $ swift -tc.file apps -sites.file amd.xml paintgrid.swift -runDir=$PWD -params=genpoints.params -image=data.0001.tiny
+
+
+
+TODO:
+
+Execution scenarios:
+
+ run.local.sh
+
+ run.beagle.sh
+
+ run.midway.sh
+
+ run.midway+beagle.sh
+
+ run.blues.sh
+
+ run.orthros.sh
+
+
+Data transfer models:
+
+- provider staging without caching
+
+- provider staging with ad-hoc caching of big common files
+
+- gridftp staging
+
+- ssh (scp) staging
+
+- wrapper staging with scp and caching
+
Added: SwiftApps/Scattering/paintgrid/docs/build_docs.sh
===================================================================
--- SwiftApps/Scattering/paintgrid/docs/build_docs.sh (rev 0)
+++ SwiftApps/Scattering/paintgrid/docs/build_docs.sh 2013-08-08 18:08:53 UTC (rev 6803)
@@ -0,0 +1,4 @@
+#!/bin/bash -e
+
+# This assumes that docs will be built on communicado or bridled
+~davidk/asciidoc-8.6.4/asciidoc -a toc -a toplevels=2 -a stylesheet=~davidk/swift-0.94/cog/modules/swift/docs/stylesheets/asciidoc.css -a max-width=800px -o tutorial.html README
Deleted: SwiftApps/Scattering/paintgrid/run-paintgrid
===================================================================
--- SwiftApps/Scattering/paintgrid/run-paintgrid 2013-08-08 17:03:00 UTC (rev 6802)
+++ SwiftApps/Scattering/paintgrid/run-paintgrid 2013-08-08 18:08:53 UTC (rev 6803)
@@ -1,9 +0,0 @@
-#! /bin/sh
-
-PATH=/clhome/WILDE/swift/rev/swift-0.94.1/bin:$PATH
-
-export GLOBUS_HOSTNAME=localhost
-export GLOBUS_TCP_PORT_RANGE=59900,59909
-export GLOBUS_TCP_SOURCE_RANGE=59900,59909
-
-swift paintgrid.swift -params=genpoints.params -image=data.0001.tiny -runTime=0.0001
Deleted: SwiftApps/Scattering/paintgrid/run-remote-persist.sh
===================================================================
--- SwiftApps/Scattering/paintgrid/run-remote-persist.sh 2013-08-08 17:03:00 UTC (rev 6802)
+++ SwiftApps/Scattering/paintgrid/run-remote-persist.sh 2013-08-08 18:08:53 UTC (rev 6803)
@@ -1,52 +0,0 @@
-swift$ cat run-persist.sh
-#! /bin/sh
-
-# Start a persistent swift coaster service, capturing its port # in portfile
-
-portfile=$(mktemp portfile.XXXX)
-coaster-service -portfile $portfile -nosec >& coaster.log &
-coasterpid=$!
-sleep 5 # Wait for the service to record its port
-port=$(cat $portfile)
-
-# Report the coaster service port and process id
-
-echo coaster pid: $coasterpid port: $port
-echo $coasterpid >coasterpid
-echo started coaster service:
-ps -p $coasterpid --ppid $coasterpid -H -j # display parent shell and java child processes
-
-# Create a sites file pointing to the service we just started
-
-cat >sites.xml <
-
-
- 4
- 16
- 1
- 1
- 1
- 00:01:00
- 600
- 100
- 100
- sandyb
- 1.00
- 10000
- single
- true
-
- /scratch/midway/{env.USER}/work
-
-
-END
-
-# Now run several Swift scripts, one at a time, just like MD_String will do
-
-NUM_SWIFT_RUNS=$1
-for ((i=0;i<$NUM_SWIFT_RUNS;i++)); do
- set -x
- swift -sites.file sites.xml -tc.file tc.intelmpi -config cf mpicatnap.swift -n=${2:-1} -t=${3:-2}
- set +x
-done
Deleted: SwiftApps/Scattering/paintgrid/sites-used
===================================================================
--- SwiftApps/Scattering/paintgrid/sites-used 2013-08-08 17:03:00 UTC (rev 6802)
+++ SwiftApps/Scattering/paintgrid/sites-used 2013-08-08 18:08:53 UTC (rev 6803)
@@ -1,3 +0,0 @@
-#! /bin/sh
-
-for l in $(/bin/ls -1t paint*.log) ; do echo $l; ./stats $l | grep -v localhost ; done | more
Deleted: SwiftApps/Scattering/paintgrid/start-tunnels
===================================================================
--- SwiftApps/Scattering/paintgrid/start-tunnels 2013-08-08 17:03:00 UTC (rev 6802)
+++ SwiftApps/Scattering/paintgrid/start-tunnels 2013-08-08 18:08:53 UTC (rev 6803)
@@ -1,11 +0,0 @@
-ssh -n -N login1.beagle.ci.uchicago.edu \
- -R 59900:orthros.xray.aps.anl.gov:59900 \
- -R 59901:orthros.xray.aps.anl.gov:59901 \
- -R 59902:orthros.xray.aps.anl.gov:59902 \
- -R 59903:orthros.xray.aps.anl.gov:59903 \
- -R 59904:orthros.xray.aps.anl.gov:59904 \
- -R 59905:orthros.xray.aps.anl.gov:59905 \
- -R 59906:orthros.xray.aps.anl.gov:59906 \
- -R 59907:orthros.xray.aps.anl.gov:59907 \
- -R 59908:orthros.xray.aps.anl.gov:59908 \
- -R 59909:orthros.xray.aps.anl.gov:59909
Deleted: SwiftApps/Scattering/paintgrid/tssh
===================================================================
--- SwiftApps/Scattering/paintgrid/tssh 2013-08-08 17:03:00 UTC (rev 6802)
+++ SwiftApps/Scattering/paintgrid/tssh 2013-08-08 18:08:53 UTC (rev 6803)
@@ -1,3 +0,0 @@
-# ssh login1.beagle.ci.uchicago.edu -L 59900:localhost:59900 'sleep 1234'
- ssh -n login1.beagle.ci.uchicago.edu 'sleep 1234 & echo $!; exit'
-
From davidk at ci.uchicago.edu Thu Aug 8 13:18:20 2013
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Thu, 8 Aug 2013 13:18:20 -0500 (CDT)
Subject: [Swift-commit] r6804 - SwiftApps/Scattering/paintgrid/docs
Message-ID: <20130808181820.32B71178884@svn.ci.uchicago.edu>
Author: davidk
Date: 2013-08-08 13:18:19 -0500 (Thu, 08 Aug 2013)
New Revision: 6804
Added:
SwiftApps/Scattering/paintgrid/docs/tutorial.html
Modified:
SwiftApps/Scattering/paintgrid/docs/README
SwiftApps/Scattering/paintgrid/docs/build_docs.sh
Log:
x bit for build_docs.sh
Adding html output
Intro setup material for readme
Modified: SwiftApps/Scattering/paintgrid/docs/README
===================================================================
--- SwiftApps/Scattering/paintgrid/docs/README 2013-08-08 18:08:53 UTC (rev 6803)
+++ SwiftApps/Scattering/paintgrid/docs/README 2013-08-08 18:18:19 UTC (rev 6804)
@@ -1,15 +1,31 @@
-Demo / tutorial application to mimic the needs of the PaintGrid application
-===========================================================================
+Demo / tutorial PaintGrid application
+=====================================
-Setup:
+PaintGrid tutorial setup
+------------------------
- $ PATH=/clhome/WILDE/swift/rev/swift-0.94.1/bin:$PATH
+Check out scripts from SVN
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+To checkout the most recent PaintGrid tutorial scripts from SVN, run the following command:
-Create a test data file:
+-----
+$ svn co https://svn.ci.uchicago.edu/svn/vdl2/SwiftApps/Scattering/paintgrid
+-----
+This will create a directory called paintgrid which contains all of the scripts mentioned in this document.
- $ dd bs=8M count=1 if=/dev/zero of=data.0001.tif
- $ dd bs=800 count=1 if=/dev/zero of=data.0001.tiny
+Run setup
+~~~~~~~~~
+Once the scripts are checked out, run the following commands to perform
+the initial setup.
+-----
+$ cd paintgrid # change to the newly created paintgrid directory
+$ source setup.sh # Updates PATH and creates a configuration file in $HOME/.swift
+-----
+
+NOTE: If you disconnect from the machine, you will need to re-run source setup.sh.
+
+
Run the Swift script:
$ mkdir ~/swift
Property changes on: SwiftApps/Scattering/paintgrid/docs/build_docs.sh
___________________________________________________________________
Added: svn:executable
+ *
Added: SwiftApps/Scattering/paintgrid/docs/tutorial.html
===================================================================
--- SwiftApps/Scattering/paintgrid/docs/tutorial.html (rev 0)
+++ SwiftApps/Scattering/paintgrid/docs/tutorial.html 2013-08-08 18:18:19 UTC (rev 6804)
@@ -0,0 +1,683 @@
+
+
+
+
+
+Demo / tutorial PaintGrid application
+
+
+
+
+
+
+
+
PaintGrid tutorial setup
+
+
+
Check out scripts from SVN
+
To checkout the most recent PaintGrid tutorial scripts from SVN, run the following command:
+
+
+
$ svn co https://svn.ci.uchicago.edu/svn/vdl2/SwiftApps/Scattering/paintgrid
+
+
This will create a directory called paintgrid which contains all of the scripts mentioned in this document.
+
+
+
Run setup
+
Once the scripts are checked out, run the following commands to perform
+the initial setup.
+
+
+
$ cd paintgrid # change to the newly created paintgrid directory
+$ source setup.sh # Updates PATH and creates a configuration file in $HOME/.swift
+
+
+
+
+ Note
+ |
+If you disconnect from the machine, you will need to re-run source setup.sh. |
+
+
+
+
+
+
$ mkdir ~/swift
+$ cp swift.properties ~/.swift
+
+
+
+
$ swift -tc.file apps -sites.file amd.xml paintgrid.swift -runDir=$PWD -params=genpoints.params -image=data.0001.tiny
+
+
+
+
+
+
+
+
+
+
+
+-
+
+provider staging without caching
+
+
+-
+
+provider staging with ad-hoc caching of big common files
+
+
+-
+
+gridftp staging
+
+
+-
+
+ssh (scp) staging
+
+
+-
+
+wrapper staging with scp and caching
+
+
+
+
+
+
+
+
+
+
+
From davidk at ci.uchicago.edu Thu Aug 8 13:22:27 2013
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Thu, 8 Aug 2013 13:22:27 -0500 (CDT)
Subject: [Swift-commit] r6805 - SwiftApps/Scattering/paintgrid/docs
Message-ID: <20130808182227.90A26178884@svn.ci.uchicago.edu>
Author: davidk
Date: 2013-08-08 13:22:27 -0500 (Thu, 08 Aug 2013)
New Revision: 6805
Modified:
SwiftApps/Scattering/paintgrid/docs/tutorial.html
Log:
Set mime-type to text/html
Property changes on: SwiftApps/Scattering/paintgrid/docs/tutorial.html
___________________________________________________________________
Added: svn:mime-type
+ text/html
From davidk at ci.uchicago.edu Thu Aug 8 14:12:14 2013
From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu)
Date: Thu, 8 Aug 2013 14:12:14 -0500 (CDT)
Subject: [Swift-commit] r6806 - SwiftApps/Scattering/paintgrid/docs
Message-ID: <20130808191214.4F628178884@svn.ci.uchicago.edu>
Author: davidk
Date: 2013-08-08 14:12:14 -0500 (Thu, 08 Aug 2013)
New Revision: 6806
Modified:
SwiftApps/Scattering/paintgrid/docs/README
SwiftApps/Scattering/paintgrid/docs/tutorial.html
Log:
Section on python applications
Modified: SwiftApps/Scattering/paintgrid/docs/README
===================================================================
--- SwiftApps/Scattering/paintgrid/docs/README 2013-08-08 18:22:27 UTC (rev 6805)
+++ SwiftApps/Scattering/paintgrid/docs/README 2013-08-08 19:12:14 UTC (rev 6806)
@@ -6,13 +6,16 @@
Check out scripts from SVN
~~~~~~~~~~~~~~~~~~~~~~~~~~
-To checkout the most recent PaintGrid tutorial scripts from SVN, run the following command:
+To checkout the most recent PaintGrid tutorial scripts from SVN, run the
+following command:
-----
$ svn co https://svn.ci.uchicago.edu/svn/vdl2/SwiftApps/Scattering/paintgrid
-----
-This will create a directory called paintgrid which contains all of the scripts mentioned in this document.
+This will create a directory called paintgrid which contains all of the scripts
+mentioned in this document.
+
Run setup
~~~~~~~~~
Once the scripts are checked out, run the following commands to perform
@@ -23,44 +26,92 @@
$ source setup.sh # Updates PATH and creates a configuration file in $HOME/.swift
-----
-NOTE: If you disconnect from the machine, you will need to re-run source setup.sh.
+The setup.sh script will also create two data files that will be used for later
+processing: data.0001.tiny (800 bytes), and data.0001.tif (8 megabytes).
+NOTE: If you disconnect from the machine, you will need to re-run source
+setup.sh.
-Run the Swift script:
+Mock PaintGrid applications
+---------------------------
+There are two python applications that serve as standins for the real science
+applications: genpoints.py and processpoints.py.
- $ mkdir ~/swift
- $ cp swift.properties ~/.swift
+genpoints.py
+~~~~~~~~~~~~
+The genpoints.py script generates a series of point values based on an input
+parameter file. The command line arguments are:
- $ swift -tc.file apps -sites.file amd.xml paintgrid.swift -runDir=$PWD -params=genpoints.params -image=data.0001.tiny
+.genpoints.py arguments
+[options="header"]
+|======================
+|Argument number|Description
+|1 |Name of parameter file to read
+|2 |Run directory - location to create output files
+|======================
+The input parameter file determines how point value files get created. Below
+is a table with a list of parameters and a brief description.
+.genpoints.py parameter file
+[options="header"]
+|===========================
+|Parameter|Description
+|minx|Minimum x value
+|maxx|Maximum x value
+|miny|Minimum y value
+|maxy|Maximum y value
+|minz|Minimum x value
+|maxz|Maximum z value
+|incr|Increment size
+|tuplesPerFile|Number of tuples per file
+|filePrefix|Prefix of point value files
+|outDir|Output directory name, relative to run directory
+|==========================
-TODO:
+Below is an example genpoints.py parameter file called onepoint.params that with max values
+at 10 and increments at 10, causing the generation of only one point file
+-----
+minx=0.0
+maxx=10.0
+miny=0.0
+maxy=10.0
+minz=0.0
+maxz=10.0
+incr=10.0
+-----
-Execution scenarios:
+Here is an example putting everything together and viewing the output:
+-----
+$ ./genpoints.py onepoint.params /clhome/DAVIDK/Scattering/paintgrid
+out/seq/seq00000
+$ cat out/seq/seq00000
+0.0 0.0 0.0
+-----
- run.local.sh
+processpoints.py
+~~~~~~~~~~~~~~~~
+The processpoints.py script simulates an image processing application. It reads
+a point file generated by genpoints.py, counts the number of bytes in a given
+data file, and then sleeps for a requested amount of time.
- run.beagle.sh
+.processpoints.py arguments
+[options="header"]
+|==========================
+|Parameter|Description
+|1 | Name of image file to read
+|2 | Name of point file to read
+|3 | Time to sleep (floating point seconds)
+|==========================
- run.midway.sh
+Here is an example of manually running processpoints.py:
+-----
+$ ./processpoints.py data.0001.tiny out/seq/seq00000 1
+Data file has 800 pixels
+Processing 1 points in model space
+Runtime is 1.0 seconds per point
+ 0.000000 0.000000 0.000000 0.000000
+-----
- run.midway+beagle.sh
- run.blues.sh
-
- run.orthros.sh
-
-
-Data transfer models:
-
-- provider staging without caching
-
-- provider staging with ad-hoc caching of big common files
-
-- gridftp staging
-
-- ssh (scp) staging
-
-- wrapper staging with scp and caching
-
+swift -tc.file apps -sites.file amd.xml paintgrid.swift -runDir=$PWD -params=genpoints.params -image=data.0001.tiny
Modified: SwiftApps/Scattering/paintgrid/docs/tutorial.html
===================================================================
--- SwiftApps/Scattering/paintgrid/docs/tutorial.html 2013-08-08 18:22:27 UTC (rev 6805)
+++ SwiftApps/Scattering/paintgrid/docs/tutorial.html 2013-08-08 19:12:14 UTC (rev 6806)
@@ -581,12 +581,14 @@
Check out scripts from SVN
-
To checkout the most recent PaintGrid tutorial scripts from SVN, run the following command:
+
To checkout the most recent PaintGrid tutorial scripts from SVN, run the
+following command:
$ svn co https://svn.ci.uchicago.edu/svn/vdl2/SwiftApps/Scattering/paintgrid
-
This will create a directory called paintgrid which contains all of the scripts mentioned in this document.
+
This will create a directory called paintgrid which contains all of the scripts
+mentioned in this document.
Run setup
@@ -597,78 +599,181 @@
$ cd paintgrid # change to the newly created paintgrid directory
$ source setup.sh # Updates PATH and creates a configuration file in $HOME/.swift
+The setup.sh script will also create two data files that will be used for later
+processing: data.0001.tiny (800 bytes), and data.0001.tif (8 megabytes).
Note
|
-If you disconnect from the machine, you will need to re-run source setup.sh. |
+If you disconnect from the machine, you will need to re-run source
+setup.sh. |
-
-
+
+
+
+
+
Mock PaintGrid applications
+
+
There are two python applications that serve as standins for the real science
+applications: genpoints.py and processpoints.py.
+
+
genpoints.py
+
The genpoints.py script generates a series of point values based on an input
+parameter file. The command line arguments are:
+
+
+Table 1. genpoints.py arguments
+
+
+
+
+Argument number |
+Description |
+
+
+
+
+1 |
+Name of parameter file to read |
+
+
+2 |
+Run directory - location to create output files |
+
+
+
+
+
The input parameter file determines how point value files get created. Below
+is a table with a list of parameters and a brief description.
+
+
+Table 2. genpoints.py parameter file
+
+
+
+
+Parameter |
+Description |
+
+
+
+
+minx |
+Minimum x value |
+
+
+maxx |
+Maximum x value |
+
+
+miny |
+Minimum y value |
+
+
+maxy |
+Maximum y value |
+
+
+minz |
+Minimum x value |
+
+
+maxz |
+Maximum z value |
+
+
+incr |
+Increment size |
+
+
+tuplesPerFile |
+Number of tuples per file |
+
+
+filePrefix |
+Prefix of point value files |
+
+
+outDir |
+Output directory name, relative to run directory |
+
+
+
+
+
Below is an example genpoints.py parameter file called onepoint.params that with max values
+at 10 and increments at 10, causing the generation of only one point file
+
-
$ mkdir ~/swift
-$ cp swift.properties ~/.swift
+
minx=0.0
+maxx=10.0
+miny=0.0
+maxy=10.0
+minz=0.0
+maxz=10.0
+incr=10.0
-
+
Here is an example putting everything together and viewing the output:
+
-
$ swift -tc.file apps -sites.file amd.xml paintgrid.swift -runDir=$PWD -params=genpoints.params -image=data.0001.tiny
+
$ ./genpoints.py onepoint.params /clhome/DAVIDK/Scattering/paintgrid
+out/seq/seq00000
+$ cat out/seq/seq00000
+0.0 0.0 0.0
-
-
-
+
+
+
processpoints.py
+
The processpoints.py script simulates an image processing application. It reads
+a point file generated by genpoints.py, counts the number of bytes in a given
+data file, and then sleeps for a requested amount of time.
+
+
+Table 3. processpoints.py arguments
+
+
+
+
+Parameter |
+Description |
+
+
+
+
+1 |
+Name of image file to read |
+
+
+2 |
+Name of point file to read |
+
+
+3 |
+Time to sleep (floating point seconds) |
+
+
+
+
+
Here is an example of manually running processpoints.py:
+
-
run.local.sh
+
$ ./processpoints.py data.0001.tiny out/seq/seq00000 1
+Data file has 800 pixels
+Processing 1 points in model space
+Runtime is 1.0 seconds per point
+ 0.000000 0.000000 0.000000 0.000000
-
-
-
-
-
-
-
--
-
-provider staging without caching
-
-
--
-
-provider staging with ad-hoc caching of big common files
-
-
--
-
-gridftp staging
-
-
--
-
-ssh (scp) staging
-
-
--
-
-wrapper staging with scp and caching
-
-
-
+
swift -tc.file apps -sites.file amd.xml paintgrid.swift -runDir=$PWD -params=genpoints.params -image=data.0001.tiny
@@ -676,7 +781,7 @@