[Swift-commit] r7662 - SwiftApps/cloud

Sun Mar 9 13:39:43 CDT 2014

Author: ketan
Date: 2014-03-09 13:39:43 -0500 (Sun, 09 Mar 2014)
New Revision: 7662

Modified:
   SwiftApps/cloud/README
   SwiftApps/cloud/coaster-service.conf
Log:
add to README and FORMAT

Modified: SwiftApps/cloud/README
===================================================================

--- SwiftApps/cloud/README	2014-03-09 16:54:34 UTC (rev 7661)
+++ SwiftApps/cloud/README	2014-03-09 18:39:43 UTC (rev 7662)
@@ -1,62 +1,102 @@
-Toolkit to run Swift applications over Amazon EC2 cloud
-========================================================
+Swift Cloud Provisioning Toolkit
+=================================
 
 EC2 setup
 ----------
 
-Installation and setup of EC2 commandline tools is assumed. If you do not have already, please find instructions of getting EC2 api from here:
-http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/SettingUp_CommandLine.html
+Installation and setup of EC2 commandline tools is assumed. If you do not have
+already, find instructions for setting the EC2 CLI API from
+http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/SettingUp_CommandLine.html[here].
 
 Install Swift
 --------------
-To install Swift, follow instructions on swift-lang.org. Note that Swift installation is required only on the submit host and not on cloud nodes.
+To install Swift, follow instructions found http://swift-lang.org/downloads/index.php[here]. Note that Swift
+installation is required only on the submit host and not on cloud nodes. To checkout the current package from svn repository:
 
+----
+svn co https://svn.ci.uchicago.edu/svn/vdl2/SwiftApps/cloud swift-cloud
+cd swift-cloud
+----
+
 Configure
 ----------
 
 environment variables
 ~~~~~~~~~~~~~~~~~~~~~~
 
-Make sure GLOBUS_TCP_PORT_RANGE and GLOBUS_HOSTNAME are defined. For example:
+Make sure `GLOBUS_TCP_PORT_RANGE` and `GLOBUS_HOSTNAME` are defined. For example:
 
+----
 export GLOBUS_TCP_PORT_RANGE=50000,51000
 export GLOBUS_HOSTNAME=churn.mcs.anl.gov
+----
 
 EC2
 ~~~~
 
-ami-id, security group, keypair set up.
+Make sure you have access to an EC2 image `ami-id`, have `security-group` and `keypair` set up. The `security-group` must be set such that the ports defined in `GLOBUS_TCP_PORT_RANGE` are open for both incoming and outgoing traffic. More information on setting up a security group can be found http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html[here].
 
 coaster-service.conf
 ~~~~~~~~~~~~~~~~~~~~~
 
-Configure coaster-service.conf. An example coaster-service.conf can be
-found in the current directory. Make sure to set the EC2XXX variables to
-the correct values as per your account.  Set other environment variables
+Configure `coaster-service.conf` to your environment. An example `coaster-service.conf` can be
+found in the `swift-cloud` directory. Make sure to set the EC2XXX variables to
+the correct values as per your EC2 allocation. For example:
+
+----
+export EC2KEYPAIR=ketan
+export EC2AMI=ami-cf0953a6
+export EC2SECGRP=Swift
+----
+
+Set other environment variables
 e.g:
 
-export WORKER_LOCATION=/home/ubuntu/worker export WORKER_USERNAME=ubuntu
-export LOG_DIR=logs export SHARED_FILESYSTEM=no export SSH_TUNNELING=yes
+----
+export WORKER_LOCATION=/home/ubuntu/worker
+export WORKER_USERNAME=ubuntu
+export LOG_DIR=logs
+export SHARED_FILESYSTEM=no
+export SSH_TUNNELING=yes
+----
 
 Start coaster service
 ----------------------
 
 Start the coaster service using the following command:
 
+----
 start-coaster-service
+----
 
-Above command will generate an apps and sites configuration file. Use these configuration files to run Swift command as shown in the next section.
+Above command will generate an `apps` and `sites.xml` configuration file. Use these configuration files to run Swift command as shown in the next section. Make sure `coaster-service.conf` configuration is available in the current working directory or use the `-conf` option to use config file located elsewhere.
 
-Make sure coaster-service.conf is available in the current working directory or use the "-conf" option to use config file located elsewhere.
+Typical output of the above `start-coaster-service` will look as follows:
 
+----
+Start-coaster-service...
+Configuration: coaster-service.conf
+Service address: localhost
+Starting coaster-service
+Service port: 50000
+Local port: 50001
+Generating sites.xml
+----
+
 Start Swift run
 ----------------
 
-Start swift run. Assuming your Swift scripts, applications and data are set, run a Swift script using swift command as follows:
+Start swift run. Assuming your Swift script, applications and data are set, run using swift command as follows:
 
-   swift -sites.file sites.xml -tc.file apps -config cf catsnsleep.swift -s=10 -n=20
+----
+swift -sites.file sites.xml -tc.file apps -config cf catsnsleep.swift -s=10 -n=20
+----
 
-The example catsnsleep.swift and cf files are provided with the package.
+The example `catsnsleep.swift` and `cf` properties files used in the above
+command are provided with the package. Additionally, an executable `catnap.sh`
+and a small `data.txt` files are provided with the package used by the
+`catsnsleep.swift` script. As a result of the above command the script will run
+20 `app` instances with a sleep time of 10 seconds in each call.
 
 Manage resources
 -----------------
@@ -64,24 +104,78 @@
 Add EC2 compute instances
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Add more resources via addnodes. Use -n to add by number and -a to add by ip address.
+Add more resources via addnodes. Use -n to add by number and -a to add by ip address. For example, the following commandline will add 10 cloud instances to the currently executing run:
 
+----
+./addnodes -n 10
+----
+
+Note that the above command will trigger a create instance command on your EC2
+account and can take upto 2 minutes before the newly created instances can
+actually join the existing Swift execution.
+
+The following command will add two nodes with ip `54.243.24.104` and `54.243.24.154`:
+
+----
+./addnodes -a "54.243.24.104 54.243.24.154"
+----
+
+
 Remove EC2 compute instances
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Remove EC2 resources via remnodes.
+While the execution is on, you can remove EC2 resources via the  `remnodes` commands with similar switches as the `addnodes` command. See the examples below:
 
+----
+./remnodes -n 5
+----
+
+The above command will remove 5 ec2 nodes from the current run. Currently running `apps` will fail and will be retried automatically by Swift on other available nodes.
+
+----
+./remnodes -a "54.243.24.104"
+----
+
+The above command will remove the node with IP `54.243.24.104` from the current run. Note that in place of IP, you can use public DNS name of the instance eg:
+
+----
+./remnodes -a "ec2-50-99-41-60.compute-1.amazonaws.com ec2-50-99-41-56.compute-1.amazonaws.com"
+----
+
+The above command will remove instances
+`ec2-50-99-41-60.compute-1.amazonaws.com` and
+`ec2-50-99-41-56.compute-1.amazonaws.com` from the current run. Note that the
+`./remnodes` command will terminate the instances provided as arguments and
+will terminate the top `n` instances provided as number `n` with its `-n`
+switch.
+
 Monitor resources
 ~~~~~~~~~~~~~~~~~
 
-Monitor Swift tasks and EC2 instances while jobs are running.
+Monitor Swift tasks and EC2 instances while jobs are running with the monitor command.
 
+----
+./monitor
+----
+
 Shut down
 ----------
+Use the stop-coaster-service command to terminate the coaster service and remote worker agents:
 
+----
 stop-coaster-service
-Use the stop-coaster-service command to stop everything.
+----
 
+Typical output of the above command will look as follows:
+
+----
+Stop-coaster-service...
+Configuration: /homes/ketan/cloud/coaster-service.conf
+Ending coaster processes..
+Killing process 23005
+Done
+----
+
 Troubleshoot
 ------------- 
 

Modified: SwiftApps/cloud/coaster-service.conf
===================================================================
--- SwiftApps/cloud/coaster-service.conf	2014-03-09 16:54:34 UTC (rev 7661)
+++ SwiftApps/cloud/coaster-service.conf	2014-03-09 18:39:43 UTC (rev 7662)
@@ -51,7 +51,7 @@
 #app cat=/bin/cat
 #app bash=/bin/bash
 #app echo=/bin/echo
-export WORKERURL="http://localhost:50003"
+export WORKERURL="http://localhost:50001"
 export PID_FILE="/homes/ketan/.swift/.coaster-service-pids"
 export LOG="start-coaster-service.log"
-export LOCAL_PORT="50003"
+export LOCAL_PORT="50001"


