[Swift-commit] r6543 - trunk/docs/siteguide

[ketan at ci.uchicago.edu]

Author: ketan
Date: 2013-06-13 11:31:22 -0500 (Thu, 13 Jun 2013)
New Revision: 6543

Added:
   trunk/docs/siteguide/ec2
Log:
first cut for running Swift on EC2 resources

Added: trunk/docs/siteguide/ec2
===================================================================
--- trunk/docs/siteguide/ec2	                        (rev 0)
+++ trunk/docs/siteguide/ec2	2013-06-13 16:31:22 UTC (rev 6543)
@@ -0,0 +1,116 @@
+Amazon EC2 Cloud
+----------------
+Amazon EC2 offers virtualized resources on demand from its cloud data centers.
+This section describes how to use Swift to run applications on EC2 resources.
+More information on EC2 can be found link:https://aws.amazon.com/ec2[here].
+
+Getting Access
+~~~~~~~~~~~~~~
+
+==Users with EC2 Allocation
+
+If you already have an EC2 allocation, you can use either the web console or command-line interface to manage resources.
+
+Before using EC2 for Swift you must create a security group in order to allow Swift communicate with EC2 instances via open ports. In order to create a security group, follow instructions link:http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#creating-security-group[here]. Preferably, open ports between 50000 and 51000.
+
+Paste your credentials from the EC2 web console into AWS_ACCESS_KEY and AWS_SECRET_KEY and export them in your .bashrc as follows:
+----
+export AWS_ACCESS_KEY=<your access key>
+export AWS_SECRET_KEY=<your secret key>
+----
+
+A public AMI for Swift user is available with id: ami-332e585a.
+
+Create instances from the above AMI as follows:
+
+----
+ec2run -k <keypair_name> -g <sec_group_name> -n <count> ami-332e585a
+----
+
+==Users without EC2 Allocation
+
+Configuring coaster-service.conf
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+To run on EC2, you will need a file called coaster-service.conf.
+This file contains many options to control how things run. Here is
+an example of a working coaster-service.conf for EC2.
+
+-----
+# Where to copy worker.pl on the remote machine for sites.xml
+export WORKER_LOCATION=/tmp
+
+# How to launch workers: local, ssh, cobalt
+export WORKER_MODE=ssh
+
+# Do all the worker nodes you're using have a shared filesystem? (yes/no)
+export SHARED_FILESYSTEM=no
+
+# Username to use on worker nodes
+export WORKER_USERNAME=root
+
+# Enable SSH tunneling? (yes/no)
+export SSH_TUNNELING=yes
+
+# Directory to keep log files, relative to working directory when launching start-coaster-service
+export LOG_DIR=logs
+
+# Swift information for creating sites.xml
+export WORK=/tmp
+
+# Application locations for tc.data
+#app convert=/usr/bin/convert
+-----
+
+Add a list of any applications you want to run in the format "#app myapp=/path/to/app".
+
+Starting the Coaster Service Script
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Now that everything is configured, change to the location of the coaster-service.conf file
+and run this command to start the coaster service:
+
+-----
+$ start-coaster-service
+-----
+
+This command will start the VMs, start the required processes on the worker nodes,
+and generate Swift configuration files for you to use. The configuration files
+will be generated in your current directory. These files are sites.xml, tc.data,
+and cf.
+
+Running Swift
+~~~~~~~~~~~~~
+Now that you have all of your configuration files generated, run the following command:
+
+-----
+$ swift -sites.file sites.xml -tc.file tc.data -config cf <yourscript.swift>
+-----
+
+If you like to create a custom tc and/or cf file for repeated use, rename it to something other
+than tc.data/cf to prevent it from being overwritten. The sites.xml however will need to be
+regenerated every time you start the coaster service. If you need to repeatedly modify some
+sites.xml options, you may edit the template in Swift's etc/sites/persistent-coasters. You
+may also create your own custom tc files with the hostname of persistent-coasters. More
+information about this can be found in the Swift userguide at
+http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html.
+
+Stopping the Coaster Service Script
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+To stop the coaster service, run the following command:
+-----
+$ stop-coaster-service
+-----
+
+This will kill the coaster service, kill the worker scripts on remote systems.
+
+Terminate the virtual machines that were created for this work as follows:
+
+----
+ec2din|grep running|awk '{print $2}'|xargs ec2kill
+----
+
+More Help
+~~~~~~~~~
+The best place for additional help is the Swift user mailing list. You can subscribe to this list at
+http://mail.ci.uchicago.edu/mailman/listinfo/swift-user. When submitting information, send
+your sites.xml file, your tc.data, and any error messages you run into.
+