[Swift-commit] r7277 - trunk/docs/siteguide

Thu Nov 7 10:36:53 CST 2013

Author: davidk
Date: 2013-11-07 10:36:53 -0600 (Thu, 07 Nov 2013)
New Revision: 7277

Modified:
   trunk/docs/siteguide/ec2
   trunk/docs/siteguide/futuregrid
Log:
Link to cloud tutorial for ec2 and futuregrid


Modified: trunk/docs/siteguide/ec2
===================================================================

--- trunk/docs/siteguide/ec2	2013-11-07 16:14:13 UTC (rev 7276)
+++ trunk/docs/siteguide/ec2	2013-11-07 16:36:53 UTC (rev 7277)
@@ -1,177 +1,4 @@
 Amazon EC2 Cloud
 ----------------
-Amazon EC2 offers virtualized resources on demand from its cloud data centers.
-This section describes how to use Swift to run applications on EC2 resources.
-More information on EC2 can be found link:https://aws.amazon.com/ec2[here].
-
-Getting Access
-~~~~~~~~~~~~~~
-
-Users with EC2 Allocation
-^^^^^^^^^^^^^^^^^^^^^^^^^
-
-If you already have an EC2 allocation, you can use either the web console or command-line interface to manage resources.
-
-Before using EC2 for Swift you must create a security group in order to allow
-Swift communicate with EC2 instances via open ports. In order to create a
-security group, follow instructions
-link:http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#creating-security-group[here].
-Preferably, open ports between 50000 and 51000.
-
-Paste your credentials from the EC2 web console into AWS_ACCESS_KEY and AWS_SECRET_KEY and export them in your .bashrc as follows:
-
-----
-export AWS_ACCESS_KEY=<your access key>
-export AWS_SECRET_KEY=<your secret key>
-----
-
-Command-line tools are pre-installed on +swift.rcc.uchicago.edu+. In order to use them add the following lines to your .bashrc:
-
-----
-export EC2_HOME=/home/maheshwari/.ec2
-export PATH=$PATH:$EC2_HOME/bin
-----
-
-A public AMI on EC2 is available to create instance. Everything needed to run
-Swift is preinstalled on this image. The id of this AMI is: ami-332e585a.
-
-Create instances from the above AMI as follows:
-
-----
-ec2run -k <keypair_name> -g <sec_group_name> -n <count> ami-332e585a
-----
-
-Users without EC2 Allocation
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Please request access to prepared instances and PKI credentials by mailing: email at mcs.anl.gov.
-
-
-Configuring coaster-service.conf
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-To run on EC2, you will need a file called coaster-service.conf.
-This file contains many options to control how things run. Here is
-an example of a working coaster-service.conf for EC2.
-
------
-#Location of SWIFT. If empty, PATH is referenced
-export SWIFT=
-
-# Where to place/launch worker.pl on the remote machine for sites.xml
-export WORKER_LOCATION=/root
-
-# How to launch workers: local, ssh, or cobalt
-export WORKER_MODE=ssh
-
-# Worker logging setting passed to worker.pl for sites.xml
-export WORKER_LOGGING_LEVEL=DEBUG
-
-export SSH_TUNNELING=yes
-
-# User name to use for all systems
-export WORKER_USERNAME=root
-
-# Worker host names for ssh
-# export WORKER_HOSTS="crush thwomp" 
-
-# Directory to keep log files,
-#relative to working directory when launching start-coaster-service
-export LOG_DIR=logs
-export WORKER_LOG_DIR=/root
-
-# Manually define ports. If not specified, ports will be automatically generated
-export LOCAL_PORT=50100
-export SERVICE_PORT=50200
-
-# Set shared filesystem to no since work will be done in local /sandbox directory
-export SHARED_FILESYSTEM=no
-
-# Below are various settings to give information about how to create sites.xml
-export WORK=/tmp/swift.work
-export JOBSPERNODE=4
-export JOBTHROTTLE=$( echo "scale=5; ($JOBS_PER_NODE * $( echo $WORKER_HOSTS|wc -w ))/100 - 0.00001"|bc )
-
-# Swift applications
-#app cat=/bin/cat
-#app bash=/bin/bash
------
-Add a list of any applications you want to run in the format "#app myapp=/path/to/app".
-
-Starting the Coaster Service Script
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Log in to the +swift.rcc.uchicago.edu+ and copy the swift_cloud directory to your home:
-
-----
-cp -r ~maheshwari/swift_cloud $HOME/
-----
-
-Make sure Swift module is loaded and your credentials are added as follows:
-
-----
-module load swift
-----
-
-----
-ssh-add
-----
-
-Replace the hosts.txt found in +swift_cloud+ directory by adding the address/DNS of instances.
-
-With the ec2 commandline, the list of ip addresses of running instances can be obtained and added to hosts.txt as follows:
-
-----
-ec2din | grep running | awk '{print $15}' > hosts.txt
-----
-
-Now that everything is configured, run this command to start the coaster service:
-
------
-source setup
------
-
-This command will add the instances to coaster configuration, start the
-required processes on the worker nodes, and generate Swift configuration files
-for you to use. The configuration files will be generated in your current
-directory. These files are sites.xml, tc.data, and cf.
-
-Running Swift
-~~~~~~~~~~~~~
-
-
-
-Now that you have all of your configuration files generated, run the following command:
-
------
-$ swift -sites.file sites.xml -tc.file tc.data -config cf catsn.swift
------
-
-If you like to create a custom tc and/or cf file for repeated use, rename it to something other
-than tc.data/cf to prevent it from being overwritten. The sites.xml however will need to be
-regenerated every time you start the coaster service. If you need to repeatedly modify some
-sites.xml options, you may edit the template in Swift's etc/sites/persistent-coasters. You
-may also create your own custom tc files with the hostname of persistent-coasters. More
-information about this can be found in the Swift userguide at
-http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html.
-
-Stopping the Coaster Service Script
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-To stop the coaster service, run the following command:
------
-$ stop-coaster-service
------
-
-This will kill the coaster service, kill the worker scripts on remote systems.
-
-Terminate the virtual machines that were created for this work as follows:
-
-----
-ec2din|grep running|awk '{print $2}'|xargs ec2kill
-----
-
-More Help
-~~~~~~~~~
-The best place for additional help is the Swift user mailing list. You can subscribe to this list at
-http://mail.ci.uchicago.edu/mailman/listinfo/swift-user. When submitting information, send
-your sites.xml file, your tc.data, and any error messages you run into.
-
+Please refer to the
+link:../../../tutorials/cloud/tutorial.html[Swift Tutorial for Cloud and Ad hoc Resources].

Modified: trunk/docs/siteguide/futuregrid
===================================================================
--- trunk/docs/siteguide/futuregrid	2013-11-07 16:14:13 UTC (rev 7276)
+++ trunk/docs/siteguide/futuregrid	2013-11-07 16:36:53 UTC (rev 7277)
@@ -1,139 +1,4 @@
 Futuregrid Cloud
 ----------------
-The NSF-funded FutureGrid cloud is administered by Indiana University. It
-offers a variety of resources via a multitude of interfaces. Currently, it
-offers cloud resources via three different interfaces: Eucalyptus,
-Nimbus (www.nimbusproject.org), and
-OpenStack (www.openstack.org). The total number of resources at
-FutureGrid is close to 5000 CPU cores and 220~TB of storage from more than six
-physical clusters. We use the resources offered by one such cluster via the
-Nebula middleware.
-
-More information on futuregrid can be found link:https://portal.futuregrid.org/[here].
-
-Requesting Futuregrid Access
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-If you do not already have a futuregrid account, you can follow the
-instructions link:https://portal.futuregrid.org/gettingstarted[here] to get started.
-This page provides information on how to create an account, how to join
-a project, how to set up your SSH keys, and how to create a new project.
-
-Downloading Swift VM Tools
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-A set of scripts based around cloudinitd are used to easily start virtual
-machines. To download, change to your home directory and run the
-following command:
-
------
-$ svn co https://svn.ci.uchicago.edu/svn/vdl2/usertools/swift-vm-boot swift-vm-boot
------
-
-Download your Credentials
-~~~~~~~~~~~~~~~~~~~~~~~~~
-Run the following commands to retrieve your credentials:
-
------
-$ cd swift-vm-boot
-$ scp yourusername at hotel.futuregrid.org:nimbus_creds.tar.gz .
-$ tar xvfz nimbus_creds.tar.gz
------
-
-When you extract your credential file, look at the file called
-hotel.conf. Near the bottom of this file will be two settings
-called vws.repository.s3id and vws.repository.s3key. Copy these
-values for the next step.
-
-Configuring coaster-service.conf
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-To run on futuregrid, you will need a file called coaster-service.conf.
-This file contains many options to control how things run. Here is
-an example of a working coaster-service.conf on futuregrid.
-
------
-# Where to copy worker.pl on the remote machine for sites.xml
-export WORKER_LOCATION=/tmp
-
-# How to launch workers: local, ssh, cobalt, or futuregrid
-export WORKER_MODE=futuregrid
-
-# Do all the worker nodes you're using have a shared filesystem? (yes/no)
-export SHARED_FILESYSTEM=no
-
-# Username to use on worker nodes
-export WORKER_USERNAME=root
-
-# Enable SSH tunneling? (yes/no)
-export SSH_TUNNELING=yes
-
-# Directory to keep log files, relative to working directory when launching start-coaster-service
-export LOG_DIR=logs
-
-# Location of the swift-vm-boot scripts
-export SWIFTVMBOOT_DIR=$HOME/swift-vm-boot
-
-# Futuregrid settings
-export FUTUREGRID_IAAS_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXX
-export FUTUREGRID_IAAS_SECRET_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
-export FUTUREGRID_HOTEL_NODES=0
-export FUTUREGRID_SIERRA_NODES=2
-export FUTUREGRID_CPUS_PER_NODE=1
-
-# Swift information for creating sites.xml
-export WORK=/tmp
-export JOBS_PER_NODE=$FUTUREGRID_CPUS_PER_NODE
-export JOB_THROTTLE=$( echo "scale=5; ($JOBS_PER_NODE * $(($FUTUREGRID_HOTEL_NODES + $FUTUREGRID_SIERRA_NODES)))/100 - 0.00001"|bc )
-
-# Application locations for tc.data
-#app convert=/usr/bin/convert
------
-
-Paste your credentials from the hotel.conf file into the FUTUREGRID_IAAS_ACCESS_KEY and FUTUREGRID_IAAS_SECRET_KEY fields.
-Adjust the number of nodes you would like to allocate here by changing the values of FUTUREGRID_HOTEL_NODES and FUTUREGRID_SIERRA_NODES.
-Add a list of any applications you want to run in the format "#app myapp=/path/to/app".
-
-Starting the Coaster Service Script
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Now that everything is configured, change to the location of the coaster-service.conf file
-and run this command to start the coaster service:
-
------
-$ start-coaster-service
------
-
-This command will start the VMs, start the required processes on the worker nodes,
-and generate Swift configuration files for you to use. The configuration files
-will be generated in your current directory. These files are sites.xml, tc.data,
-and cf.
-
-Running Swift
-~~~~~~~~~~~~~
-Now that you have all of your configuration files generated, run the following command:
-
------
-$ swift -sites.file sites.xml -tc.file tc.data -config cf <yourscript.swift>
------
-
-If you like to create a custom tc and/or cf file for repeated use, rename it to something other
-than tc.data/cf to prevent it from being overwritten. The sites.xml however will need to be
-regenerated every time you start the coaster service. If you need to repeatedly modify some
-sites.xml options, you may edit the template in Swift's etc/sites/persistent-coasters. You
-may also create your own custom tc files with the hostname of persistent-coasters. More
-information about this can be found in the Swift userguide at
-http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html.
-
-Stopping the Coaster Service Script
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-To stop the coaster service, run the following command:
------
-$ stop-coaster-service
------
-
-This will kill the coaster service, kill the worker scripts on remote systems and terminate
-the virtual machines that were created during start-coaster-service.
-
-More Help
-~~~~~~~~~
-The best place for additional help is the Swift user mailing list. You can subscribe to this list at
-http://mail.ci.uchicago.edu/mailman/listinfo/swift-user. When submitting information, send
-your sites.xml file, your tc.data, and any error messages you run into.
-
+Please refer to the 
+link:../../../tutorials/cloud/tutorial.html[Swift Tutorial for Cloud and Ad hoc Resources].


