[Swift-commit] r5468 - trunk/docs/siteguide
davidk at ci.uchicago.edu
davidk at ci.uchicago.edu
Fri Dec 23 22:52:58 CST 2011
Author: davidk
Date: 2011-12-23 22:52:58 -0600 (Fri, 23 Dec 2011)
New Revision: 5468
Modified:
trunk/docs/siteguide/fusion
trunk/docs/siteguide/intrepid
trunk/docs/siteguide/mcs
trunk/docs/siteguide/pads
Log:
Commit siteguide changes from 0.93 to trunk
Modified: trunk/docs/siteguide/fusion
===================================================================
--- trunk/docs/siteguide/fusion 2011-12-24 04:32:25 UTC (rev 5467)
+++ trunk/docs/siteguide/fusion 2011-12-24 04:52:58 UTC (rev 5468)
@@ -1,13 +1,49 @@
Fusion
-----
+------
+Fusion is a 320-node computing cluster for the Argonne
+National Laboratory community. The primary goal of the LCRC is to
+facilitate mid-range computing in all of the scientific programs of
+Argonne and the University of Chicago.
-Fusion Quickstart
-~~~~~~~~~~~~~~~
This section will walk you through running a simple Swift script
on Fusion.
+Requesting Access
+~~~~~~~~~~~~~~~~~
+If you do not already have a Fusion account, you can request one at
+https://accounts.lcrc.anl.gov/request.php. Email support at lcrc.anl.gov
+for additional help.
+
+Projects
+~~~~~~~~
+In order to run a job on a Fusion compute node, you must first be associated
+with a project.
+
+Each project has one or more Primary Investigators, or PIs. These PIs are
+responsible for adding and removing users to a project. Contact the PI of
+your project to be added.
+
+More information on this process can be found at
+http://www.lcrc.anl.gov/info/Projects.
+
+SSH Keys
+~~~~~~~~
+Before accessing Fusion, be sure to have your SSH keys configured correctly.
+SSH keys are required to access fusion. You should see information about
+this when you request your account. Email support at lcrc.anl.gov for
+additional help.
+
+Connecting to a login node
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+Once your keys are configured, you should be able to access a Fusion login
+node with the following command:
+
+-----
+ssh yourusername at fusion.lcrc.anl.gov
+-----
+
Creating sites.xml
-^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~
Swift relies on various configuration files to determine how to
run. This section will provide a working configuration file which
you can copy and paste to get running quickly. The sites.xml file
@@ -29,7 +65,7 @@
-----
Creating tc.data
-^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~
The tc.data configuration file gives information about the applications
that will be called by Swift. More information about the format
of tc.data can be found in the Swift User's guide.
@@ -41,8 +77,7 @@
-----
Copy a Swift Script
-^^^^^^^^^^^^^^^^^^^
-
+~~~~~~~~~~~~~~~~~~~~
Within the Swift directory is an examples directory which contains
several introductory Swift scripts. The example we will use in this
section is called catsn.swift. Copy this script to the same directory
@@ -55,8 +90,7 @@
TIP: The location of your swift directory may vary depending on how you installed it. Change this to the examples/misc directory of your installation as needed.
Run Swift
-^^^^^^^^^
-
+~~~~~~~~~
Finally, run the script
-----
$ swift -sites.file sites.xml -tc.file tc.data catsn.swift
@@ -65,40 +99,6 @@
You should see 10 new text files get created, named catsn*.out. If
you see these files, then you have succesfully run Swift on Fusion!
-Requesting Access
-~~~~~~~~~~~~~~~~~
-If you do not already have a Fusion account, you can request one at
-https://accounts.lcrc.anl.gov/request.php. Email support at lcrc.anl.gov
-for additional help.
-
-SSH Keys
-~~~~~~~~
-Before accessing Fusion, be sure to have your SSH keys configured correctly.
-SSH keys are required to access fusion. You should see information about
-this when you request your account. Email support at lcrc.anl.gov for
-additional help.
-
-Connecting to a login node
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-Once your keys are configured, you should be able to access a Fusion login
-node with the following command:
-
------
-ssh yourusername at fusion.lcrc.anl.gov
------
-
-Projects
-~~~~~~~~
-In order to run a job on a Fusion compute node, you must first be associated
-with a project.
-
-Each project has one or more Primary Investigators, or PIs. These PIs are
-responsible for adding and removing users to a project. Contact the PI of
-your project to be added.
-
-More information on this process can be found at
-http://www.lcrc.anl.gov/info/Projects.
-
Queues
~~~~~~
Fusion has two queues: shared and batch. The shared queue has a maximum 1
Modified: trunk/docs/siteguide/intrepid
===================================================================
--- trunk/docs/siteguide/intrepid 2011-12-24 04:32:25 UTC (rev 5467)
+++ trunk/docs/siteguide/intrepid 2011-12-24 04:52:58 UTC (rev 5468)
@@ -1,5 +1,8 @@
Intrepid
--------
+Intrepid is an IBM Blue Gene/P supercomputer located at the Argonne Leadership
+Computing Facility. More information on Intrepid can be found at
+http://www.alcf.anl.gov/.
Requesting Access
~~~~~~~~~~~~~~~~~
Modified: trunk/docs/siteguide/mcs
===================================================================
--- trunk/docs/siteguide/mcs 2011-12-24 04:32:25 UTC (rev 5467)
+++ trunk/docs/siteguide/mcs 2011-12-24 04:52:58 UTC (rev 5468)
@@ -1,5 +1,7 @@
MCS Workstations
----------------
+This sections describes how to use the general use compute servers for
+the MCS division of Argonne National Laboratory.
Create a coaster-service.conf
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -10,79 +12,6 @@
include::../../tests/providers/mcs/coaster-service.conf[]
-----
-SSH Configuration
-~~~~~~~~~~~~~~~~~
-For this configuration to work, you must be able to log into each of the MCS workstations
-using your SSH keys. Copy your id_rsa and id_rsa.pub files into the .ssh/ directory
-on login.mcs.anl.gov. Run the following commands to make sure your keys are private:
-
------
-chmod 600 id_rsa
-chmod 644 id_rsa.pub
------
-
-Once your SSH keys are in place, create a file in .ssh/ called auth.defaults. Paste the
-following text into the file:
-
------
-crush.mcs.anl.gov.type=key
-crush.mcs.anl.gov.username=yourusername
-crush.mcs.anl.gov.key=/home/yourusername/.ssh/id_rsa
-crush.mcs.anl.gov.passphrase=your passphrase here
-
-thwomp.mcs.anl.gov.type=key
-thwomp.mcs.anl.gov.username=yourusername
-thwomp.mcs.anl.gov.key=/home/yourusername/.ssh/id_rsa
-thwomp.mcs.anl.gov.passphrase=your passphrase here
-
-stomp.mcs.anl.gov.type=key
-stomp.mcs.anl.gov.username=yourusername
-stomp.mcs.anl.gov.key=/home/yourusername/.ssh/id_rsa
-stomp.mcs.anl.gov.passphrase=your passphrase here
-
-crank.mcs.anl.gov.type=key
-crank.mcs.anl.gov.username=yourusername
-crank.mcs.anl.gov.key=/home/yourusername/.ssh/id_rsa
-crank.mcs.anl.gov.passphrase=your passphrase here
-
-steamroller.mcs.anl.gov.type=key
-steamroller.mcs.anl.gov.username=yourusername
-steamroller.mcs.anl.gov.key=/home/yourusername/.ssh/id_rsa
-steamroller.mcs.anl.gov.passphrase=your passphrase here
-
-grind.mcs.anl.gov.type=key
-grind.mcs.anl.gov.username=yourusername
-grind.mcs.anl.gov.key=/home/yourusername/.ssh/id_rsa
-grind.mcs.anl.gov.passphrase=your passphrase here
-
-churn.mcs.anl.gov.type=key
-churn.mcs.anl.gov.username=yourusername
-churn.mcs.anl.gov.key=/home/yourusername/.ssh/id_rsa
-churn.mcs.anl.gov.passphrase=your passphrase here
-
-trounce.mcs.anl.gov.type=key
-trounce.mcs.anl.gov.username=yourusername
-trounce.mcs.anl.gov.key=/home/yourusername/.ssh/id_rsa
-trounce.mcs.anl.gov.passphrase=your passphrase here
-
-thrash.mcs.anl.gov.type=key
-thrash.mcs.anl.gov.username=yourusername
-thrash.mcs.anl.gov.key=/home/yourusername/.ssh/id_rsa
-thrash.mcs.anl.gov.passphrase=your passphrase here
-
-vanquish.mcs.anl.gov.type=key
-vanquish.mcs.anl.gov.username=yourusername
-vanquish.mcs.anl.gov.key=/home/yourusername/.ssh/id_rsa
-vanquish.mcs.anl.gov.passphrase=your passphrase here
------
-
-Replace the username, key path, and passphrase with your values.
-Change the permissions on the file to keep it private.
-
------
-chmod 600 auth.default
------
-
Starting the Coaster Service
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Modified: trunk/docs/siteguide/pads
===================================================================
--- trunk/docs/siteguide/pads 2011-12-24 04:32:25 UTC (rev 5467)
+++ trunk/docs/siteguide/pads 2011-12-24 04:52:58 UTC (rev 5468)
@@ -1,17 +1,41 @@
PADS
----
+PADS is a petabyte-scale, data intense computing resource located
+at the joint Argonne National Laboratory/University of Chicago
+Computation Institute. More information about PADS can be found
+at http://pads.ci.uchicago.edu.
-PADS Quickstart
-~~~~~~~~~~~~~~~
-This section will walk you through running a simple Swift script
-on PADS. It will explain how to setup software packages, how
-to create the required Swift configuration files, and finally
-how to run Swift.
+Requesting Access
+~~~~~~~~~~~~~~~~~
+If you do not already have a Computation Institute account, you can request
+access at https://www.ci.uchicago.edu/accounts. This page will give you a list
+of resources you can request access to. Be sure that PADS is selected. If
+you already have an existing CI account, but do not have access to PADS,
+send an email to support at ci.uchicago.edu to request access.
+SSH Keys
+~~~~~~~~
+Before accessing PADS, be sure to have your SSH keys configured correctly.
+There is some basic information about SSH and how to generate your key at
+http://www.ci.uchicago.edu/wiki/bin/view/Resources/SshKeys. Once you have
+followed those instructions, you can add your key at
+https://www.ci.uchicago.edu/support/sshkeys/.
+
+Connecting to a login node
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+Once your keys are configured, you should be able to access a PADS login
+node with the following command:
+
+-----
+ssh yourusername at login.pads.ci.uchicago.edu
+-----
+
Adding Software Packages
-^^^^^^^^^^^^^^^^^^^^^^^^
-In your home directory, edit a file called .soft and add these
-lines (in this order):
+~~~~~~~~~~~~~~~~~~~~~~~~
+Softenv is a system used for managing applications. In order to run Swift,
+the softenv environment will have to be modified slightly. Softenv is
+configured by a file in your home directory called .soft. Edit this file
+to look like this:
-----
+java-sun
+maui
@@ -21,6 +45,37 @@
Log out of PADS, and log back in for these changes to take effect.
+Which project(s) are you a member of?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+PADS requires that you are a member of a project. You can determine this by
+running the following command:
+
+-----
+$ projects --available
+
+The following projects are available for your use
+
+Project PI Title
+CI-CCR000013 Michael Wilde The Swift Parallel Scripting System
+-----
+
+If you are not a member of a project, you must first request access
+to a project at http://www.ci.uchicago.edu/hpc/projects.
+
+You should make sure that you have a project set as default. Run
+the projects command with no arguments to determine if you have a default.
+
+------
+$ projects
+You have no default project set.
+-----
+
+To set your default project, use projects --set
+------
+$ projects --set CI-CCR000013 --all
+Your default project for all CI clusters has been set to CI-CCR000013.
+-----
+
Creating sites.xml
^^^^^^^^^^^^^^^^^^
Swift relies on various configuration files to determine how to
@@ -33,7 +88,7 @@
The first step is to paste the text below into a file named sites.xml.
-----
-include::../../tests/providers/pads/coasters/sites.template.xml[]
+include::../../tests/providers/PADS/coasters/sites.template.xml[]
-----
This file will require just a few customizations. First, create a
@@ -52,12 +107,11 @@
Paste the following example into a file named tc.data
-----
-include::../../tests/providers/pads/coasters/tc.template.data[]
+include::../../tests/providers/PADS/coasters/tc.template.data[]
-----
Copy a Swift Script
^^^^^^^^^^^^^^^^^^^
-
Within the Swift directory is an examples directory which contains
several introductory Swift scripts. The example we will use in this
section is called catsn.swift. Copy this script to the same directory
@@ -69,6 +123,7 @@
-----
TIP: The location of your swift directory may vary depending on how you installed it. Change this to the examples/misc directory of your installation as needed.
+
Run Swift
^^^^^^^^^
@@ -77,99 +132,21 @@
$ swift -sites.file sites.xml -tc.file tc.data catsn.swift
-----
-You should see 10 new text files get created, named catsn*.out. If
-you see these files, then you have succesfully run Swift on PADS!
+You should see several new files being created, called catsn.0001.out, catsn.0002.out, etc. Each of these
+files should contain the contents of what you placed into data.txt. If this happens, your job has run
+successfully on PADS!
TIP: Make sure your default project is defined. Read on for more information.
Read on for more detailed information about running Swift on PADS.
-Requesting Access
-~~~~~~~~~~~~~~~~~
-If you do not already have a Computation Institute account, you can request
-one at https://www.ci.uchicago.edu/accounts. This page will give you a list
-of resources you can request access to. Be sure that PADS is selected. If
-you already have an existing CI account, but do not have access to PADS,
-send an email to support at ci.uchicago.edu to request access.
-SSH Keys
-~~~~~~~~
-Before accessing PADS, be sure to have your SSH keys configured correctly.
-There is some basic information about SSH and how to generate your key at
-http://www.ci.uchicago.edu/wiki/bin/view/Resources/SshKeys. Once you have
-followed those instructions, you can add your key at
-https://www.ci.uchicago.edu/support/sshkeys/.
+Queues
+^^^^^^
-Connecting to a login node
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-Once your keys are configured, you should be able to access a PADS login
-node with the following command:
+As you run more application in the future, you will likely need
+to change queues.
------
-ssh yourusername at login.pads.ci.uchicago.edu
------
-
-Configuring softenv
-~~~~~~~~~~~~~~~~~~~
-Softenv is a system used for managing applications. In order to run Swift,
-the softenv environment will have to be modified slightly. Softenv is
-configured by a file in your home directory called .soft. Edit this file
-to look like this:
------
-+java-sun
-+maui
-+torque
- at default
------
-
-Swift Work Directory
-^^^^^^^^^^^^^^^^^^^^
-The Swift work directory is a directory which Swift uses for processing work.
-This directory needs to be writable and available to all worker nodes on
-a shared filesystem.
-
-.Examples
------
-/home/username/swiftwork
-/home/username/work
-/autonfs/gpfs-pads/projects/CI-CCR000013/myusername
------
-
-Which project(s) are you a member of?
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-PADS requires that you are a member of a project. You can determine this by
-running the following command:
-
------
-$ projects --available
-
-The following projects are available for your use
-
-Project PI Title
-
-CI-CCR000013 Michael Wilde The Swift Parallel Scripting System
------
-
-If you are not a member of a project, you must first request access
-to a project at http://www.ci.uchicago.edu/hpc/projects.
-
-You should make sure that you have a project set as default. Run
-the projects command with no arguments to determine if you have a default.
-
-------
-$ projects
-You have no default project set.
------
-
-To set your default project, use projects --set
-------
-$ projects --set CI-CCR000013 --all
-Your default project for all CI clusters has been set to CI-CCR000013.
------
-
-Determine your Queue
-^^^^^^^^^^^^^^^^^^^^
-
PADS has several different queues you can submit jobs to depending on
the type of work you will be doing. The command "qstat -q" will print
the most up to date list of this information.
@@ -185,59 +162,13 @@
|long |-- |-- |24:00:00|-- |232|130|-- | E R
|=========================================================
-Generating Configuration Files
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Now that you know what queue to use, your project, and your work directory, it is time to
-set up Swift. Swift uses a configuration file called sites.xml to determine how it should run.
-There are two methods you can use for creating this file. You can manually edit
-the configuration file, or generate it with a utility called gensites.
+When you determine your computing requirements, modify this line in your
+sites.xml:
-Manually Editing sites.xml
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-Below is the template that is used by Swift's test suite for running on PADS.
-
-----
-include::../../tests/providers/pads/coasters/sites.template.xml[]
+<profile key="queue" namespace="globus">fast</profile>
-----
-Before you can use this configuration file, you will need to modify \_WORK_ to a directory writable by you,
-and have your default project defined. Copy this template, replace \_WORK_, and then save as sites.xml.
-
-If you wanted to use a different queue, replace "fast" with the desired queue name.
-
-Manually Editing tc.data
-~~~~~~~~~~~~~~~~~~~~~~~~
-Below is the tc.data file used by Swift's test suite for running on PADS.
-
------
-include::../../tests/providers/pads/coasters/tc.template.data[]
-------
-
-Copy these commands and save it as tc.data.
-
-Catsn.swift
-~~~~~~~~~~~
-The swift script we will run is called catsn.swift. It simply cats a file and saves the result. This is
-a nice simple test to ensure jobs are running correctly. Create a file called data.txt which contains
-some simple input - a "hello world" will do the trick.
-
------
-include::../../examples/misc/catsn.swift[]
------
-
-
-Running Swift
-~~~~~~~~~~~~~
-Now that everything is in place, run Swift with the following command:
-
------
-swift -sites.file sites.xml -tc.file tc.data catsn.swift -n=10
------
-
-You should see several new files being created, called catsn.0001.out, catsn.0002.out, etc. Each of these
-files should contain the contents of what you placed into data.txt. If this happens, your job has run
-successfully on PADS!
-
More Help
~~~~~~~~~
The best place for additional help is the Swift user mailing list. You can subscribe to this list at
More information about the Swift-commit
mailing list