[Swift-commit] r4303 - test/asciidoc-manual

wozniak at ci.uchicago.edu wozniak at ci.uchicago.edu
Thu Apr 7 14:22:27 CDT 2011 

Author: wozniak
Date: 2011-04-07 14:22:27 -0500 (Thu, 07 Apr 2011)
New Revision: 4303

Modified:
   test/asciidoc-manual/README.txt
   test/asciidoc-manual/manual.txt
Log:
Add in CDM section


Modified: test/asciidoc-manual/README.txt
===================================================================
--- test/asciidoc-manual/README.txt	2011-04-07 15:11:47 UTC (rev 4302)
+++ test/asciidoc-manual/README.txt	2011-04-07 19:22:27 UTC (rev 4303)
@@ -10,4 +10,11 @@
 asciidoc manual.txt
 which generates manual.html
 
-Word-wrap at column 70 via emacs ESC-q (fill-paragraph)
+Word-wrap at column 70 via emacs ESC-q (fill-paragraph).
+
+To get word-wrap to work well in bullet and numbered lists, you must
+provide a blank line between the items.
+
+Numbered lists are not working right in some cases after a code
+listing block- that's why there are some uses of the [start=*]
+attribute.

Modified: test/asciidoc-manual/manual.txt
===================================================================
--- test/asciidoc-manual/manual.txt	2011-04-07 15:11:47 UTC (rev 4302)
+++ test/asciidoc-manual/manual.txt	2011-04-07 19:22:27 UTC (rev 4303)
@@ -25,3 +25,276 @@
 of input and output files to and from the chosen sites and remote
 execution of program code.
 
+Collective Data Management
+--------------------------
+
+Overview
+~~~~~~~~
+
+The basic idea with Collective Data Management (CDM) is to reconfigure
+the way a Swift script accesses data by:
+
+1. Moving user data using efficient, possibly site-specific
+techniques; and
+
+2. Dynamically renaming user data files so that the the Swift script
+works without modifying it.
+
+The data movement technique is called the CDM "policy".  Renaming and policy
+selection are handled in an optional file called the CDM file.
+
+Key usage points
+~~~~~~~~~~~~~~~~
+
+1. The user specifies a CDM policy in a file, conventionally called
+   +fs.data+.
+
+2. +fs.data+ is given to Swift on the command line.
+
+3. At job launch time, the Swift job launch procedures query the
+   CDM policy,
+
+   .. altering the file staging phase, and
+   .. sending +fs.data+ to the compute site.
+
+4. At job run time, the Swift wrapper script
+
+   .. consults a Perl script to obtain policy, and
+   .. uses wrapper extensions to modify data movement.
+
+5. Similarly, stage out can be changed.
+
+Example command line
+^^^^^^^^^^^^^^^^^^^^
+---------------------------------------------------------------------------
+swift -sites.file sites.xml -tc.file tc.data -cdm.file fs.data stream.swift
+---------------------------------------------------------------------------
+
+CDM policy file format
+~~~~~~~~~~~~~~~~~~~~~~
+
+Example:
+
+------------------------------------------------
+# Describe CDM for my job
+property GATHER_LIMIT 1
+rule .*input.txt DIRECT /gpfs/homes/wozniak/data
+rule .*xfile*.data BROADCAST /dev/shm
+rule .* DEFAULT
+------------------------------------------------
+
+The lines contain:
+
+1. A directive, either rule or property
+   .. A rule has:
+   ... A regular expression
+   ... A policy token
+   .. Additional policy-specific arguments
+   .. A property has
+   ... A policy property token
+   ... The token value
+2. Comments with +#+ .
+3. Blank lines are ignored.
+
+Notes:
+
+1. The policy file is used as a lookup database by Swift and Perl methods.
+
+2. For example, a lookup with the database above given the argument
+input.txt would result in the +DIRECT+ policy.
+
+3. If the lookup does not succeed, the result is +DEFAULT+.
+
+4. Policies are listed as subclasses of
++org.globus.swift.data.policy.Policy+.
+
+Policy descriptions
+~~~~~~~~~~~~~~~~~~~
+
++DEFAULT+
+^^^^^^^^^
+
+* Just use file staging as provided by Swift/CoG.  Identical to
+  behavior if no CDM file is given.
+
++BROADCAST+
+^^^^^^^^^^^
+-------------------------------------
+rule .*xfile*.data BROADCAST /dev/shm
+-------------------------------------
+
+* The input files matching the pattern will be stored in the given
+  directory, an LFS location, with links in the job directory.
+
+* On the BG/P, this will make use of the +f2cn+ tool.
+
+* On machines not implementing an efficient broadcast, we will just
+  ensure correctness.  For example, on a workstation, the local location
+  could be in a +/tmp+ RAM FS, and we would just execute a shell function
+  to get the data there via +dd+.
+
++DIRECT+
+^^^^^^^^
+---------------------------------------------
+rule .*input.txt DIRECT /gpfs/scratch/wozniak
+---------------------------------------------
+
+* Allows for direct I/O to the parallel FS without staging.
+
+* The input files matching the pattern must already exist in the given
+  directory, a GFS location.  Links will be placed in the job
+  directory.
+
+* The output files matching the pattern will be stored in the given
+  directory, with links in the job directory.
+
+* Example: In the rule above, the Swift-generated file name
+  +./data/input.txt+ would be accessed by the user job in
+  +/gpfs/homes/wozniak/data/input.txt+ .
+
++LOCAL+
+^^^^^^^
+----------------------------------------------------------
+rule .*input.txt LOCAL dd /gpfs/homes/wozniak/data obs=64K
+----------------------------------------------------------
+
+* Allows for client-directed input copy to the compute node.
+
+* The user may specify +cp+ or +dd+ as the input transfer program.
+
+* The input files matching the pattern must already exist in the given
+  directory, a GFS location.  Copies will be placed in the job
+  directory.
+
+* Argument list: [tool] [GFS directory] [tool arguments]*
+
++GATHER+
+^^^^^^^^
+-------------------------------------------------------
+property GATHER_LIMIT 500000000 # 500 MB
+property GATHER_DIR /dev/shm/gather
+property GATHER_TARGET /gpfs/wozniak/data/gather_target
+rule .*.output.txt GATHER
+-------------------------------------------------------
+
+* The output files matching the pattern will be present to tasks in
+  the job directory as usual but noted in a +_swiftwrap+ shell array
+  +GATHER_OUTPUT+.
+
+* The +GATHER_OUTPUT+ files will be cached in the +GATHER_DIR+, an LFS
+  location.
+
+* The cache will be flushed when a job ends if a +du+ on +GATHER_DIR+
+  exceeds +GATHER_LIMIT+.
+
+* As the cache fills or on stage out, the files will be bundled into
+  randomly named tarballs in +GATHER_TARGET+, a GFS location.
+
+* If the compute node is an SMP, +GATHER_DIR+ is a shared resource.  It
+  is protected by the link file +GATHER_DIR/.cdm.lock+ .
+
+* Unpacking the tarballs in +GATHER_TARGET+ will produce the
+  user-specified filenames.
+
+Internal mechanism notes
+++++++++++++++++++++++++
+
+Summary:
+
+   1. Files are created by application
+   2. Acquire lock
+   3. Move files to cache
+   4. Check cache size
+   5. If limit exceeded, move all cache files to outbox
+   6. Release lock
+   7. If limit was exceeded, stream outbox as tarball to target
+
+* +GATHER+ required quite a bit of shell functionality to manage the
+  lock, etc. This is placed in +cdm_lib.sh+.
+
+* +vdl_int.k+ needed an additional task submission (+cdm_cleanup.sh+)
+  to perform the final flush at workflow completion time.  This task
+  also uses +cdm_lib.sh+.
+
+CDM Internals
+~~~~~~~~~~~~~
+
+VDL/Karajan processing
+^^^^^^^^^^^^^^^^^^^^^^
+
+1. CDM functions are available in Karajan via the +cdm+ namespace.
+
+2. These functions are defined in +org.globus.swift.data.Query+.
+
+3. If CDM is enabled, VDL skips file staging for files unless the
+policy is +DEFAULT+.
+
+
+Swift wrapper CDM routines
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+1. The +cdm.pl+ script is shipped to the compute node if CDM is
+enabled.
+
+2. When linking in inputs, CDM is consulted by
++_swiftwrap:cdm_lookup()+.
+
+3. The +cdm_action()+ shell function handles CDM methods.
+
+Test cases
+~~~~~~~~~~
+
+(See +About.txt+ for more information.)
+
+1. Simple test cases are in:
+   ** https://svn.mcs.anl.gov/repos/wozniak/collab/cdm/scripts/cdm-direct and
+   ** https://svn.mcs.anl.gov/repos/wozniak/collab/cdm/scripts/all-pairs
+2. Do a
+---------------------------------------------------------------
+mkdir cdm
+cd cdm
+svn co https://svn.mcs.anl.gov/repos/wozniak/collab/cdm/scripts
+---------------------------------------------------------------
+[start=3]
+. In +cdm-direct/+, run
+-----------------------------------
+source ./setup.sh local local local
+-----------------------------------
+[start=4]
+. Run workflow:
+---------------------------------------------------------------------------
+swift -sites.file sites.xml -tc.file tc.data -cdm.file fs.data stream.swift
+---------------------------------------------------------------------------
+[start=5]
+. Note in the log that staging is skipped for +input.txt+:
+-----------------------------------------------------
+policy: file://localhost/input.txt : DIRECT
+FILE_STAGE_IN_START file=input.txt ...
+FILE_STAGE_IN_SKIP file=input.txt policy=DIRECT
+FILE_STAGE_IN_END file=input.txt ...
+-----------------------------------------------------
+[start=6]
+. In the wrapper output, the input file is handled by CDM functionality:
+---------------------------------------------------------
+Progress  2010-01-21 13:50:32.466572727-0600  LINK_INPUTS
+CDM_POLICY: DIRECT /homes/wozniak/cdm/scripts/cdm-direct
+CDM: jobs/t/cp_sh-tkul4nmj input.txt DIRECT /homes/wozniak/cdm/scripts/cdm-direct
+CDM[DIRECT]: Linking jobs/t/cp_sh-tkul4nmj/input.txt to /homes/wozniak/cdm/scripts/cdm-direct/input.txt
+Progress  2010-01-21 13:50:32.486016708-0600  EXECUTE
+-----------------------------------------------------------------------
+[start=7]
+. +all-pairs+ is quite similar but uses more policies.
+
+PTMap case
+~~~~~~~~~~
+
+1. Start with vanilla PTMap:
+-----------------------------------------------------------
+cd cdm
+mkdir apps
+cd apps
+https://svn.mcs.anl.gov/repos/wozniak/collab/cdm/apps/ptmap
+-----------------------------------------------------------
+[start=2]
+. Source +setup.sh+.
+. Run +start.sh+.

More information about the Swift-commit mailing list