[Swift-commit] r6852 - in SwiftTutorials/CIC_2013-08-09: . doc
wilde at ci.uchicago.edu
wilde at ci.uchicago.edu
Thu Aug 15 10:55:03 CDT 2013
Author: wilde
Date: 2013-08-15 10:55:03 -0500 (Thu, 15 Aug 2013)
New Revision: 6852
Added:
SwiftTutorials/CIC_2013-08-09/doc/
SwiftTutorials/CIC_2013-08-09/doc/README
SwiftTutorials/CIC_2013-08-09/doc/asciidoc.css
SwiftTutorials/CIC_2013-08-09/doc/build_docs.sh
SwiftTutorials/CIC_2013-08-09/doc/p1.png
SwiftTutorials/CIC_2013-08-09/doc/p2.png
SwiftTutorials/CIC_2013-08-09/doc/p3.png
SwiftTutorials/CIC_2013-08-09/doc/p4.png
SwiftTutorials/CIC_2013-08-09/doc/p5.png
SwiftTutorials/CIC_2013-08-09/doc/p6.png
SwiftTutorials/CIC_2013-08-09/doc/p7.png
SwiftTutorials/CIC_2013-08-09/doc/p8.png
SwiftTutorials/CIC_2013-08-09/doc/p9.png
SwiftTutorials/CIC_2013-08-09/doc/push.sh
Log:
for cic
Copied: SwiftTutorials/CIC_2013-08-09/doc/README (from rev 6845, SwiftTutorials/CIC_2013-08-09/README)
===================================================================
--- SwiftTutorials/CIC_2013-08-09/doc/README (rev 0)
+++ SwiftTutorials/CIC_2013-08-09/doc/README 2013-08-15 15:55:03 UTC (rev 6852)
@@ -0,0 +1,881 @@
+Swift CIC Tutorial - 2013.0815
+==============================
+
+Workflow tutorial setup
+-----------------------
+
+Check out scripts from SVN
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+To checkout the most recent ATPESC tutorial scripts from SVN, run the following
+command:
+
+-----
+$ svn co https://svn.ci.uchicago.edu/svn/vdl2/SwiftTutorials/TUTORIAL-NAME
+-----
+
+This will create a directory called TUTORIAL-NAME which contains all of the
+scripts mentioned in this document.
+
+Run setup
+~~~~~~~~~
+Once the scripts are checked out, run the following commands to perform
+the initial setup.
+
+-----
+$ cd TUTORIAL-NAME # change to the newly created TUTORIAL-NAME directory
+$ source setup.sh <SITE> # sets swift config files in $HOME/.swift and selects
+ # a site from the following: ( cloud, uc3, midway )
+$ swift -version # verify that Swift 0.94 is in your $PATH and functional
+-----
+
+NOTE: If you disconnect from the machine, you will need to re-run source setup.sh.
+
+Mock "science applications" for the workflow tutorial
+-----------------------------------------------------
+There are two shell scripts included that serve a very simple stand-ins for science application:
+simulation.sh and stats.sh
+
+simulation.sh
+~~~~~~~~~~~~~
+The simulation.sh script is a simple substitute for a scientific simulation application. It generates and prints a set of one or more random integers in the range 0-29,999 as controlled by its optional arguments, which are:
+
+.simulation.sh arguments
+[width="80%",cols="^2,10",options="header"]
+
+|=======================
+|Argument number|Description
+|1 |runtime: sets run time of simulation.sh in seconds
+|2 |range: limits generated values to the range [0,range-1]
+|3 |biasfile: add the integer contained in this file to each value generated
+|4 |scale: multiplies each generated value by this integer
+|5 |count: number of values to generate in the simulation
+|=======================
+
+With no arguments, simulate.sh prints 1 number in the range of 1-100. Otherwise it generates n numbers of the form R * scale + bias.
+
+-----
+$ ./simulate.sh
+96
+-----
+
+stats.sh
+~~~~~~~~
+The stats.sh script reads a file containing n numbers and prints the average
+of those numbers to stdout.
+
+Introductory exercises
+----------------------
+Parts 1-6 (p1.swift - p6.swift) run locally and serve as examples of the Swift language.
+Parts 7-9 (p7.swift - p9.swift) submit jobs to the site specified the setup stage
+
+p1 - Run an application under Swift
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The first swift script, p1.swift, runs simulate.sh to generate a single random
+number. It writes the number to a file.
+
+image:p1.png[]
+
+.p1.swift
+-----
+type file;
+
+app (file o) mysim ()
+{
+ simulate stdout=@filename(o);
+}
+
+file f = mysim();
+-----
+
+To run this script, run the following command:
+-----
+$ cd part01
+$ swift p1.swift
+-----
+
+The simulate application gets translated to simulate.sh within the 'apps' file.
+
+NOTE: Since the file you created is not named, swift will generate a random
+name for the file in a directory called _concurrent. To view the created
+output, run "cat _concurrent/*"
+
+To cleanup the directory and remove all outputs, run:
+-----
+$ ./clean.sh
+------
+
+p2 - Mapping (naming) output files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The second swift script shows an example of naming the file. The output is now
+in a file called sim.out.
+
+image:p2.png[]
+
+.p2.swift
+-----
+type file;
+
+app (file o) mysim ()
+{
+ simulate stdout=@filename(o);
+}
+
+file f <"sim.out">;
+f = mysim();
+-----
+
+To run the script:
+-----
+$ cd ../part02
+$ swift p2.swift
+-----
+
+p3 - Parallel loops with foreach
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The p3.swift script introduces the foreach loop. This script runs many
+simulations. Output files are named here by Swift and will get created
+in the _concurrent directory.
+
+image:p3.png[]
+
+.p3.swift
+----
+type file;
+
+app (file o) mysim ()
+{
+ simulate stdout=@filename(o);
+}
+
+foreach i in [0:9] {
+ file f = mysim();
+}
+----
+
+To run:
+----
+$ cd part03
+$ swift p3.swift
+----
+
+p4 - Mapping arrays to files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+p4.swift gives an example of naming multiple files within a foreach loop.
+
+image:p4.png[]
+
+.p4.swift
+----
+type file;
+
+app (file o) mysim ()
+{
+ simulate stdout=@filename(o);
+}
+
+foreach i in [0:9] {
+ file f <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
+ f = mysim();
+}
+----
+
+To run:
+----
+$ swift p4.swift
+----
+
+Output files will be named output/sim_N.out.
+
+p5 - merging/reducing the results of a parallel foreach loop
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+p5.swift introduces a postprocessing step. After all the parallel simulations have completed, the files
+created by simulation.sh will be averaged by stats.sh.
+
+image:p5.png[]
+
+.p5.swift
+----
+type file;
+
+app (file o) mysim ()
+{
+ simulate stdout=@filename(o);
+}
+
+app (file o) analyze (file s[])
+{
+ stats @filenames(s) stdout=@filename(o);
+}
+
+file sims[];
+
+int nsim = @toInt(@arg("nsim","10"));
+
+foreach i in [0:nsim-1] {
+ file simout <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
+ simout = mysim();
+ sims[i] = simout;
+}
+
+file stats<"output/average.out">;
+stats = analyze(sims);
+----
+
+To run:
+----
+$ swift p5.swift
+----
+
+p6 - Sending arguments to applications
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+p6.swift introduces command line arguments. The script sets a variable called
+"steps" here, which determines the length of time that the simulation.sh
+will run for. It also defines a variable called nsim, which determines the
+number of simulations to run.
+
+image:p6.png[]
+
+.p6.swift
+----
+type file;
+
+app (file o) mysim (int timesteps)
+{
+ simulate timesteps stdout=@filename(o);
+}
+
+app (file o) analyze (file s[])
+{
+ stats @filenames(s) stdout=@filename(o);
+}
+
+file sims[];
+int nsim = @toInt(@arg("nsim","10"));
+int steps = @toInt(@arg("steps","1"));
+
+foreach i in [0:nsim-1] {
+ file simout <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
+ simout = mysim(steps);
+ sims[i] = simout;
+}
+
+file stats<"output/average.out">;
+stats = analyze(sims);
+----
+
+Use the command below to specify the time for each simulation.
+----
+$ cd ../part06
+$ swift p6.swift -steps=3 # each simulation takes 3 seconds
+----
+
+p7 - Running on the remote site nodes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+p7.swift is the first script that will submit jobs to remote site nodes for analysis.
+It is similar to earlier scripts, with a few minor exceptions. To generalize the script
+for other types of remote execution (e.g., when no shared filesystem is available to the compute nodes), the application simulate.sh
+will get transferred to the worker node by Swift, in the same manner as any other input data file.
+
+image:p7.png[]
+
+.p7.swift
+-----
+type file;
+
+# Application to be called by this script
+
+file simulation_script <"simulate.sh">;
+
+# app() functions for application programs to be called:
+
+app (file out) simulation (file script, int timesteps, int sim_range)
+{
+ sh @filename(script) timesteps sim_range stdout=@filename(out);
+}
+
+# Command line params to this script:
+
+int nsim = @toInt(@arg("nsim", "10")); # number of simulation programs to run
+int range = @toInt(@arg("range", "100")); # range of the generated random numbers
+
+# Main script and data
+
+int steps=3;
+
+tracef("\n*** Script parameters: nsim=%i steps=%i range=%i \n\n", nsim, steps, range);
+
+foreach i in [0:nsim-1] {
+ file simout <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
+ simout = simulation(simulation_script, steps, range);
+}
+-----
+
+To run:
+----
+$ cd ../part07
+$ swift p7.swift
+----
+
+p8 - Running the stats summary step on the remote site
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+p8.swift will also stage in and run stats.sh to calculate averages. It adds a
+trace statement so you can see the order in which things execute.
+
+image:p8.png[]
+
+.p8.swift
+-----
+type file;
+
+# Applications to be called by this script
+
+file simulation_script <"simulate.sh">;
+file analysis_script <"stats.sh">;
+
+# app() functions for application programs to be called:
+
+app (file out) simulation (file script, int timesteps, int sim_range, file bias_file, int scale, int sim_count)
+{
+ sh @filename(script) timesteps sim_range @filename(bias_file) scale sim_count stdout=@filename(out);
+}
+
+app (file out) analyze (file script, file s[])
+{
+ sh @script @filenames(s) stdout=@filename(out);
+}
+
+# Command line params to this script:
+
+int nsim = @toInt(@arg("nsim", "10")); # number of simulation programs to run
+int steps = @toInt(@arg("steps", "1")); # number of "steps" each simulation (==seconds of runtime)
+int range = @toInt(@arg("range", "100")); # range of the generated random numbers
+int count = @toInt(@arg("count", "10")); # number of random numbers generated per simulation
+
+# Main script and data
+
+tracef("\n*** Script parameters: nsim=%i steps=%i range=%i count=%i\n\n", nsim, steps, range, count);
+
+file sims[]; # Array of files to hold each simulation output
+file bias<"bias.dat">; # Input data file to "bias" the numbers:
+ # 1 line: scale offset ( N = n*scale + offset)
+foreach i in [0:nsim-1] {
+ file simout <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
+ simout = simulation(simulation_script, steps, range, bias, 100000, count);
+ sims[i] = simout;
+}
+
+file stats<"output/stats.out">; # Final output file: average of all "simulations"
+stats = analyze(analysis_script,sims);
+-----
+
+To run:
+----
+$ cd ../part08
+$ swift p8.swift
+----
+
+p9 - A more complex workflow pattern: multiple parallel pipelines
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+p9.swift adds another app function called genrand. Genrand will produce a random
+number that will be used to determine how long each simulation app will run.
+
+image:p9.png[]
+
+.p9.swift
+-----
+type file;
+
+# Applications to be called by this script
+
+file simulation_script <"simulate.sh">;
+file analysis_script <"stats.sh">;
+
+# app() functions for application programs to be called:
+
+app (file out) genrand (file script, int timesteps, int sim_range)
+{
+ sh @filename(script) timesteps sim_range stdout=@filename(out);
+}
+
+app (file out) simulation (file script, int timesteps, int sim_range, file bias_file, int scale, int sim_count)
+{
+ sh @filename(script) timesteps sim_range @filename(bias_file) scale sim_count stdout=@filename(out);
+}
+
+app (file out) analyze (file script, file s[])
+{
+ sh @script @filenames(s) stdout=@filename(out);
+}
+
+# Command line params to this script:
+int nsim = @toInt(@arg("nsim", "10")); # number of simulation programs to run
+int range = @toInt(@arg("range", "100")); # range of the generated random numbers
+int count = @toInt(@arg("count", "10")); # number of random numbers generated per simulation
+
+# Main script and data
+
+tracef("\n*** Script parameters: nsim=%i range=%i count=%i\n\n", nsim, range, count);
+
+file bias<"dynamic_bias.dat">; # Dynamically generated bias for simulation ensemble
+
+bias = genrand(simulation_script, 1, 1000);
+
+file sims[]; # Array of files to hold each simulation output
+
+foreach i in [0:nsim-1] {
+
+ int steps = readData(genrand(simulation_script, 1, 5));
+ tracef(" for simulation[%i] steps=%i\n", i, steps+1);
+
+ file simout <single_file_mapper; file=@strcat("output/sim_",i,".out")>;
+ simout = simulation(simulation_script, steps+1, range, bias, 100000, count);
+ sims[i] = simout;
+}
+
+file stats<"output/stats.out">; # Final output file: average of all "simulations"
+stats = analyze(analysis_script,sims);
+-----
+
+To run:
+----
+$ cd ../part09
+$ swift p9.swift
+----
+
+
+
+
+
+
+Running Swift scripts on Cloud resources
+----------------------------------------
+
+Setting up the Cloud exercises
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A number of preconfigured Amazon EC2 nodes are set up for running Swift.
+
+Run the Cloud exercises
+~~~~~~~~~~~~~~~~~~~~~~
+
+* Change to cloud dir:
+-----
+ cd ~/cloud
+-----
+* Copy the private key tutorial.pem to your .ssh dir:
+-----
+ cp tutorial.pem ~/.ssh/
+-----
+* Source the setup script on command line:
+-----
+ source ./setup
+-----
+* Run the catsn Swift script:
+-----
+ ./run.catsn
+-----
+* Run the Cloud versions of the Swift scripts p7, p8,and p9.swift:
+-----
+ swift -sites.file sites.xml -config cf -tc.file tc p7.swift
+ swift -sites.file sites.xml -config cf -tc.file tc p8.swift
+ swift -sites.file sites.xml -config cf -tc.file tc p9.swift
+-----
+
+////
+* Add cloud resources to existing examples:
+-----
+ ./addcloud.sh <dir> #where dir is a tutorial script directory
+ e.g.
+ ./addcloud.sh ../part01 #will create a new site pool "both.xml" in ../part01
+-----
+////
+
+* Finally, to clean up the log files, kill agent and shutdown the coaster service:
+-----
+ ./cleanme
+-----
+
+
+Notes on the Cloud exercises
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The run.catsn shell script contains the full command line to call Swift scripts with configuration files. This script runs swift as follows:
+
+swift -sites.file sites.xml -tc.file tc -config cf catsn.swift -n=10
+
+To learn more about the configuration files, see Swift user-guide:
+http://www.ci.uchicago.edu/swift/guides/release-0.94/userguide/userguide.html
+
+////
+
+Running Swift/T on Vesta with Python and R integration
+------------------------------------------------------
+
+Normally it is difficult to run scripted programs such as Python and
+R on the Blue Gene/Q because of the limited OS environment. However,
+Swift/T allows you to create a composite application that is linked
+into one Cobalt job at run time. Thus, you can write a scripted
+program that coordinates calls from Swift to C, C++, Fortran, Python,
+R, or Tcl. All data movement is handled by the Swift/T runtime over
+MPI, removing the overhead of file access.
+
+=== p11 - Scripted parallel numerics with Python and R
+
+As shown on the slides, this example calls the
+http://www.numpy.org[Numpy] numerical libraries via Python as well as
+the http://www.r-project.org[R language] for statistics. In this
+example, we use Numpy to construct matrices, perform matrix
+arithmetic, and compute determinants. Since determinant is _O(n^3^)_,
+we compute each determinant in parallel using the Swift +foreach+
+loop. All result are collected and passed to the R +max()+ function.
+
+==== Scripts
+
+This example is designed to run on Vesta.
+
+To compile and run the script, run:
+
+.run-dets.sh
+----
+include::part11-swift-py-r/code/run-dets.sh.txt[]
+----
+
+As a reference, an equivalent plain Python code is provided:
+
+.dets.py
+----
+include::part11-swift-py-r/code/dets.py[]
+----
+
+The Swift script is:
+
+.dets.swift
+----
+include::part11-swift-py-r/code/dets.swift[]
+----
+
+==== Analysis
+
+The Turbine run time script creates a +TURBINE_OUTPUT+ directory and
+reports it. Standard output from the job is in +output.txt+.
+
+Use +grep dets+ to find the results printed by the Swift script. You
+will see each determinant as inserted into the Swift array, and the
+maximal result.
+
+Turbine was launched with +-n 10+; that is, 10 total processes, 8 of
+which are workers, ranks 1-8. Much of the output is created by rank
+0, the Turbine engine. (Only one engine is used in this case,
+although Swift/T supports multiple engines for scalability.) The ADLB
+server runs on rank 9 and produces almost no output. (Swift/T and
+ADLB support multiple servers.)
+
+You may +grep+ for +python: expression:+ to see the Python expressions
+evaluated on the workers. Note the rank numbers.
+
+For production cases, you may disable logging by setting
++TURBINE_LOG=0+ in the environment.
+
+=== More information
+
+For more information about Swift/T, see:
+
+* https://sites.google.com/site/exmcomputing/swift-t[Swift/T Overview]
+* http://www.mcs.anl.gov/exm/local/guides/swift.html[Swift/T Guide]
+* http://www.mcs.anl.gov/exm/local/guides/turbine-sites.html[Sites Guide]
+ — notes for running Swift/T on various systems
+
+
+Running MPI apps under Swift
+----------------------------
+
+Modis - Satellite image data processing
+---------------------------------------
+
+In this section we will use swift to process data from a large dataset of
+files that categorize the Earth's surface, derived from the MODIS sensor
+instruments that orbit the Earth on two NASA satellites of the Earth Observing System.
+
+The dataset we use (for 2002, named +mcd12q1+) consists of 317 "tile" files
+that categorize every 250-meter square of non-ocean surface of the Earth into
+one of 17 "land cover" categories (for example, water, ice, forest, barren, urban).
+Each pixel of these data files has a value of 0 to 16, describing one square
+of the Earth's surface at a specific point in time. Each tile file has
+approximately 5 million 1-byte pixels (5.7 MB), covering 2400x2400 250-meter
+squares, based on a specific map projection.
+
+image:sinusoidal_v5.gif[]
+
+modis01 - Process 1 image
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The first modis example defines one app function called getLandUse.
+This app takes a satellite image (data/modis/2002/h00v09.pgm.gz) as
+input. getLandUse creates a text file called landuse/h00v08.landuse.byfreq
+that counts the frequency of each land type defined in the input image.
+
+.modis01.swift
+-----
+type imagefile;
+type landuse;
+
+app (landuse output) getLandUse (imagefile input)
+{
+ getlanduse @filename(input) stdout=@filename(output);
+}
+
+imagefile modisImage <"data/global/h00v09.rgb">;
+landuse result <"landuse/h00v09.landuse.byfreq">;
+result = getLandUse(modisImage);
+-----
+
+
+To run modis01.swift:
+-----
+$ cd modis/modis01/
+$ swift modis01.swift
+-----
+
+modis02 - Process multiple images in parallel
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The second modis example expands upon the first example by running getLandUse
+with multiple (317) input files. Ouptut files are stored in the landuse directory.
+In order to map several input files we will use the ext mapper here to specify
+the mapper script, location a suffix to identify matching files.
+TODO: More on mappers
+
+.modis02.swift
+-----
+type imagefile;
+type landuse;
+
+app (landuse output) getLandUse (imagefile input)
+{
+ getlanduse @filename(input) stdout=@filename(output);
+}
+
+# Constants and command line arguments
+int nFiles = @toInt(@arg("nfiles", "1000"));
+string MODISdir = @arg("modisdir", "data/global");
+
+# Input Dataset
+imagefile geos[] <ext; exec="../bin/modis.mapper", location=MODISdir, suffix=".rgb", n=n\
+Files>;
+
+# Compute the land use summary of each MODIS tile
+landuse land[] <structured_regexp_mapper; source=geos, match="(h..v..)", transform=@strc\
+at("landuse/\\1.landuse.byfreq")>;
+
+foreach g,i in geos {
+ land[i] = getLandUse(g);
+}
+-----
+
+To run modis02.swift
+-----
+$ cd modis/modis02/
+$ swift modis02.swift
+-----
+
+
+modis03 - Analyse the processed images
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The third modis example builds on the previous example. It defines a new app function
+called analyzeLandUse. The analyzeLandUse app examines the data generated by getLandUse
+and creates two summary files called topselected.txt and selectedtiles.txt. These
+files contain information about the top 10 urban areas.
+
+In the previous example, you have noticed that running all 317 input files on
+your laptop, even with 4 tasks a time, is not very efficient.
+
+TODO : In the next example, instead of running locally, we will use a cluster called midway at the University of Chicago to improve performance.
+
+
+.modis03.swift
+-----
+type file;
+type imagefile;
+type landuse;
+
+app (landuse output) getLandUse (imagefile input)
+{
+ getlanduse @filename(input) stdout=@filename(output);
+}
+
+app (file output, file tilelist) analyzeLandUse (landuse input[], string usetype, int ma\
+xnum)
+{
+ analyzelanduse @output @tilelist usetype maxnum @input;
+}
+
+# Constants and command line arguments
+int nFiles = @toInt(@arg("nfiles", "1000"));
+int nSelect = @toInt(@arg("nselect", "10"));
+string landType = @arg("landtype", "urban");
+string MODISdir = @arg("modisdir", "data/global");
+
+# Input Dataset
+imagefile geos[] <ext; exec="../bin/modis.mapper", location=MODISdir, suffix=".rgb", n=n\
+Files>;
+
+# Compute the land use summary of each MODIS tile
+landuse land[] <structured_regexp_mapper; source=geos, match="(h..v..)", transform=@strc\
+at("landuse/\\1.landuse.byfreq")>;
+
+foreach g,i in geos {
+ land[i] = getLandUse(g);
+}
+
+# Find the top N tiles (by total area of selected landuse types)
+file topSelected <"topselected.txt">;
+file selectedTiles <"selectedtiles.txt">;
+(topSelected, selectedTiles) = analyzeLandUse(land, landType, nSelect);
+
+-----
+
+To run modis03.swift
+-----
+$ cd modis/modis03/
+$ swift modis03.swift
+-----
+
+
+modis04 - Mark the top N tiles on a map
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The fourth modis example adds another app called markMap that looks at selectedtiles.txt
+and highlights the selected areas on a map. It will create a new image called
+gridmap.png which marks the top N tiles on a sinusoidal gridded map.
+
+.modis04.swift
+-----
+type file;
+type imagefile;
+type landuse;
+
+app (landuse output) getLandUse (imagefile input)
+{
+ getlanduse @filename(input) stdout=@filename(output);
+}
+
+app (file output, file tilelist) analyzeLandUse (landuse input[], string usetype, int ma\
+xnum)
+{
+ analyzelanduse @output @tilelist usetype maxnum @input;
+}
+
+app (imagefile grid) markMap (file tilelist)
+{
+ markmap @tilelist @grid;
+}
+
+# Constants and command line arguments
+int nFiles = @toInt(@arg("nfiles", "1000"));
+int nSelect = @toInt(@arg("nselect", "10"));
+string landType = @arg("landtype", "urban");
+string MODISdir = @arg("modisdir", "data/global");
+
+# Input Dataset
+imagefile geos[] <ext; exec="../bin/modis.mapper", location=MODISdir, suffix=".rgb", n=n\
+Files>;
+
+# Compute the land use summary of each MODIS tile
+landuse land[] <structured_regexp_mapper; source=geos, match="(h..v..)", transform=@s\
+trcat("landuse/\\1.landuse.byfreq")>;
+
+foreach g,i in geos {
+ land[i] = getLandUse(g);
+}
+
+# Find the top N tiles (by total area of selected landuse types)
+file topSelected <"topselected.txt">;
+file selectedTiles <"selectedtiles.txt">;
+(topSelected, selectedTiles) = analyzeLandUse(land, landType, nSelect);
+
+# Mark the top N tiles on a sinusoidal gridded map
+imagefile gridmap <"gridmap.png">;
+gridmap = markMap(topSelected);
+
+-----
+
+To run modis04.swift
+-----
+$ cd modis/modis04/
+$ swift modis04.swift
+-----
+
+modis05 - Create multi-color tile images
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The fifth modis example extends the previous examples by adding the colorModis app
+to create multi-color images for all tiles.
+
+.modis05.swift
+-----
+type file;
+type imagefile;
+type landuse;
+
+app (landuse output) getLandUse (imagefile input)
+{
+ getlanduse @filename(input) stdout=@filename(output);
+}
+
+app (file output, file tilelist) analyzeLandUse (landuse input[], string usetype, int maxnum)
+{
+ analyzelanduse @output @tilelist usetype maxnum @input;
+}
+
+app (imagefile grid) markMap (file tilelist)
+{
+ markmap @tilelist @grid;
+}
+
+app (imagefile output) colorModis (imagefile input)
+{
+ colormodis @input @output;
+}
+
+# Constants and command line arguments
+int nFiles = @toInt(@arg("nfiles", "1000"));
+int nSelect = @toInt(@arg("nselect", "10"));
+string landType = @arg("landtype", "urban");
+string MODISdir = @arg("modisdir", "data/global");
+
+# Input Dataset
+imagefile geos[] <ext; exec="../bin/modis.mapper", location=MODISdir, suffix=".rgb", n=nFiles>;
+
+# Compute the land use summary of each MODIS tile
+landuse land[] <structured_regexp_mapper; source=geos, match="(h..v..)", transform=@strcat("landuse/\\1.landuse.byfreq")>;
+
+foreach g,i in geos {
+ land[i] = getLandUse(g);
+}
+
+# Find the top N tiles (by total area of selected landuse types)
+file topSelected <"topselected.txt">;
+file selectedTiles <"selectedtiles.txt">;
+(topSelected, selectedTiles) = analyzeLandUse(land, landType, nSelect);
+
+# Mark the top N tiles on a sinusoidal gridded map
+imagefile gridmap <"gridmap.png">;
+gridmap = markMap(topSelected);
+
+# Create multi-color images for all tiles
+imagefile colorImage[] <structured_regexp_mapper; source=geos, match="(h..v..)", transform=@strcat("colorImages/\\1.color.rgb")>;
+
+foreach g, i in geos {
+ colorImage[i] = colorModis(g);
+}
+
+-----
+
+
+To run modis05.swift:
+-----
+$ cd modis/modis05/
+$ swift modis05.swift
+-----
+
+////
Added: SwiftTutorials/CIC_2013-08-09/doc/asciidoc.css
===================================================================
--- SwiftTutorials/CIC_2013-08-09/doc/asciidoc.css (rev 0)
+++ SwiftTutorials/CIC_2013-08-09/doc/asciidoc.css 2013-08-15 15:55:03 UTC (rev 6852)
@@ -0,0 +1,2 @@
+a:link { color:navy; }
+a:visited { color:navy; }
Copied: SwiftTutorials/CIC_2013-08-09/doc/build_docs.sh (from rev 6845, SwiftTutorials/CIC_2013-08-09/build_docs.sh)
===================================================================
--- SwiftTutorials/CIC_2013-08-09/doc/build_docs.sh (rev 0)
+++ SwiftTutorials/CIC_2013-08-09/doc/build_docs.sh 2013-08-15 15:55:03 UTC (rev 6852)
@@ -0,0 +1,14 @@
+#!/bin/bash -e
+
+if false; then
+ echo skipping
+pushd part11-swift-py-r/code >& /dev/null
+# Strip comments, blank lines; prepend shell prompt ($)
+grep -A 20 stc run-dets.sh | \
+ grep -v -e "^$\|#" | \
+ sed 's/^/$ /' > run-dets.sh.txt
+popd >& /dev/null
+fi
+
+asciidoc -a toc -a toplevels=2 -a stylesheet=$PWD/asciidoc.css -a max-width=800px -o cic-tutorial.html README
+
Copied: SwiftTutorials/CIC_2013-08-09/doc/p1.png (from rev 6845, SwiftTutorials/CIC_2013-08-09/p1.png)
===================================================================
(Binary files differ)
Copied: SwiftTutorials/CIC_2013-08-09/doc/p2.png (from rev 6845, SwiftTutorials/CIC_2013-08-09/p2.png)
===================================================================
(Binary files differ)
Copied: SwiftTutorials/CIC_2013-08-09/doc/p3.png (from rev 6845, SwiftTutorials/CIC_2013-08-09/p3.png)
===================================================================
(Binary files differ)
Copied: SwiftTutorials/CIC_2013-08-09/doc/p4.png (from rev 6845, SwiftTutorials/CIC_2013-08-09/p4.png)
===================================================================
(Binary files differ)
Copied: SwiftTutorials/CIC_2013-08-09/doc/p5.png (from rev 6845, SwiftTutorials/CIC_2013-08-09/p5.png)
===================================================================
(Binary files differ)
Copied: SwiftTutorials/CIC_2013-08-09/doc/p6.png (from rev 6845, SwiftTutorials/CIC_2013-08-09/p6.png)
===================================================================
(Binary files differ)
Copied: SwiftTutorials/CIC_2013-08-09/doc/p7.png (from rev 6845, SwiftTutorials/CIC_2013-08-09/p7.png)
===================================================================
(Binary files differ)
Copied: SwiftTutorials/CIC_2013-08-09/doc/p8.png (from rev 6845, SwiftTutorials/CIC_2013-08-09/p8.png)
===================================================================
(Binary files differ)
Copied: SwiftTutorials/CIC_2013-08-09/doc/p9.png (from rev 6845, SwiftTutorials/CIC_2013-08-09/p9.png)
===================================================================
(Binary files differ)
Added: SwiftTutorials/CIC_2013-08-09/doc/push.sh
===================================================================
--- SwiftTutorials/CIC_2013-08-09/doc/push.sh (rev 0)
+++ SwiftTutorials/CIC_2013-08-09/doc/push.sh 2013-08-15 15:55:03 UTC (rev 6852)
@@ -0,0 +1,12 @@
+#! /bin/sh
+
+scp cic-tutorial.html *png login.ci.uchicago.edu:/ci/www/projects/swift/links
+
+exit
+
+www=/ci/www/projects/swift/ATPESC
+
+cp --backup=numbered tutorial.html $www
+cp --backup=numbered *.png $www
+chmod g+w,a+r $www/tutorial.html $www/*png
+ls -l $www
Property changes on: SwiftTutorials/CIC_2013-08-09/doc/push.sh
___________________________________________________________________
Added: svn:executable
+ *
More information about the Swift-commit
mailing list