[Swift-commit] r6761 - SwiftTutorials/ATPESC_2013-08-06

[yadunandb at ci.uchicago.edu]

Author: yadunandb
Date: 2013-08-05 13:32:53 -0500 (Mon, 05 Aug 2013)
New Revision: 6761

Modified:
   SwiftTutorials/ATPESC_2013-08-06/README
Log:

Modis02-04 added



Modified: SwiftTutorials/ATPESC_2013-08-06/README
===================================================================
--- SwiftTutorials/ATPESC_2013-08-06/README	2013-08-05 18:29:00 UTC (rev 6760)
+++ SwiftTutorials/ATPESC_2013-08-06/README	2013-08-05 18:32:53 UTC (rev 6761)
@@ -522,9 +522,9 @@
 Modis - Satellite image data processing
 ---------------------------------------
 
-In the section we will process data from a large dataset of files that 
-categorize the Earth's surface, derived from the MODIS sensor instruments
-that orbit the Earth on two NASA satellites of the Earth Observing System.
+In the section we will use swift to process data from a large dataset of 
+files that categorize the Earth's surface, derived from the MODIS sensor 
+instruments that orbit the Earth on two NASA satellites of the Earth Observing System.
 
 The dataset we use (for 2002, named mcd12q1) files that categorize every
 250-meter square of non-ocean surface of the Earth into one of 17 "land 
@@ -535,17 +535,202 @@
 250-meter squares, based on a specific map projection.
 
 
-modis01.swift
-~~~~~~~~~~~~~
+modis01 - Process 1 file
+~~~~~~~~~~~~~~~~~~~~~~~~
 
 The first modis example defines one app function called getLandUse.
 This app takes a satellite image (data/modis/2002/h00v09.pgm.gz) as
 input. getLandUse creates a text file called landuse/h00v08.landuse.byfreq
 that counts the frequency of each land type defined in the input image.
 
+.modis01.swift
+-----
+type imagefile;
+type landuse;
+
+app (landuse output) getLandUse (imagefile input)
+{
+  getlanduse @filename(input) stdout=@filename(output);
+}
+
+imagefile modisImage <"data/global/h00v09.rgb">;
+landuse result <"landuse/h00v09.landuse.byfreq">;
+result = getLandUse(modisImage);
+-----
+
+
 To run modis01.swift:
 -----
 $ cd modis/modis01/
 $ swift modis01.swift
 -----
 
+modis021 - Process multiple files in paralle
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The second modis example expands upon the first example by running getLandUse
+with multiple (317) input files. Ouptut files are stored in the landuse directory.
+In order to map several input files we will use the ext mapper here to specify
+the mapper script, location a suffix to identify matching files.
+TODO: More on mappers
+
+.modis02.swift
+-----
+type imagefile;
+type landuse;
+
+app (landuse output) getLandUse (imagefile input)
+{
+  getlanduse @filename(input) stdout=@filename(output);
+}
+
+# Constants and command line arguments
+int nFiles       = @toInt(@arg("nfiles", "1000"));
+string MODISdir  = @arg("modisdir", "data/global");
+
+# Input Dataset
+imagefile geos[] <ext; exec="../bin/modis.mapper", location=MODISdir, suffix=".rgb", n=n\
+Files>;
+
+# Compute the land use summary of each MODIS tile
+landuse land[] <structured_regexp_mapper; source=geos, match="(h..v..)", transform=@strc\
+at("landuse/\\1.landuse.byfreq")>;
+
+foreach g,i in geos {
+    land[i] = getLandUse(g);
+}
+-----
+
+To run modis02.swift
+-----
+$ cd modis/modis02/
+$ swift modis02.swift
+-----
+
+
+modis03 - Multi-level dataflow
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The third modis example builds on the previous example. It defines a new app function
+called analyzeLandUse. The analyzeLandUse app examines the data generated by getLandUse
+and creates two summary files called topselected.txt and selectedtiles.txt. These
+files contain information about the top 10 urban areas.
+
+In the previous example, you have noticed that running all 317 input files on
+your laptop, even with 4 tasks a time, is not very efficient. 
+
+TODO : In the next example, instead of running locally, we will use a cluster called midway at the University of Chicago to improve performance.
+
+
+.modis03.swift
+-----
+type file;
+type imagefile;
+type landuse;
+
+app (landuse output) getLandUse (imagefile input)
+{
+  getlanduse @filename(input) stdout=@filename(output);
+}
+
+app (file output, file tilelist) analyzeLandUse (landuse input[], string usetype, int ma\
+xnum)
+{
+  analyzelanduse @output @tilelist usetype maxnum @input;
+}
+
+# Constants and command line arguments
+int nFiles       = @toInt(@arg("nfiles", "1000"));
+int nSelect      = @toInt(@arg("nselect", "10"));
+string landType  = @arg("landtype", "urban");
+string MODISdir  = @arg("modisdir", "data/global");
+
+# Input Dataset
+imagefile geos[] <ext; exec="../bin/modis.mapper", location=MODISdir, suffix=".rgb", n=n\
+Files>;
+
+# Compute the land use summary of each MODIS tile
+landuse land[] <structured_regexp_mapper; source=geos, match="(h..v..)", transform=@strc\
+at("landuse/\\1.landuse.byfreq")>;
+
+foreach g,i in geos {
+    land[i] = getLandUse(g);
+}
+
+# Find the top N tiles (by total area of selected landuse types)
+file topSelected <"topselected.txt">;
+file selectedTiles <"selectedtiles.txt">;
+(topSelected, selectedTiles) = analyzeLandUse(land, landType, nSelect);
+
+-----
+
+To run modis03.swift
+-----
+$ cd modis/modis03/
+$ swift modis03.swift
+-----
+
+
+modis04 - More complex Multi-level workflow
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The fourth modis example adds another app called markMap that looks at selectedtiles.txt
+and highlights the selected areas on a map. It will create a new image called
+markedGrid.ppm.
+
+.modis04.swift
+-----
+type file;
+type imagefile;
+type landuse;
+
+app (landuse output) getLandUse (imagefile input)
+{
+  getlanduse @filename(input) stdout=@filename(output);
+}
+
+app (file output, file tilelist) analyzeLandUse (landuse input[], string usetype, int ma\
+xnum)
+{
+  analyzelanduse @output @tilelist usetype maxnum @input;
+}
+
+app (imagefile grid) markMap (file tilelist)
+{
+  markmap @tilelist @grid;
+}
+
+# Constants and command line arguments
+int nFiles       = @toInt(@arg("nfiles", "1000"));
+int nSelect      = @toInt(@arg("nselect", "10"));
+string landType  = @arg("landtype", "urban");
+string MODISdir  = @arg("modisdir", "data/global");
+
+# Input Dataset
+imagefile geos[] <ext; exec="../bin/modis.mapper", location=MODISdir, suffix=".rgb", n=n\
+Files>;
+
+# Compute the land use summary of each MODIS tile
+landuse land[]    <structured_regexp_mapper; source=geos, match="(h..v..)", transform=@s\
+trcat("landuse/\\1.landuse.byfreq")>;
+
+foreach g,i in geos {
+    land[i] = getLandUse(g);
+}
+
+# Find the top N tiles (by total area of selected landuse types)
+file topSelected <"topselected.txt">;
+file selectedTiles <"selectedtiles.txt">;
+(topSelected, selectedTiles) = analyzeLandUse(land, landType, nSelect);
+
+# Mark the top N tiles on a sinusoidal gridded map
+imagefile gridmap <"gridmap.png">;
+gridmap = markMap(topSelected);
+
+-----
+
+To run modis04.swift
+-----
+$ cd modis/modis04/
+$ swift modis04.swift
+-----