[Swift-commit] r3933 - in text/parco10submission: . code

[noreply at svn.ci.uchicago.edu]

Author: wilde
Date: 2011-01-10 12:07:47 -0600 (Mon, 10 Jan 2011)
New Revision: 3933

Modified:
   text/parco10submission/code/glass.swift
   text/parco10submission/paper.tex
Log:
Added prio performance data. Minor correttions elsewhere.

Modified: text/parco10submission/code/glass.swift
===================================================================
--- text/parco10submission/code/glass.swift	2011-01-10 16:40:03 UTC (rev 3932)
+++ text/parco10submission/code/glass.swift	2011-01-10 18:07:47 UTC (rev 3933)
@@ -28,7 +28,7 @@
       "--cradius" rad "--ccoord" centerstring arctimestring;
 }
 
-CreateGlassSystem()
+GlassRun()
 {
   string temp=@arg("temp","2.0");
   string steps=@arg("steps","10");
@@ -77,4 +77,4 @@
   }
 }
 
-CreateGlassSystem();
\ No newline at end of file
+GlassRun();
\ No newline at end of file

Modified: text/parco10submission/paper.tex
===================================================================
--- text/parco10submission/paper.tex	2011-01-10 16:40:03 UTC (rev 3932)
+++ text/parco10submission/paper.tex	2011-01-10 18:07:47 UTC (rev 3933)
@@ -1317,14 +1317,14 @@
 Lines 1-5 define the mapped file types; these files are used to compose input and output structures at lines 7-19.
 These structure reflect the fact that the simulation is restartable in 1-2 hour increments, and that it works together with the Swift script to create a simple but powerful mechanism for managing checkpoint/restart across a long-running large-scale simulation campaign.
 
-The single application called by this script is the {\tt glassRun} program wrapped in the app function at lines 21-29. Note that rather than defining main program logic in ``open" (top-level) code, the script places all the program logic in the function {\tt CreateGlassSystem}, invoked by the single statement at line 80.  This enables the simulation script to be defined in a library that can be imported into other Swift scripts to perform entire campaigns or campaign subsets.
+The single application called by this script is the {\tt glassRun} program wrapped in the app function at lines 21-29. Note that rather than defining main program logic in ``open" (top-level) code, the script places all the program logic in the function {\tt GlassRun}, invoked by the single statement at line 80.  This enables the simulation script to be defined in a library that can be imported into other Swift scripts to perform entire campaigns or campaign subsets.
 
-The {\tt CreateGlassSystem} function starts by extracting a large set of science parameters from the Swift command line  at lines 33-48 using the {\tt @arg()} function. It uses the built-in function {\tt readData} at lines 42-43 to read prepared lists of molecular radii and centroids from parameter files to define the primary physical dimensions of the simulation space.
+The {\tt GlassRun} function starts by extracting a large set of science parameters from the Swift command line  at lines 33-48 using the {\tt @arg()} function. It uses the built-in function {\tt readData} at lines 42-43 to read prepared lists of molecular radii and centroids from parameter files to define the primary physical dimensions of the simulation space.
 A selectable energy function to used by the simulation application is specified as a parameter at line 48.
 
 At lines 57 and 61, the script leverages Swift flexible dynamic arrays to create a 3D array for input and an 4D array of structures for outputs. These data structures, whose leaf elements consist entirely of mapped files, are set using the external mappers specified for the input array at lines 57-59 and for the output array of structures at 61-63.  Note that many of the science parameters are passed to the mappers, which in turn are used by the input  mapper to locate files within the large multi-level directory structure of the campaign, and by the output mapper to create new directory and file naming conventions for the campaign outputs. The mappers apply the common, useful practice of using scientific metadata to determine directory and file names.
 
-The entire body of the {\tt CreateGlassSystem} is a four-level nesting of \verb|foreach| statements at lines 65-77. These loops perform a parallel parameter sweep over all combinations of radius, centroid, model, and job number within the simulation space. A single run of the script immediately expands to an independent parallel invocation of the simulation application for each point in the space - 1,670 jobs for the minimum case of a 7 x 27 x 10 x 1 space. Note that the {\tt if} statement at line 69 causes the simulation execution to be skipped if it has already been performed, as determine by a ``\verb|NULL|'' file name returned by the mapper for the output of a given job in the simulation space. In the current campaign the fourth dimension ({\tt nsub}) of the simulation space is fixed at one. This value could be increased to define sub-configurations that would perform better Monte Carlo averaging, with a multiplicative increase in the number of jobs. This is currently se
 t to one because there are ample starting configurations, but if this was not the case (as in earlier campaigns) the script could run repeated simulations with different random seeds.
+The entire body of the {\tt GlassRun} is a four-level nesting of \verb|foreach| statements at lines 65-77. These loops perform a parallel parameter sweep over all combinations of radius, centroid, model, and job number within the simulation space. A single run of the script immediately expands to an independent parallel invocation of the simulation application for each point in the space - 1,670 jobs for the minimum case of a 7 x 27 x 10 x 1 space. Note that the {\tt if} statement at line 69 causes the simulation execution to be skipped if it has already been performed, as determine by a ``\verb|NULL|'' file name returned by the mapper for the output of a given job in the simulation space. In the current campaign the fourth dimension ({\tt nsub}) of the simulation space is fixed at one. This value could be increased to define sub-configurations that would perform better Monte Carlo averaging, with a multiplicative increase in the number of jobs. This is currently set to one 
 because there are ample starting configurations, but if this was not the case (as in earlier campaigns) the script could run repeated simulations with different random seeds.
 
 The advantages of managing a simulation campaign in this manner are borne out well by Hocky's experience: the expression of the campaign is a well-structured high-level script, devoid of details about file naming, synchronization of parallel tasks, location and state of remote computing resources, or explicit explicit data transfer. Hocky was able to leverage local cluster resources on many occasions, but at any time could count on his script acquiring on the order of 1,000 compute cores from 6 to 18 sites of the Open Science Grid. When executing on the OSG, he leveraged Swift's capability to replicate jobs that were waiting in queues at more congested sites, and automatically send them to sites where resources were available and jobs were being processed at better rates. All of these capabilities would be a huge distraction from his primary scientific simulation campaign were he to use or script lower-level abstractions where parallelism and remote distribution were the man
 ual responsibility of the programmer.
 
@@ -1415,6 +1415,53 @@
   \end{center}
 \end{figure}
 
+
+\subsection{Prior performance measures}
+\mikenote{Remove above caption}
+
+Published measurements of Swift performance provide evidence that its parallel distributed programming model can be implemented with sufficient scalability and efficiency to make it a practical tool for large-scale parallel application scripting.
+
+The performance of Swift, submitting jobs over the wide area network from UChicago to the TeraGrid Ranger cluster at TACC, as published in \cite{CNARI_2009}are shown in figure \ref{SEMplots}, which shows an SEM workload of 131,072 jobs for 4 brain regions and two experimental conditions. This workflow completed in approximately 3 hours.  The logs from the {\tt swift\_plot\_log} utility show the high degree of concurrent overlap between job execution and input and output file staging to remote computing resources. 
+The workflows were developed on and submitted (to Ranger) from a single-core Linux workstation at UChicago running an Intel¨ Xeonª 3.20 GHz CPU. Data staging was performed using the Globus GridFTP protocol and job execution was performed over the Globus GRAM 2 protocol.
+During the third hour of the workflow, Swift achieved very high utilization of the 2,048 allocated processor cores and a steady rate of input and output transfers. The first two hours of the run were more bursty, due to fluctuating grid conditions and data server loads.
+
+
+\begin{figure}
+  \begin{center}
+  {\footnotesize
+    \begin{tabular}{p{14 cm}}
+      \includegraphics[scale=.4]{plots/SEM_left}
+    \end{tabular}
+    }
+    \caption{128K-job SEM fMRI application execution on the Ranger Constellation. Red=active compute jobs, blue=data stage in, green=stage out.}
+    \label{SEMplots}
+  \end{center}
+\end{figure}
+
+Prior work also showed Swift's ability to achieve ample task rates for local and remote submission to high performance clusters\cite{PetascaleScripting_2009}. These prior results are shown in figure \ref{TaskPlots}.
+
+The left figure shows the PTMap application running  the stage 1 processing of the E.coli K12 genome (4,127 sequences) on 2,048 Intrepid cores. The lower plot shows processor utilization as time progresses; Overall, the average per task execution time was 64 seconds, with a standard deviation of 14 seconds. These 4,127 tasks consumed a total of 73 CPU hours, in a span of 161 seconds on 2,048 processor cores, achieving 80 percent utilization.
+
+The right figure below shows performance of Swift running structural equation modeling problem at large scale using on the Ranger Constellation to model neural pathway connectivity from experimental fMRI data\cite{CNARI_2009}. The left figure shows the active jobs for a larger version of the problem type shown in figure \ref{SEMplots}.  This shows an  SEM script executing ~ 418,000 jobs. The red line represents job execution on Ranger; 
+
+\begin{figure}
+  \begin{center}
+  {\footnotesize
+    \begin{tabular}{p{7cm}p{7cm}}
+      \includegraphics[scale=.3]{plots/PTMap_top} &
+      \includegraphics[scale=.3]{plots/SEM_top} \\
+      \includegraphics[scale=.3]{plots/PTMap_bottom} &
+      \includegraphics[scale=.3]{plots/SEM_bottom} \\
+      A. PTMap application on 2,048 nodes of the Blue Gene/P &
+      B. SEM application on varying-size processing allocations on Ranger\\
+    \end{tabular}
+    }
+    \caption{Swift task rates for PTMap and SEM applications on the Blue Gene/P and Ranger}
+    \label{TaskPlots}
+  \end{center}
+\end{figure}
+
+
 %% \begin{figure*}[htbp]
 %%   \begin{center}
 %%     \includegraphics[scale=\plotscale]{plots/sleep}
@@ -1716,7 +1763,7 @@
 Leadership Computing Facility, TeraGrid, the Open Science Grid, the UChicago / Argonne Computation Institute
 Petascale Active Data Store, and the Amazon Web Services Education allocation program.
 
-The quantum glass example in the article is the work of Glen Hocky of the Reichman Lab of the Columbia University Department of Chemistry. We thank Glen for his contributions to the text and code of Sec. 4 and valuable feedback to the Swift project. We gratefully acknowledge the contributions of current and former Swift team members, collaborators, and users: Sarah Kenny, Allan Espinosa, Zhao Zhang, Luiz Gadelha, David Kelly, Milena Nokolic, Jon Monette, Aashish Adhikari, Marc Parisien, Michael Andric, Steven Small, John Dennis, Mats Rynge,  Michael Kubal, Tibi Stef-Praun, Xu Du, Zhengxiong Hou, and Xi Li. The initial implementation of Swift was the work of Yong Zhao and Mihael Hategan; Karajan was designed and implemented by Hategan. We thank Tim Armstrong for helpful comments on the text.
+The glass cavity simulation example in this article is the work of Glen Hocky of the Reichman Lab of the Columbia University Department of Chemistry. We thank Glen for his contributions to the text and code of Sec. 4 and valuable feedback to the Swift project. We gratefully acknowledge the contributions of current and former Swift team members, collaborators, and users: Sarah Kenny, Allan Espinosa, Zhao Zhang, Luiz Gadelha, David Kelly, Milena Nokolic, Jon Monette, Aashish Adhikari, Marc Parisien, Michael Andric, Steven Small, John Dennis, Mats Rynge,  Michael Kubal, Tibi Stef-Praun, Xu Du, Zhengxiong Hou, and Xi Li. The initial implementation of Swift was the work of Yong Zhao and Mihael Hategan; Karajan was designed and implemented by Hategan. We thank Tim Armstrong for helpful comments on the text.
 
 %% \section{TODO}