[Swift-commit] r2386 - text/hpdc09submission

[noreply at svn.ci.uchicago.edu]

Author: benc
Date: 2009-01-01 05:36:24 -0600 (Thu, 01 Jan 2009)
New Revision: 2386

Modified:
   text/hpdc09submission/paper.latex
Log:
a few more bits of rambling

Modified: text/hpdc09submission/paper.latex
===================================================================
--- text/hpdc09submission/paper.latex	2008-12-30 15:43:21 UTC (rev 2385)
+++ text/hpdc09submission/paper.latex	2009-01-01 11:36:24 UTC (rev 2386)
@@ -454,6 +454,7 @@
 \hline
 \end{tabular}
 \caption{SwiftScript built-in mappers}
+\label{mappertable}
 \end{table}
 
 \subsection{The execution environment for component programs}
@@ -513,8 +514,21 @@
 value of that variable has not yet been computed, the filename where
 that value will go is already known.
 
+TODO comment (here?) about how this model appears somewhat constrained
+but provides a well defined atomicity that can be used for various
+reliability mechanisms, site portability, on-site efficiency tuning.
+multi-site and reliabilty are already discussed; but the on-site
+efficiency tuning (eg using GPFS and laying out files in a way that is
+sympathetic to that, potentially using Collective IO fs, and using a
+workernode local filesystem) - that discussion could go into the
+'executing efficiently' section, or a different 'executing efficiently'
+section (change titles...)
+
 \section{Execution}
 
+TODO could briefly describe the execution layer here? how much depth is
+interesting?
+
 \subsection{Executing on a remote site}
 
   With the above restrictions, execution of a unix program on a remote
@@ -547,10 +561,20 @@
 the submitting system to the remote system; and with GRAM\cite{GRAM} and a local
 resource manager (LRM) providing an execution mechanism.
 
-Parameterisation of sites occurs in the \emph{site catalog}, an example
-of which is:
+Sites are defined in the \emph{site catalog}, which contains descriptions
+of each site:
 
-TODO site catalog example here
+ \begin{verbatim}
+  <pool handle="tguc">
+    <gridftp
+      url="gsiftp://tg-gridftp.uc.teragrid.org" /> 
+    <execution provider="gt4" jobmanager="PBS" 
+      url="tg-grid.uc.teragrid.org" />
+    <workdirectory>
+      /home/benc/swifttest
+    </workdirectory>
+  </pool>
+ \end{verbatim}
 
 This file may be constructed by hand or mechanically from some
 pre-existing database (such as a grid's existing discovery system).
@@ -628,7 +652,7 @@
   When any of those jobs begins executing, all other replicas will be
 cancelled.
 
-\subsection{Executing efficiently}
+\subsection{Avoiding job submission penalties}
 
 In many applications, the overhead of job submission through commonly
 available mechanisms, such as through GRAM into an LRM, can dominate
@@ -680,6 +704,20 @@
 way to talk about the 'underlying submission system, that underlies
 coasters and clustering' ?
 
+\subsection{Avoiding filesystem inefficiency}
+
+When running a large number of jobs on a site at once, access to the
+shared filesystem on that site can be a bottleneck.
+
+On large systems, the shared file system is commonly provided by
+GPFS\cite{GPFS}. This can scale well but when used na\"ively can
+exhibit pathological behaviour. Early versions of Swift triggered this
+behaviour by targetting too much file system activity at a single
+working directory, so that GPFS lock contention came to dominate execution
+time.
+
+TODO more...
+
 \section{Example applications}
 
 TODO: two or three applications in brief. discuss both the application
@@ -792,6 +830,9 @@
 
 TODO reference the VDC from VDS\cite{VDS}
 
+TODO write about the stuff on provenance db that I did before - that whole
+document of notes...
+
 \subsection{Workflow GUIs as generators of SwiftScript programs - LONI
 Pipeline}
 
@@ -846,8 +887,8 @@
 \subsection{Debugging}
 
  TODO: debugging of distributed system - can have a non-futures section
-on what is available now, as well as mentioning CEDPS\cite{CEDPS} as
-somewhat promising(?) for the future.
+on what is available now - logprocessing module, as well as
+mentioning CEDPS\cite{CEDPS} as somewhat promising(?) for the future.
 
 \section{Implementation status}
 
@@ -866,7 +907,7 @@
 
   Reference Swift as a follow-on project to VDL in VDS; how does XDTM fit
     into this? Is it of any interest other than as part of the
-    project history?
+    project history? And is history of this project interesting? maybe so...
 
 
   Acknowledgement of all developers names?
@@ -926,7 +967,14 @@
 
 \bibitem{MAPREDUCE} - mapreduce
 
+\bibitem{TERAGRID} - teragrid
 
+\bibitem{OSG} - open science grid
+
+\bibitem{ReSS} - ress
+
+\bibitem{GPFS} - GPFS
+
 \end{thebibliography}
 \end{document}