[Swift-commit] r3386 - text/internals/trunk

[noreply at svn.ci.uchicago.edu]

Author: aespinosa
Date: 2010-06-17 14:42:58 -0500 (Thu, 17 Jun 2010)
New Revision: 3386

Modified:
   text/internals/trunk/internals.tex
Log:
Mappers, scheduler and coasters

Modified: text/internals/trunk/internals.tex
===================================================================
--- text/internals/trunk/internals.tex	2010-06-17 16:47:39 UTC (rev 3385)
+++ text/internals/trunk/internals.tex	2010-06-17 19:42:58 UTC (rev 3386)
@@ -415,12 +415,158 @@
 uParallel(1,2) -> (1,2) or (2,1)
   - saves on memory usage
 
-\section{Replication groups}
+\section{Mappers}
 
-\section{Restarting}
+No distinction between input and output mappers.
 
+remap(v, f)
+isRemappable()
+
+map(): V -> F
+Rmap(): F -> V
+
+existing: -> { x | x \in V} - invoked at the beginning of Swift.  If it has no
+dependencies, the mapper gets resolved when swift starts.  Occasionally gets
+called in dependencies.  Normally invoked for input variables.  Invoked before
+starting apps generally.
+
+Exception that hold existing up: fixed array mapper as a result of some shell
+command.  Equivalent to use an ext mapper.
+
+exists(): V -> boolean  i.e. exits("a[1]")
+
+Map() is invoked whenever its needed.
+
+Ext mapper: ``pregrown }
+
+potatoes''.  You don't have to use every mapping.
+
+Isomorphism in the space of variables and the space of files.
+
+There is no atomic mapping operation.  All mapping is done on demand.
+
+f(x) = sin(x)
+
+vs
+
+f(\pi) = 0
+f(2\pi) = 1
+
+same but you can't get an infinite mapping that easily.
+
+existing() needs to gets invoked after the application is done.  existing()
+doesn't get called in a stageout.
+
+get the type system.
+
+Concurrent by definition is remappable.
+
+Remappable - aliasing.
+
+file f,g;
+f = g;
+
+dest \ src &   false     & true
+false      & file copy   & src.remap()
+true       & des.remap() & 
+
+file s <concurrent>;
+file dest <"a.txt">;
+s = app();
+dest = s; // dealt with before the application is run since swift constructs the
+          // graph first.
+
+Swift has nominal type equivalence
+
+type img;
+type hdr;
+
+img f;
+hdr g;
+
+cannot do an f = g.
+
+Leaves get mapped.
+
 \section{Scheduler}
 
+Scheduling in general is NPC.
+
+Swift schedules one job at a time from a list of ready jobs.  Coasters and
+clusters make a time window.  Time window is all in the provider and clustering
+site and completely separated.  
+
+Score modifiers:
+
+1.  sending a job decreases the score a little bit (more loaded).  a leveling
+mechanism.
+  * Load is included in the calculation of the score.
+2.  A job finishes - increase number of jobs sent to this site since it runs
+jobs quicker.
+
+Similar from increasing a credit limit in credit cards.
+
+unscaled score and scaled score as a sigmoid function
+
+scaled \in [1/T, T]
+
+Fast ramp up of the score if a site finishes jobs and then converge to a
+maximum to avoid blowing up the instability of the algorithm.
+
+A discrete time feedback system.  Process control
+
+WeightedHostScheduler class - contains the implementation of the sigmoid
+function.  Almost contained in one place.
+
+Throttle of jobs and tranfers
+ * Jobs - jobThrottle and initialScore
+
+\section{Replication}
+
+Resending queued jobs somewhere else.  Deleting other queued jobs when one of
+the jobs in the replica jobs are in the active state.
+
+\section{Clustering}
+
+A lot of small jobs chunked into 1 request.  Class (?) - swift specific
+(VDSAdaptiveScheduler in the logs).  Amortizing the queueing time.
+
+\section{Restarting}
+The original Baker and Hewitt paper described implicit futures, which are naturally supported in the Actor model of computation and pure object-oriented programming languages like Smalltalk. The Friedman and Wise paper described only explicit futures, probably reflecting the difficulty of efficiently implementing implicit futures on stock hardware. The difficulty is that stock hardware does not deal with futures for primitive data types like integers. For example, an add instruction does not know how to deal with 3 + future factorial(100000). In pure object or Actor languages this problem can be solved by sending future factorial(100000) the message +[3], which asks the future to add 3 to itself and return the result. Note that the message passing approach works regardless of when factorial(100000) finishes computation and that no stinging/forcing is required.
 \section{Coasters}
 
+Large workflow 
+
+Minimal software stack.  No need to install stuff on the remote stack.
+
+Client -> Server -> Worker
+
+Problem of GRAM2 callbacks.
+
+Choices:
+1.  keep the connection
+2.  callbacks or polling
+
+PERL's event-based system for the workers
+
+overAllocation factor - determine block's walltime
+ * lowOverAllocation
+ * highOverAllocation
+
+ - creates a decreasing exponential curve
+
+nodeGranularity - width of block (multiple)
+
+workersPerNode - 
+
+overAllocation on largest job
+
+Spread - 
+
+Area of the box is equal to total cpu job hours
+
+\section{CDM}
+
+Disabling file transfer/ data staging stuff at runtime.
+
 \end{document}