[Swift-devel] Associative array in Swift [GSoC]

[Mihael Hategan]

On Fri, 2011-06-24 at 11:47 +0530, Yadu Nand wrote:
> Hi,
> 
> >> > I think the typical example (if I remember correctly) used in the
> >> > map/reduce paper is a reasonable case: count the number of occurrences
> >> > of each word in a set of files.
> 
> As far as I know and understand (which is very little), MapReduce
> as used in Hadoop works because it runs on top of HDFS. It is not
> the ability to compute in a distributed fashion that gives advantage
> but the idea of the operation going to the system which holds the
> data. This is handled by HDFS doing replication and other stuff.
> 
> It is the huge size of the data repositories, and their distributed
> storage structure that allowed MapReduce to work its magic. Do
> we have that  ? Or could we have that on swift ?
> 

I'd say many things conspired to make m/r work: came from google, simple
concept, efficiently distribut-able, etc. But yes, for I/O bound things
minimizing data movement is an important aspect. However, that is a
scheduling issue which is somewhat orthogonal to the language issue. It
is not currently done by swift, but we previously explored data location
biasing for site selection, but we never got to actually writing
committable code for it.