[Swift-devel] MapReduce and swift

[Yadu Nand]

Hi,

I have a couple of questions about how mapreduce would look like if it were
implemented in Swift. It would be great to hear ideas on these :

1) Google's MapReduce as well as Hadoop's makes certain assumptions about
the data which are generated/used at each stage of processing . Mostly
key-value
pairs are used and the keys play a role in the reduce stage. Does this make
sense
in swift, or does the functional programming style make more sense here ?
Swift
mostly remains agnostic on file contents, so does it matter what is
the format of the
content of the files ?

2) Currently Swift would stage-in the results from a map job, and then
stage-out to the
reducers, while this could be shorted by just moving the results directly
from the map
nodes to the reduce nodes. What if we had semantics which would retrieve
file locations
rather than actual files and then choose to de-reference only when we
needed to?
(I do not think we have a method to do this already)

say,
type file_ptrs[];                                // array of file pointers
file_ptrs = map(foo, array);           // here, we just get an array of
file pointers
result     = reduce(bar, file_ptrs);  // reduction would involve getting
the actual files

Maybe the above could be wrapped up as :
result = mapreduce(foo, bar, array);

-- 
Thanks and Regards,
Yadu Nand B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20120902/3e7c8526/attachment.html>