[Swift-devel] scalability updates

[Michael Wilde]

Very nice! These look very promising. One interesting test would be 
doing a million localhost echos in a simple foreach loop on a 
range-initialized array, and looking at the memory needs.

These 2 enhancements seem to pave the way to making streamed (or 
on-demand) mappers useful. For those, I think we need a mapper paradigm 
design adjustment discussion.

But I think the next thing to work on in scalability would be the 
Condor-G provider, so we can run large coaster runs with more 
concurrency. The multi-cpu coaster allocator might be a workaround to 
re-consider if a condor-G provider is too far off.

Assuming (or when) there's agreement that this is the best solution for 
coaster scalability, I'd like to propose that as the next big task on 
your to-do list.

On 3/9/09 3:25 PM, Mihael Hategan wrote:
> I've committed two main things today:
> 
> 1. A foreach thread limiting patch, which limits the maximum number of
> concurrent threads that a foreach can have at any time. The default is
> at 1024, but it is configurable in swift.properties. For scripts whose
> main memory hog is large numbers of iterations in foreach loops, this
> should allow things to run with considerably less memory.
> 
> 2. A lazy range function (the [x:y] operator). The previous one was
> silly. Simply writing [0:1000000] would cause swift to run out of memory
> because it was trying to create a swift array with 1000000 elements
> before running a single iteration on it.
> 
> In principle, these two would roughly translate into the following:
> - a likely demise to several of our swift-runs-out-of-memory scenarios.
> Though there's still a bit to go here, because arrays in general in
> swift keep too much things in memory.
> - skenny type scripts (foreach i in [1:65535] { doStuff(); }) will not
> see that 5 minute delay before the first job is submitted.
> - Ben's provenance stuff may break if it relies on items returned by
> range() reporting a path-from-root containing the array itself (as array
> elements are roots themselves).
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel