[Swift-devel] scalability updates

[Mihael Hategan]

I've committed two main things today:

1. A foreach thread limiting patch, which limits the maximum number of
concurrent threads that a foreach can have at any time. The default is
at 1024, but it is configurable in swift.properties. For scripts whose
main memory hog is large numbers of iterations in foreach loops, this
should allow things to run with considerably less memory.

2. A lazy range function (the [x:y] operator). The previous one was
silly. Simply writing [0:1000000] would cause swift to run out of memory
because it was trying to create a swift array with 1000000 elements
before running a single iteration on it.

In principle, these two would roughly translate into the following:
- a likely demise to several of our swift-runs-out-of-memory scenarios.
Though there's still a bit to go here, because arrays in general in
swift keep too much things in memory.
- skenny type scripts (foreach i in [1:65535] { doStuff(); }) will not
see that 5 minute delay before the first job is submitted.
- Ben's provenance stuff may break if it relies on items returned by
range() reporting a path-from-root containing the array itself (as array
elements are roots themselves).