[Swift-devel] scalability updates

[Mihael Hategan]

On Tue, 2009-03-10 at 00:01 -0500, Michael Wilde wrote:
> Very nice! These look very promising. One interesting test would be 
> doing a million localhost echos in a simple foreach loop on a 
> range-initialized array, and looking at the memory needs.
> 
> These 2 enhancements seem to pave the way to making streamed (or 
> on-demand) mappers useful. For those, I think we need a mapper paradigm 
> design adjustment discussion.

I think we thought out the mappers to be procedural (i.e. not hold
state) from the beginning, so the problem does not seem to be in the
design of the mappers. Rather, it's the implementation of some of the
mappers and the implementation of swift data structures (parts of an
array cannot be garbage-collected independently).

> 
> But I think the next thing to work on in scalability would be the 
> Condor-G provider, so we can run large coaster runs with more 
> concurrency. The multi-cpu coaster allocator might be a workaround to 
> re-consider if a condor-G provider is too far off.
> 
> Assuming (or when) there's agreement that this is the best solution for 
> coaster scalability, I'd like to propose that as the next big task on 
> your to-do list.

Yes. Sounds reasonable. Now, only if I could find a condor/condor-g
installation that I have access to...