[Swift-devel] scalability updates

[Michael Wilde]

On 3/10/09 12:07 AM, Mihael Hategan wrote:
> On Tue, 2009-03-10 at 00:01 -0500, Michael Wilde wrote:
>> Very nice! These look very promising. One interesting test would be 
>> doing a million localhost echos in a simple foreach loop on a 
>> range-initialized array, and looking at the memory needs.
>>
>> These 2 enhancements seem to pave the way to making streamed (or 
>> on-demand) mappers useful. For those, I think we need a mapper paradigm 
>> design adjustment discussion.
> 
> I think we thought out the mappers to be procedural (i.e. not hold
> state) from the beginning, so the problem does not seem to be in the
> design of the mappers. Rather, it's the implementation of some of the
> mappers and the implementation of swift data structures (parts of an
> array cannot be garbage-collected independently).

By that I meant the functionality issues in mappers (ie, ability to map 
various user patterns easily, like the things I ran into in the OOPS 
app). Other than the streaming thing, I wasnt concerned with the 
performance of the mappers.
> 
>> But I think the next thing to work on in scalability would be the 
>> Condor-G provider, so we can run large coaster runs with more 
>> concurrency. The multi-cpu coaster allocator might be a workaround to 
>> re-consider if a condor-G provider is too far off.
>>
>> Assuming (or when) there's agreement that this is the best solution for 
>> coaster scalability, I'd like to propose that as the next big task on 
>> your to-do list.
> 
> Yes. Sounds reasonable. Now, only if I could find a condor/condor-g
> installation that I have access to...

Great, we'll find you one. The local ITB site that Suchandra maintains 
is a good choice, and I think we can get Greg to install the client 
package on TeraGrid if its not already there, or Ti on communicado.