[Swift-devel] scheduler stuff for Google Summer of Code 2009

[Ian Foster]

Mihael:

I think we are exploring the limits of email as a communication  
vehicle :-)

I wasn't talking about GRAM at all.

I had understood someone to say that we can't run 1M tasks because  
each task needs 10KB (or similar), and 1M*10KB is a lot.

I was observing that a workflow of 1M tasks can still be run if a  
smaller number are active at a time. That may seem like splitting  
hairs, but in fact that was a big reason for Swift's design. We would  
have applications that had multiple phases, each with thousands of  
tasks. As a DAG, this would not fit in memory (and was a pain to write  
of course). As a Swift program, we might have:

foreach(i in 1:10,000) f()
foreach(i in 1:10,000) g()
etc.

Ian.



On Feb 11, 2009, at 4:49 PM, Mihael Hategan wrote:

> ----- Ian Foster <foster at anl.gov> wrote:
>> Mihael:
>>
>> I don't know how to parse your comment.
>>
>> If I write a program that performs a series of operations on many
>> files, involving 1M tasks during its execution, but with only 10,000
>> active at each step, why is that not interesting? Or are you re-
>> defining the problem to "have 1M tasks active at once"? That is a
>> useful thing to be able to do, I am sure, but that does not mean that
>> the former is not useful also.
>
> In the context in which the engine cannot reasonably support 1M tasks,
> it seems fairly pointless to say that, e.g. a faster GRAM is better  
> for
> running 1M tasks with Swift. It makes no difference.