[Swift-devel] scheduler stuff for Google Summer of Code 2009

[Michael Wilde]

These all sound good.

Another scheduler-related set of projects relates to algorithms around 
coasters:

- how to manage (grow and shrink) the size of coaster pools
- how to size the time requests for the workers (perhaps dynamically)
- how the current dynamic throttle works for coasters
- and probably many more.

Some more areas:

- a detailed evaluation and tuning of the throttle heuristics for 
many-site workflows

- automatically clustering more diverse dags, and pipelining

- running Swift on clouds like E2C and Azure

- scaling swift to 1M+ task workflows, efficiently (streaming the mappers)

- service oriented workflows

- extending CIO techniques back to grid environments

- creating a lightweight "embedded swift" VM for running workflows from 
*within* perl, python, R, etc.

These cover a wide space of useful-to-crazy, easy-to-hard, etc.  Just 
tossing them out.

- Mike




On 2/11/09 5:39 AM, Ben Clifford wrote:
> Google Summer of Code 2009 mentor applications open in a couple of weeks. 
> dev.globus likely will be applying again. I have a few ideas for Swift 
> projects within dev.globus that I'd like to mentor.
> 
> One idea that I'm kinda fuzzy on but there might be interesting work to do 
> is implementing more interesting scheduler behaviour.
> 
> Various people have talked in the past about these, that I think have some 
> decent level of merit:
> 
>  a) changing ordering of execution of swift-level jobs based on how many
>      other swift-level jobs depend on that first job
> 
>  b) reordering stageins and stageouts so to allow (in addition to the 
>      present as-they-come (I think) policy) "prefer stageins" (which would 
>      get more jobs going sooner, but incurring an expense in that 
>      stageouts would happen slower, and in our present restart model 
>      reduce the speed as which jobs complete enough for restart), and
>      "prefer stageouts", which would get completed results out to submit 
>      side faster, at the expense of job execution speed.
> 
>  c) data affinity - there was some messing round with this, but it
>      resulted in code that did not work (which is fine for that project, 
>      as it was not production code oriented, but not for committing to
>      the codebase). So potentially this could be reimplemented or the 
>      existing code tidied up as part of this.
> 
> Comments and additional ideas...
>