[Swift-devel] Re: Throttling issues for coasters on OSG

[Mihael Hategan]

On Thu, 2010-10-21 at 11:24 -0600, Michael Wilde wrote:

> Change 1: bias scheduling based on job start rather than job
> completion. We have discussed this many times in the past, and it
> seems more optimal in all cases I can imagine. Do you agree? Can you
> point us to code modules where we can try this and evaluate it?
> 
> Change 2: for coasters, use knowledge of how many worker slots are
> registered on each site to always keep the throttle set to exactly
> this value, and hence to ensure that, given sufficient ready tasks,
> all coaster slots are always filled?
> 
> I think that for Allan's workload (many hundred thousand 5 minute
> tasks), change 2 will be most effective and in almost all cases
> desirable.

I dare say not almost, but all cases.

> 
> A third issue relates to staging, and is perhaps a non-issue if change
> 2 works. The issue is: if we use provider staging for this workload,
> which is likely desirable, will data staging be done at the time a
> worker pulls a job, or will it be done in advance of that, and
> asynchronously with job start, just as it would be with ordinary
> staging?

Staging is done after the job is sent to the worker. This is unlike it
would be done with ordinary staging.

Mihael