[Swift-devel] Questions on coaster behavior

[Mihael Hategan]

On Fri, 2012-02-24 at 08:38 -0600, Michael Wilde wrote:
> Hi Mihael, All,
> 
> I wanted to confirm some aspects of Coaster behavior that are still unclear to me after re-reading the UCC paper:
> 
> Scheduling: the coaster provider scheduler starts a number of worker blocks that are sized (in time and nodes) based on the size of its queue when it computes a schedule. This queue consists of jobs that were emitted by Swift to the provider based on the site throttle.
> 
> (note that by "job" here I mean the app() execution, not the LRM job).
> 
> But the coaster provider does not actually launch a job on a free
> coaster slot until the slot is available, right?

That is correct. Jobs are queued by the coaster service, blocks are
submitted and killed based on the shape of the queued jobs, and once the
blocks are running, jobs are sent to them.

>  Ie, there is no tight connection between the coaster slot that a
> job's time estimate contributed to, and the worker that the job is
> actually run on, right?

That's right. Only the totals are tightly connected.

>  Jobs are placed on workers at the last possible moment, and thus when
> a worker can take a job, it can get *any* job that is queued for that
> site.

Jobs are placed on workers when workers don't have anything else to do.
Each worker will get the longest job that it can fit.

>  Is all this correct?  The key point behind this question being "is
> the coaster scheduler dynamic enough to hand cases where the job
> runtime estimates were conservative, and make best use of all
> available worker cores"? 

Yes. That's the basic idea.

> 
> Staging: There are no cases in which the coaster provider staging mechanism pre-stages input data, right?

If by pre-staging you mean staging before the job makes it to the
worker, then no. The worker initiates staging.