[Swift-devel] Re: replication vs site score

[Ioan Raicu]

Does a batch-queue prediction service help things in any way?
https://portal.teragrid.org/gridsphere/gridsphere?cid=queue-prediction

I've always wondered how the Swift scheduler would behave differently if 
it had statistical information about queue times. Qin, have you compared 
your job replication strategy with one that was cognizant of the 
expected wait queue time, in order to meet deadlines? On the surface, 
assuming that the batch queue prediction is accurate, it would seem that 
scheduling with known queue times might solve the same deadline 
cognizant scheduling problem, but without wasting resources by 
unnecessary replication. The place where the queue prediction doesn't 
help, is when there is a bad node which causes an application to be slow 
or fail. In this case, replication is probably the better recourse to 
guarantee meeting deadlines.

Here is their latest paper on this: 
http://www.springerlink.com/content/7552901360631246/fulltext.pdf. The 
system is deployed on the TeraGrid, and has been for a few years now. As 
far as I have heard, it is quite robust and accurate.

Cheers,
Ioan

Ben Clifford wrote:
> On Wed, 8 Apr 2009, Mihael Hategan wrote:
>
> This:
>
>   
>> planning the whole workflow buys us little in a (very) dynamic
>> environment in which submitting a job one minute later may mean the
>> difference between 1 minute of queue time and one hour of queue time
>>     
>
> and this:
>
>   
>> You need some SLA/QOS to address that.
>>     
>
> seem to be significant characteristics that make the environments we run 
> on not amenable to scheduling in the traditional sense. The lack of any 
> meaningful guarantees about almost anything time-related makes everything 
> basically opportunistic rather than scheduled.
>
>   

-- 
===================================================
Ioan Raicu, Ph.D.
===================================================
Distributed Systems Laboratory
Computer Science Department
University of Chicago
1100 E. 58th Street, Ryerson Hall
Chicago, IL 60637
===================================================
Email: iraicu at cs.uchicago.edu
Web:   http://www.cs.uchicago.edu/~iraicu
http://dev.globus.org/wiki/Incubator/Falkon
http://dsl-wiki.cs.uchicago.edu/index.php/Main_Page
===================================================
===================================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20090408/58cb81d1/attachment.html>