[Swift-devel] Re: [Swft] Q about throttling

[Mihael Hategan]

On Fri, 2007-06-22 at 09:39 -0500, Mike Wilde wrote:
> Is there a configurable retry delay after failure?
> 
> I think you need to examine the overall workflow dependency structure.
> 
> Also, I recall from older perf charts that there's an option to enable/disable 
> pipelining.  With pipelining disabled, it seems that Swift will wait for an 
> entire dataset/foreach or procedure to finish before starting any tasks that 
> depend on the foreach or procedure.

I don't think that was in swift.

> 
> Mihael, can you look at some of these issues when you are back online and rested?

You say that as if I normally don't :)

> 
> - Mike
> 
> Ioan Raicu wrote, On 6/22/2007 9:06 AM:
> > No, I didn't keep track of this info, unless Swift does this through 
> > some of its logs. 
> > 
> > Over the last week, my observations have been the following: Swift is 
> > more than capable and willing to send out many tasks as long as they are 
> > independent (as can be seen in this graph where probably 6800 tasks got 
> > submitted), but thereafter, it had no other burst of task submission, 
> > although I believe it could have send out more.  For example, there were 
> > 2500+ tasks that failed in the middle of those 6800 tasks (which were 
> > all independent), why were 2500 tasks not resubmitted all at once... 
> > they were each about 200 seconds long, so most of them should have 
> > certainly showed up in the wait queue.
> > 
> > Ioan
> > 
> > Ben Clifford wrote:
> >>> kept busy, and the Falkon queue length was relatively at 0... so this means
> >>> that Swift was not submitting fast enough to keep all the executors busy.
> >>>     
> >>
> >> interesting. though around t=1000 there is a rapid burst of submission 
> >> getting the queue length up to about 6000 in a few minutes.
> >>
> >> Do you know what the cpu time usage of the swift submitting JVM was over 
> >> that time period?
> >>
> >>   
> > 
> > -- 
> > ============================================
> > Ioan Raicu
> > Ph.D. Student
> > ============================================
> > Distributed Systems Laboratory
> > Computer Science Department
> > University of Chicago
> > 1100 E. 58th Street, Ryerson Hall
> > Chicago, IL 60637
> > ============================================
> > Email: iraicu at cs.uchicago.edu
> > Web:   http://www.cs.uchicago.edu/~iraicu
> >        http://dsl.cs.uchicago.edu/
> > ============================================
> > ============================================
> > 
>