[Swift-devel] Re: [Swft] Q about throttling

[Mike Wilde]

Is there a configurable retry delay after failure?

I think you need to examine the overall workflow dependency structure.

Also, I recall from older perf charts that there's an option to enable/disable 
pipelining.  With pipelining disabled, it seems that Swift will wait for an 
entire dataset/foreach or procedure to finish before starting any tasks that 
depend on the foreach or procedure.

Mihael, can you look at some of these issues when you are back online and rested?

- Mike

Ioan Raicu wrote, On 6/22/2007 9:06 AM:
> No, I didn't keep track of this info, unless Swift does this through 
> some of its logs. 
> 
> Over the last week, my observations have been the following: Swift is 
> more than capable and willing to send out many tasks as long as they are 
> independent (as can be seen in this graph where probably 6800 tasks got 
> submitted), but thereafter, it had no other burst of task submission, 
> although I believe it could have send out more.  For example, there were 
> 2500+ tasks that failed in the middle of those 6800 tasks (which were 
> all independent), why were 2500 tasks not resubmitted all at once... 
> they were each about 200 seconds long, so most of them should have 
> certainly showed up in the wait queue.
> 
> Ioan
> 
> Ben Clifford wrote:
>>> kept busy, and the Falkon queue length was relatively at 0... so this means
>>> that Swift was not submitting fast enough to keep all the executors busy.
>>>     
>>
>> interesting. though around t=1000 there is a rapid burst of submission 
>> getting the queue length up to about 6000 in a few minutes.
>>
>> Do you know what the cpu time usage of the swift submitting JVM was over 
>> that time period?
>>
>>   
> 
> -- 
> ============================================
> Ioan Raicu
> Ph.D. Student
> ============================================
> Distributed Systems Laboratory
> Computer Science Department
> University of Chicago
> 1100 E. 58th Street, Ryerson Hall
> Chicago, IL 60637
> ============================================
> Email: iraicu at cs.uchicago.edu
> Web:   http://www.cs.uchicago.edu/~iraicu
>        http://dsl.cs.uchicago.edu/
> ============================================
> ============================================
> 

-- 
Mike Wilde
Computation Institute, University of Chicago
Math & Computer Science Division
Argonne National Laboratory
Argonne, IL   60439    USA
tel 630-252-7497 fax 630-252-1997