[Swift-devel] Re: Is there a way to make swift send 256 tasks at a time

[Michael Wilde]

Lets experiment and measure with the current settable properties before 
we consider any changes.

Note that we already have a tail effect on most workloads; perhaps the 
Swift dynamics will have little effect on that. They might even have a 
positive effect by biasing work towards psets that are returning work 
faster (due to the fact that they wound up with shorter jobs). Im not 
sure if that happens at the moment.

In real usage, users may request longer provisions and take advantage of 
free cycles in the "tails" for real work. And/or they may set 
provisioned resources to free up for other users as their own usage 
tails off.

- Mike

On 7/23/08 9:12 AM, Ioan Raicu wrote:
> 
> 
> Michael Wilde wrote:
>> ...
>>
>> But, I suspect you want *exactly* 256 to keep all the cores in each 
>> pset busy without over-committing jobs. So what I think the best you 
>> can do is start with the throttle at 3, and the score at 85 which will 
>> give you 3 * 85 + 2 = 257 jobs. If the throttle creeps up for good 
>> behavior, then you'll wind up with a little overcomiting, which isnt 
>> bad. It wont go over 302 with a throttle of 3.
> Can the throttle.score.job.factor be set to a float?  Then setting it to 
> 2.54 will give you exactly 256, assuming all tasks come back successful.
>>
>> I suspect we could try a patch that prevents the score from changing.
>>
>> Can you tell from Falkon logs as we ramp up testing of this, whether 
>> overcomiting is a problem? I suspect at worst it will exacerbate 
>> tail-effects as the workflow is winding down and some resources sit 
>> idle while others are overcomited.
> Yes, it is the tail end of the runs that will be most affected.
>>
>> Lastly, note that the fact that the score can drop might eventually 
>> prove useful in handling psets that go bad in the middle of long runs. 
> Right!
> Ioan
>