[Swift-user] Re: [Swift-devel] How to the maximum number of concurrent jobs allowed on a site to a fixed size?

[Allan Espinosa]

the formula is 5 = 2 + 100 * throttle
so here the throttle should be 0.03.

another approach is to create a pbs queue / reservation with only 5
nodes available then just set swift to submit as much jobs as it can.

2009/8/6 Yi Zhu <yizhu at cs.uchicago.edu>:
> Hi, Allan
>
> Thanks for you reply.
>
> I've tried the way you suggested, but it doesn't work as I expected.
>
>
> Suppose I have a PBS cluster with 10 worker nodes,(btw. I don't have the
> privilege to offline/shutdown those worker nodes), Now I want to use some
> portion of workers nodes rather than all of them to do jobs that are
> submitted by swift. The way i think is setting the value of
> "maximum number of concurrent jobs allowed on a site" to  number of workers
> nodes I wish to run. (e.g. If I only want 5 out 10 workers node keep busy, I
> can choose to set the  value of maximum number of concurrent jobs to 5 ,so
> there are only 5 jobs can be run on pbs concurrently)
>
> since I can not just set that value (i.e. swift use a formula to dynamically
> calculate the value), I need find another way to sort it out.
>
> Change the value of "foreach.max.threads" doesn't  work as we expected:
>
> when i set it to 1, swift just freeze when started.
> when i set it to 2, it seems submit job one by one.
> when i set it to 3, swift submit 4 jobs at once.
> when i set it to 4, swift submit 9 jobs at  once.
> when i set it to 5, swift submit 16 jobs at  once.
> when i set it to 10,swift submit 81 jobs at once.
>
> -Yi
>
>
>
>
>
>
> Allan Espinosa wrote:
>>
>> hi yi,
>>
>> in swift.properites file you set
>>
>> foreach.max.threads=1024
>>
>> to
>>
>> foreach.max.threads=N
>>
>> where N is the max number concurrent jobs you want per swift session.
>> Also when you set score to be ridicuosly high (ie 10000) you always
>> get the maximum theoretical number of jobs you want based on the
>> throttling parameters.
>>
>> -Allan
>>
>> 2009/8/6 Yi Zhu <yizhu at cs.uchicago.edu>:
>>>
>>> Hi, all
>>>
>>> As we've already know, Swift dynamically change the maximum number of
>>> concurrent jobs allowed on a site based on the performance history of
>>> that
>>> site. According to swift Document: Each site is assigned a score
>>> (initially
>>> 1), which can increase or decrease based on whether the site yields
>>> successful or faulty job runs. The score for a site can take values in
>>> the
>>> (0.1, 100) interval. The number of allowed jobs is calculated using the
>>> following formula:
>>>
>>> 2 + score*throttle.score.job.factor
>>>
>>> We can change the throttle.score.job.factor in sites.xml or
>>> swift.properties
>>> files, but since the "score" value can be increased/decreased during the
>>> execution, It seems that we can not really set the maximum  number of
>>> concurrent jobs allowed on a site to a fixed number. Anyone have any idea
>>> of
>>> that?