[Swift-user] Re: [Swift-devel] How to the maximum number of concurrent jobs allowed on a site to a fixed size?

[Yi Zhu]

Hi, Allan

Thanks for you reply.

I've tried the way you suggested, but it doesn't work as I expected.


Suppose I have a PBS cluster with 10 worker nodes,(btw. I don't have the 
privilege to offline/shutdown those worker nodes), Now I want to use 
some portion of workers nodes rather than all of them to do jobs that 
are submitted by swift. The way i think is setting the value of
"maximum number of concurrent jobs allowed on a site" to  number of 
workers nodes I wish to run. (e.g. If I only want 5 out 10 workers node 
keep busy, I can choose to set the  value of maximum number of 
concurrent jobs to 5 ,so there are only 5 jobs can be run on pbs 
concurrently)

since I can not just set that value (i.e. swift use a formula to 
dynamically calculate the value), I need find another way to sort it out.

Change the value of "foreach.max.threads" doesn't  work as we expected:

when i set it to 1, swift just freeze when started.
when i set it to 2, it seems submit job one by one.
when i set it to 3, swift submit 4 jobs at once.
when i set it to 4, swift submit 9 jobs at  once.
when i set it to 5, swift submit 16 jobs at  once.
when i set it to 10,swift submit 81 jobs at once.

-Yi






Allan Espinosa wrote:
> hi yi,
> 
> in swift.properites file you set
> 
> foreach.max.threads=1024
> 
> to
> 
> foreach.max.threads=N
> 
> where N is the max number concurrent jobs you want per swift session.
> Also when you set score to be ridicuosly high (ie 10000) you always
> get the maximum theoretical number of jobs you want based on the
> throttling parameters.
> 
> -Allan
> 
> 2009/8/6 Yi Zhu <yizhu at cs.uchicago.edu>:
>> Hi, all
>>
>> As we've already know, Swift dynamically change the maximum number of
>> concurrent jobs allowed on a site based on the performance history of that
>> site. According to swift Document: Each site is assigned a score (initially
>> 1), which can increase or decrease based on whether the site yields
>> successful or faulty job runs. The score for a site can take values in the
>> (0.1, 100) interval. The number of allowed jobs is calculated using the
>> following formula:
>>
>> 2 + score*throttle.score.job.factor
>>
>> We can change the throttle.score.job.factor in sites.xml or swift.properties
>> files, but since the "score" value can be increased/decreased during the
>> execution, It seems that we can not really set the maximum  number of
>> concurrent jobs allowed on a site to a fixed number. Anyone have any idea of
>> that?
>>
>>
>> Many Thanks.
>>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>