[Swift-user] Limiting concurrent executing jobs

[Ketan Maheshwari]

Hi Bill,

Your finding is right. To control the number of concurrent jobs on cluster, you will need to set the jobThrottle parameter in the sites file. The following formula decides the throttle value:
 
Number of concurrent jobs = ( jobthrottle + 0.01 ) X 100

Which means that if you want say 10 concurrent jobs to run in your cluster at any time, the job throttle needs to be 0.09

Furthermore, if you want to control the number of concurrent threads spawned by Swift engine, you need to set the foreach.max.threads in config file to a value you desire the numbers of threads to be.

Hope that helps,

Ketan


On May 2, 2011, at 5:49 PM, Gustafson, William I wrote:

> Hi,
> 
> I am trying to use Swift to execute a Perl script on my local cluster. The script is very disk intensive and the Lustre disk system bottlenecks if I run more than ~10 instances of the script. Therefore, I need to make sure that Swift only submits up to a certain number of jobs. I also need this ability so that I do not take over the whole cluster, preventing other users from getting access. What setting do I need to change so that I can have this functionality? I read through the Users Guide and the closest I have found is setting the jobThrottle setting. But, this either sets a very low hard limit when jobThrottle=0, or else allows the concurrently running jobs to slowly grow. For example, if I set jobThrottle=1 I initially get two jobs running, but the running job count slowly grows up to 25+ if I let it.
> 
> Also, it would be nice to have a setting to indicate that I can use more than the initial default of 2 jobs right from the beginning of the run. This would prevent some wasted time at the beginning of the run. Is this possible?
> 
> I am using localhost for the Swift pool, and the job submission actually happens in the Perl script. My cluster is setup with SLURM so I cannot use Swift’s built-in job submission protocols.
> 
> Thanks for the help.
> 
> -Bill
> 
> _______________________________________________
> William I. Gustafson Jr., Ph.D.
> Scientist
> ATMOSPHERIC SCIENCES AND GLOBAL CHANGE DIVISION
> 
> Pacific Northwest National Laboratory
> http://www.pnl.gov/atmospheric/staff/staff_info.asp?staff_num=5716
> http://www.researcherid.com/rid/A-7732-2008
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user