[Swift-user] Limiting concurrent executing jobs

[Gustafson, William I]

Hi,

I am trying to use Swift to execute a Perl script on my local cluster. The script is very disk intensive and the Lustre disk system bottlenecks if I run more than ~10 instances of the script. Therefore, I need to make sure that Swift only submits up to a certain number of jobs. I also need this ability so that I do not take over the whole cluster, preventing other users from getting access. What setting do I need to change so that I can have this functionality? I read through the Users Guide and the closest I have found is setting the jobThrottle setting. But, this either sets a very low hard limit when jobThrottle=0, or else allows the concurrently running jobs to slowly grow. For example, if I set jobThrottle=1 I initially get two jobs running, but the running job count slowly grows up to 25+ if I let it.

Also, it would be nice to have a setting to indicate that I can use more than the initial default of 2 jobs right from the beginning of the run. This would prevent some wasted time at the beginning of the run. Is this possible?

I am using localhost for the Swift pool, and the job submission actually happens in the Perl script. My cluster is setup with SLURM so I cannot use Swift’s built-in job submission protocols.

Thanks for the help.

-Bill

_______________________________________________
William I. Gustafson Jr., Ph.D.
Scientist
ATMOSPHERIC SCIENCES AND GLOBAL CHANGE DIVISION

Pacific Northwest National Laboratory
http://www.pnl.gov/atmospheric/staff/staff_info.asp?staff_num=5716
http://www.researcherid.com/rid/A-7732-2008