[Swift-user] Limiting concurrent executing jobs

[Mihael Hategan]

You can prevent the ramp-up by setting initialScore (in sites.xml) to
some large number (>1000).

You could also try to specify a scratch directory, such that all jobs
will be run there instead of lustre.

Mihael

On Mon, 2011-05-02 at 15:49 -0700, Gustafson, William I wrote:
> Hi,
> 
> I am trying to use Swift to execute a Perl script on my local cluster. The script is very disk intensive and the Lustre disk system bottlenecks if I run more than ~10 instances of the script. Therefore, I need to make sure that Swift only submits up to a certain number of jobs. I also need this ability so that I do not take over the whole cluster, preventing other users from getting access. What setting do I need to change so that I can have this functionality? I read through the Users Guide and the closest I have found is setting the jobThrottle setting. But, this either sets a very low hard limit when jobThrottle=0, or else allows the concurrently running jobs to slowly grow. For example, if I set jobThrottle=1 I initially get two jobs running, but the running job count slowly grows up to 25+ if I let it.
> 
> Also, it would be nice to have a setting to indicate that I can use more than the initial default of 2 jobs right from the beginning of the run. This would prevent some wasted time at the beginning of the run. Is this possible?
> 
> I am using localhost for the Swift pool, and the job submission actually happens in the Perl script. My cluster is setup with SLURM so I cannot use Swift’s built-in job submission protocols.
> 
> Thanks for the help.
> 
> -Bill
> 
> _______________________________________________
> William I. Gustafson Jr., Ph.D.
> Scientist
> ATMOSPHERIC SCIENCES AND GLOBAL CHANGE DIVISION
> 
> Pacific Northwest National Laboratory
> http://www.pnl.gov/atmospheric/staff/staff_info.asp?staff_num=5716
> http://www.researcherid.com/rid/A-7732-2008
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user