[Swift-devel] Swift throttling

[Mihael Hategan]

On Mon, 2008-02-04 at 16:17 +0000, Ben Clifford wrote:
> 
> On Mon, 4 Feb 2008, Michael Wilde wrote:
> 
> > hopefully the slow-start sensing
> > algorithms will sense that resource is already under strain and stay at a low
> > submission rate.
> 
> I don't think it does that at all.

Actually, yes. There's the submit throttle which limits the submission
parallelism. When remote site is under load and accepts jobs slowly, the
client will invariably submit slower.

And now that I think of it, the maxSubmitRate looks a lot like it could
be integrated here.

> 
> > - we modify the code to enable explicit limits on the algorithm to be set by
> > the user, eg:
> >  throttle.host.submitlimit - max # jobs that can be queued to a host
> >  throttle.host.submitrate - max #jobs/sec that can be queued to a host
> >                             (float)
> 
> parameters that control thsoe exist already, I think, for the whole 
> workflow. In the single site case, site specific ones aren't needed 
> because of that. If they were being implemented, it would probably be 
> better to make them settable in the sites catalog so that they can be 
> defined differently for each site.
> 
> throttle.scote.job.factor limits the number of concurrent jobs to 2 + 
> 100*throttle.score.job.factor   (so to achieve a limit of 52, set 
> throttle.score.job.factor to 0.5)

Unfortunately that's an int. So it won't work. I'll make it a float.

> 
> There's a per-site profile setting:
> maxSubmitRate - limits the maximum rate of job submission, in jobs per 
> second.
>