[Swift-devel] submitting jobs to the queue

[Mihael Hategan]

On Wed, 2007-03-07 at 16:30 -0600, Veronika V. Nefedova wrote:
> Hi,
> 
> I've noticed one very strange behavior. For example, I have 68 jobs to be 
> submitted to the remote host simultaneously. Swift submits at first just 26 
> jobs. I checked that several times - its always 26 jobs. Then, when at 
> least one job out of those 26 is finished - swift goes ahead and submits 
> the rest (all of those left - 42 in my case).
> Is it a bug or a feature?

Feature. Although it should probably be tamed down in the one site case.
Each site has a score that changes based on how it behaves. If a site
completes jobs ok, it gets a higher score in time. If jobs fail on it,
it gets a lower score.

Now, let's consider the following scenario: 2 sites, one fast one slow.
With no scores and no limitations, half of the jobs would go to one, and
half to the other. The workflow finishes when the slow site finishes
half the jobs.
What happens however, is that Swift limits the number of initial jobs,
and does "probing". This allows it to infer some stuff about the sites
by the time it gets to submit lots of jobs. It should yield better
performance on larger workflows with imbalanced sites, which is, I'm
guessing, our main scenario.

> 
> Nika
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>