[Swift-devel] submitting jobs to the queue

[Veronika V. Nefedova]

OK, Here is my another question.
Teragrid allows the user to have 385 jobs in a queue. If I run my complete 
workflow (244 molecules), on stage four I'll have 80 times 244 jobs (i.e. 
close to 20K). How do I set the limit for the number of submitted jobs to 
the queue to 385 ? I remember that condor had a specific parameter to 
condor_submit that was managing exactly that...

Nika

At 04:36 PM 3/7/2007, Mihael Hategan wrote:
>On Wed, 2007-03-07 at 16:30 -0600, Veronika V. Nefedova wrote:
> > Hi,
> >
> > I've noticed one very strange behavior. For example, I have 68 jobs to be
> > submitted to the remote host simultaneously. Swift submits at first 
> just 26
> > jobs. I checked that several times - its always 26 jobs. Then, when at
> > least one job out of those 26 is finished - swift goes ahead and submits
> > the rest (all of those left - 42 in my case).
> > Is it a bug or a feature?
>
>Feature. Although it should probably be tamed down in the one site case.
>Each site has a score that changes based on how it behaves. If a site
>completes jobs ok, it gets a higher score in time. If jobs fail on it,
>it gets a lower score.
>
>Now, let's consider the following scenario: 2 sites, one fast one slow.
>With no scores and no limitations, half of the jobs would go to one, and
>half to the other. The workflow finishes when the slow site finishes
>half the jobs.
>What happens however, is that Swift limits the number of initial jobs,
>and does "probing". This allows it to infer some stuff about the sites
>by the time it gets to submit lots of jobs. It should yield better
>performance on larger workflows with imbalanced sites, which is, I'm
>guessing, our main scenario.
>
> >
> > Nika
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> >