[Swift-devel] Needs for site selection and job scheduling enhancements

[wilde at mcs.anl.gov]

Mihael,

Below is a proposal for Swift scheduling features that will need a fair amount of deliberation. This email is intended to start the process. I can move this to a bugzila enhancement to start the process, so long as you agree that the discussion makes sense to have.

Allan, Dan and I have been re-examining the SCEC workflow that Allan is working on.

Doing it efficiently on OSG raises scheduling aspects that Swift still doesn't handle well. We propose to address these issues in two phases:

I. Use simple workflows that group more work into single scripts to achieve the job affinities needed for reasonable performance. Provide scheduling hints to Swift.

II. Determine how Swift could automatically achieve the same scheduling decisions.

Phase II is pretty complex as far as we can tell, so lets defer its discussion.

To do phase I, we want to ask if any of the following capabilities could be added, and which ones are both reasonable and of "affordable" cost and make sense to try.

Most of these involve enabling a Swift script to specify scheduling "hints" on individual app() invocations.

1. Hint to bias a job to a specific set of sites (by pool name?) 

app myapp (file f, int v1, int v2)
{
  myapp @f sitebias(UNL=v1,Clemson=v2,UChicago=v2);
}

2. Hint to bias a job to site(s) that already have designated input parameter files cached.

  myapp @f1 @f2 filebias(f2=v3);


3. A prio hint to cause a job to be scheduled earlier than lower prio jobs

  myapp @f1 schedulebias(f2=v3);

We'd like to permit multiple hints to be specified on a single app call:

  myapp @f sitebias(UNL=v1,Clemson=v2,UChicago=v2) schedulebias(f2=v3);

And we might need a feature (perhaps a swift.properties setting) to tell Swift to defer initial scheduling decisions for N seconds or until J jobs have been queued by the script, so that a sufficiently large number of jobs are in the queue before scheduling decisions are made (probably delay for say a minute on a multi-hour script run).

In addition, we're wondering how easy (and desirable) any/all of the following language extensions could be done:

- select statement to work on string values and/or ranges

- elseif clause to achieve the above in a multi-branch if statement

- function pointers to select a function dynamically, eg from an array

- ability to set the app program name from a variable

These enhancements would enable us to manually code in the scheduling hints by providing multiple pool groups with different throttle settings and to manually force jobs to different pools.

If the easiest way to set the hints requested above on an individual job is to pass an env var on the command line, then that capability might be a useful alternative to setting env vars with one-value-for-all method that we currently employ with the ENV profile.  This could be considered as a useful enhancement separate from the question of how scheduling hints are set.

Lastly, in phase I we will be testing the performance of having the jobs "pull" files via wget in a pre-staged manner, within the applicaton script.  For Phase II we'd like to consider having Swift do that in the worker: Have the coaster worker "pull" files in via wget or similar command/function, asynchronously pre-staging files for jobs that have been queued/assigned to a site. But that can be deferred for a later discussion.

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory