[Swift-devel] Re: question about Swift scheduling and execution

[Mihael Hategan]

Such questions should go to swift-devel.

On Mon, 2008-12-29 at 17:45 -0600, Zhengxiong Hou wrote:
> Hi Mihael,
>      Could you please spend some time to help to answer the following 
> questions about the Swift scheduling and execution mechanism?
> 
>   *  (1) How to set the policy, POLICY_WEIGHTED_RANDOM or 
> POLICY_BEST_SCORE ?*
>          ------in WeightedHostScoreScheduler.java-------------
>                if (POLICY.equals(name)) {
>                 if (value instanceof String) {
>                     value = ((String) value).toLowerCase();
>                 }
>                 if ("random".equals(value)) {
>                     policy = POLICY_WEIGHTED_RANDOM;
>                 }
>                 else if ("best".equals("value")) {
>                     policy = POLICY_BEST_SCORE;
>                 }
>                 else {
>                     throw new KarajanRuntimeException("Unknown policy 
> type: " + value);
>                 }
>             }

You can set that in libexec/scheduler.xml, by adding <property
name="policy" value="random/best"/> in <project>/<scheduler>.

> 
>  *  (2) How to decide whether a site is Overloaded or NOT? Where is the 
> load information source?*
> ------in OverloadedHostMonitor.java-------------
>    WeightedHost wh = (WeightedHost) i.next();
>                                                         if 
> (wh.isOverloaded() == 0) {

Take a look at the WeightedHost.isOverloaded() method.

The logic is basically overloaded = (load <= maxLoad), where maxLoad =
jobThrottle * tscore + 1.

>                                                                 
> whss.removeOverloaded(wh);
>                                                                 i.remove();
>                                                         }
> 
>  * (3)  What's the specific constrains ? *
>  ------in WeightedHostScoreScheduler.java-------------
>     protected WeightedHostSet constrain(WeightedHostSet s, 
> ResourceConstraintChecker rcc,
>             TaskConstraints tc) {
>         if (rcc == null) {
>             return s;
>         }

The only one I know being used is the executable (so that swift will
schedule only on sites that have it specified in tc.data), but it is
meant to allow specification of architecture, OS, etc.

> ------in TaskConstraints.java-------------
>         private synchronized Map getMap() {
>                 if (map == null) {
>                         map = new HashMap();
>                 }
>                 return map;
>         }
> 
>         public void addConstraint(String name, Object value) {
>                 getMap().put(name, value);
>         }
> 
>         public Object getConstraint(String name) {
>                 return getMap().get(name);
>         }
> 
>   (4)  In the standard output information,  i.e.
>              Firstly:         Sorted: [FLTECH:*0.000(1.000):0/1* 
> overload: 0, AGLT2:*0.000(1.000*):1/1 overload: 0]
>              Finally:         Sorted: [AGLT2:*229.804(93.821):19/19* 
> overload: 0]
> 
>       I think the score is the (*1.000*) and *(93.821). *This score 
> should be less than scoreHighCap (= 100). 
>       So, *what's the meaning of 0.000 and **229.804?

Raw score. score = e^(B*arctan(C*rawScore)), where B and C are
constants. 

While not exactly the same thing, that function is similar in principle
and purpose to this:

http://en.wikipedia.org/wiki/Gompertz_curve

>       *And *what's the exact meaning of **0/1, 1/1, 19/19 ? *( I guess 
> that it means the running jobs. There are 0 out of scheduled 1 job is 
> running; 1 out 1 scheduled job is running; 19 out of 19 scheduled jobs 
> are running. Is that right?)

No. The first number shows the number of currently running jobs on that
site, and the second one shows the maximum number of concurrent jobs
that will be allowed on that site (after which it becomes overloaded).
This is all fairly clear if you look at WeightedHost.toString().