[Swift-devel] nearly, but not quite optimal performance on ranger from ranger under coasters

[Glen Hocky]

Hi Everyone,
Today I tried to run jobs of length ~45 minutes. I ran on Ranger from 
Ranger with
    <profile namespace="karajan" key="initialScore">50</profile>
    <profile namespace="karajan" key="jobThrottle">5</profile>

This submitted 51 jobs to the queue, of which 38 went to the run state, 
meaning there should be 608 available coasters. I  had
> Progress:  Selecting site:2569  Active:431  Finished successfully:9
For ~30 minutes until some jobs started finishing. Then I had a stead 
decline of activity, see below.
I assume this is happening because the extra 177 coasters that were 
available timed out durning this 45 minutes and new ones did not become 
available, but that could be wrong.


How should I tune my settings to fix this?

p.s. logs in CI-HOME:   
/home/hockyg/public_html/swift_logs/ranger_from_ranger/oops-20090429-1519-03iyqi74.log

> Progress:  Selecting site:2569  Active:431  Finished successfully:9
> Progress:  Selecting site:2569  Active:431  Finished successfully:9
> Progress:  Selecting site:2569  Active:431  Finished successfully:9
> Progress:  Selecting site:2569  Active:431  Finished successfully:9
> Progress:  Selecting site:2569  Active:431  Finished successfully:9
> Progress:  Selecting site:2569  Active:431  Finished successfully:9
> Progress:  Selecting site:2569  Active:430  Checking status:1  
> Finished successfully:9
> Progress:  Selecting site:2569  Active:430  Finished successfully:10
> Progress:  Selecting site:2569  Active:430  Finished successfully:10
> Progress:  Selecting site:2569  Active:430  Finished successfully:10
> Progress:  Selecting site:2569  Active:429  Checking status:1  
> Finished successfully:10
> Progress:  Selecting site:2569  Active:429  Finished successfully:11
> Progress:  Selecting site:2569  Active:429  Finished successfully:11
...
> Progress:  Selecting site:2516  Stage in:53  Active:196  Checking 
> status:3  Stage out:194  Finished successfully:47
> Progress:  Selecting site:2516  Stage in:53  Active:196  Checking 
> status:3  Stage out:193  Finished successfully:48
> Progress:  Selecting site:2516  Stage in:53  Active:196  Checking 
> status:3  Stage out:192  Finished successfully:49
> Progress:  Selecting site:2516  Stage in:53  Active:196  Checking 
> status:3  Stage out:192  Finished successfully:49
> Progress:  Selecting site:2516  Stage in:53  Active:195  Checking 
> status:4  Stage out:192  Finished successfully:49
> Progress:  Selecting site:2516  Stage in:53  Active:166  Checking 
> status:33  Stage out:190  Finished successfully:51
> Progress:  Selecting site:2516  Stage in:53  Active:166  Checking 
> status:33  Stage out:189  Finished successfully:52
> Progress:  Selecting site:2516  Stage in:53  Active:165  Checking 
> status:34  Stage out:189  Finished successfully:52
> Progress:  Selecting site:2516  Stage in:53  Active:155  Checking 
> status:44  Stage out:188  Finished successfully:53
> Progress:  Selecting site:2516  Stage in:53  Active:152  Checking 
> status:47  Stage out:187  Finished successfully:54
> Progress:  Selecting site:2516  Stage in:53  Active:151  Checking 
> status:48  Stage out:186  Finished successfully:55
> Progress:  Selecting site:2516  Stage in:53  Active:99  Checking 
> status:100  Stage out:184  Finished successfully:57
> Progress:  Selecting site:2516  Stage in:53  Active:99  Checking 
> status:100  Stage out:184  Finished successfully:57
> Progress:  Selecting site:2516  Stage in:53  Active:99  Checking 
> status:100  Stage out:183  Finished successfully:58
> Progress:  Selecting site:2510  Stage in:59  Active:90  Checking 
> status:109  Stage out:182  Finished successfully:59
> Progress:  Selecting site:2509  Stage in:60  Active:90  Checking 
> status:109  Stage out:181  Finished successfully:60
> Progress:  Selecting site:2509  Stage in:60  Active:90  Checking 
> status:109  Stage out:179  Finished successfully:62
> Progress:  Selecting site:2509  Stage in:60  Active:90  Checking 
> status:109  Stage out:178  Finished successfully:63
> Progress:  Selecting site:2509  Stage in:60  Active:90  Checking 
> status:109  Stage out:177  Finished successfully:64
> Progress:  Selecting site:2509  Stage in:60  Active:90  Checking 
> status:109  Stage out:176  Finished successfully:65
> Progress:  Selecting site:2508  Stage in:61  Active:90  Checking 
> status:109  Stage out:176  Finished successfully:65
> Progress:  Selecting site:2508  Stage in:61  Active:90  Checking 
> status:109  Stage out:174  Finished successfully:67
> Progress:  Selecting site:2508  Stage in:61  Active:90  Checking 
> status:109  Stage out:174  Finished successfully:67
> Progress:  Selecting site:2508  Stage in:61  Active:90  Checking 
> status:109  Stage out:173  Finished successfully:68
> Progress:  Selecting site:2508  Stage in:61  Active:90  Checking 
> status:109  Stage out:172  Finished successfully:69
> Progress:  Selecting site:2508  Stage in:61  Active:90  Checking 
> status:109  Stage out:171  Finished successfully:70
> Progress:  Selecting site:2508  Stage in:61  Active:90  Checking 
> status:109  Stage out:170  Finished successfully:71
> Progress:  Selecting site:2503  Stage in:66  Active:40  Checking 
> status:159  Stage out:169  Finished successfully:72
> Progress:  Selecting site:2503  Stage in:66  Active:40  Checking 
> status:159  Stage out:168  Finished successfully:73
> Progress:  Selecting site:2503  Stage in:66  Active:40  Checking 
> status:159  Stage out:167  Finished successfully:74