[Swift-devel] scheduler scoring with file transfer

[Allan Espinosa]

i observed in swift logs that there are scheduler score updates after
FILE_OPERATIONs.  As we can see below in the nostagein workflow, there
are less submitted jobs than the one with stageins.

Does this mean i have to match my file transfer throttles with job
submission throttles?

same score and throttling parameters in sites.xml file:
<config>
        <pool handle="TERAPORT" sysinfo="INTEL32::LINUX">
    <execution provider="coaster" url="none"  jobmanager="local:pbs" />
                <!--<execution provider="coaster" url="tp-grid1.uchicago.edu"-->
        <!--jobmanager="gt2:gt2:pbs" />-->
                <gridftp url="local://localhost" />
    <workdirectory >/home/aespinosa/workflows/activelog/workdir</workdirectory>

                <profile namespace="karajan" key="initialScore">2.02</profile>
                <profile namespace="karajan" key="jobThrottle">1.98</profile>

                <profile namespace="globus" key="queue">fast</profile>
                <profile namespace="globus" key="slots">4</profile>
                <profile namespace="globus" key="workersPerNode">2</profile>
    <profile namespace="globus" key="maxnodes">64</profile>
    <profile namespace="globus" key="maxwalltime">00:06:00</profile>
    <profile namespace="globus" key="maxtime">3600</profile>
        </pool>
</config>


[aespinosa at tp-login1 blast]$ ./demoblast.sh (blast.swift with coasters)
Swift svn swift-r2949 cog-r2406

RunID: out.run_000
Progress:
Progress:
Progress:
Progress:
Progress:
Progress:  uninitialized:1
Progress:  Initializing:1022  Selecting site:1
Progress:  Selecting site:1022  Initializing site shared directory:1
Progress:  Selecting site:1011  Stage in:12
Progress:  Selecting site:1010  Stage in:13
Progress:  Selecting site:1005  Stage in:18
Progress:  Selecting site:998  Stage in:25
Progress:  Selecting site:989  Stage in:34
Progress:  Selecting site:988  Stage in:35
Progress:  Selecting site:984  Stage in:39
Progress:  Selecting site:983  Stage in:40
Progress:  Selecting site:974  Stage in:49
Progress:  Selecting site:973  Submitting:49  Submitted:1
Progress:  Selecting site:973  Submitted:50
Progress:  Selecting site:973  Submitted:50


ACTIVE JOBS--------------------
JOBNAME            USERNAME      STATE  PROC   REMAINING            STARTTIME


     0 Active Jobs     174 of  200 Processors Active (87.00%)
                        94 of  100 Nodes Active      (94.00%)

IDLE JOBS----------------------
JOBNAME            USERNAME      STATE  PROC     WCLIMIT            QUEUETIME

1101631            aespinosa       Idle    23    00:54:00  Thu Jun 18 12:47:56


066-many.swift (no stageins)

[aespinosa at tp-login1 activelog]$ ./runtest.sh
Swift svn swift-r2949 cog-r2406

RunID: activelog
Progress:
Progress:  uninitialized:1
Progress:  Initializing:1022  Selecting site:1
Progress:  Selecting site:1022  Initializing site shared directory:1
Progress:  Selecting site:1013  Submitting:9  Submitted:1
Progress:  Selecting site:1013  Submitted:10
...


-- 
Allan M. Espinosa <http://allan.88-mph.net/blog>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>