<div>Hey everyone,</div><div>I've been trying to run some short jobs in the "fast" queue on pads. That means I need to keep the wall time under 1 hour, and my tasks are around 20 min. What's been happening, at least for a smallish number of jobs, is that swift decreases the number of jobs submitted to the queue as the number of tasks is reduced and at the end, some tasks remain unfinished while no jobs are in the queue, and this continues indefinately.</div>
<div><br></div><div>The following is one sites entry where I reproducibly had this problem for 70 tasks</div><div><br></div><div><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<execution provider="coaster" url="none" jobManager="local:pbs"/></blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<!--<profile namespace="globus" key="queue">fast</profile>--></blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<profile namespace="globus" key="maxtime">3600</profile></blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<profile namespace="globus" key="maxwalltime">00:25:00</profile></blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<profile namespace="globus" key="workersPerNode">1</profile></blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<profile namespace="globus" key="internalHostname">172.5.86.5</profile></blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<profile namespace="globus" key="slots">120</profile></blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<profile namespace="globus" key="nodeGranularity">1</profile></blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<profile namespace="globus" key="maxNodes">1</profile></blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<profile namespace="karajan" key="jobThrottle">0.99</profile></blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<profile namespace="karajan" key="initialScore">10000</profile></blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<profile namespace="globus" key="project">CI-CCR000013</profile></blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<gridftp url="local://localhost" /></blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<scratch>/tmp</scratch></blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<workdirectory>/home/hockyg/reichman/glassy_dynamics/code/swift/run/real</workdirectory></blockquote><div><br></div><div><br></div><div>There are also some of this type of error </div><div><div>Exception caught while unregistering channel</div>
<blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
org.globus.cog.karajan.workflow.service.channels.ChannelException: Trying to bind invalid channel (2027063355: {}) to 60652275: {}</blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
at org.globus.cog.karajan.workflow.service.channels.MetaChannel.bind(MetaChannel.java:67)</blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
at org.globus.cog.karajan.workflow.service.channels.ChannelManager.unregisterChannel(ChannelManager.java:401)</blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
at org.globus.cog.karajan.workflow.service.channels.ChannelManager.shutdownChannel(ChannelManager.java:411)</blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
at org.globus.cog.karajan.workflow.service.channels.ChannelManager.handleChannelException(ChannelManager.java:284)</blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
at org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel.handleChannelException(AbstractStreamKarajanChannel.java:83)</blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
at org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel$Sender.run(AbstractStreamKarajanChannel.java:257)</blockquote><div>but i'm not sure that's related... </div></div><div><br>
</div>
<div><br></div><div>Running with "Swift svn swift-r3432 (swift modified locally) cog-r2829"</div><div><br></div><div>Swift output went something like</div><div>Progress: Submitted:69 Active:1 Finished successfully:1</div>
<div>Progress: Submitted:67 Active:3 Finished successfully:1</div><div>Progress: Submitted:66 Active:4 Finished successfully:1</div><div>Progress: Submitted:65 Active:5 Finished successfully:1</div><div>Progress: Submitted:64 Active:6 Finished successfully:1</div>
<div>Progress: Submitted:61 Active:9 Finished successfully:1</div><div>Progress: Submitted:58 Active:12 Finished successfully:1</div><div>Progress: Submitted:57 Active:13 Finished successfully:1</div><div>Progress: Submitted:54 Active:16 Finished successfully:1</div>
<div>Progress: Submitted:52 Active:18 Finished successfully:1</div><div>Progress: Submitted:51 Active:19 Finished successfully:1</div><div>Progress: Submitted:50 Active:20 Finished successfully:1</div><div>Progress: Submitted:49 Active:21 Finished successfully:1</div>
<div>Progress: Submitted:48 Active:22 Finished successfully:1</div><div>Progress: Submitted:41 Active:29 Finished successfully:1</div><div>Progress: Submitted:38 Active:32 Finished successfully:1</div><div>Progress: Submitted:37 Active:33 Finished successfully:1</div>
<div>Progress: Submitted:35 Active:35 Finished successfully:1</div><div>Progress: Submitted:31 Active:39 Finished successfully:1</div><div>Progress: Submitted:30 Active:40 Finished successfully:1</div><div>Progress: Submitted:26 Active:44 Finished successfully:1</div>
<div>Progress: Submitted:26 Active:44 Finished successfully:1</div><div>Progress: Submitted:26 Active:44 Finished successfully:1</div><div>Progress: Submitted:26 Active:44 Finished successfully:1</div><div>Progress: Submitted:26 Active:44 Finished successfully:1</div>
<div>Progress: Submitted:26 Active:44 Finished successfully:1</div><div>Progress: Submitted:26 Active:43 Checking status:1 Finished successfully:1</div><div>Progress: Submitted:26 Active:43 Finished successfully:2</div>
</div><div>Progress: Submitted:26 Active:42 Checking status:1 Finished successfully:2</div><div><div>Progress: Submitted:25 Active:42 Checking status:1 Finished successfully:3</div><div>Progress: Submitted:25 Active:41 Checking status:1 Finished successfully:4</div>
<div>Progress: Submitted:25 Active:41 Finished successfully:5</div><div>Progress: Submitted:25 Active:40 Checking status:1 Finished successfully:5</div><div>Progress: Submitted:25 Active:39 Checking status:1 Finished successfully:6</div>
<div>Progress: Submitted:24 Active:40 Finished successfully:7</div><div>Progress: Submitted:24 Active:39 Checking status:1 Finished successfully:7</div><div>Progress: Submitted:24 Active:38 Checking status:1 Finished successfully:8</div>
<div>Progress: Submitted:24 Active:38 Finished successfully:9</div><div>Progress: Submitted:24 Active:37 Checking status:1 Finished successfully:9</div></div><div>Progress: Submitted:24 Active:35 Checking status:1 Finished successfully:11</div>
<div><div>Progress: Submitted:23 Active:35 Checking status:1 Finished successfully:12</div><div>Progress: Submitted:22 Active:35 Checking status:1 Finished successfully:13</div><div>Progress: Submitted:22 Active:35 Finished successfully:14</div>
<div>Progress: Submitted:22 Active:34 Checking status:1 Finished successfully:14</div><div>Progress: Submitted:21 Active:34 Checking status:1 Finished successfully:15</div><div>Progress: Submitted:21 Active:34 Finished successfully:16</div>
<div>Progress: Submitted:21 Active:33 Checking status:1 Finished successfully:16</div><div>Progress: Submitted:21 Active:33 Finished successfully:17</div><div>Progress: Submitted:20 Active:32 Checking status:1 Finished successfully:18</div>
<div>Progress: Submitted:20 Active:32 Finished successfully:19</div><div>Progress: Submitted:20 Active:31 Checking status:1 Finished successfully:19</div><div>Progress: Submitted:19 Active:31 Finished successfully:21</div>
<div>Progress: Submitted:19 Active:30 Checking status:1 Finished successfully:21</div><div>Progress: Submitted:18 Active:30 Checking status:1 Finished successfully:22</div><div>Progress: Submitted:18 Active:29 Checking status:1 Finished successfully:23</div>
<div>Progress: Submitted:18 Active:28 Checking status:1 Finished successfully:24</div><div>Progress: Submitted:17 Active:29 Finished successfully:25</div><div>Progress: Submitted:17 Active:29 Finished successfully:25</div>
<div>Progress: Submitted:17 Active:28 Checking status:1 Finished successfully:25</div><div>Progress: Submitted:17 Active:27 Checking status:1 Finished successfully:26</div><div>Progress: Submitted:17 Active:26 Checking status:1 Finished successfully:27</div>
<div>Progress: Submitted:17 Active:25 Checking status:1 Finished successfully:28</div><div>Progress: Submitted:17 Active:24 Checking status:1 Finished successfully:29</div><div>Progress: Submitted:16 Active:25 Finished successfully:30</div>
<div>Progress: Submitted:16 Active:24 Checking status:1 Finished successfully:30</div><div>Progress: Submitted:15 Active:24 Checking status:1 Finished successfully:31</div><div>Progress: Submitted:15 Active:24 Finished successfully:32</div>
<div>Progress: Submitted:15 Active:23 Checking status:1 Finished successfully:32</div><div>Progress: Submitted:14 Active:24 Finished successfully:33</div><div>Progress: Submitted:14 Active:23 Checking status:1 Finished successfully:33</div>
<div>Progress: Submitted:14 Active:22 Checking status:1 Finished successfully:34</div><div>Progress: Submitted:14 Active:22 Finished successfully:35</div><div>Progress: Submitted:14 Active:21 Checking status:1 Finished successfully:35</div>
<div>Progress: Submitted:13 Active:22 Finished successfully:36</div><div>Progress: Submitted:13 Active:22 Finished successfully:36</div><div>Progress: Submitted:13 Active:20 Checking status:1 Finished successfully:37</div>
<div>Progress: Submitted:12 Active:21 Finished successfully:38</div><div>Progress: Submitted:12 Active:20 Checking status:1 Finished successfully:38</div><div>Progress: Submitted:12 Active:19 Checking status:1 Finished successfully:39</div>
<div>Progress: Submitted:12 Active:19 Finished successfully:40</div><div>Progress: Submitted:12 Active:18 Checking status:1 Finished successfully:40</div><div>Progress: Submitted:12 Active:17 Checking status:1 Finished successfully:41</div>
<div>Progress: Submitted:11 Active:17 Checking status:1 Finished successfully:42</div><div>Progress: Submitted:11 Active:17 Finished successfully:43</div><div>Progress: Submitted:11 Active:16 Checking status:1 Finished successfully:43</div>
<div>Progress: Submitted:11 Active:15 Checking status:1 Finished successfully:44</div><div>Progress: Submitted:10 Active:16 Finished successfully:45</div></div><div><div>Progress: Submitted:3 Active:22 Finished successfully:46</div>
<div>Progress: Submitted:3 Active:21 Checking status:1 Finished successfully:46</div></div><div><div>Progress: Submitted:3 Active:19 Finished successfully:49</div><div>Progress: Submitted:3 Active:19 Finished successfully:49</div>
<div>Progress: Submitted:2 Active:20 Finished successfully:49</div><div>Progress: Submitted:1 Active:21 Finished successfully:49</div></div><div>.</div><div>.</div><div>.</div><div><div>Progress: Submitted:1 Active:15 Finished successfully:55</div>
<div>Progress: Submitted:1 Active:15 Finished successfully:55</div><div>Progress: Submitted:1 Active:15 Finished successfully:55</div><div>Progress: Submitted:1 Active:15 Finished successfully:55</div><div>Progress: Submitted:1 Active:14 Checking status:1 Finished successfully:55</div>
<div>Progress: Submitted:1 Active:14 Finished successfully:56</div><div>Progress: Submitted:1 Active:13 Checking status:1 Finished successfully:56</div><div>Progress: Submitted:1 Active:12 Checking status:1 Finished successfully:57</div>
<div>Progress: Submitted:1 Active:12 Finished successfully:58</div><div>Progress: Submitted:1 Active:11 Checking status:1 Finished successfully:58</div><div>Progress: Submitted:1 Active:10 Checking status:1 Finished successfully:59</div>
<div>Progress: Submitted:1 Active:10 Finished successfully:60</div><div>Progress: Submitted:1 Active:10 Finished successfully:60</div><div>Progress: Submitted:1 Active:10 Finished successfully:60</div><div>Progress: Submitted:1 Active:8 Checking status:1 Finished successfully:61</div>
<div>Progress: Submitted:1 Active:8 Finished successfully:62</div><div>Progress: Submitted:1 Active:7 Checking status:1 Finished successfully:62</div><div>Progress: Submitted:1 Active:7 Finished successfully:63</div>
<div>Progress: Submitted:1 Active:6 Checking status:1 Finished successfully:63</div><div>Progress: Submitted:1 Active:4 Checking status:1 Finished successfully:65</div></div><div><div>Progress: Submitted:1 Active:3 Checking status:1 Finished successfully:66</div>
<div>Progress: Submitted:1 Active:3 Finished successfully:67</div><div>Progress: Submitted:1 Active:3 Finished successfully:67</div><div>Progress: Submitted:1 Active:2 Checking status:1 Finished successfully:67</div>
<div>Progress: Submitted:1 Active:2 Finished successfully:68</div><div>Progress: Submitted:1 Active:1 Checking status:1 Finished successfully:68</div></div><div><div>Progress: Submitted:1 Finished successfully:70</div>
<div>Progress: Submitted:1 Finished successfully:70</div><div>Progress: Submitted:1 Finished successfully:70</div><div>Progress: Submitted:1 Finished successfully:70</div><div>Progress: Submitted:1 Finished successfully:70</div>
<div>Progress: Submitted:1 Finished successfully:70</div><div>Progress: Submitted:1 Finished successfully:70</div><div>Progress: Submitted:1 Finished successfully:70</div><div>Progress: Submitted:1 Finished successfully:70</div>
<div>Progress: Submitted:1 Finished successfully:70</div><div>Progress: Submitted:1 Finished successfully:70</div><div>Progress: Submitted:1 Finished successfully:70</div><div>Progress: Submitted:1 Finished successfully:70</div>
<div>Progress: Submitted:1 Finished successfully:70</div><div>Progress: Submitted:1 Finished successfully:70</div></div><div><br></div><div>etc</div>