Command(16119, SHUTDOWN): handling reply timeout; sendReqTime=110823-113714.545, sendTime=110823-113714.545, now=110823-113917.674
  Submitted:80  Stage out:3  Finished successfully:16078
  time: Tue, 23 Aug 2011 11:39:17 -0500  Initializing:1  Selecting site:3  Stage in:9  Submitting:2  Submitted:80  Stage out:3  Finished successfully:16078Progress:
  time: Tue, 23 Aug 2011 11:39:20 -0500  Selecting site:3  Stage in:10  Submitting:1  Submitted:81  Stage out:2  Finished successfully:16079
Progress:  time: Tue, 23 Aug 2011 11:39:20 -0500  Selecting site:3  Stage in:10  Submitting:1  Submitted:81  Stage out:2  Finished successfully:16079
Progress:  time: Tue, 23 Aug 2011 11:39:20 -0500  Selecting site:3  Stage in:10  Submitting:1  Submitted:81  Stage out:2  Finished successfully:16079
Command(16119, SHUTDOWN)fault was: Reply timeout
org.globus.cog.karajan.workflow.service.ReplyTimeoutException
    at org.globus.cog.karajan.workflow.service.commands.Command.handleReplyTimeout(Command.java:288)
    at org.globus.cog.karajan.workflow.service.commands.Command$Timeout.run(Command.java:293)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)
Progress:  time: Tue, 23 Aug 2011 11:39:22 -0500  Selecting site:2  Stage in:10  Submitting:1  Submitted:82  Stage out:1  Finished successfully:16080
Failed to shut down block: Block 0823-161151-000000 (8x1200.000s)
org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: Failed to cancel task. qdel returned with an exit code of 153
    at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.cancel(AbstractExecutor.java:205)
    at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.cancel(AbstractJobSubmissionTaskHandler.java:85)
    at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.cancel(AbstractTaskHandler.java:69)
    at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:102)
    at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:91)
    at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.cancel(BlockTaskSubmitter.java:46)
    at org.globus.cog.abstraction.coaster.service.job.manager.Block.forceShutdown(Block.java:309)
    at org.globus.cog.abstraction.coaster.service.job.manager.Block$1.run(Block.java:300)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)
Command(16119, SHUTDOWN)fault was: Invalid channel: 995824187: {}
org.globus.cog.karajan.workflow.service.ReplyTimeoutException
    at org.globus.cog.karajan.workflow.service.commands.Command.handleReplyTimeout(Command.java:288)
    at org.globus.cog.karajan.workflow.service.commands.Command$Timeout.run(Command.java:293)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)

Failed to shut down block: Block 0823-161151-000000 (8x1200.000s)
org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: Failed to cancel task. qdel returned with an exit code of 153
    at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.cancel(AbstractExecutor.java:205)
    at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.cancel(AbstractJobSubmissionTaskHandler.java:85)
    at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.cancel(AbstractTaskHandler.java:69)
    at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:102)
    at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:91)
    at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.cancel(BlockTaskSubmitter.java:46)
    at org.globus.cog.abstraction.coaster.service.job.manager.Block.forceShutdown(Block.java:309)
    at org.globus.cog.abstraction.coaster.service.job.manager.Node.errorReceived(Node.java:98)
    at org.globus.cog.karajan.workflow.service.commands.Command.errorReceived(Command.java:230)
    at org.globus.cog.karajan.workflow.service.commands.Command.reexecute(Command.java:256)
    at org.globus.cog.karajan.workflow.service.commands.Command.reexecute(Command.java:274)
    at org.globus.cog.karajan.workflow.service.commands.Command.reexecute(Command.java:274)
    at org.globus.cog.karajan.workflow.service.commands.Command.handleReplyTimeout(Command.java:288)
    at org.globus.cog.karajan.workflow.service.commands.Command$Timeout.run(Command.java:293)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)
Java heap space